Data Services
Ascend's Data Services streamline data pipeline management, offering robust permission controls, advanced data configurations, and efficient workflows across stages. It supports seamless data integration and processing across platforms like BigQuery, Databricks, and Snowflake.
Data Service Overview
A Data Service in Ascend represents a collection of Dataflows with a shared security model. It serves as a container for organizing and managing data pipelines. When you design Dataflows within a Data Service, you are essentially creating a blueprint that will be executed by the Ascend Control Plane, automating the data processing tasks.
Dataflows are continuously running data pipelines. They form the core building blocks of a Data Service, enabling the automation of data processing tasks. By designing Dataflows, you define the flow of data and transformations that should be applied to it. These dataflows are then executed by Ascend's Control Plane.
Data Service Settings
A Data Service consists of several important entities that contribute to its functionality and flexibility. Understanding these entities is crucial for effectively working with Data Services in Ascend.
Connections are configuration settings that establish connectivity between a Data Service and external data sources or systems. They can be specific to a single Data Service or span across multiple Data Services. By defining connections, you enable dataflows within a Data Service to interact with external data sources, enabling seamless data integration and processing.
You can manage permissions within a Data Service. Members are individual users given permissions in a Data Service, whereas teams are collections of these members, grouped for efficient access control through inherited permissions.
Members are individual users who are added to a specific Data Service and granted permissions to access and work with its dataflows. They play a vital role in collaborating and managing dataflows within the Data Service. By adding members to a Data Service, you can control their level of access and ensure that the right individuals have the necessary privileges.
Teams are collections of one or more members within a Data Service. They provide a convenient way to group users based on their roles or responsibilities. Permissions can be assigned to teams, allowing you to manage access control more efficiently. By assigning permissions to a team, you can ensure that all team members automatically inherit the specified privileges.
Credentials
Credentials are authentication details used to access external data sources or systems. Similar to connections, credentials can be specific to a single Data Service or shared across multiple Data Services. By managing credentials within a Data Service, you ensure that dataflows have the necessary authorization to securely access external resources.
Each Data Service can be configured to your Data Cloud service with Data Planes. A Data Plane in Ascend's platform is the fundamental infrastructure that supports data processing and movement, including services like BigQuery, Databricks, and Snowflake. It allows the distribution of workloads across more infrastructure and aligns Ascend teams with their access to various resources, hosting the workloads for the Data Flows within a Data Service.
The Data Cloud types are:
Default Behavior of Components
Within the Data Plane Configuration, you have the ability to control the starting states of all new components created within a Data Service. By default, Ascend begins running the component as soon as the component created. You can override this by setting the default behavior to Paused. If you want to override the override when you create the component, you can still do that, too.
Data Shares
Data Shares are not impacted by setting the default behavior of components. Data Shares pass data to their corresponding Data Share Connector based on the Data Refresh Mode of the Connector. In Continuous mode, the connector runs whenever new data is produced by the Data Share. In Scheduled mode, the connector checks for new data at the specified interval.
Ascend provides a webhook notification system that operates at the Data Service level. This system allows you to monitor the status and performance of dataflows within a Data Service. You can configure notifications to receive alerts and updates about the execution and health of your dataflows, ensuring timely awareness of any issues or successes.
Service Accounts are designed for external or automated systems to securely access Ascend without requiring authentication tied to a specific person. These accounts are beneficial for integrating Ascend with other tools or platforms, enabling seamless automation and data interaction. Service accounts offer a secure and efficient way to connect external systems with your Data Service.
Query Statistics
Query Statistics is a query management tool available within each Data Service. It allows individuals with Super Admin or Site Admin privileges to monitor the queries being executed through the Query tab, specifically within their associated Dataflows.
Admins can use this tool to view critical information about the active queries. This includes the identity of the user executing the query, the duration of the query run, among other data points. Moreover, this tool also provides administrators the ability to terminate a running query if necessary.
Organizational Patterns: Development, Staging, and Production Dataflows
A common approach in Ascend is to create three separate Data Services to maintain Development, Staging, and Production Dataflows. This pattern provides a structured environment for managing dataflows across different stages of the data lifecycle.By organizing dataflows into Development, Staging, and Production Data Services, you can maintain clear separation between environments and ensure smooth progression from development to production. This separation minimizes the risk of impacting critical data processes while allowing teams to iterate and test dataflows effectively.
Updated about 1 year ago