A Data Service in Ascend represents a collection of Dataflows with a shared security model. It serves as a container for organizing and managing data pipelines. When you design Dataflows within a Data Service, you are essentially creating a blueprint that will be executed by the Ascend Control Plane, automating the data processing tasks.
Dataflows are continuously running data pipelines. They form the core building blocks of a Data Service, enabling the automation of data processing tasks. By designing Dataflows, you define the flow of data and transformations that should be applied to it. These dataflows are then executed by Ascend's Control Plane.
A Data Service consists of several important entities that contribute to its functionality and flexibility. Understanding these entities is crucial for effectively working with Data Services in Ascend.
Connections are configuration settings that establish connectivity between a Data Service and external data sources or systems. They can be specific to a single Data Service or span across multiple Data Services. By defining connections, you enable dataflows within a Data Service to interact with external data sources, enabling seamless data integration and processing.
You can manage permissions within a Data Service. Members are individual users given permissions in a Data Service, whereas teams are collections of these members, grouped for efficient access control through inherited permissions.
Members are individual users who are added to a specific Data Service and granted permissions to access and work with its dataflows. They play a vital role in collaborating and managing dataflows within the Data Service. By adding members to a Data Service, you can control their level of access and ensure that the right individuals have the necessary privileges.
Teams are collections of one or more members within a Data Service. They provide a convenient way to group users based on their roles or responsibilities. Permissions can be assigned to teams, allowing you to manage access control more efficiently. By assigning permissions to a team, you can ensure that all team members automatically inherit the specified privileges.
Credentials are authentication details used to access external data sources or systems. Similar to connections, credentials can be specific to a single Data Service or shared across multiple Data Services. By managing credentials within a Data Service, you ensure that dataflows have the necessary authorization to securely access external resources.
Each Data Service can be configured to your Data Cloud service with Data Planes. A Data Plane in Ascend's platform is the fundamental infrastructure that supports data processing and movement, including services like BigQuery, Databricks, and Snowflake. It allows the distribution of workloads across more infrastructure and aligns Ascend teams with their access to various resources, hosting the workloads for the Data Flows within a Data Service.
The Data Cloud types are:
Ascend provides a webhook notification system that operates at the Data Service level. This system allows you to monitor the status and performance of dataflows within a Data Service. You can configure notifications to receive alerts and updates about the execution and health of your dataflows, ensuring timely awareness of any issues or successes.
Service Accounts are designed for external or automated systems to securely access Ascend without requiring authentication tied to a specific person. These accounts are beneficial for integrating Ascend with other tools or platforms, enabling seamless automation and data interaction. Service accounts offer a secure and efficient way to connect external systems with your Data Service.
Query Statistics is a query management tool available within each Data Service. It allows individuals with Super Admin or Site Admin privileges to monitor the queries being executed through the Query tab, specifically within their associated Dataflows.
Admins can use this tool to view critical information about the active queries. This includes the identity of the user executing the query, the duration of the query run, among other data points. Moreover, this tool also provides administrators the ability to terminate a running query if necessary.
A common approach in Ascend is to create three separate Data Services to maintain Development, Staging, and Production Dataflows. This pattern provides a structured environment for managing dataflows across different stages of the data lifecycle.By organizing dataflows into Development, Staging, and Production Data Services, you can maintain clear separation between environments and ensure smooth progression from development to production. This separation minimizes the risk of impacting critical data processes while allowing teams to iterate and test dataflows effectively.
Updated 5 months ago