Data Feeds (Legacy)

🚧

Legacy Feature

Data Feeds are an integral part of the Gen1 infrastructure, offering efficient data sharing and transformation capabilities.

However, we highly recommend upgrading to Gen2, as it introduces the advanced Data Shares feature, providing more robust, versatile, and secure data sharing options for enhanced collaboration and streamlined data operations.

Data Feeds are unique stages within Dataflows, designed to interconnect different data flows at the data level. They allow for real-time access to a Dataflow's data without granting direct access to the Dataflow itself.

Dataflows utilize a publisher-subscriber model where a Dataflow can publish a Data Feed that other Dataflows can subscribe to. The link between these entities is dynamic, ensuring that any fresh data is instantaneously available to all subscribers. Subsequently, the downstream components initiate data processing.

Data Feeds cater to a multitude of scenarios:

• Sharing data without permitting edits to the original Dataflow.
• Creating diverse branches of a project to eventually merge them in a final Dataflow.
• Controlling access to specific Dataflows and data.

You can either create (publish) new Data Feeds or connect (subscribe) to existing ones.

Data Feed Permissions

Data Feed permissions determine the access level for Data Services, teams, and individuals. They play a crucial role in sharing data while enforcing stringent governance and access controls, supervised by Ascend.

The permission options are:

1. Allow all Teams in all Data Services to access this Data Feed: Enables all system users/teams to view and subscribe to the Data Feed.
2. Only selected teams in selected Data Services can access this Data feed: Allows customized access to the Data Feed for selected teams in chosen Data Services.

How to Create a Data Feed Connector

Follow these steps to create a Data Feed Connector:

1. In the Deliver section of the BUILD panel, select on Create Data Feed.
2. Assign a descriptive name to the Data Feed and optionally, provide a description.
3. Choose the upstream transform for the Data Feed from the dropdown list, which displays all transforms in your Dataflow. Alternatively, from the graph, right-click the desired Transform, and select Create new data feed.
4. Select the necessary permissions for this Data Feed.
5. Select on Create to finish creating the Data Feed.

Post creation, users with necessary permission can subscribe to the Data Feed.

📘

If Data Feed Connectors are subscribing to a Data Feed, the Data Feed cannot be deleted until all associated Data Feed Connectors are removed.

How to Connect to an Existing Data Feed Connector

Follow these steps to connect to an existing Data Feed Connector:

1. In the Deliver section of the BUILD panel, select on Connect to Data Feed.
2. The display shows a catalog of Data Feeds. Search by name and use the Filter By option to refine your search.
3. Select the desired Data Feed, view the preview, and click SUBSCRIBE.
4. The Dataflow adds the subscription, making this Component available for further transformation.

🚧

A Data Feed Connector directly mirrors the upstream transform and will be affected by any changes in that transform. This could potentially invalidate any transforms built upon the Data Feed Connector if the schema of the upstream transform changes.