Data Feeds are a special type of stages that are used to link different Dataflow at data level.
They give live access to the data of a Dataflow without giving access to the Dataflow itself.
One Dataflow "publishes" a Data Feed that other Dataflows subscribe to. These two ends are linked together live, such that if any new data is processed, it will be automatically reflected to all subscribers -- and the components linked downstream will start processing as well.
Data Feeds can be useful for many different use cases:
- giving access to data without letting anyone edit your own Dataflow
- building different branches of the same project and then join them in a final Dataflow
- manage access to specific Dataflows and data
You can create new Data Feeds (publish) or connect to existing Data Feeds (subscribe).
The permissions of a Data Feed determine which Data Services and teams / individuals are allowed to subscribe to this Data Feed. In this way, Data Feeds allow for sharing of data with strict governance and access controls managed by Ascend.
There are two permission options:
- Allow all Teams in all Data Services to access this Data Feed: This option allows every user / team in the system to see and subscribe to the Data Feed.
- Only selected teams in selected Data Services can access this Data feed: This options allows for fine-grain customization of access to the Data Feed.
- Click Create Data Feed in the Integrate section of the BUILD panel
- Provide a name for the Data Feed. As a best practice, this name should clearly describe the nature of the data. Optionally provide a description.
- Select the upstream transform for the Data Feed. From the drop-down you'll see a list of all the transforms in your Dataflow to choose from. Alternatively, you can right-click the Transform you want to create a Data Feed Connector from, and select Create new data feed.
- Select the appropriate Permissions for this Data Feed.
- Click Create to finish creating the Data Feed.
Once a Data Feed is created, users with the required permission are able to subscribe to it Connecting to an existing Data Feed Connector.
If one or more Data Feed Connectors are subscribing to a Data Feed, you won't be able to delete the Data Feed until those Data Feed Connectors are all deleted first.
- Click Connect to Data Feed in the Read section of the _BUILD panel.
- This view displays a catalog of Data Feeds. You can search by name and use Filter By to help narrow your search.
- Select the Data Feed you would like to subscribe to, view the preview, and click SUBSCRIBE.
- The Dataflow adds the subscription and this Component is now available for further transformation.
A Data Feed Connector is a direct mirror of the upstream transform and as such will be impacted by that transform. This implies that any transforms built upon the Data Feed Connector may become invalid if the schema of the upstream transform changes.
Updated 7 months ago