Ascend Developer Hub

Dataflows

Dataflows contain multiple resources. These resources include Read Connectors, Transforms, Write Connectors, Data Feeds, and Groups.

What is a Dataflow?

Dataflows are continuously running data pipelines. You design Dataflow with a combination of declarative configurations, SQL, and Python. As you design Dataflows, you are creating a blueprint that is sent to the Ascend Dataflow Control Plane to be automated.

What belongs to a Dataflow?

Dataflows have multiple child objects:

  • Read Connectors: Read Connectors are how Ascend connects to and synchronizes data coming into a Dataflow. Ascend supports many out-of-the-box connectors, as well as a Custom Read Connectors framework for writing your own.
  • Transforms: Transforms are how data is modified within Ascend. Transforms can be written in either SQL or PySpark.
  • Write Connectors: Write Connectors are how Ascend connects to and synchronizes data coming out of a Dataflow.
  • Data Feeds: Data Feeds are how you connect multiple Dataflows together, as well as, expose "data as an API" to external systems such as Jupyter, Tableau, and Zeppelin.

Creating a Dataflow

Select a Data Service

  • Select the Data Service dropdown at the top left corner of your browser, and select the Data Service where you would like to create your new Dataflow.

Create a new Dataflow

  • Select the Dataflow dropdown next to the Data Service dropdown, and then click on + Create a new Dataflow.

This will bring up a panel where you give a name to this new Dataflow (required). You can also enter more details to describe your Dataflow (optional).

Click the Create button and a new dataflow is generated in Ascend!

Advanced Dataflow Topics

Once you've set up your initial Dataflow(s), we suggest coming back here to explore some advanced Dataflow topics, such as:


What's Next

Now let's make your dataflow interesting by adding some components. Common next actions to take from here include:

Read Connectors
Transforms
Data Feeds

Dataflows


Dataflows contain multiple resources. These resources include Read Connectors, Transforms, Write Connectors, Data Feeds, and Groups.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.