Queryable Dataflows
Combining the Interactivity of Warehouses with the Scale of Pipelines
Queryable Dataflows
Using Queryable dataflows all stages of all Dataflows are queryable without switching tools or disrupting your development process.
Interactive Query Editor
Figure 1
The figure 1 above shows the interactive query editor that lets you interact with Connectors and Transforms in Dataflows, as well as Data Feeds broadcasting from other Data Services, as though they are read-only tables in a SQL database. This embedded editor has built-in auto-completion of SQL keywords, as well as table and column names, for writing queries faster.
In the left part of the screen, in the Schema menu, there is a summary of all the elements of the examined Dataflow. For each block you can check fields and data types. Clicking the eye icon next to an element name gives you the possibility to run a sample query on the desired Component's data.
On the right side of the screen you can easily submit queries and get results. Queries can also be submitted asynchronously; you can either wait for the query to finish, start a new query, or navigate back through the list of previous queries and review past results without rerunning the query.
Entering a query
Click the Interactive Query icon in the upper left-side of the screen. Enter your SQL query in the editor and click RUN when finished.
Figure 2
Select a cluster pool to run query
For Gen 2 Spark Dataplane, you can select a cluster pool to run query, if not selected, the query will run on the default cluster pool of this Data Service.
Figure 3
Query to Transform stage
Once you obtained the desired result, it can be moved directly into a Dataflow Transform Component by clicking on the TO TRANSFORM button. The latter will open a new dialog box to finalize and create the new block as shown in Figure 3 below.
Figure 4
Share Query
Teams collaborating on the same Dataflow can even share their queries (and the resulting datasets) with each other. To share the query just click on the SHARE button and share the link obtained with your team or collaborators.
Figure 5
Query History
Ascend automatically saves the history of queries performed in a dataflow. By accessing the History section from the Action panel, it is possible to view, interact and filter the old queries by User and State. Once a query is selected, its result is also shown.
Figure 6
Updated 8 months ago