Transforms

Creating and Updating Transforms

An Ascend Transform is created and connected to a Read Connectors , Data Feeds, or another Transform component. A Transform component performs operations like cleaning, filtering, joining, and/or aggregating across data sets. These operations are written in SQL, PySpark, and Scala / Java .

Metadata Columns

Transforms have access to the metadata of their inputs, exposed through metadata columns.

For SQL Transforms, the metadata column name is referenced directly in the SQL statement -- it will be automatically detected and included.

For PySpark Transforms, as well as Scala / Java Transforms, the user-supplied function must implement an additional method to request the specified metadata column in the inputs list. Please refer to the PySpark Transforms or Scala & Java Transforms documentation pages for the interfaces.

Supported Metadata Columns:

Column Name

Description

__ASCEND__URL

The full original filename of the input partition. This column must be used directly downstream of a Read Connector and requires the transformation to be a 'mapping' transform (see partitioning strategies for more detail). In all other cases, an attempt to use this column will error.

Updated 11 days ago


Transforms


Creating and Updating Transforms

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.