An Ascend Transform is created and connected to a Read Connectors , Data Feeds, or another Transform component. A Transform component performs operations like cleaning, filtering, joining, and/or aggregating across data sets. These operations are written in SQL, PySpark, and Scala / Java .
Transforms have access to the metadata of their inputs, exposed through metadata columns.
For SQL Transforms, the metadata column name is referenced directly in the SQL statement -- it will be automatically detected and included.
For PySpark Transforms, as well as Scala / Java Transforms, the user-supplied function must implement an additional method to request the specified metadata column in the inputs list. Please refer to the PySpark Transforms or Scala & Java Transforms documentation pages for the interfaces.
The full original filename of the input partition. This column must be used directly downstream of a Read Connector and requires the transformation to be a 'mapping' transform (see partitioning strategies for more detail). In all other cases, an attempt to use this column will error.
Updated 11 days ago