Adding a Write Connector

• click +ADD and then click Connector to bring up Connectors panel.

The Connectors panel displays a variety of Write Connectors supported by Ascend.

Select by clicking on the WRITE icon for the connector you want to connect to:

This will bring up the CREATE NEW WRITE CONNECTOR panel where you can select the upstream for the write connector.

Configuring Write Connectors

While there are a variety of connectors, many share similar configuration settings.

Location

File/Object based Write Connectors such as S3 and GCS have a location setting comprised of:

• Bucket: The S3 bucket name, e.g. my_data_bucket. Do not put in any folder information in this box.
• Object Prefix: The S3 folders hierarchical prefix, e.g. good_data/my_awesome_table. Do not include any leading forward slashes.
• Partition Folder Pattern: The folder pattern that will be generated by Ascend based on the values from the upstream partition column, e.g. you can use ${at_hour_ts(yyyy-MM-dd/HH)} where column at_hour_ts is a partition column from the upstream transform.

IAM or Credential Configuration

Enter the Access Key and Secret Key (or other credentials) for the IAM User which has write access to the path specified.

Testing Connection

Use Test Connection to check whether all permissions are correctly configured.

Selecting a formatter

Ascend supports 3 popular data formats for S3 and GCS write connectors: JSON, Parquet and XSV.

XSV formatter: Supports 3 different delimiters and 9 different line terminators and allows specifying whether a Header Row should be included. The XSV generated is RFC4180 compliant.

🚧

Important

The XSV Formatter will NOT replace newline characters within values. Replace newline characters in the upstream transform if you require XSV files to contain only single line records.

JSON formatter: Will generate a file where each line in the file is a valid JSON object representing one data row from upstream.

🚧

Important

The JSON Formatter will automatically replace new line characters in column values to \n so that the output JSON file is guaranteed to have single line records.

Parquet formatter: Will apply snappy compression automatically to the output files automatically.