Ascend Developer Hub

Read Connectors

Read from any data source with any format

Adding a Read Connector

  • click + ADD and then click Connector to bring up Connectors panel.

The Connectors panel displays a variety of Read Connectors supported by Ascend.

Select by clicking on the READ icon for the connector you want to connect to:

Configuring Read Connectors

While there are a variety of connectors, many share similar configuration settings.

Location

File/Object based Read Connectors such as S3 and GCS have a location setting comprised of

  • Bucket: The bucket name, such as ascend-io-sample-read-data.
  • Pattern: The pattern used to identify eligible files.
    • Match: Matches the pattern precisely character-for-character.
    • Glob: Glob applies a pattern matching algorithm.
    • Regex: Regex applies a pattern matching algorithm.

IAM access

Enter the Access Key and Secret Key for the IAM User which has write access to the path specified.

Test Connection

Use Test Connection to check whether all permissions are correctly configured.

Parsers & Schema

Data formats currently available are: Avro, Grok, JSON, Parquet and XSV. However, you can create your own parser functions or define a UDP (User Defined Parser) to process a file format.

Schema information will automatically be fetched for JSON, Parquet and XSV files with a header row.

Here's an example:

Here is an example of an XSV file:

Connector Refresh Schedule

Choose how often you want to check for new data in the configured source. Never - Manual refresh only; Every N Minutes; Hourly; Every N hours; Daily, Weekly; System Default.

Processing Priority (optional)

When resources are constrained, Processing Priority will be used to determine which components to schedule first. Higher priority numbers are scheduled before lower ones. Increasing the priority on a component also causes all its upstream components to be prioritized higher. Negative priorities can be used to postpone work until excess capacity becomes available.


Read Connectors


Read from any data source with any format

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.