- click + ADD and then click Connector to bring up Connectors panel.
The Connectors panel displays a variety of Read Connectors supported by Ascend.
Select by clicking on the READ icon for the connector you want to connect to:
While there are a variety of connectors, many share similar configuration settings.
- Bucket: The bucket name, such as
- Pattern: The pattern used to identify eligible files.
Enter the Access Key and Secret Key for the IAM User which has write access to the path specified.
Use Test Connection to check whether all permissions are correctly configured.
Data formats currently available are: Avro, Grok, JSON, Parquet and XSV. However, you can create your own parser functions or define a UDP (User Defined Parser) to process a file format.
Schema information will automatically be fetched for JSON, Parquet and XSV files with a header row.
Here's an example:
Here is an example of an XSV file:
Choose how often you want to check for new data in the configured source. Never - Manual refresh only; Every N Minutes; Hourly; Every N hours; Daily, Weekly; System Default.
When resources are constrained, Processing Priority will be used to determine which components to schedule first. Higher priority numbers are scheduled before lower ones. Increasing the priority on a component also causes all its upstream components to be prioritized higher. Negative priorities can be used to postpone work until excess capacity becomes available.
Updated 11 months ago