Read Connector
After you have an Ascend Read Connection set up, then you continue to set up the Read Connector in your dataflow.
Create New Read Connector
Now we can use the established connection to create a Read Connector to ingest the data from our Kinesis Data Stream. To create a Read Connector:
- Click
Use
on the connection we created. You should now see a list of data streams associated with that connection (Figure 1). - Click
Select
on the data stream you want to use (Figure 2).
Connector Configuration
- BROWSE AND SELECT DATA: Click this button (Figure 3) to view all data streams associated with your Kinesis connection and select which stream you want to ingest data from.
- Configure details about your data stream (Figure 4):
- Stream Name (required): The name of your stream as it appears on your AWS Kinesis dashboard.
- Delay Second for Each Call: The delay, in seconds, between each call to the API to avoid hitting the rate limit
- Starting Position:
- At Timestamp: Start reading data from the position denoted by a time stamp entered in the Starting Date field
- Latest: Start reading just after the most recent record in the shard
- Oldest: Gets all records in data stream
Generate Schema
Once you click on the GENERATE SCHEMA button, the parser will create a schema and a data preview will be populated as in Figure 5 below.
- Add schema column: Add a custom column to the generated schema
Refresh Schedule
The refresh schedule (Figure 6) specifies how often Ascend checks the data location to see if there's new data. Ascend will automatically kick off the corresponding big data jobs once new or updated data is discovered.
Component Pausing
Update the status of the read connector by marking it either Running to mark it active or Paused to pause the connector from running.
Processing Priority (optional)
When resources are constrained, Processing Priority will be used to determine which components to schedule first.
Higher priority numbers are scheduled before lower ones. Increasing the priority on a component also causes all its upstream components to be prioritized higher. Negative priorities can be used to postpone work until excess capacity becomes available.
Updated over 2 years ago