Amazon Kinesis

Connect Ascend to your Amazon Kinesis data stream. The Amazon Kinesis connection allows Ascend to read and transform data.

Prerequisites

  • Access credentials
  • Data stream on AWS Kinesis

Connection Properties

The following table describes the fields available when creating a new Amazon Kinesis Connection. Create a Connection using the information below and these step-by-step instructions.

FieldRequiredDescription
Access TypeRequiredThe type of connection is Read-Only for Amazon Kinesis.
Connection NameRequiredInput your desired name.
Region NameRequiredThe region name, such as us-west-1.
IAM Role ARNOptionalOptional IAM role to assume.
Requires CredentialsRequiredAutomatically selected. Credentials are required for Kinesis Data Streams

IAM Role Access

In order to establish a connection and read in records from a Kinesis Data Stream, the IAM user must have access to perform the following actions on the data stream:

  • kinesis:GetShardIterator
  • kinesis:GetRecords
  • kinesis:ListStreams
  • kinesis:ListShards
  • kinesis:DescribeStreamSummary

Here's an example policy configuration for an IAM user to create a read connector for the data stream
kinesis:us-west-1:1234567890:stream/StockTradeData:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kinesis:GetShardIterator",
                "kinesis:GetRecords"
            ],
            "Resource": "arn:aws:kinesis:us-west-1:1234567890:stream/StockTradeData"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "kinesis:ListStreams",
                "kinesis:ListShards"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "kinesis:DescribeStreamSummary",
            "Resource": "arn:aws:kinesis:us-west-1:1234567890:stream/StockTradeData"
        }
    ]
}

Credential Properties

The following table describes the fields available when creating a new Amazon Kinesis credential.

Field NameRequiredDescription
Credential NameRequiredThe name to identify this credential with. This credential will be available as a selection for future use.
Credential TypeRequiredThis field will automatically populate with Amazon Kinesis Data Stream.
AWS Access Key IDRequiredThe AWS Access Key associated with the AWS account.
AWS Secret Access KeyRequiredThe Secret Key associated with the above mentioned Access Key.

Read Connector Properties

The following table describes the fields available when creating a new Amazon Kinesis Read Connector.

Field NameRequiredDescription
Stream NameRequiredThe name of your stream as it appears on your AWS Kinesis dashboard.
Delay Second for Each CallOptionalThe delay, in seconds, between each call to the API to avoid hitting the rate limit. The default delay is 1 second.
Starting PositionOptionalTimestamp: Start reading data from the position denoted by a time stamp entered in the Starting Date field

Latest: Start reading just after the most recent record in the shard

Oldest: Gets all records in data stream