Ascend Developer Hub

Structured Data Lake

Access all data in Ascend using an S3-compatible API

Using the AWS S3 CLI to access Ascend Structured Data Lake

Because SDL is exposed as an S3 endpoint, you can use most any existing S3 client, including Amazon's aws s3 command line interface.
To get started, you will need to create an access key id/secret pair in Ascend.
You can use either a Service Account or a Developer Access Key.
Once you have the key id and secret, save them in your .aws/credentials file where the AWS CLI can find them:

๐Ÿ“˜

Set up a separate profile for using Ascend

Because the Ascend SDL S3 server is not the same as the Amazon AWS S3 server, we recommend you use a separate profile so that you can continue to use the [default] profile with Amazon AWS S3 server. In this example, we set up a profile called [trial].

[trial]
aws_access_key_id=YOUR_KEY_ID
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

The Ascend SDL S3 endpoint must be accessed using Virtual-Hosted Style Requests.
To ensure the AWS CLI is using this style of request, add the following your .aws/config .

[profile trial]
source_profile = trial
s3 =
    addressing_style = virtual

Now you can use Amazon's aws s3 CLI to access any read connector, transform, or data feed in Ascend. The Integrations Panel (see above) for each component shows you the S3 URL where
the data is stored.
Don't forget to specify the --endpoint_url (also given in the Integrations Panel) and the profile for each request.

> aws s3 ls --profile trial --endpoint-url https://s3.ascend.io s3://trial/Getting_Started_with_Ascend/IoT_Device_and_Weather_Analysis/K_Means_Cluster/

Updated 5 months ago

Structured Data Lake


Access all data in Ascend using an S3-compatible API

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.