The Ascend Structured Data Lake is a way to retrieve the records from any Component in Ascend as Snappy-compressed Parquet files.
Retrieving records through this file-based interface enables:
- High performance
- Bulk retrieval
- Well-known interface that many tools are already suitable to integrate with (such as Spark processing engines and notebooks)
Regardless of which cloud provider Ascend has been deployed to, the Structured Data Lake provides the endpoint with S3 compatibility, transparently serving up files from the underlying cloud's object store.
All environments use the endpoint
The S3 Protocol URI follows the organization within Ascend:
s3://<Ascend Subdomain>/<Data Service Id>/<Dataflow ID>/<Component ID>
Each component in the UI has a section called 'Data Integrations' which includes the full S3 Protocol URI.
Because SDL is exposed as an S3 endpoint, you can use most any existing S3 client, including Amazon's
aws s3 Command Line Interface (AWS CLI).
- Create an Access Key ID + Secret pair in Ascend.
You can use either a Service Account or a Developer Access Key.
- Save the Access Key ID and Secret in your
~/.aws/credentialsfile, where the AWS CLI can find them:
[trial] aws_access_key_id=YOUR_KEY_ID aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
Set up a separate profile for using Ascend
Because the Ascend SDL S3 server is not the same as the Amazon AWS S3 server, we recommend you use a separate profile so that you can continue to use the
[default]profile with Amazon AWS S3 server. In this example, we set up a profile called
- Set the AWS CLI addressing style to virtual by adding the following configuration to
~/.aws/config. The Ascend S3 endpoint must be accessed using Virtual-Hosted Style Requests.
[profile trial] source_profile = trial s3 = addressing_style = virtual
- Make a S3 requests (such a
cp, etc) through the
aws s3CLI to access any read connector, transform, or data feed in Ascend. Specify the
profilefor each request.
> aws s3 ls --profile trial --endpoint-url https://s3.ascend.io s3://trial/Getting_Started_with_Ascend/IoT_Device_and_Weather_Analysis/K_Means_Cluster/
Updated 11 days ago