Ascend Python SDK

Installation & Usage

Requirements: Python 3.8+

pip install

You can install the package directly via pip.

pip install ascend-io-sdk

Then import the package:

import ascend

Getting Started

1. Follow the installation procedure
2. Create credentials (access key + secret key) in the Ascend UI.
3. Instantiate a client by passing in credentials to the Client.

from ascend.sdk.client import Client
from ascend.protos.api.api_pb2 import DataService

client = Client(hostname="trial.ascend.io", access_key="youraccesskey", secret_key="yoursecretkey")
client.create_data_service(DataService(id="demos", name="demos", description="test data service"))

Authorization

Credentials can be passed in to the client directly with the keyword arguments of access_key or secret_key as in the following:

from ascend.sdk.client import Client

client = Client(hostname="trial.ascend.io", access_key="youraccesskey", secret_key="yoursecretkey")

Alternatively, the client can read credentials from a config file, located in ~/.ascend/credentials. The file should have the following structure:

[trial]
ascend_access_key_id=<your_key_id_here>
ascend_secret_access_key=<your_secret_key_here>

where the [trial] should be replaced with your Ascend subdomain, [subdomain].

Then, constructing the client, you do not need to pass in the credentials directly:

from ascend.sdk.client import Client

client = Client(hostname="trial.ascend.io")

HTTP Proxy and Custom CA Certificate(s) Support

The Ascend SDK supports custom HTTP proxies and/or custom CA certificates via environment variables.

Set the environment variable(s) HTTPS_PROXY (or HTTP_PROXY) to your proxy's URL/hostname/port to enable proxy support.

Set the environment variable REQUESTS_CA_BUNDLE with the path to your CA certificate bundle (file, not directory), to specify custom certificates.
Note that client certificate validation is currently not supported.

Certificate bundles must be in PEM format, may contain more than one certificate, and are typically suffixed with file extensions such as .pem, .cer or .crt.

For example, to enable an HTTP/S proxy and custom certificate bundle on a Unix shell:

export HTTPS_PROXY=http://myproxy:8080
export REQUESTS_CA_BUNDLE=/path/to/my_ca_cert_bundle.cer

On Windows, replace export with set and ensure the path to the REQUESTS_CA_BUNDLE is defined in Windows path format.

Lower and Higher-Level SDK APIs

The SDK Python API comes in 2 flavors -

1. A low-level CRUD API to access, create, update, and delete resources in Ascend.
2. A high-level declarative API to declare the desired end-state of Dataflows and DataServices.

The use-cases for 1.) and 2.) are different, and complementary to each other.

• The low-level API is useful for iterating on a Dataflow, and accessing detailed information about components such as schema, records, partitions, lineage, etc. Resources in this API are defined as protobuf objects, and the CRUD operations are available as instance methods on the SDK Client.
• The high-level API is useful when you want to export and commit your Dataflow or DataService to source-control, or have it be part of a CICD pipeline. Resources in this API are defined as Dataflow resource definition objects. Utility classes such as DataflowApplier or DataServiceApplier are used to apply/sync these definitions with your account. Under the hood, the high-level API is implemented on top of the low-level API.

Getting started with Protobuf classes

The Ascend model classes and types are all generated from Google's protobufs. For developers that are used to protobufs, these classes should feel familiar. For developers new to Protobuf classes, however, this section provides a brief overview.

Constructing an Object

Attributes to an object can be passed into the constructor with keyword syntax, like the following:

from ascend.protos.api.api_pb2 import DataService

ds = DataService(id="demos", name="demos", description="test data service")

Alternatively, fields can be set directly through assignment:

from ascend.protos.api.api_pb2 import DataService

ds = DataService()
ds.id = "demos"
ds.name = "demos"
ds.description = "test data service"

Nested Types

Fields for a nested type can be nested in the constructor, as so:

from ascend.protos.external.external_pb2 import Aws, Credentials

b = Aws.S3.Bucket(name="mybucket", credential_id=Credentials.Id(value="id-3"))

However, it is not allowed to assign the nested field directly:

b = Aws.S3.Bucket()
b.credential_id = Credentials.Id(value="id-3") # WRONG!

In this example, to set credential_id, either set the value directly or use CopyFrom:

b = Aws.S3.Bucket()
b.credential_id.value = "id-3"
b.credential_id.CopyFrom(Credentials.Id(value="id-3"))

For more information on setting nested types, please see the Protobuf Python reference on embedded messages.

Working with Lists

Working with lists is similar to working with nested types, in that the list is already instantiated.

from ascend.protos.api.api_pb2 import Transform

t = Transform()
t.inputs.append(Transform.Input(uuid="fake-uuid")) # Append one value
t.inputs.extend([Transform.Input(uuid="fake-uuid2"), Transform.Input(uuid="fake-uuid3")]) # Append a whole list
len(t.inputs) # returns 3

For more information on working with lists, please see the Protobuf Python reference on repeated fields and repeated message fields.

📘

Review the detailed SDK Client Reference for complete client API documentation of the low-level imperative API. Or the Dataflow