Python SDK

Ascend Python SDK

Installation & Usage

Requirements: Python 3.8+

pip install

You can install the package directly via pip.

pip install ascend-io-sdk

Then import the package:

import ascend

Getting Started

  1. Follow the installation procedure
  2. Create credentials (access key + secret key) in the Ascend UI.
  3. Instantiate a client by passing in credentials to the Client.
from ascend.sdk.client import Client
from ascend.protos.api.api_pb2 import DataService

client = Client(hostname="trial.ascend.io", access_key="youraccesskey", secret_key="yoursecretkey")
client.create_data_service(DataService(id="demos", name="demos", description="test data service"))

Authorization

Credentials can be passed in to the client directly with the keyword arguments of access_key or secret_key as in the following:

from ascend.sdk.client import Client

client = Client(hostname="trial.ascend.io", access_key="youraccesskey", secret_key="yoursecretkey")

Alternatively, the client can read credentials from a config file, located in ~/.ascend/credentials. The file should have the following structure:

[trial]
ascend_access_key_id=<your_key_id_here>
ascend_secret_access_key=<your_secret_key_here>

where the [trial] should be replaced with your Ascend subdomain, [subdomain].

Then, constructing the client, you do not need to pass in the credentials directly:

from ascend.sdk.client import Client

client = Client(hostname="trial.ascend.io")

Lower and Higher-Level SDK APIs

The SDK Python API comes in 2 flavors -

  1. A low-level CRUD API to access, create, update, and delete resources in Ascend.
  2. A high-level declarative API to declare the desired end-state of Dataflows and DataServices.

The use-cases for 1.) and 2.) are different, and complementary to each other.

  • The low-level API is useful for iterating on a Dataflow, and accessing detailed information about components such as schema, records, partitions, lineage, etc. Resources in this API are defined as protobuf objects, and the CRUD operations are available as instance methods on the SDK Client.
  • The high-level API is useful when you want to export and commit your Dataflow or DataService to source-control, or have it be part of a CICD pipeline. Resources in this API are defined as Dataflow resource definition objects. Utility classes such as DataflowApplier or DataServiceApplier are used to apply/sync these definitions with your account. Under the hood, the high-level API is implemented on top of the low-level API.

Getting started with Protobuf classes

The Ascend model classes and types are all generated from Google's protobufs. For developers that are used to protobufs, these classes should feel familiar. For developers new to Protobuf classes, however, this section provides a brief overview.

Constructing an Object

Attributes to an object can be passed into the constructor with keyword syntax, like the following:

from ascend.protos.api.api_pb2 import DataService

ds = DataService(id="demos", name="demos", description="test data service")

Alternatively, fields can be set directly through assignment:

from ascend.protos.api.api_pb2 import DataService

ds = DataService()
ds.id = "demos"
ds.name = "demos"
ds.description = "test data service"

Nested Types

Fields for a nested type can be nested in the constructor, as so:

from ascend.protos.external.external_pb2 import Aws, Credentials

b = Aws.S3.Bucket(name="mybucket", credential_id=Credentials.Id(value="id-3"))

However, it is not allowed to assign the nested field directly:

b = Aws.S3.Bucket()
b.credential_id = Credentials.Id(value="id-3") # WRONG!

In this example, to set credential_id, either set the value directly or use CopyFrom:

b = Aws.S3.Bucket()
b.credential_id.value = "id-3"
b.credential_id.CopyFrom(Credentials.Id(value="id-3"))

For more information on setting nested types, please see the Protobuf Python reference on embedded messages.

Working with Lists

Working with lists is similar to working with nested types, in that the list is already instantiated.

from ascend.protos.api.api_pb2 import Transform

t = Transform()
t.inputs.append(Transform.Input(uuid="fake-uuid")) # Append one value
t.inputs.extend([Transform.Input(uuid="fake-uuid2"), Transform.Input(uuid="fake-uuid3")]) # Append a whole list
len(t.inputs) # returns 3

For more information on working with lists, please see the Protobuf Python reference on repeated fields and repeated message fields.

📘

Review the detailed SDK Client Reference for complete client API documentation of the low-level imperative API. Or the Dataflow


Did this page help you?