Requirements: Python 3.8+
You can install the package directly via
pip install ascend-io-sdk
Then import the package:
- Follow the installation procedure
- Create credentials (access key + secret key) in the Ascend UI.
- Instantiate a client by passing in credentials to the
from ascend.sdk.client import Client from ascend.protos.api.api_pb2 import DataService client = Client(hostname="trial.ascend.io", access_key="youraccesskey", secret_key="yoursecretkey") client.create_data_service(DataService(id="demos", name="demos", description="test data service"))
Credentials can be passed in to the client directly with the keyword arguments of
secret_key as in the following:
from ascend.sdk.client import Client client = Client(hostname="trial.ascend.io", access_key="youraccesskey", secret_key="yoursecretkey")
Alternatively, the client can read credentials from a config file, located in
~/.ascend/credentials. The file should have the following structure:
[trial] ascend_access_key_id=<your_key_id_here> ascend_secret_access_key=<your_secret_key_here>
[trial] should be replaced with your Ascend subdomain,
Then, constructing the client, you do not need to pass in the credentials directly:
from ascend.sdk.client import Client client = Client(hostname="trial.ascend.io")
The SDK Python API comes in 2 flavors -
- A low-level CRUD API to access, create, update, and delete resources in Ascend.
- A high-level declarative API to declare the desired end-state of Dataflows and DataServices.
The use-cases for 1.) and 2.) are different, and complementary to each other.
- The low-level API is useful for iterating on a Dataflow, and accessing detailed information about components such as schema, records, partitions, lineage, etc. Resources in this API are defined as protobuf objects, and the CRUD operations are available as instance methods on the SDK Client.
- The high-level API is useful when you want to export and commit your Dataflow or DataService to source-control, or have it be part of a CICD pipeline. Resources in this API are defined as Dataflow resource definition objects. Utility classes such as
DataServiceApplierare used to apply/sync these definitions with your account. Under the hood, the high-level API is implemented on top of the low-level API.
The Ascend model classes and types are all generated from Google's protobufs. For developers that are used to protobufs, these classes should feel familiar. For developers new to Protobuf classes, however, this section provides a brief overview.
Attributes to an object can be passed into the constructor with keyword syntax, like the following:
from ascend.protos.api.api_pb2 import DataService ds = DataService(id="demos", name="demos", description="test data service")
Alternatively, fields can be set directly through assignment:
from ascend.protos.api.api_pb2 import DataService ds = DataService() ds.id = "demos" ds.name = "demos" ds.description = "test data service"
Fields for a nested type can be nested in the constructor, as so:
from ascend.protos.external.external_pb2 import Aws, Credentials b = Aws.S3.Bucket(name="mybucket", credential_id=Credentials.Id(value="id-3"))
However, it is not allowed to assign the nested field directly:
b = Aws.S3.Bucket() b.credential_id = Credentials.Id(value="id-3") # WRONG!
In this example, to set
credential_id, either set the value directly or use
b = Aws.S3.Bucket() b.credential_id.value = "id-3" b.credential_id.CopyFrom(Credentials.Id(value="id-3"))
For more information on setting nested types, please see the Protobuf Python reference on embedded messages.
Working with lists is similar to working with nested types, in that the list is already instantiated.
from ascend.protos.api.api_pb2 import Transform t = Transform() t.inputs.append(Transform.Input(uuid="fake-uuid")) # Append one value t.inputs.extend([Transform.Input(uuid="fake-uuid2"), Transform.Input(uuid="fake-uuid3")]) # Append a whole list len(t.inputs) # returns 3
Review the detailed SDK Client Reference for complete client API documentation of the low-level imperative API. Or the Dataflow
Updated 3 months ago