Python SDK
Ascend Python SDK
Installation & Usage
Requirements: Python 3.8+
pip install
You can install the package directly via pip
.
pip install ascend-io-sdk
Then import the package:
import ascend
Getting Started
- Follow the installation procedure
- Create credentials (access key + secret key) in the Ascend UI.
- Instantiate a client by passing in credentials to the
Client
.
from ascend.sdk.client import Client
from ascend.protos.api.api_pb2 import DataService
client = Client(hostname="trial.ascend.io", access_key="youraccesskey", secret_key="yoursecretkey")
client.create_data_service(DataService(id="demos", name="demos", description="test data service"))
Authorization
Credentials can be passed in to the client directly with the keyword arguments of access_key
or secret_key
as in the following:
from ascend.sdk.client import Client
client = Client(hostname="trial.ascend.io", access_key="youraccesskey", secret_key="yoursecretkey")
Alternatively, the client can read credentials from a config file, located in ~/.ascend/credentials
. The file should have the following structure:
[trial]
ascend_access_key_id=<your_key_id_here>
ascend_secret_access_key=<your_secret_key_here>
where the [trial]
should be replaced with your Ascend subdomain, [subdomain]
.
Then, constructing the client, you do not need to pass in the credentials directly:
from ascend.sdk.client import Client
client = Client(hostname="trial.ascend.io")
HTTP Proxy and Custom CA Certificate(s) Support
The Ascend SDK supports custom HTTP proxies and/or custom CA certificates via environment variables.
Set the environment variable(s) HTTPS_PROXY
(or HTTP_PROXY
) to your proxy's URL/hostname/port to enable proxy support.
Set the environment variable REQUESTS_CA_BUNDLE
with the path to your CA certificate bundle (file, not directory), to specify custom certificates.
Note that client certificate validation is currently not supported.
Certificate bundles must be in PEM format, may contain more than one certificate, and are typically suffixed with file extensions such as .pem
, .cer
or .crt
.
For example, to enable an HTTP/S proxy and custom certificate bundle on a Unix shell:
export HTTPS_PROXY=http://myproxy:8080
export REQUESTS_CA_BUNDLE=/path/to/my_ca_cert_bundle.cer
On Windows, replace export
with set
and ensure the path to the REQUESTS_CA_BUNDLE is defined in Windows path format.
Lower and Higher-Level SDK APIs
The SDK Python API comes in 2 flavors -
- A low-level CRUD API to access, create, update, and delete resources in Ascend.
- A high-level declarative API to declare the desired end-state of Dataflows and DataServices.
The use-cases for 1.) and 2.) are different, and complementary to each other.
- The low-level API is useful for iterating on a Dataflow, and accessing detailed information about components such as schema, records, partitions, lineage, etc. Resources in this API are defined as protobuf objects, and the CRUD operations are available as instance methods on the SDK Client.
- The high-level API is useful when you want to export and commit your Dataflow or DataService to source-control, or have it be part of a CICD pipeline. Resources in this API are defined as Dataflow resource definition objects. Utility classes such as
DataflowApplier
orDataServiceApplier
are used to apply/sync these definitions with your account. Under the hood, the high-level API is implemented on top of the low-level API.
Getting started with Protobuf classes
The Ascend model classes and types are all generated from Google's protobufs. For developers that are used to protobufs, these classes should feel familiar. For developers new to Protobuf classes, however, this section provides a brief overview.
Constructing an Object
Attributes to an object can be passed into the constructor with keyword syntax, like the following:
from ascend.protos.api.api_pb2 import DataService
ds = DataService(id="demos", name="demos", description="test data service")
Alternatively, fields can be set directly through assignment:
from ascend.protos.api.api_pb2 import DataService
ds = DataService()
ds.id = "demos"
ds.name = "demos"
ds.description = "test data service"
Nested Types
Fields for a nested type can be nested in the constructor, as so:
from ascend.protos.external.external_pb2 import Aws, Credentials
b = Aws.S3.Bucket(name="mybucket", credential_id=Credentials.Id(value="id-3"))
However, it is not allowed to assign the nested field directly:
b = Aws.S3.Bucket()
b.credential_id = Credentials.Id(value="id-3") # WRONG!
In this example, to set credential_id
, either set the value directly or use CopyFrom
:
b = Aws.S3.Bucket()
b.credential_id.value = "id-3"
b.credential_id.CopyFrom(Credentials.Id(value="id-3"))
For more information on setting nested types, please see the Protobuf Python reference on embedded messages.
Working with Lists
Working with lists is similar to working with nested types, in that the list is already instantiated.
from ascend.protos.api.api_pb2 import Transform
t = Transform()
t.inputs.append(Transform.Input(uuid="fake-uuid")) # Append one value
t.inputs.extend([Transform.Input(uuid="fake-uuid2"), Transform.Input(uuid="fake-uuid3")]) # Append a whole list
len(t.inputs) # returns 3
For more information on working with lists, please see the Protobuf Python reference on repeated fields and repeated message fields.
Review the detailed SDK Client Reference for complete client API documentation of the low-level imperative API. Or the Dataflow
Updated 9 months ago