- If you would like to use your own docker image in Ascend, make sure to go through the previous article on "preparing a custom image", to have it properly set up.
First, find Runtime Settings options in a transform:
- In Pyspark and SQL Transforms, it is nested under "Advanced Settings"
- In Scala Transform, it is marked as a required field in the creation flow.
In this dropdown, you'll find the following four options:
Use Data Service Image (default)
This is the default option for Transforms if nothing is specified.
Native Ascend Spark
Once selected, you may choose one of the standard Ascend Spark runtimes.
If you have registered custom labels at the site level, they can be selected here.
Container image URL
Enter an image URL (for example,
Transformer blocks have a caching method to store Docker images. These include the Custom Image one choses to use.
Due to this reason, Ascend suggests to change the Label every time the Image is updated in order to avoid using an old Image.
To make managing custom images easier in Ascend, you may declare all needed images in the "Site admin" settings and reference them later when building Dataflows.
Only site admins for your environment may operate on this. Reach out to your site admins to help you register needed images.
- Go to the Dashboard from the top left Ascend Icon
- Click on "Admin," then "Cluster Management."
- In the "Custom Docker Images" section, click "Add a new image."
- Give the image an easily referenceable name as a label, for example, "custom-image-with-Arrow-library."
- Enter image URL and required runtime.
- Hit "Create" to finish image registration.
Once complete, you may choose the registered image easily in both Data Service settings and Transforms.
By default, all Transforms in a Data Service runs on the same initial Ascend Spark image. If you would like to default to your custom Docker image, you may:
- Click "Data Service Settings", after hitting the small "Gear" icon in the upper section of the left panel.
- Navigate to "Container Images".
- Choose a default image to be used for this Data Service. Follow the table above to make sense of the options.
- Hit "Update", and all Transforms will default to use the new image.
Register an image from a private container/image repository
It is possible that your container images are not hosted in a public container/image repository, and may require additional credentials to pull images. In this case, please reach out to Ascend support (via Intercom or Slack). We will help you set up the configuration so those images can be pulled.
NOTE: Ascend.io currently only supports private container/image repositories that use standard
docker loginstyle authentication to obtain a long-lived authentication token. Ascend.io does not currently support container/image repositories such as Amazon ECR, which uses a non-standard authentication method.
Updated 3 months ago