Developer HubDocsSDKCommunityFree Trial

CI/CD with Ascend

Suggest Edits

In software engineering, CI/CD means Continuous Integration/Continuous Deployment. Its purpose is to provide a process that manages and integrates incremental code changes made by developers to software deliverables, and automate the testing and deployment of those deliverables to one or more runtime environments.

Prerequisites

In order to fully understand how CI/CD is implemented with Ascend, one should meet the following prerequisites:

Ascend Software Development Kit (SDK)

In Ascend, the CI/CD concept relies on a strong understanding of the Ascend Software Development Kit (SDK). The SDK enables the following capabilities for CI/CD:

  1. Building and downloading Python code representations of Data Services, Dataflows, and Components in Ascend. These artifacts are represented as Python classes that can be applied to Ascend to update the state of deployed pipelines on the platform. Additionally, the SDK downloads other related artifacts, such as user-supplied Python or SQL code from Transforms.

  2. Running tests on specific blocks of code, primarily on user-written code.

By leveraging the Ascend SDK, developers can automate the building and deployment of Ascend artifacts while ensuring the integrity of their code through testing.

Ascend Development Workflow

Ascend offers two types of development patterns: UI-first and code-first.

  • UI-first: Development is UI-driven: The concept is to create data pipelines primarily by using the UI.
  • Code-first: Development is code-driven: Developers write their code using Python and integrate it in with their own CI/CD tools. The changes are then uploaded to Ascend using the SDK and the "high-level API". Every modification happens in the code, there is no UI development.

📘

It is common to start with UI-first development, and then download/generate code and switch to a code-driven workflow.

Best Practices to Apply CI/CD to Ascend

Configuration

Every CI/CD platform supports the concept of a flexible build “action”, “step” or “task”, using slightly different terminology, depending on the platform. A typical work task/chore for software engineers setting up automated build/deployment processes and pipelines is the following:

  • Configure CI/CD platform actions/steps/tasks that call Python scripts to automate deployment
  • Sequence the Python scripts that Ascend generates into the build processes using the platform’s actions/steps/tasks

Sample Workflow

The sample workflow involves developing in the Ascend UI, downloading the pipeline code, running automated tests, and merging the code from feature branches to the main and production branches after passing tests and code reviews. Python scripts are used to apply changes to the Ascend stage and production environments during deployment.

Develop

  • Start by configuring Data Services, Connections, Dataflows, and Permissions within Ascend's UI. Then, use the SDK to download the necessary pipeline code for CI/CD. Adjust this code to align with your specific build processes and testing requirements.

Build and Test

  • Commit the customized code to a development branch in your source control system. Once committed, execute automated tests to verify the integrity of the code.

Review

  • Should the tests pass, initiate a pull request targeting the main development branch. This ensures the modifications undergo a thorough review before integration.

Merge and Promote

  • Upon approval of the code review, merge the changes to the main branch. Then, merge the main branch to the staging branch to kick off staging tests and deployment.

Deploy

  • If all staging tests pass and reviews are positive, merge these changes to the production branch. Implement updates to the Ascend production environment using Python scripts. Repeat this process as needed to maintain and update your setup.

To sum up, implementing CI/CD with Ascend involves using the Ascend SDK with Python and Ascend's UI to adopt either a code-first or UI-first workflow. The process includes stages of development, testing, review, merge, promotion, and deployment. Continuous improvement is key in CI/CD, so iterating this process ensures your software remains updated and of high quality. Understanding your CI/CD tools, source control systems, and the Ascend SDK is vital for optimizing the Ascend CI/CD pipeline.

Updated 5 months ago

More Reading