CI/CD

In software engineering, CI/CD means Continuous Integration / Continuous Deployment. Its purpose is to provide a process that manages and integrates incremental code changes made by developers to software deliverables, and automate the testing and deployment of those deliverables to one or more runtime environments.

CI/CD with Ascend

Prerequisites

In order to fully understand how CI/CD is implemented with Ascend, one should meet the following prerequisites:

• Python development experience
• Experience with the Ascend.io Software Development Kit (SDK):
  • SDK Developer Docs
  • High level SDK overview video
  • Hour 2 of initial training video
• Experience with Source Control
  • Git, SVN, etc.
• Experience with CI/CD tools
  • ex. GitHub Actions, CircleCI, Jenkins, cloud vendor CI/CD such as AWS CodeBuild, etc.

Ascend Software Development Kit (SDK)

The concept of CI/CD (in Ascend) relies on a solid understanding of the Ascend Software Development Kit (SDK).

In the context of CI/CD, the SDK makes it possible to:

• build and download Data Services, Dataflows and Components in the form of Python code
  • Ascend represents these artifacts as Python classes which can then be "applied" to Ascend to update the state of pipelines deployed on the platform
  • Other dependent artifacts are downloaded along with the generated Python classes, such as
    user-supplied Python or SQL code from Transforms (for example)
• run tests on specific blocks of code - mainly, code which is written by the user

Ascend Development Workflow

Ascend offers two types of development patterns: UI-first and code-first.

• UI-first: Development is UI-driven.
  • The concept is to create data pipelines primarily by using the UI.
• Code-first: Development is code-driven.
  • Developers write their code using Python and integrate it in with their own CI/CD tools. The changes are then uploaded to Ascend using the SDK and the "high-level API". Every modification happens in the code, there is no UI development.

📘

It is common to start with UI-first development, and then download/generate code and switch to a code-driven workflow.

Best Practices to Apply CI/CD to Ascend

Configuration

Every CI/CD platform supports the concept of a flexible build “action”, “step” or “task”, using slightly different terminology, depending on the platform. A typical work task/chore for software engineers setting up automated build/deployment processes and pipelines is the following:

• Configure CI/CD platform actions/steps/tasks that call Python scripts to automate deployment
• Sequence the Python scripts that Ascend generates into the build processes using the platform’s actions/steps/tasks

Sample Workflow

1. Start by developing in the UI in Ascend
2. Download the developed pipeline code using the SDK
3. Build/configure around the downloaded code to facilitate desired build process and tests
4. Commit the code to source control feature/specific dev branch
5. Run automated tests on feature/specific dev branch
6. Submit a pull request/request for code review, requesting to merge to main dev branch
7. If tests pass and pull request/code review approved, merge to main dev branch
8. Merge (automated or manual) from main branch to stage branch to run staging tests and deploy to stage
9. Deploying to stage means to run Python scripts that to “apply” changes to Ascend stage environment
10. If tests pass, QA/testing passes in stage, release is approved, then merge from stage to production branch
11. Merging to production runs Python scripts to “apply” to Ascend production environment
12. Repeat!