CI/CD Guide
CI/CD
In software engineering, CI/CD means Continuous Integration / Continuous Deployment. Its purpose is to provide a process that manages and integrates incremental code changes made by developers to software deliverables, and automate the testing and deployment of those deliverables to one or more runtime environments.
CI/CD with Ascend
Prerequisites
In order to fully understand how CI/CD is implemented with Ascend, one should meet the following prerequisites:
- Python development experience
- Experience with the Ascend.io Software Development Kit (SDK):
- Experience with Source Control
- Git, SVN, etc.
- Experience with CI/CD tools
- ex. GitHub Actions, CircleCI, Jenkins, cloud vendor CI/CD such as AWS CodeBuild, etc.
Ascend Software Development Kit (SDK)
The concept of CI/CD (in Ascend) relies on a solid understanding of the Ascend Software Development Kit (SDK).
In the context of CI/CD, the SDK makes it possible to:
- build and download Data Services, Dataflows and Components in the form of Python code
- Ascend represents these artifacts as Python classes which can then be "applied" to Ascend to update the state of pipelines deployed on the platform
- Other dependent artifacts are downloaded along with the generated Python classes, such as
user-supplied Python or SQL code from Transforms (for example)
- run tests on specific blocks of code - mainly, code which is written by the user
Ascend Development Workflow
Ascend offers two types of development patterns: UI-first and code-first.
- UI-first: Development is UI-driven.
- The concept is to create data pipelines primarily by using the UI.
- Code-first: Development is code-driven.
- Developers write their code using Python and integrate it in with their own CI/CD tools. The changes are then uploaded to Ascend using the SDK and the "high-level API". Every modification happens in the code, there is no UI development.
It is common to start with UI-first development, and then download/generate code and switch to a code-driven workflow.
Best Practices to Apply CI/CD to Ascend
Configuration
Every CI/CD platform supports the concept of a flexible build “action”, “step” or “task”, using slightly different terminology, depending on the platform. A typical work task/chore for software engineers setting up automated build/deployment processes and pipelines is the following:
- Configure CI/CD platform actions/steps/tasks that call Python scripts to automate deployment
- Sequence the Python scripts that Ascend generates into the build processes using the platform’s actions/steps/tasks
Sample Workflow
- Start by developing in the UI in Ascend
- Download the developed pipeline code using the SDK
- Build/configure around the downloaded code to facilitate desired build process and tests
- Commit the code to source control feature/specific dev branch
- Run automated tests on feature/specific dev branch
- Submit a pull request/request for code review, requesting to merge to main dev branch
- If tests pass and pull request/code review approved, merge to main dev branch
- Merge (automated or manual) from main branch to stage branch to run staging tests and deploy to stage
- Deploying to stage means to run Python scripts that to “apply” changes to Ascend stage environment
- If tests pass, QA/testing passes in stage, release is approved, then merge from stage to production branch
- Merging to production runs Python scripts to “apply” to Ascend production environment
- Repeat!
Updated 12 months ago