In software engineering, CI/CD means Continuous Integration / Continuous Deployment. Its purpose is to provide a process that manages and integrates incremental code changes made by developers to software deliverables, and automate the testing and deployment of those deliverables to one or more runtime environments.
In order to fully understand how CI/CD is implemented with Ascend, one should meet the following prerequisites:
- Python development experience
- Experience with the Ascend.io Software Development Kit (SDK):
- Experience with Source Control
- Git, SVN, etc.
- Experience with CI/CD tools
- ex. GitHub Actions, CircleCI, Jenkins, cloud vendor CI/CD such as AWS CodeBuild, etc.
The concept of CI/CD (in Ascend) relies on a solid understanding of the Ascend Software Development Kit (SDK).
In the context of CI/CD, the SDK makes it possible to:
- build and download Data Services, Dataflows and Components in the form of Python code
- Ascend represents these artifacts as Python classes which can then be "applied" to Ascend to update the state of pipelines deployed on the platform
- Other dependent artifacts are downloaded along with the generated Python classes, such as
user-supplied Python or SQL code from Transforms (for example)
- run tests on specific blocks of code - mainly, code which is written by the user
Ascend offers two types of development patterns: UI-first and code-first.
- UI-first: Development is UI-driven.
- The concept is to create data pipelines primarily by using the UI.
- Code-first: Development is code-driven.
- Developers write their code using Python and integrate it in with their own CI/CD tools. The changes are then uploaded to Ascend using the SDK and the "high-level API". Every modification happens in the code, there is no UI development.
It is common to start with UI-first development, and then download/generate code and switch to a code-driven workflow.
Every CI/CD platform supports the concept of a flexible build “action”, “step” or “task”, using slightly different terminology, depending on the platform. A typical work task/chore for software engineers setting up automated build/deployment processes and pipelines is the following:
- Configure CI/CD platform actions/steps/tasks that call Python scripts to automate deployment
- Sequence the Python scripts that Ascend generates into the build processes using the platform’s actions/steps/tasks
- Start by developing in the UI in Ascend
- Download the developed pipeline code using the SDK
- Build/configure around the downloaded code to facilitate desired build process and tests
- Commit the code to source control feature/specific dev branch
- Run automated tests on feature/specific dev branch
- Submit a pull request/request for code review, requesting to merge to main dev branch
- If tests pass and pull request/code review approved, merge to main dev branch
- Merge (automated or manual) from main branch to stage branch to run staging tests and deploy to stage
- Deploying to stage means to run Python scripts that to “apply” changes to Ascend stage environment
- If tests pass, QA/testing passes in stage, release is approved, then merge from stage to production branch
- Merging to production runs Python scripts to “apply” to Ascend production environment
Updated 12 months ago