CI / CD · Conveyor

Reusable pipeline templates

The CI/CD engine of the Forge Platform. Reusable Azure DevOps templates so every service gets the same build, scan and deploy flow without duplicating pipeline code.

1 · What is Conveyor?

D
DRY
All common logic — Docker builds, Trivy scans, ECR pushes, Ledger updates — defined once. Application repos consume templates with ~15 lines of YAML.
S
Security embedded
Trivy scans (filesystem + image) are mandatory and non-negotiable. Immutable tags only. No ECR auto-create. OIDC authentication — no static keys.
P
Split architecture
YAML CI (build-only, in Git) + Classic Release Pipeline (promotion, ADO UI). Scripts in conveyor/scripts/ are version-controlled even though the Classic Release is not.

Open Conveyor in Azure DevOps ↗

2 · Repository structure

conveyor/
├── templates/                              # Reusable Azure DevOps YAML pipeline templates
│   ├── application-ci-pipeline.yml         # CI: build, Trivy scan, push to ECR
│   ├── application-cd-pipeline.yml         # CD: update Ledger manifest
│   ├── application-rollback-pipeline.yml   # Emergency rollback
│   ├── ec2-deploy-pipeline.yml             # EC2 deployment via SSM
│   └── languages/                          # Language-specific build steps
│       ├── dotnet.yml
│       └── python.yml
├── scripts/                                # Version-controlled CD scripts
│   ├── copy-artifact.sh                    # Cross-account ECR copy via crane
│   └── update-ledger.sh                    # Update image.tag in Ledger (optimistic locking)
└── docs/
    ├── PIPELINE_STRATEGY.md                # Full architecture rationale
    └── CLASSIC_RELEASE_SETUP_GUIDE.md      # Step-by-step ADO setup

3 · Pipeline templates

CI
application-ci-pipeline.yml

The build template. Every service uses this. Flow: checkout → Trivy filesystem scan → language build (dotnet/python) → Docker build (Build ID tag only) → Trivy image scan → ECR login → verify repo exists → push.

Parameters

ParameterDefaultDescription
imageName—ECR image name (required)
languagedockerdocker, dotnet, or python
environment—dev, qa, or prod-wealth
awsServiceConnectionawsADO service connection name
CD
application-cd-pipeline.yml

Updates image.tag in a Ledger Application manifest. Used by Classic Release Pipeline stages. Uses yq for safe YAML editing with optimistic locking and retry on push conflict.

EC2
ec2-deploy-pipeline.yml

Deploys to EC2 via SSM Run Command. Sends docker-compose.yml, runs forge-refresh-env, writes IMAGE_TAG, pulls and restarts containers. Targets EC2s by tag.

RB
application-rollback-pipeline.yml

Emergency rollback. Validates the target image exists in ECR (shows last 10 tags if not), then calls update-ledger.sh to revert image.tag.

4 · Scripts

Called by Classic Release Pipeline stages. Version-controlled in Git even though the Classic Release pipeline definition itself is not.

CP
copy-artifact.sh

Copies a Docker image from Dev ECR to another account's ECR using crane (registry-to-registry, no Docker daemon). Verifies digest after copy. Saves OIDC creds before assuming roles to avoid credential chaining issues.

./copy-artifact.sh <IMAGE_NAME> <IMAGE_TAG> <TARGET_ENV>
# Env vars: TARGET_ACCOUNT (required), SOURCE_ACCOUNT (default: dev)
UL
update-ledger.sh

Clones Ledger, updates image.tag in the specified manifest (prefers yq, falls back to sed), commits and pushes. Uses HTTPS + System.AccessToken on ADO agents. Implements optimistic locking — records remote SHA before editing, retries if another pipeline pushed concurrently.

./update-ledger.sh <IMAGE_NAME> <IMAGE_TAG> <TARGET_ENV> <MANIFEST_FILE> [RELEASE_NAME]

5 · How to use Conveyor

Application repos reference Conveyor as a repository resource. A typical CI pipeline is ~15 lines.

EKS service — CI only (Classic Release handles promotion)

trigger:
  branches:
    include: [main]

resources:
  repositories:
    - repository: conveyor
      type: git
      name: Forge/Conveyor

stages:
  - stage: BuildAndPush
    jobs:
      - template: templates/application-ci-pipeline.yml@conveyor
        parameters:
          imageName: "orders-api"
          language: "dotnet"
          environment: "dev"

A Classic Release Pipeline picks up the build artifact and promotes through environments. See Release Management for the full promotion flow.

EC2 service — CI + Deploy

stages:
  - stage: BuildAndPush
    jobs:
      - template: templates/application-ci-pipeline.yml@conveyor
        parameters:
          imageName: "dagster"
          language: "python"
          environment: "dev"

  - stage: Deploy
    dependsOn: BuildAndPush
    jobs:
      - template: templates/ec2-deploy-pipeline.yml@conveyor
        parameters:
          componentName: "dagster"
          ec2TagKey: "forge:dagster-enabled"
          environment: "dev"

6 · Authentication — OIDC hub-and-spoke

No static credentials anywhere. Conveyor uses OIDC + IAM role chaining.

AUTHENTICATION FLOW Azure DevOps OIDC token pipeline identity SECURITY-TOOLING (HUB) ConveyorExecutionRole wildcard subject trust Dev ConveyorDeployRole-dev QA ConveyorDeployRole-qa Prod-Wealth ConveyorDeployRole-prod-wealth
  1. Pipeline acquires OIDC token from Azure DevOps
  2. Assumes ConveyorExecutionRole in security-tooling (Hub) via wildcard subject trust
  3. Assumes environment-specific ConveyorDeployRole-<env> in target account (Spoke)

7 · Enforced policies

PolicyEnforcement
Immutable tagsBuild.BuildId only. No latest. ECR repos configured as immutable.
No ECR auto-createPipeline fails with clear error if repo doesn't exist. Platform team provisions via Blueprint.
Mandatory Trivy scansFilesystem + image scan on every build. Currently warn-only (exitCode: 0).
No static credentialsOIDC + short-lived assumed roles. No IAM user keys.
Optimistic lockingupdate-ledger.sh verifies remote SHA before push. Retries on conflict.

8 · Live build · #37707 · dev-dagster

[2026-05-04 12:14:02] Conveyor v3.2.1 · pipeline azure-pipelines.yml@conveyor/main
✓ Trivy filesystem scan (warn-only) 14.2s
✓ Python build · pip install 38s
✓ Docker build dev-dagster:37707 2m 22s
✓ Trivy image scan 28s
└─ 638 packages, 207 unique licenses, 307 copyleft flags
✓ ECR login · OIDC → ConveyorExecutionRole 2.1s
✓ Verify ECR repo exists 0.8s
✓ Push 290971xxxxxx.dkr.ecr.eu-west-2/dev-dagster:37707 18s
[2026-05-04 12:16:16] Build complete · artifact published
Classic Release Pipeline triggered → update-ledger.sh → Argo CD sync (28s)