Article · 30 March 2026 · 5 min read

Duct tape to trust: our DevSecOps lab, rebuilt

Version one was a week of work, one pipeline, and a lot of duct tape. It proved the concept: code goes in, a scanned image comes out, Argo CD puts it on a cluster. Version two is the same little app with almost every layer underneath replaced, restructured, or hardened. This is what changed, and why.

The app is the excuse

Anchor is a spaced-repetition scheduler we built for ourselves: give it a lesson name and a start date, it hands back the exact review dates (day 0, then 3, 7, 14, 30) with a calendar file. A few hundred lines of Streamlit. It matters here for one reason: it is the application we run through the entire DevSecOps pipeline, end to end, the way a client's service would be.

What the lab is

A GitLab pipeline that builds, scans, tests, deploys, monitors, and rolls back a containerized app across three Kubernetes environments (dev, staging, production) on a k3s cluster provisioned with Terraform. Nothing lives in one monolithic file: the app repo carries its code and manifests, the pipeline logic lives in the same public components library this website deploys through, and production is GitOps: the pipeline commits a version bump to a manifests repo, and Argo CD reconciles the cluster from it. Git is the source of truth for what runs.

The pipeline, step by step

• Before a single Docker layer: unit tests, secret detection, SAST, lint, and Dockerfile lint, all in parallel.
• Build once: the image artifact is passed downstream, so every later job works from the same bytes.
• Scan: Grype fails the pipeline on critical vulnerabilities; the report is an artifact.
• Real end-to-end: Playwright opens the app in a real browser inside Docker-in-Docker. No curl approximation: what the user sees is what we test.
• Gates: pushing the image, staging, and production are manual decisions; dev deploys automatically.
• Health checks decide: each deploy is verified by polling for the expected image tag in Ready pods. Production polls up to fifteen minutes, because the Argo CD sync happens out of band.
• Rollback without heroics: if the health check fails, the rollback job fires on its own; if it passed, the job reads the result and skips itself. No manual decision under pressure.
• Then we test again: Playwright runs against the live production URL after the sync, and an on-demand S3 backup is one click away.

What changed between v1 and v2

The instructive part is why each layer moved. GitHub Actions gave way to GitLab for the include mechanism, the built-in registry, and native manual gates. The single pipeline file was split into reusable components, because a fix should land everywhere at once. Trivy gave way to Grype after serious vulnerabilities were found in the scanner itself: a reminder that following the news and adapting is part of the job. A self-hosted Harbor registry on its own server became the GitLab registry that ships with the project: one fewer machine, one fewer attack surface. One environment became three, each with its own health check and rollback. No post-deploy verification became poll-based checks with automatic rollback. And a blind cluster got the Datadog Operator: APM traces, pod health, and a dashboard.

Honestly, what is still ahead

Policy engines, secret management, DORA metrics, canary deploys, and tighter Argo CD feedback into the pipeline are still on the list. The goal has not moved: a platform we can present, defend, and hand off. v2 is closer. It is not done, and saying so is part of the method.

The code is public: gitlab.com/project-ds1/project-ds-v2. And if you are debating one big repo versus splitting early: split early. The cost of extraction later is always higher than the cost of a second repo now.