Sample
Nordvantage Logistics SA does not exist. Every company name, service name, figure, command output, identifier and date below is invented. This is not a record of work performed for any client, real or prospective. nordvantage.example is a reserved, non-routable domain.
The fact patterns are drawn from real engagements with figures shifted and names removed. The conditions recur; the client does not.
Platform audit · Report v1.0
Nordvantage Logistics SA
A one-week review of CI/CD, infrastructure, deployments, observability, disaster recovery and bus factor. Read-only on everything in service, plus one restore drill into a disposable instance.
flochai · Florian Chaillou
Review week 11–15 May 2026 · Report issued 20 May 2026 · Check list version 2026.1
hello@flochai.com
At a glance
One page · forward this if you forward nothing else
1
High finding
notified same day
3 + 5
Medium and Low
listed in full
€7,164
Per year, attached
to nothing
3h 40m
Measured recovery
previously unknown
12/14
Plan items your
team can do alone
- 1Your GitLab is reachable from the open internet, unpatched against three published advisories, and holds six variables the production deploy job uses. Half a day to close. R-001
- 2A backup restore works, and takes 3 hours 40 minutes. Your runbooks assume faster and disagree with each other. R-009
- 3€597 a month is billed for disks, addresses and snapshots attached to nothing. Two hours to delete. §7
What this week cost, against what it found. The audit was €4,900. The first cost table is €7,164 a year and needs a decision from nobody, so the fee returns in about seven months on that table alone. The second table is a further €24,360 a year and every line of it costs engineering time to capture. Neither figure counts the High, and neither counts knowing that recovery takes 3h 40m.
I have not multiplied those together into a headline. The numbers are above; the division is yours.
Everything else is Medium or Low and none of it is on fire. Two conditions I raised during the week were killed by their own counter-argument and are printed in §8 with the argument that killed them.
Take the three-page summary with you
The PDF is the same summary that opens this report, typeset for A4. Leave an email and it is yours; it also tells us someone is reading.
Download the summary (PDF)
The form did not go through, but that is our problem, not yours: Download the summary (PDF)
§ 0What you are holding
| Engagement | Platform audit, fixed price, one review week |
| Scope | Six domains: CI/CD pipelines · infrastructure · deployments · observability · disaster recovery · bus factor |
| Method | Read-only observation of everything in service, plus one restore drill into a disposable instance. 58 pre-registered checks, sent to you on 6 May, before the week began |
| Access held | GCP project viewer on 3 projects · billing account viewer · GitLab Reporter on group nordvantage · GitLab group Owner scoped to CI-01, granted 12/05 and revoked 15/05 · Datadog read-only. Full grant and refusal record in Appendix D |
| Evidence | 71 captured items, delivered as evidence-nordvantage-2026-05.tar.age, encrypted to your key |
| Deliverables | This report: a risk register ranked by impact, a 90-day plan ordered by value against effort, and a one-hour walkthrough call |
This audit changed nothing you run. No configuration was modified and no fix was applied. Every command in this report is read-only and you can run all of them yourself. The one exception is stated plainly because it matters: the restore drill (R-009) provisioned a disposable instance, restored a backup into it, and destroyed it the same day. Nothing in service was touched at any point.
Contractually out of scope, and therefore absent: any fix, development or configuration change; application security testing; penetration testing; application code review; regulatory compliance audit. Where a check touched the edge of one of these, the boundary is stated at the check. DE-01 is the clearest case: I read which paths each merge request touched and the DDL in your migration files. I did not read application logic.
Every figure drawn from your systems carries its source inline. Counts of my own checks, and my estimates of your team's effort, are mine and carry no marker. They are the only unmarked numbers in this document.
§ 1The short version
Nordvantage runs a freight booking and tracking platform on GCP: two GKE Autopilot clusters, nv-prod and nv-staging, carrying 23 services kubectl · point-in-time 13/05, a regional Cloud SQL PostgreSQL instance, a managed Kafka cluster for carrier events, and a self-hosted GitLab instance that is both your source of truth and your deploy path. Around 40 engineers reported · call 1, releasing weekly with hotfixes as needed.
I ran 58 checks across six domains. 34 came back sound, 9 produced a finding, 2 produced a cost finding, 5 I could not determine in five days, 8 did not apply to your setup.
Of the 9 findings, 1 is ranked High, meaning I observed every step of a sequence ending in credential exposure except the initiating event itself. It is R-001, your GitLab instance, and I called Marc on 12 May rather than leaving it for this document. Closing the exposure costs half a day plus a maintenance window. The other eight are three Medium and five Low, listed in full, each with the reason it did not clear the bar.
12 of the 14 items in the 90-day plan are things your team can do without outside help. The first five, which close the High and two Mediums, come to roughly two person-days plus one maintenance window. The rest spread across the quarter and each carries its own estimate in §10. The two that need a skill you may not have in-house are named as such, with the specific capability written out.
Separately from the register, the billing review identified €7,164 a year of spend attached to resources serving nothing, and a further €24,360 a year that is worth having but needs a decision from you. Cost is not risk, so none of it is in the register; it is in §7 with the method that produced it.
Two conditions I raised on Tuesday are not in the register at all. They failed their own counter-argument and ship as rejections, because a register that only ever grows is a sales document.
§ 2Coverage
58 checks, pre-registered and sent to you on 6 May, five days before the week started. You could see the list before I could see your systems. That ordering is the point: it stops the check list being written backwards from whatever I happened to find.
| Domain | Checks | Sound | Finding | Cost | Not determined | N/A |
| CI/CD pipelines | 12 | 8 | 1 | 0 | 1 | 2 |
| Infrastructure | 14 | 6 | 3 | 2 | 1 | 2 |
| Deployments | 9 | 5 | 2 | 0 | 1 | 1 |
| Observability | 8 | 5 | 1 | 0 | 1 | 1 |
| Disaster recovery | 9 | 5 | 2 | 0 | 1 | 1 |
| Bus factor | 6 | 5 | 0 | 0 | 0 | 1 |
| Total | 58 | 34 | 9 | 2 | 5 | 8 |
Where a check sampled rather than enumerated, the population, the sample size and the selection rule were recorded at the moment of sampling. Nothing here reconstructs its own sampling rationale after the fact, because a reconstructed rationale is always a flattering one.
§ 3Working, and should not be disturbed
A report that is entirely findings is both a sales document and an inaccurate description of your platform. These are load-bearing and I would argue against changing any of them.
Every pipeline runs the same seven-stage template. 21 of 23 services extend a shared CI component from an internal library rather than carrying bespoke YAML GitLab · API, 13/05. This is the biggest single reason your CI surface is auditable at all, and why check CI-04 took twenty minutes rather than two days.
An organisation policy blocks external IP addresses on VMs in the production folder, with an allow list of three instance IDs gcloud org-policies describe, 13/05. The exception list is short enough to read, and someone has kept it that way.
Terraform state is remote, versioned and locked, and the nightly plan runs on a schedule with its output retained GitLab · scheduled pipelines, 12/05. Drift is visible the next morning without anyone remembering to look for it.
Deploys are traceable to a merge request. Every production deployment in the audit window carried a linked MR and an approver who was not the author GitLab · deployments API, 11/05 to 15/05.
Alert routing has an owner per service, and the on-call rota is current and names a second person for every slot Datadog · monitors, 14/05. Bus factor was the domain I most expected to produce a finding, and it produced none.
The runbook wiki is used, not merely present. 11 of the 14 runbooks were edited within the last 90 days GitLab wiki · page history, 14/05. That is unusual at this size and worth protecting.
§ 4Do not do this
Two things a well-meaning quarter might produce, which would make your platform worse.
Do not tighten gitlab-deploy@ to objectViewer without checking what it writes first
It is the obvious-looking cleanup and it can take production down. Before touching it, run git grep -nE 'terraform (plan|apply|init)' scoped to the deploy jobs, and read 90 days of bucket data-access logs filtered to that principal's writes. If either returns anything, the grant is load-bearing. If you do it anyway, the rollback is to re-grant objectAdmin. Note also that item 9 of the plan puts a Terraform job in CI this quarter, which would collide with the tightened grant.
Do not adopt a maturity model or a scoring dashboard for this register
Eight findings do not need a heat map, and the moment a register has a score, the score becomes the thing that gets managed. This is a list of specific conditions with named next actions, and it should stay one.
§ 5If you do only three things
| # | Item | Register | Effort | Who |
| 1 | Take the GitLab instance off its public address, or put an authenticating proxy in front of it | R-001 | half-day + window | your team |
| 2 | Patch bastion-ops and put it on a security-pocket upgrade schedule | R-003 | hours | your team |
| 3 | Clear the stuck background migration so the upgrade path opens | R-002 | days | your team |
That is one and a half days of your team's time plus one maintenance window, and it closes the only High along with two of the three Mediums.
This set is ordered differently from the short list at the end of §10, deliberately. Here the ranking is by risk reduction. There it is by what fits an afternoon. R-002 is the largest job and it sits third because item 1 does not depend on it: you can close the exposure today and upgrade in the window you already have booked.
§ 6Risk register
| Level | Count | Total effort |
| High | 1 | half-day + window |
| Medium | 3 | days, plus hours |
| Low | 5 | about 3 days |
| Total | 9 | |
Levels are derived, not chosen. Each entry states a consequence class and a trigger-evidence grade, and the level is a lookup of the two in the table below. To move an entry up, I have to produce evidence that moves it a cell. I cannot type a level.
| Consequence class ↓ / trigger → | observed-in-window | mechanism-present | hypothetical |
data-loss | high | high | medium |
credential-exposure | high | high | medium |
outage-hours | high | medium | low |
outage-minutes | medium | low | low |
degraded | medium | low | low |
toil | low | low | dropped |
latent | low | low | dropped |
observed-in-window — the full chain, including the initiating event, occurred in your data, dated.
mechanism-present — every step of the chain observed; only the initiating event absent.
hypothetical — at least one step in the chain is assumed rather than observed.
Note what the bottom two rows delete: a hypothetical annoyance is not a finding, it is nothing. Two conditions were removed by that rule this week and appear in §8.
4 of the 9 entries were already on your board, including 0 of the 1 High. On those four the contribution is the dating, the consequence and the ordering, not the discovery. 7 of the 9 are fixable by your team alone.
R-001 · infrastructure
Self-hosted GitLab answers on a public address while running a version with published unpatched advisoriesHigh
credential-exposure×mechanism-present
→High
basis measured · half-day + window · your team
NotifiedMarc D., phone, 12 May 16:20. This was live when I found it and you heard it that afternoon, not from this file.
gitlab.nordvantage.example resolves to a global forwarding rule with an external address, and a firewall rule admits 0.0.0.0/0 on 443 to the instance tag gcloud forwarding-rules; firewall-rules, 12/05. The running version is 16.9.1 GitLab /api/v4/version, 12/05, against which the vendor has published three advisories describing authenticated remote code execution, none patched in 16.9.1 GitLab release notes, read 12/05. The instance holds 14 CI/CD variables at group scope, of which 6 are referenced by name in the production deploy job GitLab · CI/CD settings metadata, 12/05.
Every step of that chain is observed. The only absent step is an attacker choosing to walk it.
One inference, named as one. That those six variables are sufficient to deploy is inferred from their names, their environment scope and the deploy job that references them. I did not read a value and could not have: the grant was metadata-only (Appendix D). If you know one of them is inert, say so on the call and the finding drops a level.
The strongest argument against this
The objection you made on the call, and it is the right one: the instance requires a login, SSO is enforced, and no anonymous route exists, so a stranger cannot simply read the variables. That is true and it is why this is not worse than it is. It does not clear the finding, because two of the three advisories need only the lowest authenticated privilege level, and your instance accepts registrations from any address that reaches it. The distance between reachable and exploited here is a published technique and an account, not a research project.
Evidence you can re-run
# 1. Reachable from outside? Run this from a host that is NOT on your network.
curl -sS -o /dev/null -w 'HTTP %{http_code} from %{remote_ip}\n' \
https://gitlab.nordvantage.example/users/sign_in
# 2. Which ingress rules create a public path (scoped: ingress only, enabled only)
gcloud compute firewall-rules list \
--filter='direction=INGRESS AND disabled=false AND sourceRanges:0.0.0.0/0' \
--format='table(name,priority,network,targetTags.list(),
allowed[].map().firewall_rule().list())'
# 3. What actually applies to the instance, not what merely exists in the project
gcloud compute instances network-interfaces get-effective-firewalls gitlab-01 \
--zone=europe-west1-b \
--format='table(type,direction,disabled,priority,sourceRanges.list())'
# 4. The six variables, metadata only. jq drops .value before anything reaches disk.
curl -sS -H "PRIVATE-TOKEN: $TOKEN" \
'https://gitlab.nordvantage.example/api/v4/groups/nordvantage/variables?per_page=100' \
| jq -r '.[] | [.key, .protected, .masked, .environment_scope] | @tsv'
What these return: command 1 is the one that proves the finding, and it is the one to run first: an HTTP status from an unauthenticated request made off your network. Commands 2 and 3 say why it answers, and 3 is the authoritative one, because it reports the rules effective on the instance rather than every rule that exists in the project. Command 4 returns four metadata columns per variable and no values.
The variable metadata argues both ways, so here it is. Of the six the deploy job references, four are masked and two are not; none is protected; all six carry environment_scope: * GitLab · CI/CD settings metadata, 12/05. Unprotected and unscoped is what makes them readable by a pipeline on any branch. Masking hides a value in job logs; it does not stop the API returning it to a session that can read the settings page.
Today, in ten minutes, if the window is not until next week. Narrow the firewall rule's source range from 0.0.0.0/0 to your office and VPN CIDRs, and turn off open registration in the admin settings. Neither is the fix and neither needs a window. Both remove the part of the chain an attacker actually walks.
Next action. Put the instance behind IAP or the VPN. That is half a day and does not depend on the upgrade. Then upgrade, which needs R-002 cleared first. Rotate the six deploy variables after the upgrade rather than before, since a rotation is only durable once the exposure is closed.
fix_by: client-alone · already on your board: no
R-002 · CI/CD
A batched background migration has been failing since 14 April, and it blocks the upgrade pathMedium
outage-hours×mechanism-present
→Medium
basis measured · days · your team
A batched background migration has been retrying and failing since 14/04 GitLab Admin · background migrations, 13/05. GitLab's upgrade path refuses to advance while a batched migration is incomplete GitLab upgrade documentation, so your next version upgrade stops part-way and the instance is unavailable for as long as diagnosis takes, inside the window rather than before it. Every step of that is observed; the absent step is someone starting the upgrade. Horizon: the first upgrade attempt, which R-001 has just made urgent.
The strongest argument against this
The objection worth making, and the one that demoted this entry: batched background migrations are post-deploy data backfills, not in-flight schema changes. The database is not half-converted, and a restore would not inherit a broken schema. That is correct. I raised this as data loss on Tuesday and it did not survive contact with what the tracking table actually shows. What remains is real and narrower: an upgrade you now need for R-001 will fail in the window unless this is cleared first.
Evidence you can re-run
# Symbolic status, and only what has not finished. status_name avoids the
# integer enum, which is what makes the raw table hard to read.
sudo gitlab-rails runner '
Gitlab::Database::BackgroundMigration::BatchedMigration
.where.not(status: :finished)
.each { |m| puts [m.id, m.job_class_name, m.table_name,
m.status_name, m.updated_at].join(" ") }'
# The failing jobs underneath one of them. `attempts` and the exception live
# here, on the jobs table, not on the migration row.
sudo gitlab-psql -c "select batched_background_migration_id as mig, status, \
attempts, left(exception_class, 48) as exception, updated_at \
from batched_background_migration_jobs \
where batched_background_migration_id = 4291 \
order by updated_at desc limit 10;"
What these return: the first lists unfinished batched migrations with a readable status; the second lists the individual job failures underneath one, with the attempt count and the exception class. Note what is not here: gitlab-rake db:migrate:status reports regular schema migrations and cannot see batched background migrations at all, so it cannot support this finding and I have not cited it. Neither command says whether a restore would succeed; that is R-009.
Next action. Pull the failing job class and its error from batched_background_migration_jobs, match it against the known-issue list for 16.9, then clear or re-run it in a maintenance window ahead of the upgrade.
fix_by: client-alone · already on your board: yes, as "GitLab upgrade blocked"
R-003 · infrastructure
bastion-ops carries 34 pending security updates, including a kernel updateMedium
credential-exposure×hypothetical
→Medium
basis measured · hours · your team
The jump host bastion-ops reports 34 pending updates, all 34 from the security pocket, and the running kernel is two ABI versions behind the newest installed package apt-check; uname -r, 13/05. The host carries SSH agent forwarding for the production project and is the documented route to the Cloud SQL instances runbook db-access, 13/05. A local privilege escalation there reaches production database credentials.
The strongest argument against this
The objection that nearly kills it: the host has no external address, it sits behind the organisation policy in §3, and reaching it requires IAP plus a Google identity in your directory. An attacker holding that already has easier paths than a kernel exploit. That is exactly why the trigger is hypothetical and not mechanism-present: the step where an attacker gets a session on this box is assumed, not observed. It stays on the register because the fix is hours, not because the chain is complete.
Evidence you can re-run
gcloud compute ssh bastion-ops --tunnel-through-iap --command \
'/usr/lib/update-notifier/apt-check --human-readable; uname -r; dpkg -l "linux-image-*" | tail -5'
What this returns: the pending and security-pending counts, the running kernel release, and the installed kernel packages. It does not establish that any pending update is being exploited anywhere, and I did not test one.
Next action. Patch and reboot in your existing Thursday window, then enable unattended-upgrades restricted to the security pocket so the count cannot drift again without someone deciding it should.
fix_by: client-alone · already on your board: no
R-004 · deployments
Production runs floating image tags, so a rollback cannot name a known-good artifactLow
degraded×mechanism-present
→Low
basis measured · half-day · your team
9 of 23 workloads reference a mutable tag rather than a digest kubectl · point-in-time 13/05. The tag is written by CI from the branch or release name, so tag to pipeline is traceable; digest to pipeline is not, because no image carries a source-revision label gcloud artifacts docker images list, 13/05. A rollback therefore re-pulls whatever that tag points at now, which is not necessarily what was running when the incident started.
This entry merges a condition I first raised separately, that promotion from staging to production is manual with no automated link from the QA-passed build to the deployed image. The two share one next action, and one entry is more useful to you than two.
The strongest argument against this
Your registry has immutable tags enabled for the release-* prefix gcloud artifacts repositories describe, 13/05, so the release path is already pinned in practice, and all nine floating references are on main-tracking services that reach production only through staging. The exposure is a rollback during an incident on a service that was mid-deploy, which is narrow. That is fair, and it is why this is Low rather than Medium.
Evidence you can re-run
kubectl get deploy,statefulset,daemonset -A -o json | jq -r '
.items[] as $w
| ($w.spec.template.spec.containers
+ ($w.spec.template.spec.initContainers // []))[]
| select(.image | test("@sha256:") | not)
| [$w.kind, $w.metadata.namespace, $w.metadata.name, .name, .image] | @tsv'
What this returns: one row per container that is referenced by tag rather than digest, across Deployments, StatefulSets and DaemonSets, with initContainers included. Per-container matters: a line-level grep drops the whole workload as soon as any one sidecar is pinned, which hides exactly the mixed cases you care about. It does not tell you whether a tag is currently immutable in the registry; the repository policy is a separate call.
Next action. Have CI write the resolved digest into the manifest at deploy time. The tag stays for humans; the digest is what the cluster pulls.
fix_by: client-alone · already on your board: yes
R-005 · infrastructure
Pod requests are one copy-pasted block, so Autopilot bills a number nobody choseLow
degraded×mechanism-present
→Low
basis measured · days · flochai or a capable in-house owner
19 of 23 workloads carry an identical CPU and memory request block kubectl · point-in-time 13/05. On GKE Autopilot the request is what is billed, not the usage GCP pricing documentation, so the reservation is the invoice. Two consequences run in opposite directions and both are invisible today: services well under their request are paid for and not used, and two services show sustained CPU throttling against that same request Cloud Monitoring · container metrics, 30 days to 13/05, which is a latency problem rather than a cost one.
The strongest argument against this
The block was almost certainly chosen deliberately once, for the first service, and copying it is how a small team keeps 23 services consistent without a platform engineer. Replacing one uniform number with 23 hand-tuned ones trades a cost problem for a maintenance problem. That is a real objection, and it is why the next action is a measured sizing rule rather than per-service tuning, and why this sits at Low.
Evidence you can re-run
kubectl get deploy,statefulset,daemonset -A -o json | jq -r '
.items[] as $w | $w.spec.template.spec.containers[]
| [(.resources.requests.cpu // "unset"),
(.resources.requests.memory // "unset")] | @tsv' \
| sort | uniq -c | sort -rn
What this returns: the distinct CPU and memory request pairs in use, and how many workload containers carry each. It counts workloads rather than pods deliberately: counting pods lets a ten-replica deployment vote ten times and inflates the apparent uniformity. It does not show utilisation, and the sizing decision needs both.
Next action. Build the 30-day utilisation view from container metrics you already collect, then size per workload against observed steady state, respecting Autopilot's request floors and its memory-to-CPU ratio limits. Lower requests where the measurement supports it and raise the two that are throttling. On limits, check before you rely on them: Autopilot normalises limits to requests unless the cluster's version and compute class support burstable workloads, in which case headroom above the request genuinely is free. On a cluster without that, the request is the only lever you have and leaving a larger limit buys nothing. Establish which you are on before sizing. A right-sizing exercise that only ever goes down is a target, not a measurement.
fix_by: flochai · already on your board: no
R-006 · observability
Container logs have no shared destination, so triage starts by finding themLow
toil×observed-in-window
→Low
basis measured · days · flochai or a capable in-house owner
Logs are readable per cluster through Cloud Logging with the default _Default sink and no aggregated destination gcloud logging sinks list, 14/05. Retention is 30 days gcloud logging buckets describe, 14/05. In the two incident threads you shared, the first message containing a log line arrived 34 and 51 minutes after the opening message Slack · INC-2026-03 and INC-2026-07, provided 12/05. That gap is the finding: it happened, it is dated, and it is in your own data. Two threads is a sample of two, and I am not extrapolating a mean from it.
The strongest argument against this
Both incidents were resolved inside the hour anyway, and 30-day retention meets the only obligation you named. Aggregation improves a process that is not currently failing. Correct, and it is why the class is toil rather than degraded: what this costs you is engineer attention per incident, not customer-visible service.
Evidence you can re-run
gcloud logging sinks list --format='table(name,destination,filter)'
gcloud logging buckets describe _Default --location=global --format='value(retentionDays)'
What these return: the configured sinks with destinations and filters, and the retention on the default bucket. They establish that no aggregated destination exists in this project. They do not measure triage time; that came from the two threads you provided.
Next action. Route both clusters into one curated BigQuery dataset with a partition expiry matching your retention obligation, and point the incident runbook at a saved query.
fix_by: flochai · already on your board: yes
R-007 · deployments
No environment exercises the carrier and payment integrations off productionLow
degraded×hypothetical
→Low
basis observed · days · your team
nv-staging runs the application but points at provider sandboxes for two of five integrations and at production endpoints for the other three ConfigMap integration-endpoints, 13/05. Changes to those three are validated against production or not at all. I did not observe a failure caused by this and I am not going to imply one.
The strongest argument against this
Two of the three have no sandbox offered by the provider, which you cannot fix, and your team compensates with a documented manual test against a live account you own runbook integration-release, 13/05. That is a real control, and it is why the trigger is hypothetical and the entry is Low.
Evidence you can re-run
kubectl -n nv-staging get configmap integration-endpoints -o yaml
What this returns: the configured endpoint per integration in staging, which shows which point at sandboxes. It does not show what traffic actually goes where, since a service could override the value at runtime.
Next action. For the one integration whose provider does offer a sandbox and where staging currently uses production, switch it. For the two with no sandbox, write the manual test into the release checklist so it is a step rather than a habit.
fix_by: client-alone · already on your board: no
R-008 · disaster recovery
No agreed recovery point for production, so each runbook assumes its ownLow
latent×mechanism-present
→Low
basis observed · hours · your team
Cloud SQL automated backups run daily with point-in-time recovery enabled and 7 days of transaction logs retained gcloud sql instances describe, 14/05. What does not exist is a written statement of how much data loss is acceptable, agreed with anyone outside engineering. In its absence three different assumptions are visible in the runbooks: one says restore to this morning, one says no loss because PITR exists, one is silent GitLab wiki, 14/05. The mechanism is complete; the absent step is a restore decision nobody has had to make yet.
The strongest argument against this
Your actual technical capability is good, and PITR with 7 days of logs is better than most companies of this size. Writing a number down changes nothing about what the platform can do. True, and that is precisely why this is Low: it is half a page of agreement, not an engineering project. What it buys is that the person making the call at 3am is not also deciding the policy.
Next action. One page, agreed with your COO, stating acceptable loss per data class, then reconcile the three runbooks against it.
fix_by: client-alone · already on your board: yes, informally
R-009 · disaster recovery
Measured recovery takes 3h 40m; every runbook assumes it is fasterMedium
outage-hours×mechanism-present
→Medium
basis measured · hours · your team
I restored your most recent production database backup into a disposable instance on Wednesday afternoon and destroyed it the same day. It worked. That is the first thing to say, and it is not a given: roughly a third of the backups I have drilled do not.
It took 3 hours 40 minutes end to end restore drill, 13/05, timings in evidence: 26 minutes to provision, 2h 51m to restore and replay transaction logs, 23 minutes to verify row counts against the source. Your three runbooks variously assume "restore to this morning" and "no loss, we have PITR" GitLab wiki, 14/05. None of them names a duration, and the number nobody had is the one that decides whether you fail over or wait.
The strongest argument against this
3h 40m is not a bad number for a database this size, and the drill ran on a disposable instance with no contention, so a real recovery under load would differ. Both true. The finding is not that the number is bad, it is that nobody had it, and a recovery-time expectation that lives in three runbooks as three different assumptions is not an expectation. This is Medium rather than High because the chain ends in hours of downtime, and the initiating event, an actual failure, has not occurred.
Next action. Put 3h 40m into the one-page objective from R-008 as the current measured recovery time, then decide separately whether it is acceptable. If it is not, the lever is a warm standby, which is a cost decision rather than an engineering one.
fix_by: client-alone · already on your board: no
One thing to accept and not fix
The nv-edi bridge does not use the shared CI template, and should not be made to. It is the legacy EDI connector, it deploys twice a year, and its pipeline is 40 lines that one person understands GitLab · .gitlab-ci.yml, 13/05. Migrating it would cost days and buy consistency on a service whose whole value is that it has not changed since 2023. Recorded here so the next person who greps for non-conforming pipelines does not spend a sprint on it.
§ 7What this costs you, and what it need not
Cost is not risk, so none of this is in the register. It is here because it is the part of the week that carries a number, and because the first table is true whatever you decide about anything else.
How these figures were produced. Every line comes from the billing export, compared month over month across the three months to 30 April GCP billing export. Where a line grew because you shipped more, that growth is separated out and is not counted as waste. Your platform cost €14,200 a month on average across that window GCP billing export.
Removable now, with no argument against it
Each of these is a resource attached to nothing, serving no request. There is no counter-argument to make, which is why it is separated from the table that follows.
One caution on the image line. R-004 found that 9 of 23 workloads pull by mutable tag, so an untargeted cleanup policy can delete an image production is about to reschedule onto. Difference the registry against the running set first and delete only unreferenced digests. That is fifteen minutes, not a decision, which is why the line stays in this table rather than the next.
| What | How I know | Per month | Per year |
| 11 persistent disks attached to nothing, 2.4 TB total, oldest detached 14 months ago | gcloud compute disks list --filter='-users:*' | €310 | €3,720 |
| 6 reserved static addresses, none attached to a resource | gcloud compute addresses list --filter='status!=IN_USE' | €33 | €396 |
| 1.8 TB of snapshots under no lifecycle policy, oldest 14 months | gcloud compute snapshots list | €47 | €564 |
| 890 GB of container images, cleanup scoped to digests no workload references | gcloud artifacts docker images list, differenced against the running set | €89 | €1,068 |
| Health-check and readiness probe lines, 38% of ingested log volume | Cloud Logging ingestion, billing export | €118 | €1,416 |
| Total | | €597 | €7,164 |
That is 4.2% of your monthly bill and about two hours of work. It is item 1 of the 90-day plan for that reason: nothing in it needs a decision from anyone.
Worth doing, but each needs a decision
These are larger and none of them is free. The argument against each one is real, and it is stated rather than left for you to find.
| What | Per month | Per year | The argument against |
nv-staging serves no traffic for 128 of 168 hours a week Cloud Monitoring, 30 days to 13/05 | €740 | €8,880 | Your engineers work irregular hours, and scaling an Autopilot cluster to zero on a schedule is real engineering, not a setting |
| Cloud SQL runs one tier above what 90 days of CPU supports: p95 is 14% Cloud Monitoring, 90 days to 13/05 | €310 | €3,720 | It was sized for the migration peak. Headroom you chose deliberately is not the same as waste, and a resize needs a restart |
| No committed-use discount against the steady-state compute floor, which has not dropped below 62% of peak in 90 days GCP billing export | €980 | €11,760 | A one-year commitment on a platform you may re-architect. The floor is stable today; the commitment outlives that certainty |
| Total | €2,030 | €24,360 | |
Not costed, deliberately
R-005 — Autopilot request sizing. The reservation is the invoice, but the direction is not uniform: two services need more. No figure until the 30-day utilisation view exists, which is plan item 11.
U-001 — Kafka provisioning. Throughput is measured; the floor set by partitions, replication and retention is not. Quoting a saving before that is answered would be quoting a number I cannot defend.
Identified is not realised. €31,524 a year appears above. None of it is saved until someone does the work, and the second table costs engineering time to capture. Treat the first table as money and the second as a shortlist of decisions.
I have not applied a multiplier, projected a run rate, or counted the same resource twice. If you want to check one line, the disk query is the fastest: it returns a list you can read in a minute.
§ 8Considered and not raised
Things I checked that turned out fine, and two conditions that died on their own counter-argument. Both were on my list on Tuesday. Neither survived Thursday.
Rejected
"CI failures reach nobody"
Pipeline results post to no chat channel, which is a real gap in the abstract. The counter-argument holds: GitLab's author email path is intact and enabled, and across the pipelines I could read, a failure was generally followed by a fixing commit from the same author the same working day GitLab · pipelines API, 11/05 to 15/05. That call filters on failed pipelines only, so it cannot show me the fix directly and I am not quoting a mean time to recovery from it. A chat notifier would be a convenience. It is not a risk reduction and it does not belong in a register.
Rejected
"The Kafka cluster is over-provisioned"
The cluster is provisioned above the throughput I measured over 30 days Confluent · metrics API, to 13/05, and the gap is real money. It is not a risk. Nothing about it can hurt you, and putting a cost condition into a register of things that can hurt you devalues both. It is item 12 of the 90-day plan instead, and U-001 records the question that has to be answered before anyone acts on it.
Checked and sound, briefly: user-managed service account keys (none) · service account key age · public bucket ACLs (none) · secret scanning across all 23 repositories · branch protection on main group-wide · Terraform drift on the nightly plan · on-call rota currency and depth · merge request approval settings · runner tag scoping · Cloud SQL public IP (disabled) · org policy inheritance on the production folder.
§ 9What I could not determine in five days
Five checks did not resolve. This section is a property of a five-day audit, not an admission. Note what is not here: whether your backups restore. That question is answered in R-009, because the drill is in scope.
U-001 — Whether the Kafka cluster's provisioning genuinely exceeds its requirement.
I measured throughput, not the floor. Partition count, replication factor and retention may set a minimum well above what throughput alone implies. What it would take: reading the topic configuration against the vendor's sizing rules, about half a day. Nobody should act on plan item 12 before this is answered.
U-002 — Whether any departed employee retains access.
I hold no leaver list, so I cannot difference it against the directory. What it would take: an HR leaver list for the last 24 months, and twenty minutes.
U-003 — Whether the nine floating-tag services have ever rolled back to a different artifact than intended.
Deployment history does not retain the resolved digest GitLab · deployments API, 13/05. What it would take: enabling digest capture from now on. The past is not recoverable.
U-004 — Whether the two throttling services in
R-005 are throttling in a way users feel.
I have container CPU metrics but no request-latency objective to correlate against Datadog · monitors, 14/05. What it would take: a latency objective on those two services, which is also the fix.
U-005 — Whether the three org policy exceptions in §3 are still justified.
I confirmed the policy and read the allow list gcloud, 13/05. I did not establish what those three instances are or whether they still need the exception. What it would take: ten minutes with whoever added them.
§ 10The 90-day plan
Ordered by value against effort. 12 of 14 items are yours to do without me. The two marked flochai name the capability rather than the company, so you can price hiring it against buying it.
Days 1 to 14 — configuration only, each closeable in an afternoon
| # | Item | Register | Effort | Who |
| 1 | Delete the unattached disks, addresses and snapshots; add cleanup and lifecycle policies; drop health-check lines from log ingestion | §7 | 2 hours | your team |
| 2 | Put GitLab behind IAP or the VPN | R-001 | half-day + window | your team |
| 3 | Patch bastion-ops, enable security-pocket unattended upgrades | R-003 | hours | your team |
| 4 | Run the leaver difference | U-002 | 20 min | your team |
| 5 | Write the recovery point objective, one page | R-008 | hours | your team + COO |
Days 15 to 45
| # | Item | Register | Effort | Who |
| 6 | Clear the stuck background migration | R-002 | days | your team |
| 7 | Upgrade GitLab past 16.9, then rotate the six deploy variables | R-001 | days + window | your team |
| 8 | Record 3h 40m as the measured recovery time and decide whether it is acceptable | R-009 | hours | your team |
| 9 | Write the resolved digest into the manifest at deploy time | R-004 | half-day | your team |
Days 46 to 90
| # | Item | Register | Effort | Who |
| 10 | Terraform plan job in CI on merge requests | — | days | your team |
| 11 | Build the 30-day utilisation view, then size per workload | R-005 | days | flochai — Autopilot request floors and ratio constraints |
| 12 | Answer U-001, then act on the Kafka provisioning if it holds | U-001 | half-day, then days | your team, then either |
| 13 | Aggregate both clusters into one curated logging dataset | R-006 | days | flochai — log routing and partition-expiry design |
| 14 | Move the one integration with a sandbox off production in staging | R-007 | days | your team |
If you cannot spend six days
Do items 1, 2 and 3. That is one afternoon plus a window. It closes the High, one Medium, and the whole of the first cost table. This is not the same set as §5, and the difference is deliberate: §5 ranks by risk reduction and this ranks by what fits an afternoon. R-002 is the larger job and its value depends on the upgrade actually being attempted, so if you have one afternoon rather than three, buy the exposure closure first.
Not in the 90 days, and why
Migrating nv-edi to the shared template (accepted, §6). A chat notifier for CI (rejected, §8). Per-service latency objectives beyond the two in U-004, which is a quarter of its own and should not start in the same quarter as the sizing work.
§ 11Method, and how to argue with this document
One person cannot have separation of duties. This is the honest substitute.
The check list was fixed before the week. 58 checks, sent 6 May. Findings could only come from that list, which stops the list being written backwards from what I found.
Severity is a lookup, not an opinion. Every entry states its two inputs. If you disagree with a level, disagree with one of the inputs, and there is then a defined thing to argue about rather than a feeling to trade.
The register froze Friday at 17:00 and access ended Monday. An unevidenced thought on Tuesday is not a finding, and nothing was added after the freeze. The 90-day plan was written after it, deliberately, so severity could not be retrofitted to fit a plan.
You argued with it on Thursday. Call #2 put the draft register in front of you while there was still a working day to act on a demotion. R-002 was demoted from High to Medium as a direct result, and the reason is written into the entry. What you did not see before the freeze: the five Low entries and the two rejections, which I raised on Thursday but did not walk through line by line.
I do not price consequences in euros, because I do not know your revenue per hour. I do price savings in euros, from your own vendor bill, and §7 is the only place that happens.
Corrections are free for 30 days after delivery. After that the register is yours. Every entry carries a command you can re-run alone, so an entry that still returns the same answer in six months, with nothing having happened and nothing having fired, means I was wrong about it. You do not need me to see that.
The walkthrough call
One hour, agenda fixed: R-001 and its window, then R-002 because item 6 depends on it, then the two flochai items so you can decide whether they are worth outside help or worth hiring for. Ten minutes at the end for the rejections, because how a finding gets killed is more useful to you than how one gets raised.
Please revoke my access at the end of that call, not before. I have committed to correcting anything you can disprove for 30 days, and I need to be able to check.
AppendixAccess, evidence and checks
Access requested, granted and refused
| Access | Requested | Granted | Note |
| GCP project viewer, 3 projects | 06/05 | 11/05 | |
| Billing account viewer | 06/05 | 11/05 | On the billing account, not the project |
GitLab Reporter, group nordvantage | 06/05 | 11/05 | |
| GitLab group Owner, scoped to CI-01 | 06/05 | 12/05, revoked 15/05 | Exception. Metadata only: key, protected, masked, environment scope. No value read, filtered at the command |
| Datadog read-only | 06/05 | 11/05 | Monitor history before 02/2026 not visible |
| BigQuery metadata viewer | 06/05 | refused | U-001 partially blocked |
| HR leaver list | 12/05 | not provided | U-002 ships open |
| Disposable instance for the restore drill | 06/05 | 12/05, destroyed 13/05 | In scope. Result in R-009 |
Deliberately not requested: production database data access; application source beyond CI configuration and migration DDL; Auth0 read:users; customer records of any kind.
Evidence
71 items, filed automatically as each command ran. Nothing unscrubbed touched disk. The scrubber runs before anything is written, and the capture tool exits if it is not present. The day-1 smoke test, run before any other command: a fake AWS key piped through the tool filed output reading AWS_KEY_REDACTED. Had it failed, nothing else would have run.
Secret values. For GCP the capture is value-free by construction, because the role held cannot read secret payloads. For GitLab that is not true: the variables endpoint returns values. The capture was filtered at the command so no value reached disk, the grant was scoped to CI-01, and it was time-boxed to 12–15 May.
Personal data. The bundle contains personal data: IAM identities, incident participants, alert routing handles. It is encrypted to your key, the local copy is destroyed at delivery plus 30 days, and the processing terms in the signed general terms apply to it.
The 58 checks — representative rows
| ID | Check | Result | Disposition |
IN-01 | Org policy on external IPs, production folder | Enforced, 3 named exceptions | sound |
IN-04 | Host patch level, all long-lived VMs | bastion-ops: 34 pending, 34 security | R-003 |
IN-10 | Public exposure of admin surfaces, per cloud | GitLab reachable from 0.0.0.0/0 on 443 | R-001 |
IN-12 | Billing export review, 3 months, growth separated | €597/mo attached to nothing | §7 |
IN-13 | Commitment and steady-state floor analysis | No discount against a 62% floor | §7 |
IN-11 | Workload request and limit distribution | 19 of 23 identical | R-005 |
CI-01 | CI/CD variable metadata, scope and protection | 14 group-scope, 6 in deploy job | R-001 |
CI-04 | Pipeline template conformance | 21 of 23 extend the shared component | sound |
CI-09 | Failure notification path | Author email intact, no chat integration | rejected |
DE-01 | MR to deployment traceability (paths and DDL only) | Every prod deploy linked, approver ≠ author | sound |
DE-03 | Image reference immutability | 9 of 23 by tag, not digest | R-004 |
OB-02 | Log sink topology and retention | No aggregated sink, 30-day retention | R-006 |
DR-02 | Backup configuration, production database | Daily plus PITR, 7 days of logs | sound |
DR-03 | Restore drill into a disposable instance | Succeeded, 3h 40m end to end | R-009 |
BF-01 | On-call rota currency and depth | Current, second name in every slot | sound |
How I am paid, and what I did about it
This audit is a fixed fee, and it credits against the first month if it becomes a retainer. That gives this document a standing incentive to make your platform look worse than it is, and you should read it knowing that.
What you can check yourself. Every level is a lookup of two stated inputs, so a level cannot be typed. The register came out one High, three Medium, five Low, and two conditions I raised on Tuesday were killed by their own counter-argument and are printed in §8 along with the argument that killed them. One finding was demoted during Thursday's call and says so in its own entry. 12 of the 14 plan items route to your team, not to me, and the two that do not name the capability so you can price hiring it against buying it.
If you think this document overstates something, §11 says exactly how to argue with it, and the correction window is open for 30 days.