flochaiThe migration that failed for four months
Every hour, on a self-managed GitLab, the same background migration woke up, tried to insert rows into a table a newer release was designed to drop, and failed. It had been doing this since April. Nobody was watching, because nothing visible was broken. Yet.
How instances rot politely
Self-hosted platforms rarely fail loudly. They accumulate: an upgrade pending because nobody scheduled it, a deprecated config key renamed two releases ago, a background job failing on a loop that no dashboard surfaces. Ours had all three. The instance needed a critical CVE closed, had drifted config since the last package upgrade, and carried a batched background migration that had been failing, silently, for four months.
Upgrade first, with the map open
GitLab upgrades are a path, not a jump: we carried the instance through the CVE fix, then to the required stop on the way to current, verifying zero pending schema migrations at each hop. The package upgrade also left two small landmines we defused on the spot: deprecated nginx configuration keys silently renamed upstream, and a Redis process the upgrade left running beside the new one.
Diagnose before you delete
The stuck migration was the interesting part. It was backfilling data into an archived table that a later release drops entirely: work whose destination was scheduled for demolition. The tempting move is to force-finalize and hope. Instead we traced it, with GitLab support, to a known upstream bug in single-record batched migrations, then verified the part that actually matters: was there any real data work left? Zero rows missing the sharding key, both constraints already validated. Only then did we finalize it from the Rails console, along with a second orphaned migration found on the way.
Then write down how you would survive
The same engagement produced the document the instance had always deserved: hourly backups giving a stated one-hour recovery point, week-deep machine snapshots, copies shipped off the instance, and two rehearsable recovery paths, one restoring to a standby, one rebuilding from a snapshot after destructive damage. Recovery went from tribal knowledge to a written procedure with a number attached.
What this buys
The instance is current and unblocked for its next required upgrades, a four-month-old failure is cleared with proof that no data was lost, and the next bad day has instructions. None of this is glamorous. All of it is the difference between an incident and a disaster.
Platforms age. Someone should be watching.
Upgrades inside the security window, background jobs someone actually reads, and recovery as a written procedure: this is the unglamorous half of the retainer. If your self-hosted anything has not been looked at in months, that is the tell.
Get in touch