Article · 1 October 2026 · 6 min read

Debug Detective: the tip-off

Every other case in this series starts with something broken. This one starts with a sentence in a meeting. There was no alert, no failing job, no log to open. My pipeline was green and had been green all along, and the thing wrong with it was the tool I had installed to tell me when things were wrong.

The case I did not solve

I did not find this. Someone told me. My mentor mentioned it out loud, in a meeting, and that is the entire discovery.

There is no screenshot for this section and there cannot be one. Every other piece in this series opens with the symptom on a screen. This one opens with a colleague saying a thing, because nothing on any screen of mine was ever going to say it. Sudo could not have helped either. She makes the hidden column visible, and this was not hidden in a column. It was not in my system at all.

What had actually happened

On 19 March 2026, someone holding stolen release credentials published a malicious Trivy v0.69.4. They force-pushed 76 of the 77 version tags in trivy-action to credential-stealing commits and replaced all seven tags in setup-trivy. Three days later they pushed poisoned images to Docker Hub, 0.69.5 and 0.69.6.

The payload read runner process memory through /proc/<pid>/mem and swept more than fifty paths for SSH keys, cloud credentials, Kubernetes tokens and Docker configs. What it collected it encrypted and shipped out. Where it could not ship, it dumped the haul into a public repository named tpcp-docs.

The windows were short. Three hours for the binary. Ten for the images.

The line in my own pipeline

latest. The tag every tutorial hands you. The tag you write when the version does not feel like the interesting part of the line.

During the incident latest pointed at the malicious build. The maintainers had to restore it to a safe version by hand afterwards.

What I did, and what I did not do

I swapped Trivy for Grype. One file, about ten minutes, committed 28 March at 09:15, six days after the second window closed.

Then I moved on.

I did not check whether I had been hit. I did not rotate anything. I treated a supply chain compromise as a dependency upgrade. That is the mistake this piece is about, and I made it while running a series on not making mistakes like it.

Two false leads, both mine

Five months later I finally asked the question, and I got it wrong twice before I got it right.

First I reasoned from commits. No commit in any of the four repositories landed inside either window, so no pipeline could have run, so I was clear. That is a proxy, and proxies are where this series keeps finding its ghosts. Pipelines also come from retries, manual runs, tags and schedules.

Second I reasoned from the job definition. The scan job declares no variables, no secrets, no id_tokens, so it holds no credentials, so there was nothing to steal. Also wrong. GitLab injects project and group variables into every job whether the job asks for them or not.

Both of those felt like checking. Neither was.

What the log actually said

Every pipeline is logged, and the windows are known to the minute. Seventeen scan jobs ran that week. One of them started at 16:44:55 UTC on 22 March, an hour after the poisoned images went up.

Then the job trace, which settles it:

16:45:55Z  Unable to find image 'aquasec/trivy:latest' locally
16:45:56Z  latest: Pulling from aquasec/trivy
16:45:58Z  Status: Downloaded newer image for aquasec/trivy:latest

Not a cached layer. A cold pull, from Docker Hub, inside the window. And the digest it pulled is published in the advisory as malicious.

I ran it. For five months I had assumed I probably had not, on the strength of never having looked.

There is no tpcp-docs repository in my namespace, which is the one piece of good news the payload leaves you. It is also the weaker half of the evidence: that repository was the fallback for when exfiltration failed, and the primary route was an encrypted upload that leaves nothing behind to find. So I rotated everything the job could reach and stopped trying to prove a negative.

What kept it small

Two things, neither of them a decision I made about supply chains.

The job ran on a shared SaaS runner, so the container was thrown away minutes later. There was no long-lived host underneath it holding SSH keys or a kubeconfig.

And the only cloud access anywhere in the pipeline goes through OIDC, exchanged for a token that dies with the job. There were no static keys in that process to take, because I had removed static keys months earlier for a completely unrelated reason.

You do not get to feel clever about it either, because you were not being clever. You were being tidy.

Detective's note

The ghost was not in the machine. It was in the news, and I was not reading it.

• Pin your tools. :latest is a standing promise that someone else chooses the bytes you run tomorrow.
• Your scanner is a dependency. Everything you install to check things is a thing that needs checking.
• Swapping a compromised tool is remediation. It is not investigation. Ask whether you were hit, and ask it the same day.
• Beware the check that feels like checking. Commits are not pipelines, and an empty job definition is not an empty environment.
• Ephemeral runners and short-lived credentials are what stand between a bad pull and a bad month.