Overview

#

I’ve written about this twice already. First about pointing a scanner at my own stack and getting a wall of red, then about learning to split it — the handful of findings that were mine, the long tail that belonged to whoever built the image. I fixed mine. I acknowledged theirs. The board went green.

That conclusion was comfortable. A bit too comfortable, in hindsight.

Because “most of it isn’t mine” is true, and it’s also a lovely place to stop looking. So this time I went the other way. I took every Dockerfile I’ve ever written — sixty-seven of them across the whole estate — and asked a narrower question:

The pile that actually is mine. How bad is that?

Worse than I thought. And the worst single thing in it, I put there myself.

The line I wrote

#

My CRM runs on Python. The image is built from python:3.12-slim, and somewhere near the bottom of the Dockerfile there was this:

RUN apt-get update && apt-get install -y --no-install-recommends \
      libpq-dev gcc \
    && rm -rf /var/lib/apt/lists/*

In the final stage. The one that ships.

I know exactly why it’s there. At some point pip install psycopg2 failed on a missing pg_config, I did what everyone does, I installed the build tools, it worked, and I moved on. Classic.

Except requirements.txt says psycopg2-binary==2.9.10. That’s a prebuilt wheel with libpq statically bundled inside it. Nothing compiles at runtime. Nothing has compiled at runtime for as long as that pin has been there.

So I deleted the line and rebuilt.

640 CVEs before. 185 after. 396 MB down to 193 MB.

Four hundred and fifty-five vulnerabilities, gone, from deleting one line that never needed to exist. No upstream to wait for. No maintainer to blame. Just me, installing a C compiler into a production container and shipping it, for months.

And it’s worse than the number, because a compiler plus dev headers in a running container is exactly the toolkit you’d want if you ever got a shell in there. I didn’t just inherit that risk. I built it, tagged it, and pushed it to my own registry.

What a base image actually costs

#

Once I’d stopped feeling clever, I did the boring part properly: pull every base image I use, and every one I might replace it with, and scan them all. Not read the docs. Not trust the marketing. Actually measure.

Before a single line of my code is added:

That’s the starting position. You type FROM python:3.12-slim, you have written nothing yet, and you are already carrying 157 known vulnerabilities. Three of them critical.

I’d been treating the FROM line as plumbing. It isn’t. It’s the single biggest dependency decision in the whole file, and I’d been making it on autopilot for years.

The part where hardened images help

#

The pitch for distroless and Chainguard images is simple enough: ship your application and the libraries it links against, and nothing else. No shell. No package manager. No perl, no tar, no curl. If it isn’t needed to run the program, it isn’t in the image.

Two things follow from that.

The obvious one is the CVE count, because most of what a scanner finds in a base image is in software your app never calls. Strip the software, strip the findings.

The less obvious one matters more to me. There is no shell in there. A lot of what turns “I found a bug in your web app” into “I own your server” is the stuff lying around after the first foothold — a shell to run, a package manager to pull tools with, curl to fetch the next stage. Distroless doesn’t defend the front door any better than slim does. It just makes the room behind it very boring to stand in.

Measured on my own services, built image to built image:

Same code. Same dependencies. Different floor to stand on.

The trap

#

Here’s the bit I’d have got wrong if I’d trusted the obvious answer.

If you go looking for “hardened Python base image”, you land on gcr.io/distroless/python3-debian12. It’s the one everything points at. So I scanned it, expecting to confirm what I already believed.

239 CVEs. Two critical, forty-four high.

Against 157 for the python:3.12-slim it was supposed to replace.

It ships Debian’s packaged Python, which lags upstream. Swapping to it would have made four of my services measurably worse while letting me tell myself I’d hardened them — which is the specific kind of wrong that’s hard to catch later, because everything about it looks like progress.

What actually works is distroless/cc-debian12 plus bringing your own interpreter:

# Bookworm so glibc matches the runtime below.
FROM python:3.12-slim-bookworm AS pybuild
RUN pip install --no-cache-dir -r requirements.txt

FROM gcr.io/distroless/cc-debian12
COPY --from=pybuild /usr/local /usr/local
USER nonroot
ENTRYPOINT ["python3", "server.py"]

Nearly identical intent. Completely different result. The only way I found out was by measuring both instead of picking the one with the right name.

The pin that became the problem

#

While I was in there: I’ve been pinning base images for the usual reason, so builds are reproducible.

nginx:1.29-alpine — the version I’d carefully pinned — measured 101 CVEs. Rolling nginx:alpine measured 20. node:20-alpine measured 77 against 36 for node:22-alpine.

The pin hadn’t protected anything. It had frozen a snapshot and then quietly rotted while I congratulated myself on being disciplined. And the funny part is I clearly half-knew, because my Hugo sites all carry this line:

RUN apk --no-cache upgrade

Patch the base at build time. Which works — those images genuinely measured zero, before and after. But it also throws away the entire point of pinning. I’d built a workaround for a problem instead of noticing the problem.

A pin without a refresh habit isn’t reproducibility. It’s just an old image with a confident-looking tag.

The part I got wrong

#

I’d rather put this here than bury it, because it’s the finding I liked least.

My distroless images use a staging step to carry the shared libraries Python links against out of the build stage:

RUN cp -aL /usr/lib/*-linux-gnu/*.so* /rootfs/pylib/

Copy the .so files, keep the glob architecture-neutral, move on. It works. The apps run.

It also copies about 152 shared objects from roughly 40 Debian packages — files only, without their package records. So the scanner looks at the finished image, reads a package database listing nine distroless packages, and reports a lovely low number.

Meanwhile 85 vulnerabilities, one critical and thirteen high, are physically inside that image and simply not being counted. CPython’s _sqlite3 links Debian’s libsqlite3.so.0. That library shows ten CVEs on the build stage and zero in the final scan. It’s the same file. It’s right there.

So the improvements above are real — the code is genuinely gone, the shell is genuinely gone — but the numbers are flattering, and I only know that because someone went looking rather than accepting a green board. Which is precisely the mistake I wrote a whole post about last time.

Narrowing that glob to what ldd actually resolves is next on my list.

What I actually did

#

Thirteen of my own applications measured properly, before and after, on the built images rather than the base images. 1,935 vulnerabilities down to 638. Seven deployed to production, one at a time, each one verified live before I touched the next.

Six of those thirteen were pure deletions. No new base image, no migration, no risk — just removing things that were never used. The compiler. An unused curl. libssl-dev installed against a dependency graph with no OpenSSL in it at all.

That’s the ratio I keep coming back to. The biggest wins weren’t clever. They were subtractions.

The point

#

The last two posts landed on a comfortable idea: most CVEs in your stack aren’t yours, so learn to tell them apart and stop panicking about the rest.

I still think that’s right. But it’s half a thought, and I stopped there because stopping there felt good.

The other half is that the pile which is yours has one enormous lever in it, and it’s the line you think about least. FROM isn’t boilerplate. It’s a dependency you adopt wholesale, on autopilot, usually copied from whatever you did last time — and it sets the floor for everything above it before you’ve written a thing.

CVE counts aren’t a to-do list. Most of them will never be exploitable in your context, and treating them as one is how people end up ignoring the number entirely. But they’re a signal about how much software you’re carrying that you don’t need, and that signal is worth reading — because the noise is what makes the real finding invisible.

I had a C compiler in a production image for months. It wasn’t hidden. It was on line 14 of a file I wrote, sitting in a report I’d already looked at and decided was mostly other people’s problem.

The wall of red wasn’t the danger.

Getting comfortable in front of it was.