The certificate’s gone. Kaput. Not just expired, but expired days ago, and nobody raised a digital eyebrow until the website started throwing browser-level tantrums. Running Caddy on a systemd-based Linux box, you figured ACME renewals were humming along. The logs, bless their heart, said nothing. Nothing at all.
Except they were. Or rather, systemd-resolved was. This charming piece of network plumbing has a peculiar habit: it lies about DNS. Not always. Not to everyone. But sometimes, under specific, maddeningly inconsistent conditions related to upstream resolvers, it just returns SERVFAIL. For certain queries. And Caddy, ever the diligent server, sees this failure and quietly aborts its certificate renewal. Poof. No new cert. Just a slow, creeping digital decay.
Here’s the kicker: your systemd-resolved status shows zilch. dig might point to 8.8.8.8 and get a perfectly reasonable answer. The problem isn’t the internet. It’s the little stub resolver in your own box, whispering sweet nothings about connectivity while actively sabotaging your applications. And it keeps its little secrets well hidden, logging nothing of consequence for your debugging pleasure.
The Unseen Saboteur
This is the kind of bug that makes you question your sanity. You check Caddy. Is Caddy broken? Nope. Is ACME broken? Nope. Is your DNS provider down? Heavens, no, everything else is working fine. It’s the silent, invisible rot that festers in the most mundane parts of your infrastructure.
The stub resolver is the one lying to your application, and it doesn’t log it anywhere useful.
That quote right there? It’s the whole damn story. It’s a perfect encapsulation of the sheer, unadulterated frustration that comes with debugging a problem that’s actively hiding from you. It’s not a loud siren; it’s a whisper in the dark.
How to Stop the Digital Decay
So, what do you do when your systemd-resolved decides to play hide-and-seek with your DNS queries? The article offers a few palliative measures. First, you can tell Caddy to just ignore the system’s DNS configuration entirely. Point it straight at Cloudflare’s 1.1.1.1 or Google’s 8.8.8.8 within your Caddyfile. Easy enough.
{
servers :443 {
dns resolver 1.1.1.1
}
}
Alternatively, you can play with Go’s GODEBUG environment variable, forcing the application’s networking stack to use its own resolver instead of the system’s questionable judgment. That’s a bit more… opaque. For the less adventurous, a simple systemctl restart systemd-resolved will clear the cobwebs. But it’s a band-aid. You’ll be back here again.
Why is This Happening?
This isn’t just a Caddy problem, mind you. This is a systemd-resolved problem. A fundamental issue with how modern Linux systems handle name resolution. We’ve layered abstractions upon abstractions, and sometimes, those layers start doing their own thing. We’re so busy congratulating ourselves on sophisticated network stacks that we forget the basics. Does DNS work? Yes. Usually. And that’s the problem.
If you’re running Caddy on systemd-resolved and you’ve experienced mysterious cert expirations, the solution isn’t a deep dive into Caddy’s ACME client. It’s a blunt investigation into your local DNS resolver. It’s the kind of thing that wastes an entire afternoon, maybe more, leaving you wondering who let the digital sheep out of the pasture.
It’s a stark reminder: even the most polished open-source tools can be tripped up by the most mundane, yet critical, system components. Especially when those components are selectively lying to you.
