OMG: bugry: nginx doesn't come up because of broken certs for tvixbolt.tvl.su

(This incident has been resolved, just opening this to document the issue.)

After updating the DNS record for tvixbolt.tvl.su and redeploying bugry, nginx exited and failed to be restarted at some later point (possibly after acme for tvixbolt.tvl.su suceeded for the first time):

Sep 02 15:08:34 bugry systemd[1]: Starting Nginx Web Server...
Sep 02 15:08:35 bugry nginx-pre-start[3992199]: nginx: [emerg] SSL_CTX_use_PrivateKey("/var/lib/acme/tvixbolt.tvl.su/key.pem") failed (SSL: error:05800074:x509 certificate routines::key values mismatch)
Sep 02 15:08:35 bugry nginx-pre-start[3992199]: nginx: configuration file /etc/nginx/nginx.conf test failed
Sep 02 15:08:35 bugry systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Sep 02 15:08:35 bugry systemd[1]: nginx.service: Failed with result 'exit-code'.
Sep 02 15:08:35 bugry systemd[1]: Failed to start Nginx Web Server.

I don't know what the precise issue is. This issue seems to be somewhat common when certificates are in the wrong order in bundle according to some googling I did.

I “resolved” this issue by simply kicking the acme cert request again:

1. Temporarily disable tvixbolt.tvl.su.
2. mv /var/lib/acme/tvixbolt.tvl.su /root/2025-09-02-broken-acme-dir-for-tvixbolt.tvl.su
3. Re-enable tvixbolt.tvl.su, deploy

I haven't deleted 2025-09-02-broken-acme-dir-for-tvixbolt.tvl.su yet, but I doubt we'll need it again.

1. sterni closed this issue at 2025-09-02T15·27+00