OMG: Google nuked TVL from GCP (including DNS)
Up until today, TVL DNS was hosted on Google Cloud, costing a handful of dollars per month which I paid via some credit card.
Earlier today I tried to change the country of my Play Store account (a futile endeavour, really, because none of the involved Google things really work at all) and it ended up with Google disabling my payment account completely because they think it's "suspicious" (which it doesn't say directly, but you can find out by Googling the weird error codes you start getting).
Anyways, Google Cloud has no grace period for this kind of stuff, so in the moment Google does something to your billing account they completely nuke all of your services.
This means we instantly lost:
- all DNS for tvl.fyi, tvl.su and nixery.dev
- whitby's backup storage
I have restored the DNS entries by moving the zones to GleSYS, but nameserver updates and stuff take time. From many perspectives it looks like tvl.fyi is mostly back, and tvl.su is still mostly down.
Followup action items:
- Remove all dependencies on Google stuff. The Restic backups need to go somewhere else, the email setup for tvl.su needs to change.
- Figure out where DNS should be permanently (GleSYS has no zone file import and setting this up was painful by hand, but it was what I had available).
Status as of this comment:
All important entries have been recreated in GleSYS for the zones tvl.fyi, tvl.su, nixery.dev, tazj.in
The NS record changes are being served already, so based on cache settings things are starting to be up again.
I verified email receiving for tvl.su and tazj.in, but not sending (DKIM and stuff is probably broken right now).
tazjin at 2021-11-19T23·20+00
Current status:
- DNS is fully back up
- Email receiving and sending (with DKIM) works
- Whitby backups are now on GleSYS object storage
That means all active problems are mitigated. There are some followup action items that are still outstanding:
These will be followed up separately.
tazjin at 2021-11-21T12·18+00
- tazjin closed this issue at 2021-11-21T12·18+00