OMG: Google nuked TVL from GCP (including DNS)

#155
Opened by tazjin at 2021-11-19T22·53+00

Up until today, TVL DNS was hosted on Google Cloud, costing a handful of dollars per month which I paid via some credit card.

Earlier today I tried to change the country of my Play Store account (a futile endeavour, really, because none of the involved Google things really work at all) and it ended up with Google disabling my payment account completely because they think it's "suspicious" (which it doesn't say directly, but you can find out by Googling the weird error codes you start getting).

Anyways, Google Cloud has no grace period for this kind of stuff, so in the moment Google does something to your billing account they completely nuke all of your services.

This means we instantly lost:

I have restored the DNS entries by moving the zones to GleSYS, but nameserver updates and stuff take time. From many perspectives it looks like tvl.fyi is mostly back, and tvl.su is still mostly down.

Followup action items:

  1. Status as of this comment:

    All important entries have been recreated in GleSYS for the zones tvl.fyi, tvl.su, nixery.dev, tazj.in

    The NS record changes are being served already, so based on cache settings things are starting to be up again.

    I verified email receiving for tvl.su and tazj.in, but not sending (DKIM and stuff is probably broken right now).

    tazjin at 2021-11-19T23·20+00

  2. Current status:

    • DNS is fully back up
    • Email receiving and sending (with DKIM) works
    • Whitby backups are now on GleSYS object storage

    That means all active problems are mitigated. There are some followup action items that are still outstanding:

    These will be followed up separately.

    tazjin at 2021-11-21T12·18+00

  3. tazjin closed this issue at 2021-11-21T12·18+00