OMG: Many local users purged from whitby

#109
Opened by tazjin at 2021-04-09T15·46+00

Due to a mistake in [cl/2905](https://cl.tvl.fyi/c/depot/+/2905/) most local users have been purged from whitby. We've been meaning to test whether whitby is rebootable so, well, we get to find out now.

The system was switched to the broken system with test not switch.

  1. Okay, while things are still serving I'm taking the following precautions:

    1. Proactive asking Hetzner to attach a KVM console.
    2. Validating that we have access to the SSH keys that are in the initrd.
    3. Validating that we have access to the disk encryption password.

    tazjin at 2021-04-09T15·49+00

  2. Sent request for attaching the console to Hetzner at Fri Apr 9 03:53:12 PM UTC 2021. Next update when they confirm console attachment.

    In the meantime, note that no service users are affected - only interactive users.

    tazjin at 2021-04-09T15·54+00

  3. Console is not yet attached, but the recovery plan is the following:

    1. Attempt to attach via the console and log into the root account - I may have the password to that. If it succeeds, restore the system to canon.

    2. If it fails, reboot the system via the console and attempt to unlock the disk on boot over SSH.

    tazjin at 2021-04-09T16·04+00

  4. Update: Nothing from Hetzner yet.

    tazjin at 2021-04-09T16·32+00

  5. Update: Console has been attached, starting on recovery plan #1.

    tazjin at 2021-04-09T16·40+00

  6. Recovery plan #1 worked, whitby is canonical!

    tazjin at 2021-04-09T16·44+00

  7. tazjin closed this issue at 2021-04-09T16·44+00