//tvix/boot/tests fail to execute on aarch64-linux

#407
Opened by flokli at 2024-06-30T11·29+00

Running any of these on a aarch64-linux machine currently fails with the following message:

Error booting VM: VmBoot(UefiLoad(UefiTooBig))

This is cloud-hypervisor complaining about a too large kernel.

Indeed it looks like we have a 17M bzImage file as kernel on x86_64-linux, and a 518M vmlinux file on aarch64-linux.

It seems this should be possible to do these days, thanks to the EFI_ZBOOT kernel option (https://lwn.net/Articles/907679/).

There's been a nixpkgs PR at https://github.com/NixOS/nixpkgs/pull/239721 starting to enable this.

However, https://github.com/NixOS/nixpkgs/blob/df708463c90db17e8e5984b4ff98094f791622eb/lib/systems/platforms.nix#L379 would also need to be set (https://github.com/NixOS/nixpkgs/pull/239721#issuecomment-1607803458), prompting a larger refactor in https://github.com/NixOS/nixpkgs/pull/244706 instead.

Both are still marked as draft and somewhat stale. We should see if they can be resurrected and updated or re-rolled.

  1. Why are you using vmlinux instead of Image (which for me is only 70M, likely in large part due to being stripped) on aarch64?

    qyliss at 2024-06-30T13·35+00

  2. If I build tvix.boot.kernel.dev, that's the only thing in the output (root), and it's much larger on aarch64 than x86.

    flokli at 2024-07-01T06·31+00

  3. That's exactly why it's in dev — it's there for debugging, etc. The normal, shrunk kernel image is in the default output, i.e. ${linux}/${stdenv.hostPlatform.linux-kernel.target}. You used to not be able to use that with x86_64, because for a while Cloud Hypervisor removed bzImage support, but to my knowledge there was never an equivalent situation for aarch64.

    qyliss at 2024-07-01T06·33+00

  4. I added this to https://cl.tvl.fyi/c/depot/+/11905, and now at least cloud-hypervisor is happy with the kernel.

    It seems the kernel still isn't happy with the our pid1 in initrd, or our virtiofs, but that's something to look into next.

    flokli at 2024-07-01T06·54+00