Using QEMU and binfmt_misc to chroot into an aarch64 file system

The initial situation

This is where I was when I started this adventure:

• I have a Raspberry Pi 4 running Arch Linux ARM. The initramfs is generated using dracut, because I’m using dracut-sshd to remotely unlock a LUKS2-encrypted volume.
• I changed something in my dracut configuration, manually regenerated the initramfs, rebooted, and was unable to SSH into the RPi.

My conclusion was that I probably messed up the command to manually create the initramfs. An easy way to fix it should be to re-install the linux-rpi package, which automatically results in an updated initramfs being generated.

So I did the following:

1. Turn off the RPi, remove the SD card and connect it to my PC using a USB adapter.
2. Mount the card’s data partition at /mnt, and the boot partition at /mnt/boot.
3. Chroot into the filesystem using arch-chroot.

When running step 3, I was greeted by this error:

chroot: failed to run command ‘/bin/bash’: Exec format error

I then realized that I was trying to chroot into a filesystem belonging to an aarch64 machine using an x64 machine. I can’t just execute the binaries in the chroot, as they were compiled for a different architecture.

The almost-solution

Searching for solutions online led me to to this StackOverflow answer, which I’ll reproduce here:

1. Install the qemu-user-static package (or whatever the name is in your Linux distribution).
2. Copy the newly installed qemu-$YOUR_ARCH_HERE-static binary to the mounted filesystem, in my case: sudo cp /usr/bin/qemu-aarch64-static /mnt/usr/bin.
3. Run sudo arch-chroot /mnt qemu-aarch64-static /bin/bash to use QEMU to run the native bash executable.

Depending on which distribution you use, this might already be all you need to do (read on for why). In my case (Arch Linux) however, the above instructions got me a working shell inside the chroot, but trying to execute any command again gave the “Exec format” error:

bash: /usr/bin/ls: cannot execute binary file: Exec format error

I suppose this is because bash spawns the process that runs the command in such a way that it ends up outside of the usermode emulation again.

You can get around this by manually using qemu-aarch64-static again, e.g. running qemu-aarch64-static /bin/ls, but this is (a) cumbersome and (b) doesn’t work if you want to run a shell script.

The solution

Luckily, the Linux kernel has a way that enables you to seamlessly run binaries compiled for other architectures, if you have the right interpreter at hand (QEMU in our case) and configure the kernel correctly. This nifty feature is called binfmt_misc.

First, check if the feature is already active on your machine by verifying the /proc/sys/fs/binfmt_misc directory exists. If it doesn’t, you can enable the feature with the following command:

mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

As we already have the interpreter installed, we just need to do the configuration part. If you’re interested in the details, feel free to read the binfmt_misc documentation linked above, I’ll just paste the script that configures everything here. I took the magic and mask values from this QEMU script. If you want use this yourself for a different architecture, you’ll need to use the correct magic, mask, and arch values.

magic='\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00'
mask='\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff'
arch='aarch64'
interpreter="/usr/bin/qemu-$arch-static"
echo ":qemu-$arch:M::$magic:$mask:$interpreter:" | sudo tee /proc/sys/fs/binfmt_misc/register

There should now be a /proc/sys/fs/binfmt_misc/qemu-$arch file with contents similar to these:

enabled
interpreter /usr/bin/qemu-aarch64-static
flags:
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

And now, we can simply run sudo arch-chroot /mnt, and everything should work as expected!

Of course, after I’d done the manual setup, I found this page in the Arch Linux Wiki, where I learnt that installing the qemu-user-binfmt package and then running sudo systemctl restart systemd-binfmt would’ve done the configuration for me.

I don’t regret doing it manually, though, because I now know a bit more about binfmt_misc that I would have otherwise, and also using the config from that package doesn’t work for me. arch-chroot always exits with this error:

chroot: failed to run command ‘/bin/bash’: No such file or directory

I suspect this is because the package’s config sets the P flag, and I didn’t do that in my manual config. This can probably be fixed somehow, but for now I’m happy with using my manual config.