Emanuele Rocca

Murder Mystery: GCC Builds Failing After sbuild Refactoring

ema@linux.it (Emanuele Rocca) — Fri, 13 Dec 2024 16:31:40 +0100

This is the story of an investigation conducted by Jochen Sprickerhof, Helmut Grohne, and myself. It was true teamwork, and we would have not reached the bottom of the issue working individually. We think you will find it as interesting and fun as we did, so here is a brief writeup. A few of the steps mentioned here took several days, others just a few minutes. What is described as a natural progression of events did not always look very obvious at the moment at all.

Let us go through the Six Stages of Debugging together.

Stage 1: That cannot happen

Official Debian GCC builds start failing on multiple architectures in late November.

The build error happens on the build servers when running the testuite, but we know this cannot happen. GCC builds are not meant to fail in case of testsuite failures! Return codes are not making the build fail, make is being called with -k, it just cannot happen.

A lot of the GCC tests are always failing in fact, and an extensive log of the results is posted to the debian-gcc mailing list, but the packages always build fine regardless.

On the build daemons, build failures take several hours.

Stage 2: That does not happen on my machine

Building on my machine running Bookworm is just fine. The Build Daemons run Bookworm and use a Sid chroot for the build environment, just like I am. Same kernel.

Debian packages are built by a network of autobuilding machines using a program called sbuild. In my last blog post I mentioned the transition from the schroot backend to a new one based on unshare.

The only obvious difference between my setup and the Debian buildds is that I am using sbuild 0.85.0 from bookworm, and the buildds have 0.86.3~bpo12+1 from bookworm-backports. Trying again with 0.86.3~bpo12+1, the build fails on my system too. The build daemons were updated to the bookworm-backports version of sbuild at some point in late November. Ha.

Stage 3: That should not happen

There are quite a few sbuild versions in between 0.85.0 and 0.86.3~bpo12+1, but looking at recent sbuild bugs shows that sbuild 0.86.0 was breaking "quite a number of packages". Indeed, with 0.86.0 the build still fails. Trying the version immediately before, 0.85.11, the build finishes correctly. This took more time than it sounds, one run including the tests takes several hours. We need a way to shorten this somehow.

The Debian packaging of GCC allows to specify which languages you may want to skip, and by default it builds Ada, Go, C, C++, D, Fortran, Objective C, Objective C++, M2, and Rust. When running the tests sequentially, the build logs stop roughly around the tests of a runtime library for D, libphobos. So can we still reproduce the failure by skipping everything except for D? With DEB_BUILD_OPTIONS=nolang=ada,go,c,c++,fortran,objc,obj-c++,m2,rust the build still fails, and it fails faster than before. Several minutes, not hours. This is progress, and time to file a bug. The report contains massive spoilers, so no link. :-)

Stage 4: Why does that happen?

Something is causing the build to end prematurely. It’s not the OOM killer, and the kernel does not have anything useful to say in the logs. Can it be that the D language tests are sending signals to some process, and that is what’s killing make ? We start tracing signals sent with bpftrace by writing the following script, signals.bt:

tracepoint:signal:signal_generate {
    printf("%s PID %d (%s) sent signal %d to PID %d\n", comm, pid, args->sig, args->pid);
}

And executing it with sudo bpftrace signals.bt.

The build takes its sweet time, and it fails. Looking at the trace output there’s a suspicious process.exe terminating stuff.

process.exe (PID: 2868133) sent signal 15 to PID 711826

That looks interesting, but we have no clue what PID 711826 may be. Let’s change the script a bit, and trace signals received as well.

tracepoint:signal:signal_generate {
    printf("PID %d (%s) sent signal %d to %d\n", pid, comm, args->sig, args->pid);
}

tracepoint:signal:signal_deliver {
    printf("PID %d (%s) received signal %d\n", pid, comm, args->sig);
}

The working version of sbuild was using dumb-init, whereas the new one features a little init in perl. We patch the current version of sbuild by making it use dumb-init instead, and trace two builds: one with the perl init, one with dumb-init.

Here are the signals observed when building with dumb-init.

PID 3590011 (process.exe) sent signal 2 to 3590014
PID 3590014 (sleep) received signal 9
PID 3590011 (process.exe) sent signal 15 to 3590063
PID 3590063 (std.process tem) received signal 9
PID 3590011 (process.exe) sent signal 9 to 3590065
PID 3590065 (std.process tem) received signal 9

And this is what happens with the new init in perl:

PID 3589274 (process.exe) sent signal 2 to 3589291
PID 3589291 (sleep) received signal 9
PID 3589274 (process.exe) sent signal 15 to 3589338
PID 3589338 (std.process tem) received signal 9
PID 3589274 (process.exe) sent signal 9 to 3589340
PID 3589340 (std.process tem) received signal 9
PID 3589274 (process.exe) sent signal 15 to 3589341
PID 3589274 (process.exe) sent signal 15 to 3589323
PID 3589274 (process.exe) sent signal 15 to 3589320
PID 3589274 (process.exe) sent signal 15 to 3589274
PID 3589274 (process.exe) received signal 9
PID 3589341 (sleep) received signal 9
PID 3589273 (sbuild-usernsex) sent signal 9 to 3589320
PID 3589273 (sbuild-usernsex) sent signal 9 to 3589323

There are a few additional SIGTERM being sent when using the perl init, that’s helpful. At this point we are fairly convinced that process.exe is worth additional inspection. The source code of process.d shows something interesting:

1221 @system unittest
1222 {
[...]
1247     auto pid = spawnProcess(["sleep", "10000"],
[...]
1260     // kill the spawned process with SIGINT
1261     // and send its return code
1262     spawn((shared Pid pid) {
1263         auto p = cast() pid;
1264         kill(p, SIGINT);

So yes, there’s our sleep and the SIGINT (signal 2) right in the unit tests of process.d, just like we have observed in the bpftrace output.

Can we study the behavior of process.exe in isolation, separatedly from the build? Indeed we can. Let’s take the executable from a failed build, and try running it under /usr/libexec/sbuild-usernsexec.

First, we prepare a chroot inside a suitable user namespace:

unshare --map-auto --setuid 0 --setgid 0 mkdir /tmp/rootfs
cd /tmp/rootfs
cat /home/ema/.cache/sbuild/unstable-arm64.tar | unshare --map-auto --setuid 0 --setgid 0 tar xf  -
unshare --map-auto --setuid 0 --setgid 0 mkdir /tmp/rootfs/whatever
unshare --map-auto --setuid 0 --setgid 0 cp process.exe /tmp/rootfs/

Now we can run process.exe on its own using the perl init, and trace signals at will:

/usr/libexec/sbuild-usernsexec --pivotroot --nonet u:0:100000:65536  g:0:100000:65536 /tmp/rootfs ema /whatever -- /process.exe

We can compare the behavior of the perl init vis-a-vis the one using dumb-init in milliseconds instead of minutes.

Stage 5: Oh, I see.

Why does process.exe send more SIGTERMs when using the perl init is now the big question. We have a simple reproducer, so this is where using strace becomes possible.

sudo strace --user ema --follow-forks -o sbuild-dumb-init.strace ./sbuild-usernsexec-dumb-init --pivotroot --nonet u:0:100000:65536  g:0:100000:65536 /tmp/dumbroot ema /whatever -- /process.exe

We start comparing the strace output of dumb-init with that of perl-init, looking in particular for different calls to kill.

Here is what process.exe does under dumb-init:

3593883 kill(-2, SIGTERM)               = -1 ESRCH (No such process)

No such process. Under perl-init instead:

3593777 kill(-2, SIGTERM <unfinished ...>

The process is there under perl-init!

That is a kill with negative pid. From the kill(2) man page:

If pid is less than -1, then sig is sent to every process in the process group whose ID is -pid.

It would have been very useful to see this kill with negative pid in the output of bpftrace, why didn’t we? The tracepoint used, tracepoint:signal:signal_generate, shows when signals are actually being sent, and not the syscall being called. To confirm, one can trace tracepoint:syscalls:sys_enter_kill and see the negative PIDs, for example:

PID 312719 (bash) sent signal 2 to -312728

The obvious question at this point is: why is there no process group 2 when using dumb-init?

Stage 6: How did that ever work?

We know that process.exe sends a SIGTERM to every process in the process group with ID 2. To find out what this process group may be, we spawn a shell with dumb-init and observe under /proc PIDs 1, 16, and 17. With perl-init we have 1, 2, and 17. When running dumb-init, there are a few forks before launching the program, explaining the difference. Looking at /proc/2/cmdline we see that it’s bash, ie. the program we are running under perl-init. When building a package, that is dpkg-buildpackage itself.

The test is accidentally killing its own process group.

Now where does this -2 come from in the test?

2363     // Special values for _processID.
2364     enum invalid = -1, terminated = -2;

Oh. -2 is used as a special value for PID, meaning "terminated". And there’s a call to kill() later on:

2694     do { s = tryWait(pid); } while (!s.terminated);
[...]
2697     assertThrown!ProcessException(kill(pid));

What sets pid to terminated you ask?

Here is tryWait:

2568 auto tryWait(Pid pid) @safe
2569 {
2570     import std.typecons : Tuple;
2571     assert(pid !is null, "Called tryWait on a null Pid.");
2572     auto code = pid.performWait(false);

And performWait:

2306         _processID = terminated;

The solution, dear reader, is not to kill.

PS: the bug report with spoilers for those interested is #1089007.

Building Debian packages The Right Way

ema@linux.it (Emanuele Rocca) — Tue, 26 Nov 2024 09:34:40 +0100

There is more than one way to do it, but it seems that The Right Way to build Debian packages today is using sbuild with the unshare backend. The most common backend before the rise of unshare was schroot.

The official Debian Build Daemons have recently transitioned to using sbuild with unshare, providing a strong motivation to consider making the switch. Additionally the new approach means: (1) no need to configure schroot, and (2) no need to run the build as root.

Here are my notes about moving to the new setup, for future reference and in case they may be useful to others.

First I installed the required packages:

apt install sbuild mmdebstrap uidmap

Then I created the following script to update my chroots every night:

#!/bin/bash

for arch in arm64 armhf armel; do
    HOME=/tmp mmdebstrap --quiet --arch=$arch --include=ca-certificates --variant=buildd unstable \
        ~/.cache/sbuild/unstable-$arch.tar http://127.0.0.1:3142/debian
done

In the script, I’m calling mmdebstrap with --quiet because I don’t want to get any output on succesful execution. The script is running in cron with email notifications, and I only want to get a message if something goes south. I’m setting HOME=/tmp for a similar reason: the unshare user does not have access to my actual home directory, and by default dpkg tries to use $HOME/.dpkg.cfg as the configuration file. By overriding HOME I avoid the following message to standard error:

dpkg: warning: failed to open configuration file '/home/ema/.dpkg.cfg' for reading: Permission denied

Then I added the following to my sbuild configuration file (~/.sbuildrc):

$chroot_mode = 'unshare';
$unshare_tmpdir_template = '/dev/shm/tmp.sbuild.XXXXXXXXXX';

The first option sets the sbuild backend to unshare, whereas unshare_tmpdir_template is needed on Bookworm to ensure that the build process runs in memory rather than on disk for performance reasons. Starting with Trixie, /tmp is by default a tmpfs so the setting won’t be needed anymore.

Packages for different architectures can now be built as follows:

# Tarball used: ~/.cache/sbuild/unstable-arm64.tar
$ sbuild --dist=unstable hello

# Tarball used: ~/.cache/sbuild/unstable-armhf.tar
$ sbuild --dist=unstable --arch=armhf hello

If you have any comments or suggestions about any of this, please let me know.

PGP keys on Yubikey, with a side of Mutt

ema@linux.it (Emanuele Rocca) — Fri, 05 Apr 2024 15:22:40 +0200

Here are my notes about copying PGP keys to external hardware devices such as Yubikeys. Let me begin by saying that the gpg tools are pretty bad at this.

MAKE A COUPLE OF BACKUPS OF ~/.gnupg/ TO DIFFERENT ENCRYPTED USB STICKS BEFORE YOU START. GPG WILL MESS UP YOUR KEYS. SERIOUSLY.

For example, would you believe me if I said that saving changes results in the removal of your private key? Well check this out.

Now that you have multiple safe, offline backups of your keys, here are my notes.

apt install yubikey-manager scdaemon

Plug the Yubikey in, see if it’s recognized properly:

ykman list
gpg --card-status

Change the default PIN (123456) and Admin PIN (12345678):

gpg --card-edit
gpg/card> admin
gpg/card> passwd

Look at the openpgp information and change the maximum number of retries, if you like. I have seen this failing a couple of times, unplugging the Yubikey and putting it back in worked.

ykman openpgp info
ykman openpgp access set-retries 7 7 7

Copy your keys. MAKE A BACKUP OF ~/.gnupg/ BEFORE YOU DO THIS.

gpg --edit-key $KEY_ID
gpg> keytocard # follow the prompts to copy the first key

Now choose the next key and copy that one too. Repeat till all subkeys are copied.

gpg> key 1
gpg> keytocard

Typing gpg --card-status you should be able to see all your keys on the Yubikey now.

Using the key on another machine

How do you use your PGP keys on the Yubikey on other systems?

Go to another system, if it does have a ~/.gnupg directory already move it somewhere else.

apt install scdaemon

Import your public key:

gpg -k
gpg --keyserver pgp.mit.edu --recv-keys $KEY_ID

Check the fingerprint and if it is indeed your key say you trust it:

gpg --edit-key $KEY_ID
> trust
> 5
> y
> save

Now try gpg --card-status and gpg --list-secret-keys, you should be able to see your keys. Try signing something, it should work.

gpg --output /tmp/x.out --sign /etc/motd
gpg --verify /tmp/x.out

Using the Yubikey with Mutt

If you’re using mutt with IMAP, there is a very simple trick to safely store your password on disk. Write down your IMAP password to a file in a safe location, say ~/.mutt_password. Then encrypt it and remove the plain-text file:

gpg --encrypt ~/.mutt_password # This will create ~/.mutt_password.gpg
rm ~/.mutt_password

Add the following to ~/.muttrc:

set imap_pass=`gpg --decrypt ~/.mutt_password.gpg`

With the above, mutt now prompts you to insert the Yubikey and type your PIN in order to connect to the IMAP server.

Enabling Kernel Settings in Debian

ema@linux.it (Emanuele Rocca) — Mon, 12 Feb 2024 09:52:40 +0200

This time it’s about enabling new kernel config options in the official Debian kernel packages. A few dependencies are needed to run the various scripts used by the Debian kernel folks, as well as to build the kernel itself:

apt install git gpg python3-debian python3-dacite
apt build-dep linux

With that in place, fetch the linux and kernel-team repos:

git clone --depth 1 https://salsa.debian.org/kernel-team/linux
git clone --depth 1 https://salsa.debian.org/kernel-team/kernel-team

So far you’ve only got the Debian-specific bits. Fetch the actual kernel sources now. In the likely case that you’re building a stable kernel, run the following from within the linux directory:

debian/bin/genorig.py https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

Use the torvalds repo if you’re building an RC version instead:

debian/bin/genorig.py https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Now generate the upstream tarball as well as debian/control. The first command will take a bit, and the second command will fail: but that’s success — just as the output says.

debian/rules orig
debian/rules debian/control

Now generate patched sources with:

debian/rules source

Time to edit the Kconfig and enable/disable whatever setting you wanted to change. Take a look around the files under debian/config/ to see where your changes should go. If it’s a setting shared among multiple architectures that may be debian/config/config. For x86-specific things, the file is debian/config/amd64/config. On aarch64 debian/config/arm64/config. If in doubt, you could try asking #debian-kernel on IRC.

It may look like you need to figure out where exactly in the file the setting should be placed. That is not the case. There’s a helpful script fixing things up for you:

../kernel-team/utils/kconfigeditor2/process.py .

The above will fail if you forgot to run debian/rules source. The debian/build/source_rt/Kconfig file is needed by the script:

Traceback (most recent call last):
  File "/tmp/linux/../kernel-team/utils/kconfigeditor2/process.py", line 19, in __init__
    menu = fs_menu[featureset or 'none']
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'rt'

During handling of the above exception, another exception occurred:
[...]
FileNotFoundError: [Errno 2] No such file or directory: './debian/build/source_rt/Kconfig'

If that happens, run:

debian/rules source

Now process.py should work fine and fix your config file.

Excellent, now the config is updated and we’re ready to build the kernel. Off we go:

export MAKEFLAGS=-j$(nproc)
export DEB_BUILD_PROFILES='pkg.linux.nokerneldbg pkg.linux.nokerneldbginfo pkg.linux.notools nodoc'
dpkg-buildpackage -b -nc -uc

Custom Debian Installer and Kernel on a USB stick

ema@linux.it (Emanuele Rocca) — Fri, 06 Oct 2023 11:29:40 +0200

There are many valid reasons to create a custom Debian Installer image. You may need to pass some special arguments to the kernel, use a different GRUB version, automate the installation by means of preseeding, use a custom kernel, or modify the installer itself.

If you have a EFI system, which is probably the case in 2023, there is no need to learn complex procedures in order to create a custom Debian Installer stick.

The source of many frustrations is that the ISO format for CDs/DVDs is read-only, but you can just create a VFAT filesystem on a USB stick, copy all ISO contents onto the stick itself, and modify things at will.

Create a writable USB stick

First create a FAT32 filesystem on the removable device and mount it. The device is sdX in the example.

$ sudo parted --script /dev/sdX mklabel msdos
$ sudo parted --script /dev/sdX mkpart primary fat32 0% 100%
$ sudo mkfs.vfat /dev/sdX1
$ sudo mount /dev/sdX1 /mnt/data/

Then copy to the USB stick the installer ISO you would like to modify, debian-testing-amd64-netinst.iso here.

$ sudo kpartx -v -a debian-testing-amd64-netinst.iso

# Mount the first partition on the ISO and copy its contents to the stick
$ sudo mount /dev/mapper/loop0p1 /mnt/cdrom/
$ sudo rsync -av /mnt/cdrom/ /mnt/data/
$ sudo umount /mnt/cdrom

# Same story with the second partition on the ISO
$ sudo mount /dev/mapper/loop0p2 /mnt/cdrom/
$ sudo rsync -av /mnt/cdrom/ /mnt/data/
$ sudo umount /mnt/cdrom

$ sudo kpartx -d debian-testing-amd64-netinst.iso
$ sudo umount /mnt/data

Now try booting from the USB stick just to verify that everything went well and we can start customizing the image.

Boot loader, preseeding, installer hacks

The easiest things we can change now are the shim, GRUB, and GRUB’s configuration. The USB stick contains the shim under /EFI/boot/bootx64.efi, while GRUB is at /EFI/boot/grubx64.efi. This means that if you want to test a different shim / GRUB version, you just replace the relevant files. That’s it. Take for example /usr/lib/grub/x86_64-efi/monolithic/grubx64.efi from the package grub-efi-amd64-bin, or the signed version from grub-efi-amd64-signed and copy them under /EFI/boot/grubx64.efi. Or perhaps you want to try out systemd-boot? Then take /usr/lib/systemd/boot/efi/systemd-bootx64.efi from the package systemd-boot-efi, copy it to /EFI/boot/bootx64.efi and you’re good to go. Figuring out the right systemd-boot configuration needed to start the Installer is left as an exercise.

By editing /boot/grub/grub.cfg you can pass arbitrary arguments to the kernel and the Installer itself. See the official Installation Guide for a comprehensive list of boot parameters.

One very commong thing to do is automating the installation using a preseed file. Add the following to the kernel command line: preseed/file=/cdrom/preseed.cfg and create a /preseed.cfg file on the USB stick. As a little example:

d-i time/zone select Europe/Rome
d-i passwd/root-password this-is-the-root-password
d-i passwd/root-password-again this-is-the-root-password
d-i passwd/user-fullname string Emanuele Rocca
d-i passwd/username string ema
d-i passwd/user-password password lol-haha-uh
d-i passwd/user-password-again password lol-haha-uh
d-i apt-setup/no_mirror boolean true
d-i popularity-contest/participate boolean true
tasksel tasksel/first multiselect standard

See Steve McIntyre’s awesome page with the full list of available settings and their description: https://preseed.einval.com/debian-preseed/.

Two noteworthy settings are early_command and late_command. They can be used to execute arbitrary commands and provide thus extreme flexibility! You can go as far as replacing parts of the installer with a sed command, or maybe wgetting an entirely different file. This is a fairly easy way to test minor Installer patches. As an example, I’ve once used this to test a patch to grub-installer:

d-i partman/early_command string wget https://people.debian.org/~ema/grub-installer-1035085-1 -O /usr/bin/grub-installer

Finally, the initrd contains all early stages of the installer. It’s easy to unpack it, modify whatever component you like, and repack it. Say you want to change a given udev rule:

$ mkdir /tmp/new-initrd
$ cd /tmp/new-initrd
$ zstdcat /mnt/data/install.a64/initrd.gz | sudo cpio -id
$ vi lib/udev/rules.d/60-block.rules
$ find . | cpio -o -H newc | zstd --stdout > /mnt/data/install.a64/initrd.gz

Custom udebs

From a basic architectural standpoint the Debian Installer can be seen as an initrd that loads a series of special Debian packages called udebs. In the previous section we have seen how to (ab)use early_command to replace one of the scripts used by the Installer, namely grub-installer. It turns out that such script is installed by a udeb, so let’s do things right and build a new Installer ISO with our custom grub udeb.

Fetch the code for the grub-installer udeb, make your changes and build it with a classic dpkg-buildpackage -rfakeroot.

Then get the Installer code and install all dependencies:

$ git clone https://salsa.debian.org/installer-team/debian-installer/
$ cd debian-installer/
$ sudo apt build-dep .

Now add the grub-installer udeb to the localudebs directory and create a new netboot image:

$ cp /path/to/grub-installer_1.198_arm64.udeb build/localudebs/
$ cd build
$ fakeroot make clean_netboot build_netboot

Give it some time, soon enough you’ll have a brand new ISO to test under dest/netboot/mini.iso.

Custom kernel

Perhaps there’s a kernel configuration option you need to enable, or maybe you need a more recent kernel version than what is available in sid.

The Debian Linux Kernel Handbook has all the details for how to do things properly, but here’s a quick example.

Get the Debian kernel packaging from salsa and generate the upstream tarball:

$ git clone https://salsa.debian.org/kernel-team/linux/
$ ./debian/bin/genorig.py https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

For RC kernels use the repo from Linus instead of linux-stable.

Now do your thing, for instance change a config setting by editing debian/config/amd64/config. Don’t worry about where you put it in the file, there’s a tool from https://salsa.debian.org/kernel-team/kernel-team to fix that:

$ /path/to/kernel-team/utils/kconfigeditor2/process.py .

Now build your kernel:

$ export MAKEFLAGS=-j$(nproc)
$ export DEB_BUILD_PROFILES='pkg.linux.nokerneldbg pkg.linux.nokerneldbginfo pkg.linux.notools nodoc'
$ debian/rules orig
$ debian/rules debian/control
$ dpkg-buildpackage -b -nc -uc

After some time, if everything went well, you should get a bunch of .deb files as well as a .changes file, linux_6.6~rc3-1~exp1_arm64.changes here. To generate the udebs used by the Installer you need to first get a linux-signed .dsc file, and then build it — with sbuild in this example:

$ /path/to/kernel-team/scripts/debian-test-sign linux_6.6~rc3-1~exp1_arm64.changes
$ sbuild --dist=unstable --extra-package=$PWD linux-signed-arm64_6.6~rc3+1~exp1.dsc

Excellent, now you should have a ton of .udebs. To build a custom installer image with this kernel, copy them all under debian-installer/build/localudebs/ and then run fakeroot make clean_netboot build_netboot as described in the previous section. In case you are trying to use a different kernel version from what is currently in sid, you will have to install the linux-image package on the system building the ISO, and change LINUX_KERNEL_ABI in build/config/common. The linux-image dependency in debian/control probably needs to be tweaked as well.

That’s it, the new Installer ISO should boot with your custom kernel!

There is going to be another minor obstacle though, as anna will complain that your new kernel cannot be found in the archive. Copy the kernel udebs you have built onto a vfat formatted USB stick, switch to a terminal, and install them all with udpkg:

~ # udpkg -i *.udeb

Now the installation should proceed smoothly.

UEFI Secure Boot on the Raspberry Pi

ema@linux.it (Emanuele Rocca) — Thu, 04 May 2023 13:29:40 +0200

UPDATE: this post unexpectedly ended up on Hacker News and I received a lot of comments. The two most important points being made are (1) that Secure Boot on the RPi as described here is not actually truly secure. An attacker who successfully gained root could just mount the firmware partition and either add their own keys to the EFI variable store or replace the firmware altogether with a malicious one. (2) The TianCore firmware cannot be used instead of the proprietary blob as I mentioned. What truly happens is that the proprietary blob is loaded onto the VideoCore cores, then TianoCore is loaded onto the ARM cores. Thanks for the corrections.

A port of the free software TianoCore UEFI firmware can be used instead of the proprietary boot blob to boot the Raspberry Pi. This allows to install Debian on the RPi with the standard Debian Installer, and it also makes it possible to use UEFI Secure Boot. Note that Secure Boot had been broken on arm64 for a while, but it’s now working in Bookworm!.

Debian Installer UEFI boot

To begin, you’ll need to download the appropriate firmware files for the RPi3 or RPi4. I’ve got a Raspberry Pi 3 Model B+ myself, so the rest of this document will assume an RPi3 is being installed.

Plug the SD card you are going to use as the RPi storage device into another system. Say it shows up as /dev/sdf. Then:

# Create an msdos partition table
$ sudo parted --script /dev/sdf mklabel msdos
# Create, format, and label a 10M fat32 partition
$ sudo parted --script /dev/sdf mkpart primary fat32 0% 10M
$ sudo mkfs.vfat /dev/sdf1
$ sudo fatlabel /dev/sdf1 RPI-FW
# Get the UEFI firmware onto the SD card
$ sudo mount /dev/sdf1 /mnt/data/
$ sudo unzip Downloads/RPi3_UEFI_Firmware_v1.38.zip -d /mnt/data/
$ sudo umount /mnt/data

At this point, the SD card can be used to boot the RPi, and you’ll get a UEFI firmware.

Download the Bookworm RC 2 release of the installer, copy it to a USB stick as described in the Installation Guide, and boot your RPi from the stick. If for some reason booting from the stick does not happen automatically, enter the firmware interface with ESC and choose the USB stick from Boot Manager.

Proceed with the installation as normal, paying attention not to modify the firmware partition labeled RPI-FW. I initially thought it would be nice to reuse the firmware partition as ESP partition as well. However, setting the esp flag on makes the RPi unbootable. Either configuring the partition as ESP in debian-installer, or manually with sudo parted --script /dev/sda set 1 esp on, breaks boot. In case you accidentally do that, set it back to off and the edk2 firmware will boot again.

What I suggest doing in terms of partitioning is: (1) leave the 10M partition created above for the firmware alone, and (2) create another 512M or so ESP partition for EFI boot.

The installation should go smoothly till the end, but rebooting won’t work. Doh. This is because of an important gotcha: the Raspberry Pi port of the TianoCore firmware we are using does not support setting UEFI variables persistently from a "High Level Operating System (HLOS)", which is the debian-installer in our case. Persistently is the keyword there: variables can be set and modified regularly — with efibootmgr or otherwise, but crucially the modifications do not survive reboot. However, changes made from the firmware interface itself are persistent. So enter the firmware with ESC right after booting the RPi, select Boot Maintenance Manager → Boot Options → Add Boot Option → Your SD card → Your ESP partition → EFI → debian → shimaa64.efi. Choose a creative name for your boot entry (eg: "debian"), save and exit the firmware interface. Bookworm should be booting fine at this point!

Enabling Secure Boot

Although the TianoCore firmware does support Secure Boot, there are no keys enrolled by default. To add the required keys, copy PK-0001.der, DB-0001.der, DB-0002.der, KEK-0001.der, and KEK-0002.der to a FAT32 formatted USB stick.

Here’s a summary of the Subject field for each of the above:

PK-0001.der.pem
        Subject: O = Debian, CN = Debian UEFI Secure Boot (PK/KEK key), emailAddress = debian-devel@lists.debian.org
DB-0001.der.pem
        Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Microsoft Windows Production PCA 2011
DB-0002.der.pem
        Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Microsoft Corporation UEFI CA 2011
KEK-0001.der.pem
        Subject: O = Debian, CN = Debian UEFI Secure Boot (PK/KEK key), emailAddress = debian-devel@lists.debian.org
KEK-0002.der.pem
        Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Microsoft Corporation KEK CA 2011

Plug the stick into the RPi, boot and enter the firmware interface with ESC. Select Device Manager → Secure Boot Configuration → Secure Boot Mode → choose Custom Mode → Custom Secure Boot Options → PK Options → Enroll PK → choose PK-0001.der. Do the same for DB Options, this time choose DB-0001.der and DB-0002.der. As you may have guessed by now, the same must be done for KEK Options, but adding KEK-0001.der and KEK-0002.der. Save, exit, reboot. If everything went well, your RPi now has booted with Secure Boot enabled.

See https://wiki.debian.org/SecureBoot for the details on how to check whether Secure Boot has been enabled correctly and much more.

EFI and Secure Boot Notes

ema@linux.it (Emanuele Rocca) — Sun, 26 Mar 2023 07:18:40 +0200

To create a bootable EFI drive to use with QEMU, first make a disk image and create a vfat filesystem on it.

$ dd if=/dev/zero of=boot.img bs=1M count=512
$ sudo mkfs.vfat boot.img

By default, EFI firmwares boot a specific file under /efi/boot/. The name of such file depends on the architecture: for example, on 64 bit x86 systems it is bootx64.efi, while on ARM it is bootaa64.efi.

Copy /usr/lib/grub/x86_64-efi/monolithic/grubx64.efi from package grub-efi-amd64-bin to /efi/boot/bootx64.efi on the boot image, and that should be enough to start GRUB.

# mount boot.img /mnt/
# mkdir -p /mnt/efi/boot/
# cp /usr/lib/grub/x86_64-efi/monolithic/grubx64.efi /mnt/efi/boot/bootx64.efi
# umount /mnt/

Now get the x86 firmware from package ovmf and start qemu:

$ cp /usr/share/OVMF/OVMF_CODE.fd /tmp/code.fd
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -cdrom boot.img

GRUB looks fine, but it would be good to have a kernel to boot. Let’s add one to boot.img.

# mount boot.img /mnt
# cp vmlinuz-6.1.0-7-amd64 /mnt/vmlinuz
# umount /mnt/

Boot with qemu again, but this time pass -m 1G. The default amount of memory is not enough to boot.

$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -cdrom boot.img -m 1G

At the grub prompt, type the following to boot:

grub> linux /vmlinuz
grub> boot

The kernel will start and reach the point of trying to mount the root fs. This is great but it would now be useful to have some sort of shell access in order to look around. Let’s add an initrd!

# mount boot.img /mnt
# cp initrd.img-6.1.0-7-amd64 /mnt/initrd
# umount /mnt/

There’s the option of starting qemu in console, let’s try that out. Start qemu with -nographic, and append console=ttyS0 to the kernel command line arguments.

$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -cdrom boot.img -m 1G -nographic
grub> linux /vmlinuz console=ttyS0
grub> initrd /initrd
grub> boot

If all went well we are now in the initramfs shell. We can now run commands! At this point we can see that the system has Secure boot disabled:

(initramfs) dmesg | grep secureboot
[    0.000000] secureboot: Secure boot disabled

In order to boot with Secure boot, we need:

a signed shim, grub, and kernel
the right EFI variables for Secure boot

The package shim-signed provides a shim signed with Microsoft’s key, while grub-efi-amd64-signed has GRUB signed with Debian’s key.

The signatures can be shown with sbverify --list:

$ sbverify --list /usr/lib/shim/shimx64.efi.signed
warning: data remaining[823184 vs 948768]: gaps between PE/COFF sections?
signature 1
image signature issuers:
 - /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
image signature certificates:
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Windows UEFI Driver Publisher
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation Third Party Marketplace Root

Similarly for GRUB and the kernel:

$ sbverify --list /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed
signature 1
image signature issuers:
 - /CN=Debian Secure Boot CA
image signature certificates:
 - subject: /CN=Debian Secure Boot Signer 2022 - grub2
   issuer:  /CN=Debian Secure Boot CA
$ sbverify --list /mnt/vmlinuz
signature 1
image signature issuers:
 - /CN=Debian Secure Boot CA
image signature certificates:
 - subject: /CN=Debian Secure Boot Signer 2022 - linux
   issuer:  /CN=Debian Secure Boot CA

Let’s use the signed shim and grub in the boot image:

# mount boot.img /mnt
# cp /usr/lib/shim/shimx64.efi.signed /mnt/efi/boot/bootx64.efi
# cp /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed /mnt/efi/boot/grubx64.efi
# umount /mnt

And start QEMU with the appropriate EFI variables for Secure boot:

$ cp /usr/share/OVMF/OVMF_VARS.ms.fd /tmp/vars.fd
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -drive file=/tmp/vars.fd,format=raw,if=pflash -cdrom boot.img -m 1G -nographic

We can double-check in the firmware settings if Secure boot is indeed enabled. At the GRUB prompt, type fwsetup:

grub> fwsetup

Check under "Device Manager" → "Secure Boot Configuration" that "Attempt Secure Boot" is selected, then boot from GRUB as before. If all went well, the kernel should confirm that we have booted with Secure boot:

(initramfs) dmesg | grep secureboot
[    0.000000] secureboot: Secure boot enabled

Disposable Debian VMs with debvm

ema@linux.it (Emanuele Rocca) — Wed, 15 Mar 2023 07:18:40 +0200

Some notes on using debvm, an amazing piece of software I’ve started using only recently.

Create a new virtual machine:

$ debvm-create

You now have a virtual machine with Debian Sid of your host native architecture (probably amd64). The image file is called rootfs.ext4. You’ve got 1G of disk space in the VM.

You can now just run the VM! You will be automatically logged is as root.

$ debvm-run

Experiment in the VM, run all the sort of tests you have in mind. For example, one thing I commonly do is verifying that things work in a clean environment, as opposed to "on my machine".

If anything goes wrong, or if you just want to repeat the test: shutdown the guest, remove rootfs.ext4, and start again with debvm-create.

Now, especially if you intend creating and recreating VMs multiple times, it helps to use a local apt mirror as a cache to avoid fetching all the required packages from the internet over and over again. Install apt-cacher-ng on your host machine and point debvm-create at it:

$ debvm-create -- http://10.0.3.1:3142/debian

The additional options after -- are passed to mmdebstrap. In this case we’re specifying http://10.0.3.1:3142/debian as the URL of our local apt mirror. It is going to be used both for image creation and as the only entry in /etc/apt/sources.list on the guest. This is the reason for using 10.0.3.1, the IP address of the lxcbr0 interface used by qemu, instead of 127.0.0.1: to make sure that the guest VM has access to it too.

For a slightly more advanced example, we now want to:

run a arm64 VM
have more disk space availably, say 2G
install additional packages (curl and locales)
allow SSH as root with the given public keys, in this example the authorized_keys installed on the host
start the VM in the background with no console output

$ debvm-create -a arm64 -o sid-arm64.ext4 -z 2G -k ~/.ssh/authorized_keys -- http://10.0.3.1:3142/debian --include curl,locales

Start the VM:

$ debvm-run -i sid-arm64.ext4 -s 2222 -g -- -display none &

SSH into the guest:

$ ssh -o NoHostAuthenticationForLocalhost=yes -p 2222 root@127.0.0.1

Enjoy!

Running Any Distro on Debian

ema@linux.it (Emanuele Rocca) — Sat, 07 Jan 2023 07:18:40 +0200

All graybeard hackers know that: (1) Docker is silly, and (2) containers are very useful. This post is about running pretty much any Linux distribution on top of a Debian host using chroots, which technically speaking aren’t containers but for my purposes pretty much are. The chroot can be used in a disposable fashion or it can be instructed to persist any modifications made to the filesystem. The tools used for this are schroot, debootstrap, rinse, and alpine-chroot-install.

schroot

Back in 2005 when Docker was not a thing yet, lots of the things that Gartner Inc. likes about containers were very much possible with chroot. For example, if all you care about is running commands on a different distro without messing up your own file system, a chroot will do.

Container is just a fancy name for chroot with namespaces FFS.

Anyways, to configure schroot you need to edit /etc/schroot/schroot.conf. For example:

[bullseye]
type=directory
directory=/srv/chroots/bullseye

[fedora-36]
type=file
file=/srv/chroots/fedora-36.tar

The main commands to remember are just two: schroot -c $name and schroot -l: the former starts a shell in the chroot named $name (eg: bullseye in the configuration above). The latter lists all available chroots. That’s it.

ema@pinolo:~$ schroot -l
chroot:bullseye
chroot:fedora-36
ema@pinolo:~$ sudo schroot -c fedora-36
[root@pinolo ema]# cat /etc/redhat-release
Fedora release 36 (Thirty Six)
[root@pinolo ema]#

There are two main types of chroots: directory or file. A directory chroot like bullseye in the example above can be modified, meaning that the results of any command executed within it will persist after leaving the chroot. Conversely the second example is a file chroot (fedora-36). file chroots, are ephemeral: you can use them to try whatever disruptive action you like, and after logging out the chroot will be back to its original stage.

Creating pretty much any distro

One great thing about Docker is that with dockerhub you can fetch a pre-made image of whatever distro you care about. If you’re not careful you end up with a monero miner on your laptop, but I digress.

Luckily for us, Debian has all the tools to bootstrap whatever distro you fancy in a safe way.

Use debootstrap to create a Debian distribution (or derivative, including Ubuntu). For example, to create a directory chroot with Debian Bullseye under /srv/chroots/bullseye:

sudo debootstrap bullseye /srv/chroots/bullseye http://deb.debian.org/debian

Similarly, for a directory chroot with Ubuntu Kinetic:

sudo debootstrap kinetic /srv/chroots/kinetic http://archive.ubuntu.com/ubuntu

See ls /usr/share/debootstrap/scripts/ for the list of all distros supported by debootstrap.

There’s a more modern and under certain aspects better version of debootstrap called mmdebstrap. See the man page for details, but you can create a file chroot with Debian Sid with this simple command:

sudo mmdebstrap sid /srv/chroots/sid.tar

It may be obvious to my astute readers, but let me say it out loud: you can make a file chroot out of a directory chroot with tar:

sudo tar -cvf /srv/chroots/kinetic.tar -C /srv/chroots/kinetic .

To create RedHat and derivatives, use rinse:

sudo rinse --distribution fedora-36 --arch amd64 --directory /srv/chroots/fedora-36

See ls -l /etc/rinse/*.packages for the list of all distros supported by rinse. There’s CentOS, OpenSuse, you name it.

Alpine Linux can be created with alpine-chroot-install as follows:

sudo alpine-chroot-install -d /srv/chroots/alpine/

Technically, pacstrap should be able to bootstrap an Arch Linux installation, but I did not manage to get it to work. If you know how to make it do the right thing, let me know! First I tried:

sudo pacstrap /srv/chroots/arch/
==> Creating install root at /srv/chroots/arch/
==> Installing packages to /srv/chroots/arch/
error: no usable package repositories configured.
==> ERROR: Failed to install packages to new root

To fix that I’ve added the following to pacman.conf:

[core]
Server = https://mirrors.kernel.org/archlinux/$repo/os/$arch

Trying the command again however I just got a lot of failed dependencies, and finally:

:: unable to satisfy dependency 'archlinux-keyring' required by base
==> ERROR: Failed to install packages to new root

That’s when I gave up and followed the Arch wiki, which suggests downloading pre-made tarballs. Hopefully without monero miners.

Testing Debian Python Packages with Autopkgtest

ema@linux.it (Emanuele Rocca) — Thu, 06 Oct 2022 07:18:40 +0200

In the past few years autopkgtest established itself as the system for automated testing of Debian packages.

Although there is good documentation available, both in terms of the command itself and the details about writing tests, I often have to look around and copy-paste code to add automated testing to Python libraries. For this reason, I’m dropping a few notes here. They’re surely going to be useful for me, and hopefully for others too!

Adding tests to a packaged Python library

If the package you’re working with already has a debian/tests/ directory, then congratulations! It already ships some autopkgtests, so you can just look around there. Otherwise, go ahead and create the directory. Then, create a file called debian/tests/control with the following contents:

Test-Command: for py in $(py3versions -s); do echo "[*] testing on $py:"; $py -c 'import module_name' ; done

Replace module_name with the name of the Python module you want to test, and you’re done writing some basic automated testing for your Debian Python package. What the test does is simply trying to import the module, with the added bonus of doing so for all supported python3 versions on the system.

You likely want to test something more than just importing the library. Simply write your test code as a python script (eg: debian/tests/smoke.py) and change debian/tests/control as follows:

Test-Command: for py in $(py3versions -s); do echo "[*] testing on $py:"; $py -c 'import module_name' ; done

Test-Command: for py in $(py3versions -s); do echo "[*] testing on $py:"; $py debian/tests/smoke.py' ; done

By default, autopkgtest installs all the packages generated by the source package containing the tests. If you want to install only one specific package, or if the tests to be executed need another set of packages, use the Depends: field:

Test-Command: for py in $(py3versions -s); do echo "[*] testing on $py:"; $py -c 'import module_name' ; done
Depends: python3-module_name

Test-Command: for py in $(py3versions -s); do echo "[*] testing on $py:"; $py debian/tests/smoke.py' ; done
Depends: python3-module_name, curl

A lot of Debian packages are using autopkgtest. Get inspired by taking a look at some of them with the Debian Code Search query Depends path:debian/tests/control.

Running the tests

There are many ways to execute the tests with autopkgtest on your development machine. The system supports various virtualization servers, depending on the level of isolation you want. LXC, Docker, and QEMU are some of the supported virtualization servers. QEMU is the easiest way I found, and the one providing the best isolation. See isolation-machine in the specifications.

An autopkgtest virtual machine image for QEMU can be created as follows:

sudo autopkgtest-build-qemu unstable /var/tmp/autopkgtest-unstable.img

Once the image is in place, you can execute your tests from within the package source directory with:

autopkgtest -- qemu /var/tmp/autopkgtest-unstable.img

That’s it!

There are multiple knobs to play with, for example in terms of the resources to assign to the VM. For example, you can specify the number of virtual CPUs and the amount of memory:

autopkgtest -- qemu --cpus=8 --ram-size=4096 /var/tmp/autopkgtest-unstable.img

Additionally, you may want to get dropped into a shell in case of tests failure to investigate the situation. That can be done with --shell-fail:

autopkgtest --shell-fail -- qemu --cpus=8 --ram-size=4096 /var/tmp/autopkgtest-unstable.img

See man autopkgtest-virt-qemu(1) for the fine details, and happy testing!

Debugging production: the LIAR method

ema@linux.it (Emanuele Rocca) — Mon, 22 Mar 2021 09:38:00 +0100

The release of DTrace in 2005 marked a turning point in the world of UNIX systems when it comes to production analysis, and more in general it greatly helped the process of understanding what’s going on under the covers by means of dynamic tracing.

Various systems inspired by DTrace have been developed throughout the years, including SystemTap and bpftrace. Although these systems differ in several important aspects, they all allow to ask questions about a running system by following a similar workflow. The List, Instrument, Aggregate, and Report (LIAR) acronym can be used to describe such workflow. The rest of this article will illustrate the methodology using bpftrace, taking as an example the problem of identifying which programs are sending data over TCP sockets, and how much data they are sending.

List

Without specific knowledge about Linux kernel internals, we can try and List all available probe points that have 'tcp' and 'send' in their name:

$ sudo bpftrace -l '*tcp*send*'
tracepoint:tcp:tcp_send_reset
kprobe:__traceiter_tcp_send_reset
kprobe:tcp_send_mss
kprobe:do_tcp_sendpages
kprobe:tcp_sendpage_locked
kprobe:tcp_sendpage
kprobe:tcp_sendmsg_locked
kprobe:tcp_sendmsg
[...]

The function tcp_sendmsg seems interesting, and indeed it is the one responsible for gathering up data to be written to a TCP socket. Its signature is:

int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)

The 'size' argument is what we are interested in, and we are going to print it together with the PID next.

Instrument

At the Instrument step, we need to write a simple bpftrace program that prints a line every time the tcp_sendmsg function is called. Given that 'size' is the third argument of tcp_sendmsg, and that bpftrace allows to print function arguments using the arg0, …, argN keywords, we are going to access it by using 'arg2'. We are going to print the builtin 'pid' variable too.

$ sudo bpftrace -e 'kprobe:tcp_sendmsg {
    printf("pid=%d: size=%d\n", pid, arg2)
}'
Attaching 1 probe...
pid=764374: size=36
pid=764374: size=36
pid=633506: size=43
pid=633506: size=566
pid=633506: size=600
pid=633506: size=58
pid=633506: size=819
^C

The output looks reasonable, we can now work on aggregating the 'size' data to get further insights on the TCP sending behavior of this system.

Aggregate

The Map Functions provided by bpftrace allow to Aggregate data in various useful ways. For example, the stats(): function returns the count, average, and total for a given value.

Let’s use it:

$ sudo bpftrace -e 'kprobe:tcp_sendmsg {
    @bytes[pid] = stats(arg2);
    print(@bytes);
}'
Attaching 1 probe...
@bytes[770234]: count 1, average 77, total 77

@bytes[770234]: count 1, average 77, total 77
@bytes[770237]: count 1, average 77, total 77

We have defined a BPF map called 'bytes', used the PID as the key, and aggregated the 'size' argument using stats(). Every time tcp_sendmsg is called, we print the values obtained so far.

Report

As the final step, we need to Report our findings. Instead of printing the values every time tcp_sendmsg is called, it would be nicer to do that only once at program termination. Just like DTrace, bpftrace defaults to automatically printing aggregation results when the program exits. This is to say that, in bpftrace terminology, all populated maps are printed automatically.

The full program is thus simply:

$ sudo bpftrace -e 'kprobe:tcp_sendmsg {
    @bytes[pid] = stats(arg2);
}'
Attaching 1 probe...
^C

@bytes[769042]: count 1, average 75, total 75
@bytes[769047]: count 1, average 75, total 75
@bytes[769052]: count 1, average 75, total 75
@bytes[769057]: count 1, average 75, total 75
@bytes[633506]: count 13, average 378, total 4915

All values of the 'bytes' map are printed by default when the user issues a CTRL-C. If we want to instead make the program exit after 10 seconds, we can use the 'interval' probe as follows:

$ sudo bpftrace -e 'kprobe:tcp_sendmsg {
    @bytes[pid] = stats(arg2);
}

interval:s:10 {
    exit();
}'

Conclusion

This article introduced the LIAR method of production debugging, a workflow in four steps that can be followed using DTrace-like systems to inspect various aspects of running production systems. See the original DTrace paper, and the bpftrace reference guide for more!

certbot renew: An authentication script must be provided

ema@linux.it (Emanuele Rocca) — Mon, 01 Mar 2021 06:52:40 +0100

For a personal project of mine I’ve been using a wildcard TLS certificate issued by Let’s Encrypt. certbot made the process of creating the certificate extremely easy, I just had to apt install certbot and then run the following command, duly copy-pasted from the Internet:

certbot --manual certonly --agree-tos --email ema@example.org --preferred-challenges=dns -d '*.example.org'

The command gave this output:

[...]
Please deploy a DNS TXT record under the name
_acme-challenge.example.org with the following value:

05REcrGYWuv_fTBQ3QQYTxmNm3f_LU2cN8JNf_f458z

Being a happy Gandi customer and Terraform user, I deployed the change with this configuration:

resource "gandi_livedns_record" "acme" {
    zone = "example.org"
    name = "_acme-challenge"
    type = "TXT"
    ttl = 300
    values = [
        "05REcrGYWuv_fTBQ3QQYTxmNm3f_LU2cN8JNf_f458z"
    ]
}

This was enough to issue the certificate, and I completely forgot about it till the nice Let’s Encrypt Expiry Bot emailed saying the certificate would expire in 20 days.

Didn’t I configure auto-renewal of the cert? I thought I did but then again who knows, weeks have gone by! It turns out I did, with this wonderful /etc/cron.weekly/certbot-renew script:

#!/bin/sh

certbot renew

Other than not being very advanced, the script was also failing claiming that The manual plugin is not working and that An authentication script must be provided with --manual-auth-hook:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/example.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Non-interactive renewal: random delay of 293 seconds
Could not choose appropriate plugin: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.')
Attempting to renew cert (example.org) from /etc/letsencrypt/renewal/example.org.conf produced an unexpected error: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.'). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/example.org/fullchain.pem (failure)

Unfortunately certbot did not guess that — even though the certificate was created manually — I wanted renewals to happen automatically. Luckily certbot also didn’t guess my Gandi API key.

When the certificate was issued, this /etc/letsencrypt/renewal/example.org.conf was created for me:

# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/example.org
cert = /etc/letsencrypt/live/example.org/cert.pem
privkey = /etc/letsencrypt/live/example.org/privkey.pem
chain = /etc/letsencrypt/live/example.org/chain.pem
fullchain = /etc/letsencrypt/live/example.org/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = redacted
pref_challs = dns-01,
authenticator = manual
manual_public_ip_logging_ok = True
server = https://acme-v02.api.letsencrypt.org/directory

The important line is authenticator = manual in the renewalparams section. We need to replace that with a plugin for the Gandi DNS service. Thankfully Yohann Leon was nice enough to write one, and Unit193 packaged it for Debian.

What I did to get certbot to automatically renew my wildcard certificate was:

1) installing the plugin with apt install python3-certbot-dns-gandi

2) replacing authenticator = manual with authenticator = certbot-plugin-gandi:dns

3) adding certbot_plugin_gandi:dns_credentials = /etc/letsencrypt/gandi.ini to tell the plugin where to find my credentials

4) creating /etc/letsencrypt/gandi.ini with dns_gandi_api_key=REDACTED

Almost. Now certbot renew failed saying that it cannot find certbot_plugin_gandi:dns_api_key:

Attempting to renew cert (example.org) from /etc/letsencrypt/renewal/example.org.conf produced an unexpected error: Missing property in credentials configuration file /etc/letsencrypt/gandi.ini:
 * Property "certbot_plugin_gandi:dns_api_key" not found (should be API key for Gandi account).. Skipping.

So apparently something was wrong with step (4). Though the docs use dns_gandi_api_key as the property name, it seems it should be certbot_plugin_gandi:dns_api_key instead.

And that’s it, here is the final configuration.

/etc/letsencrypt/renewal/example.org.conf

# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/example.org
cert = /etc/letsencrypt/live/example.org/cert.pem
privkey = /etc/letsencrypt/live/example.org/privkey.pem
chain = /etc/letsencrypt/live/example.org/chain.pem
fullchain = /etc/letsencrypt/live/example.org/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = certbot-plugin-gandi:dns
certbot_plugin_gandi:dns_credentials = /etc/letsencrypt/gandi.ini
account = redacted
pref_challs = dns-01,
server = https://acme-v02.api.letsencrypt.org/directory

/etc/letsencrypt/gandi.ini

certbot_plugin_gandi:dns_api_key=REDACTED

Lessons learned from a 27 years old UNIX book

ema@linux.it (Emanuele Rocca) — Tue, 16 Feb 2021 22:06:00 +0100

One of the Amazon reviewers of "Sun Performance and Tuning: Java and the Internet" gave it 3/5 stars. While still a nice introduction, the book by Adrian Cockcroft has become dated — claimed Roland in 2003, which believe it or not was 18 years ago. The book Roland reviewed was published in 1998, and it is a second edition. The first edition (1994) has a significantly different title: "Sparc & Solaris", is the focus instead of "Java and the Internet". Other than reflecting a major shift in Sun’s focus happened between '94, and '98, the two editions also have a different price tag. "Java and the Internet" costs 15.97 EUR, while one can get "Sparc & Solaris" used in allegedly Very Good conditions for 1.94. If I’m buying an outdated book I’m gonna get the most outdated version possible, I figured, plus imagine the embarrassment if someone would find a book about Java in my bookshelf. Shipping is 3 bucks, for a grand total of 4.94 EUR the book is in my hands 15 days after placing the order.

"The Porsche Book" is quick to explain the analogy between UNIX and the Porsche 911 featured on the cover: both were created in the '60s, in a form similar to the current one. Both represent the latest and greatest of products still around in 1994, and indeed 2021, despite their quirks.

A valid question at this point clearly is: why do you care about a book published 30 years ago when there’s so much recent and surely more useful material? One reason is that I find the history of Sun Microsystems particularly interesting. I’ve read and very much enjoyed High Noon as well as Sunburst — dealing with recent history there’s the privilege of having plenty of primary sources readily available, which is great. When it comes to the Porsche book, I find it fascinating to read about the most common usage scenarios of Sun servers in the '90s, and I can’t help but feel sorry for the admin who didn’t know that for SunOS 4, only filenames up to 14 characters long are cached in the Directory Name Lookup Cache (DNLC). In this article, however, I want to focus on the information contained in the book which is still valid today. In a world that moves as quickly as the world of computing, you could argue that all books are outdated to begin with. If some of the information in "Sun Performance and Tuning" was valid in 1994 and is still valid in 2021, there’s a good chance that it will still be valid 30 years from now.

It seems to me that we can categorize tech knowledge as follows: theoretical knowledge, generic practical information, and implementation details. Of the three, theoretical knowledge is the most likely to stay relevant with time, followed by generic practical information which might still apply some years later, and finally implementation details which are very likely to become irrelevant as time goes by. Those details are far from being useless however: in fact they provide the distinction between novice and experienced practitioner. To give an example: the chapter about Networks opens with a description of the built-in le network interface, pointing out that it shared its DMA connection with the SCSI interface. Crucially, the network interface had higher priority: heavy network activity was likely to reduce disk throughput. In 1994 you would have needed to see multiple workstations with bad disk performance before concluding that network traffic had anything to do with it — or read the book.

But what is the content that stood the test of time?

Configuration Factors and Levels

When performing a set of tests, there are multiple settings and knobs one can work with, each having various options to choose from. Which filesystem are we using? What’s the version of the application we’re testing? Kernel buffer size?

Factor	Level
Filesystem type	ext4, xfs, btrfs,...
OS version	5.10.16, 5.11, ...
vm.dirty_ratio kernel setting	20, 40, 60, ...

To measure every combination of 6 different factors with 4 levels each would take 4^6 = 4096 measurements, which clearly is madness. By reducing the levels we consider to 2, still evaluating all 6 different factors in all possible combination, we get down to 2^6 = 64 measurements instead.

Throughput, Response Time, Utilization

Throughput — the amount of work performed in a given amount of time — was a thing in the '60s, it’s the same thing today, and it will still be work/time in 2050. Your web server might have a maximum throughput of 10000 requests per second (rps), that’s something you can use to compare it with another one. Response time (latency) is defined as the amount of time the user has to wait. For example, how long did a given database transaction take. Utilization is how much of the computer’s resources were used to do the work. The values reported by sar and iostat, two tools very much still in use today, are an example of Utilization measures.

The author suggests taking multiple sar Utilization measurements at various load levels and combine them with awk to get an overview of how the system behaves. We do have Prometheus, Grafana, and friends nowadays, but the sar | awk idea is a useful reminder that quick, simple measurements can go a long way too!

vmstat, sar and iostat Rules

vmstat and sar are described in detail, with a focus on runnable queue, blocked queue, swap usage, and so forth. The Linux versions of vmstat and sar from procps and sysstat are a bit different than the Solaris ones, but the main ideas still apply.

The three commands are also used in the Rules and Tunables Quick Reference Tables section. For each subsystem (Disk, Network, Memory, CPU), the author specified a list of Rules involving the commands and the action to take if the rule applies. Rules consist in the name of the command with any options, then a "." followed by the name of the variable to take into account. For example, the following indicates a disk bottleneck: 35% < iostat-D30.util < 65%. The action to take is to try balance the I/O load on other disks too, or get more disks.

A contemporary version of those rules for Linux would be great to have.

Interrupt Distribution Tuning

Interrupts in Solaris 2.2 were load-balanced among all available CPUs. The drawback was that, in case of heavy interrupt load from a given device, cache hit rate would suffer: the OS allowed to statically assign IRQs to CPUs as an improvement instead.

Nowadays a very similar problem occurs with the irqbalance daemon on Linux, which on systems doing lots of network activity should not be used. SMP affinity should instead be configured to ensure the interrupts of multiqueue network cards are statically mapped among CPUs.

There’s more information in "Sun Performance and Tuning" that can still provide valuable lessons today, as well as some true historical gems for those into such things. Go check it out on the Internet Archive if you’re interested!

The alternative to wpa_supplicant

ema@linux.it (Emanuele Rocca) — Wed, 02 Dec 2020 16:52:40 +0100

As of December 2020, there is an alternative to wpa_supplicant. It works. It’s called iwd, iNet Wireless Daemon, and you can use it on Debian systems today.

Now, if you use Linux and you’ve never heard of wpa_supplicant before, you are a lucky person. Starting some weeks ago, all my videoconferences got interrupted after a few minutes, ping example.org said "not today", and I was sad. Looking at the the logs I could always see lines like this, a few seconds before the connection dropped:

Nov 30 18:56:36 orion wpa_supplicant[728]: wlan0: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-50 noise=9999 txrate=115600

I am however sitting very close to the access point, and my laptop is the only device in the house having issues. So I blame wpa_supplicant. Suddenly iwd came to mind, a project I had heard about at All Systems Go! 2019. Here is the video of the talk.

Here’s how it goes:

apt install iwd
systemctl enable iwd

As you don’t want to see wpa_supplicant ever again, mask its unit:

systemctl mask wpa_supplicant

Then you need to tell Network Manager to use iwd as a supplicant, which boils down to adding the following to /etc/NetworkManager/NetworkManager.conf:

[device]
wifi.backend=iwd

Reboot. Rejoice! wpa_supplicant is gone and your wifi connection is up nonetheless. My connection stayed up for the whole duration of a meeting. I am a happy man.

A brief introduction to SystemTap

ema@linux.it (Emanuele Rocca) — Mon, 14 Oct 2019 17:18:40 +0100

SystemTap allows to instrument Linux systems at runtime. By using it, you can gather insights about running programs, including the Linux kernel itself, without invoking them in specific ways, modifying them, or indeed even having access to their source code.

On Debian systems and derivatives, including Ubuntu, get started with SystemTap by installing the systemtap package as well as the Linux kernel headers:

apt install systemtap linux-headers-$(uname -r)

To verify that SystemTap is working correctly you can try this hello world one-liner:

$ sudo stap -v -e 'probe oneshot { println("hello world") }'
Pass 1: parsed user script and 476 library scripts using 101404virt/87992res/6960shr/81096data kb, in 150usr/30sys/218real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 102988virt/89752res/7152shr/82680data kb, in 10usr/0sys/7real ms.
Pass 3: using cached /root/.systemtap/cache/28/stap_2871d732a80a0612c98a3e7e9d7dc4a2_973.c
Pass 4: using cached /root/.systemtap/cache/28/stap_2871d732a80a0612c98a3e7e9d7dc4a2_973.ko
Pass 5: starting run.
hello world
Pass 5: run completed in 10usr/20sys/494real ms.

Give it another go without -v for more terse and less exciting output.

User space instrumentation

Now that SystemTap is installed and working on your machine, let’s use it to give a peek at what ls is doing under the hood. In order for SystemTap to inspect the behavior of a given program it needs to have access to its debugging symbols, which in the case of ls on Debian are provided by the coreutils-dbgsym package. Once the package is installed, you can ask SystemTap to list all available probe points, which to simplify a little we can say are equivalent to the functions ls can call. Let’s list as an example all functions that might have something to do with usernames:

$ sudo stap -L 'process("/bin/ls").function("*user*")'
process("/bin/ls").function("format_user@src/ls.c:3955") $u:uid_t $width:int $stat_ok:_Bool
process("/bin/ls").function("format_user_or_group@src/ls.c:3927") $name:char const* $id:long unsigned int $width:int
process("/bin/ls").function("format_user_or_group_width@src/ls.c:3973") $id:long unsigned int
process("/bin/ls").function("format_user_width@src/ls.c:3991") $u:uid_t
process("/bin/ls").function("getuser@lib/idcache.c:69") $uid:uid_t $match:struct userid*

The format_user function seems interesting. We can see that it takes three arguments: u, width, and stat_ok. Let’s print all invocations of it, as well as the value of u:

$ sudo stap -e 'probe process("/bin/ls").function("format_user") { printf("format_user(uid=%d)\n", $u) }'

If SystemTap complains about a Build-id mismatch, try again passing -DSTP_NO_BUILDID_CHECK on the command line. In case you’re curious, read man error::buildid to find out more about this.

Now try running ls -l /etc/passwd, and you should see the following output from SystemTap:

format_user(uid=0)

Try running ls /etc/passwd without -l and notice that SystemTap produces no output, indicating that ls does not call the format_user function in that case.

SystemTap is a fully-fledged programming language with variables, loops, conditionals and so forth. One-liners on the shell are fine for exploration and simple examples like the ones above, but for longer scripts you might want to save your work to a file. Let’s do that by creating ls_non_root.stp, a file that slightly changes our previous example by only printing format_user calls for files owned by non-root users:

// SystemTap example: ls_non_root.stp
probe process("/bin/ls").function("format_user") {
    if ($u != 0) {
        printf("format_user(uid=%d)\n", $u)
    }
}

Run the script with stap -v ls_non_root.stp and you should now see some output only when running ls -l on files owned by users other than root.

Kernel instrumentation

How about the Linux kernel? Just as we did before for coreutils, we need to install the debugging symbols for the kernel currently running. What is different though, is that the kernel debug symbols are huge. For example, in the case of the 4.19 kernel the debug symbols are about 5G. Make sure you’ve got plenty of disk space available and the patience needed while waiting for apt install linux-image-$(uname -r)-dbg to do its thing.

Now let’s look for available Linux kernel probe points matching the *icmp*reply pattern:

$ sudo stap -L 'kernel.function("*icmp*reply*")'
kernel.function("icmp_push_reply@./net/ipv4/icmp.c:367") $icmp_param:struct icmp_bxm* $fl4:struct flowi4* $ipc:struct ipcm_cookie* $rt:struct rtable**
kernel.function("icmp_reply@./net/ipv4/icmp.c:402") $icmp_param:struct icmp_bxm* $skb:struct sk_buff* $ipc:struct ipcm_cookie $fl4:struct flowi4
kernel.function("icmpv6_echo_reply@./net/ipv6/icmp.c:670") $skb:struct sk_buff* $tmp_hdr:struct icmp6hdr $fl6:struct flowi6 $msg:struct icmpv6_msg $ipc6:struct ipcm6_cookie

Interesting! Let’s run ping localhost in one terminal and see if, as you’d expect, the icmp_reply function gets called:

stap -ve 'probe kernel.function("icmp_reply") { println("reply") }'

Silence. Ha! localhost resolves to ::1 here, and icmp_reply deals with ICMPv4. The function we’re looking for is icmpv6_echo_reply. We can extend the script as follows, and see when the kernel is sending both v4 and v6 echo replies.

// v4/v6 echo reply
probe kernel.function("icmp_reply") {
    println("Sending v4 echo reply")
}

probe kernel.function("icmpv6_echo_reply") {
    println("Sending v6 echo reply")
}

Some of SystemTap requirements such as large debug packages and GCC are undesirable on production systems. Luckily, SystemTap is designed so that you can develop and compile your probes on a build host (eg. your laptop, or a designated build server), and run them on production hosts with minimal dependencies. For example, to compile the hello world probe on a development machine:

$ sudo stap -e 'probe oneshot { println("hello word") }' -m hello -p4
hello.ko

The command above generates a kernel module named hello.ko, which can be copied to a production host and run with staprun hello.ko. Only systemtap-runtime needs to be installed on the target machine. If the kernels running on the build and target hosts differ, you need to specify the target Linux kernel version with -r. For example, to target the 4.19.0-3-amd64 kernel:

$ sudo stap -e 'probe oneshot { println("hello word") }' -m hello -p4 -r 4.19.0-3-amd64

In this introduction we have always executed stap as root. On production systems you might want to add your user to the staprun group instead.

We’ve just scratched the surface of what SystemTap can do. Go ahead and read the documentation, play with it, and have fun.

ema@linux.it (Emanuele Rocca) — Mon, 01 Jan 0001 00:00:00 +0000

Contacts, LOL so 90s.

E-mail: ema@linux.it. My PGP key is 0x50FDB7A3, and this is the full fingerprint if you want to verify it: 9545 9696 1928 1C17 FFAC 5891 D508 5A07 50FD B7A3.

IRC: ema on OFTC and Libera

Mastodon: @ema @ fosstodon.org Twitter: @realEmaRocca

GitHub: https://github.com/ema/

ema@linux.it (Emanuele Rocca) — Mon, 01 Jan 0001 00:00:00 +0000

Emanuele Rocca. Packages Not Using The Default Build Flags: A Taxonomy. DebConf25, IRISA, Jul 2025, Brest, France. pp.4.
G. Pierre, T. Kielmann, E. Rocca, K. Razavi, B. IJff, H. Fernandez, R. Figueiredo, A. Uta, A. Vintila, A. Oprescu, T. Schuett, M. Berlin, M. Artac, A. Cernivec, ConPaaS: An Integrated Runtime Environment for Elastic Cloud Applications, in: HPDC (Poster), 2013.

My Erdős number is 5 thanks to the ConPaaS publication: Thilo Kielmann -> Willem Jan Fokkink -> Robbert J. Fokkink -> Jeffrey O. Shallit -> Paul Erdős.

Further, I have performed as an extra in a movie featuring Maximilian Dirr, meaning that my Bacon number is 3: Maximilian Dirr -> Donald Sutherland -> Kevin Bacon.

At parties, I can thus impress people by saying that my Erdős–Bacon number is 8.

ema@linux.it (Emanuele Rocca) — Mon, 01 Jan 0001 00:00:00 +0000

Packages not using the default build ﬂags: a taxonomy (slides, video)
2025-07-15, Debconf 25, Brest, France
Enabling Architectural Features in Debian: PAC and BTI on arm64 (slides, video)
2025-02-02, FOSDEM 25, Brussels, Belgium
Debian Live (slides, video)
2024-10-12, Mini-debconf, Cambridge, United Kingdom
Hardware Enablement in Debian: Lenovo Thinkpad X13s (slides, video)
2023-11-25, Mini-debconf, Cambridge, United Kingdom
Traffic Switchover 2018 vs 2020 (slides)
2020-09-21, Wikimedia SRE Session, The Internet
Go and Wikipedia’s CDN (slides)
2019-11-11, Golang Users Berlin, Zalando Lounge HQ Berlin, Germany
Wikipedia’s CDN, A Day In The Life (slides)
2019-10-31, Wikimedia SRE Session, The Internet
SystemTap markers in ATS (video)
2019-10-10, Apache Traffic Server Summit, Sunnyvale, California, USA
Serving Wikipedia with ATS (slides, video)
2019-10-08, Apache Traffic Server Summit, Sunnyvale, California, USA
Wikipedia’s CDN (slides)
2018-03-26, DIBRIS, Università degli studi di Genova, Italy
Running Wikipedia.org (slides, video)
2016-06-17, VarnishCon 2016, Amsterdam, Netherlands
Terraform (slides)
2015-10-05, ZenMate, Berlin, Germany
Buildbot (slides)
2015-06-18, ZenMate, Berlin, Germany
Varnish Cache (slides)
2015-06-11, ZenMate, Berlin, Germany
A brief history of Unix (slides, video)
2015-03-19, ZenMate, Berlin, Germany
ConPaaS Architecture (slides, video)
2013-06-13, ConPaaS Workshop, Vrije Universiteit Amsterdam, Netherlands
Developer support in a federated PaaS environment (slides)
2012-06-14, MSc thesis discussion, Vrije Universiteit Amsterdam, Netherlands
Cross-site request forgery (slides)
2011-12-13, Vrije Universiteit Amsterdam, Netherlands
Scalable Applications: design, refactor, host (slides)
2011-10-14, Vrije Universiteit Amsterdam, Netherlands
Bigtable: A Distributed Storage System for Structured Data (slides)
2011-03-16, Vrije Universiteit Amsterdam, Netherlands
[Italian] Notifica e consultazione del dossier clinico (slides)
2010-01-20, E.O. Ospedali Galliera, Genova, Italy
[Italian] SystemTap: applicazioni alla sicurezza informatica (slides)
2009-10-17, BSc thesis discussion, DISI, Università degli studi di Genova, Italy
[Italian] Debian Security Team (slides)
2005-12-16, DISI, Università degli studi di Genova, Italy