We are using FAI tools to install Debian10 Buster and Debian12 Bookworm clients with backported kernel versions.
We use the latest FAI CD ISO 6.2 <https://fai-project.org/fai-cd/faicd64-small_6.2.2.iso> for Debian 12 Bookworm clients. We installed Debian12 with dedicated separate kernel versions using backported kernels 6.5 and 6.7 linux-image-6.5.0-0.deb12.4-amd64 linux-image-6.7.12+bpo-amd64 Everything was working fine until last week. As you know, backported kernels 6.5 and 6.7 have been removed from the official Debian repositories and replaced with the new backported kernel 6.9, which is in testing. Debian -- Details of package linux-image-6.9.7+bpo-amd64 in bookworm-backports <https://packages.debian.org/bookworm-backports/kernel/linux-image-6.9.7+bpo-amd64> To manage these backported features in FAI, we created shell scripts. Since the backported kernels are no longer available on the remote site, we used the Debian Snapshot snapshot.debian.org to retrieve old kernel packages based on dates and version numbers. I have verified this manually and was able to get the required older kernel versions on the FAI server. *Here is what I have tried to achieve this:* 1. As backported kernel versions are removed from the remote, we cannot get kernels 6.5 and 6.7, only the latest kernel 6.9 is able to get. root@faiserver:~# apt-cache search linux-image-6.5 root@faiserver:~# apt-cache search linux-image-6.7 root@faiserver:~# apt-cache search linux-image-6.9 linux-headers-6.9.7+bpo-amd64 - Header files for Linux 6.9.7+bpo-amd64 linux-headers-6.9.7+bpo-cloud-amd64 - Header files for Linux 6.9.7+bpo-cloud-amd64 linux-headers-6.9.7+bpo-rt-amd64 - Header files for Linux 6.9.7+bpo-rt-amd64 linux-image-6.9.7+bpo-amd64-dbg - Debug symbols for linux-image-6.9.7+bpo-amd64 linux-image-6.9.7+bpo-amd64-unsigned - Linux 6.9 for 64-bit PCs linux-image-6.9.7+bpo-cloud-amd64-dbg - Debug symbols for linux-image-6.9.7+bpo-cloud-amd64 linux-image-6.9.7+bpo-cloud-amd64-unsigned - Linux 6.9 for x86-64 cloud linux-image-6.9.7+bpo-rt-amd64-dbg - Debug symbols for linux-image-6.9.7+bpo-rt-amd64 linux-image-6.9.7+bpo-rt-amd64-unsigned - Linux 6.9 for 64-bit PCs, PREEMPT_RT linux-image-6.9.7+bpo-amd64 - Linux 6.9 for 64-bit PCs (signed) linux-image-6.9.7+bpo-cloud-amd64 - Linux 6.9 for x86-64 cloud (signed) linux-image-6.9.7+bpo-rt-amd64 - Linux 6.9 for 64-bit PCs (signed), PREEMPT_RT 2. Update the FAI server sources file for the Debian mirror. *Default sources list:* root@faiserver:~# cat /etc/apt/sources.list # These lines should work for many sites deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware deb http://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware #deb http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware # repository that may contain newer fai packages for bookworm deb http://fai-project.org/download bookworm koeln *Updated sources file:* root@faiserver:~/Debian/Manufacturing/fai/scripts# cat /etc/apt/sources.list # These lines should work for many sites deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware deb http://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware deb http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware # repository that may contain newer fai packages for bookworm deb http://fai-project.org/download bookworm koeln # Debian Snapshot deb [check-valid-until=no] https://snapshot.debian.org/archive/debian/20240516T145256Z/ bookworm-backports main 3. Create a `99proxy` file under `/etc/apt/apt.conf.d/`: Ref https://snapshot.debian.org/#:~:text=To%20access%20snapshots%20of,update%20for%20this%20purpose . cat /etc/apt/apt.conf.d/99proxy Acquire::https::snapshot.debian.org::Proxy "DIRECT"; 4. Execute `apt-get update` to update the package lists. 5. Now I am able to get the available kernel versions from the remote site: root@faiserver:~/Debian/Manufacturing/fai/scripts# apt-cache search linux-image-6.5 linux-headers-6.5.0-0.deb12.4-amd64 - Header files for Linux 6.5.0-0.deb12.4-amd64 linux-headers-6.5.0-0.deb12.4-cloud-amd64 - Header files for Linux 6.5.0-0.deb12.4-cloud-amd64 linux-image-6.5.0-0.deb12.4-amd64-dbg - Debug symbols for linux-image-6.5.0-0.deb12.4-amd64 linux-image-6.5.0-0.deb12.4-amd64-unsigned - Linux 6.5 for 64-bit PCs linux-image-6.5.0-0.deb12.4-cloud-amd64-dbg - Debug symbols for linux-image-6.5.0-0.deb12.4-cloud-amd64 linux-image-6.5.0-0.deb12.4-cloud-amd64-unsigned - Linux 6.5 for x86-64 cloud linux-image-6.5.0-0.deb12.1-amd64 - Linux 6.5 for 64-bit PCs (signed) linux-image-6.5.0-0.deb12.1-cloud-amd64 - Linux 6.5 for x86-64 cloud (signed) linux-image-6.5.0-0.deb12.1-rt-amd64 - Linux 6.5 for 64-bit PCs, PREEMPT_RT (signed) linux-image-6.5.0-0.deb12.4-amd64 - Linux 6.5 for 64-bit PCs (signed) linux-image-6.5.0-0.deb12.4-cloud-amd64 - Linux 6.5 for x86-64 cloud (signed) root@faiserver:~/Debian/Manufacturing/fai/scripts# apt-cache search linux-image-6.7 linux-headers-6.7.12+bpo-amd64 - Header files for Linux 6.7.12+bpo-amd64 linux-headers-6.7.12+bpo-cloud-amd64 - Header files for Linux 6.7.12+bpo-cloud-amd64 linux-headers-6.7.12+bpo-rt-amd64 - Header files for Linux 6.7.12+bpo-rt-amd64 linux-image-6.7.12+bpo-amd64-dbg - Debug symbols for linux-image-6.7.12+bpo-amd64 linux-image-6.7.12+bpo-amd64-unsigned - Linux 6.7 for 64-bit PCs linux-image-6.7.12+bpo-cloud-amd64-dbg - Debug symbols for linux-image-6.7.12+bpo-cloud-amd64 linux-image-6.7.12+bpo-cloud-amd64-unsigned - Linux 6.7 for x86-64 cloud linux-image-6.7.12+bpo-rt-amd64-dbg - Debug symbols for linux-image-6.7.12+bpo-rt-amd64 linux-image-6.7.12+bpo-rt-amd64-unsigned - Linux 6.7 for 64-bit PCs, PREEMPT_RT 6. We have a custom shell script to prepare the FAI server setup for client installations. Sample script to configure NFS server: # Configure NFS Server configure_nfs() { /etc/init.d/nfs-kernel-server stop sed -i "s/linux-image-amd64/#/g" /etc/fai/NFSROOT sed -i '/AMD64/a\linux-image-6.7.12+bpo-amd64-unsigned' /etc/fai/NFSROOT run sed -i "s/NFSROOT_ETC_HOSTS/#NFSROOT_ETC_HOSTS/g" /etc/fai/nfsroot.conf run sed --in-place '/bookworm-backports/d' /etc/fai/apt/sources.list "echo 'deb http://deb.debian.org/debian bookworm-backports main contrib non-free' | tee -a /etc/fai/apt/sources.list } Updated `/etc/fai/NFSROOT` file: root@faiserver:~# cat /etc/fai/NFSROOT # package list for creating the NFSROOT PACKAGES install-norec FULL nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile numactl udns-utils netcat-traditional nmap pxelinux syslinux-common ca-certificates # for get-config-dir-https and apt-transport-https usbutils pciutils ssh screen tmux netselect mdadm cryptsetup #git # git consumes a lot of disk space on the FAI CD PACKAGES install-norec # dracut replaces live-boot and initramfs-tools dracut live-boot- initramfs-tools- dracut-config-generic dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg # some network cards needs firmware PACKAGES install-norec NONFREE firmware-bnx2 firmware-bnx2x firmware-realtek firmware-cavium firmware-myricom firmware-netronome firmware-misc-nonfree # you should not edit the lines below # architecture dependent list of packages that are installed PACKAGES install-norec I386 grub-pc efibootmgr linux-image-686-pae PACKAGES install-norec AMD64 linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr ##/bookworm-backports # if you want to use a newer kernel PACKAGES install-norec ARM64 grub-efi-arm64 efibootmgr linux-image-arm64 PACKAGES install-norec ARMHF grub-efi-arm efibootmgr linux-image-armmp I've hardcoded the package name for `linux-image-6.7.12+bpo-amd64-unsigned` in the `NFSROOT` file. but it is still failing with the error: `E: Unable to locate package linux-image-6.7.12+bpo-amd64-unsigned` but it is still available on snapshot. Classes are set to NFSROOT FULL AMD64 DEBIAN DEBIAN_12 Adding additional packages to /srv/fai/nfsroot: nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile numactl udns-utils netcat-traditional nmap pxelinux syslinux-common ca-certificates usbutils pciutils ssh screen tmux netselect mdadm cryptsetup dracut live-boot- initramfs-tools- dracut-config-generic dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr install_packages: reading config files from directory /etc/fai install_packages: read config file NFSROOT install_packages: executing chroot /srv/fai/nfsroot apt-get clean install_packages: executing chroot /srv/fai/nfsroot apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew --allow-change-held-packages --fix-missing install --no-install-recommends nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile numactl udns-utils netcat-traditional nmap pxelinux syslinux-common ca-certificates usbutils pciutils ssh screen tmux netselect mdadm cryptsetup dracut live-boot- initramfs-tools- dracut-config-generic dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr Reading package lists... Building dependency tree... Reading state information... E: Unable to locate package linux-image-6.7.12+bpo-amd64-unsigned E: Couldn't find any package by glob 'linux-image-6.7.12+bpo-amd64-unsigned' E: Couldn't find any package by regex 'linux-image-6.7.12+bpo-amd64-unsigned' ERROR: 25600 25600 ERROR: chroot /srv/fai/nfsroot apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew --allow-change-held-packages --fix-missing install --no-install-recommends nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile numactl udns-utils netcat-traditional nmap pxelinux syslinux-common ca-certificates usbutils pciutils ssh screen tmux netselect mdadm cryptsetup dracut live-boot- initramfs-tools- dracut-config-generic dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr return code 100 install_packages: executing chroot /srv/fai/nfsroot apt-get clean install_packages: executing chroot /srv/fai/nfsroot dpkg --configure --pending install_packages: executing chroot /srv/fai/nfsroot dpkg -C install_packages: executing chroot /srv/fai/nfsroot apt-get clean 1 errors during executing of install_packages ERROR: install_packages had exit code: 3 However, if I try it manually, it is found: root@faiserver:~/Debian/Manufacturing/fai/scripts# apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew --allow-change-held-packages --fix-missing install --no-install-recommends linux-image-6.7.12+bpo-amd64-unsigned Reading package lists... Done Building dependency tree... Done Reading state information... Done Suggested packages: linux-doc-6.7 debian-kernel-handbook Recommended packages: firmware-linux-free apparmor The following NEW packages will be installed: linux-image-6.7.12+bpo-amd64-unsigned 0 upgraded, 1 newly installed, 0 to remove and 56 not upgraded. Need to get 99.8 MB of archives. After this operation, 103 MB of additional disk space will be used. Get:1 https://snapshot.debian.org/archive/debian/20240516T145256Z bookworm-backports/main amd64 linux-image-6.7.12+bpo-amd64-unsigned amd64 6.7.12-1~bpo12+1 [99.8 MB] Fetched 99.8 MB in 17s (5,744 kB/s) Selecting previously unselected package linux-image-6.7.12+bpo-amd64-unsigned. (Reading database ... 25696 files and directories currently installed.) Preparing to unpack .../linux-image-6.7.12+bpo-amd64-unsigned_6.7.12-1~bpo12+1_amd64.deb ... Unpacking linux-image-6.7.12+bpo-amd64-unsigned (6.7.12-1~bpo12+1) ... Setting up linux-image-6.7.12+bpo-amd64-unsigned (6.7.12-1~bpo12+1) ... I: /vmlinuz is now a symlink to boot/vmlinuz-6.7.12+bpo-amd64 I: /initrd.img is now a symlink to boot/initrd.img-6.7.12+bpo-amd64 /etc/kernel/postinst.d/initramfs-tools: update-initramfs: Generating /boot/initrd.img-6.7.12+bpo-amd64 I: The initramfs will attempt to resume from /dev/sda5 I: (UUID=663da36d-e9d7-4559-bbb4-a2d5141bd99c) I: Set the RESUME variable to override this. /etc/kernel/postinst.d/zz-update-grub: Generating grub configuration file ... Found linux image: /boot/vmlinuz-6.7.12+bpo-amd64 Found initrd image: /boot/initrd.img-6.7.12+bpo-amd64 Found linux image: /boot/vmlinuz-6.1.0-18-amd64 Found initrd image: /boot/initrd.img-6.1.0-18-amd64 Found memtest86+x64 image: /boot/memtest86+x64.bin done I suspect there may be an issue with the sources list because my `/etc/fai/apt/sources.list` might not be updated correctly. I am still debugging this further. If anyone has a better suggestion for obtaining older kernel versions for client installations or any other alternative approach, please share it with me. Thanks in advance. Regards. Samadhan India
