BTW, I am wondering what would be the corresponding BTRFS config. Does something like this work as expected or not (mixing / and /tmp this time) :
``` disk_config disk1 disklabel:gpt bootable:1 fstabkey:uuid primary - 2G - - primary - 70% - - primary - 0- - - disk_config disk2 sameas:disk1 disk_config btrfs fstabkey:uuid btrfs raid1 /boot/efi disk1.1,disk2.1 auto,ssd,lazytime,rw,nosuid,nodev,exec,subvol=@boot btrfs raid1 / disk1.4,disk2.4 subvol=@root,defaults,noatime,compress=zstd btrfs raid1 /home disk1.3,disk2.3 subvol=@home,defaults,noatime,compress=zstd ``` ?? Le mar. 5 déc. 2023 à 13:15, Rémy Dernat <remy...@gmail.com> a écrit : > > Hi, > > After digging some time around this issue, I am finally able to boot > the RAID1 partition with EFI boot enabled on it (using this example: > https://fai-project.org/download/misc/disk_config/lvm-on-luks-on-raid). > > Here is my final disk_config file: > > ```# example of new config file for setup-storage > # > # <type> <mountpoint> <size> <fs type> <mount options> <misc options> > > disk_config disk1 disklabel:gpt bootable:1 fstabkey:uuid > primary - 2G - - > primary - 0- - - > > disk_config disk2 sameas:disk1 > > disk_config raid > raid1 /boot/efi disk1.1,disk2.1 vfat rw,noatime > mdcreateopts="--verbose --metadata=0.90" > raid1 - disk1.2,disk2.2 - - > > disk_config lvm > vg vg_system md1 > vg_system-swap swap 16G swap sw > vg_system-root / 60G-200G ext4 rw,noatime > vg_system-tmp /tmp 20G-50G ext2 rw,noatime,nosuid,nodev > vg_system-home /home 30%- ext4 rw,noatime,nosuid,nodev > > ``` > > Best regards, > Rémy > > Le ven. 1 déc. 2023 à 12:11, Rémy Dernat <remy...@gmail.com> a écrit : > > > > Hi Andrew, > > > > Thanks for suggestions. > > > > Now, my file "disk_config/GRUB_EFI_RAID1" is [*]: > > > > ``` > > # example of new config file for setup-storage > > # > > # <type> <mountpoint> <size> <fs type> <mount options> <misc options> > > > > disk_config disk1 disklabel:gpt fstabkey:uuid bootable:1 > > primary - 2G - - > > primary /boot/efi 1G vfat rw createopts="-F32" > > primary - 2G - - > > primary - 0- - - > > > > #disk_config disk2 sameas:disk1 > > disk_config disk2 disklabel:gpt fstabkey:uuid > > primary - 2G - - > > primary /boot/efi2 1G vfat rw createopts="-F32" > > primary - 2G - - > > primary - 0- - - > > > > disk_config raid fstabkey:uuid > > raid1 /boot disk1.1,disk2.1 ext4 rw,noatime mdcreateopts="--verbose > > --metadata=0.90" > > raid1 swap disk1.3,disk2.3 swap sw > > raid1 - disk1.4,disk2.4 - - > > > > disk_config lvm fstabkey:uuid > > vg vg_system md2 > > vg_system-root / 60G-200G ext4 rw,noatime > > vg_system-tmp /tmp 20G-50G ext2 rw,noatime,nosuid,nodev > > ##vg_system-scratch /scratch 30%- ext2 rw,noatime > > vg_system-home /home 30%- ext4 rw,noatime,nosuid,nodev > > ``` > > > > => Even if the installation succeeded, grub does not seem to work. > > > > Anyway, I am wondering how you set this in the disk_config file (syntax ?): > > > > > I like to put a 100MB partition at the end of disks in a RAID pair to > > > allow > > > for slightly differing size of a replacement disk. Although currently it > > > seems that any new disk we get is many many GB bigger than the old failed > > > one. > > > > Thanks again, > > > > Best regards, > > Rémy > > > > > > > > [*] there was an error for /boot partition which was not formatted the > > right way (vfat does not allow symbolic links). vfat should be used > > for /boot/EFI only. > > > > Le ven. 1 déc. 2023 à 05:43, Andrew Ruthven <and...@etc.gen.nz> a écrit : > > > > > > Nice, good work. > > > > > > A couple of suggestions... > > > > > > I'd put swap on RAID1, if you have a disk go offline while the system is > > > running, you've lost anything that had been swapped out to the swap on the > > > failed disk. > > > > > > I like to put a 100MB partition at the end of disks in a RAID pair to > > > allow > > > for slightly differing size of a replacement disk. Although currently it > > > seems that any new disk we get is many many GB bigger than the old failed > > > one. > > > > > > I've found that some servers allow the EFI partition to be RAID1, the > > > current servers we're using support that. So no need to copy files around. > > > > > > Cheers, > > > Andrew > > > > > > On Thu, 2023-11-30 at 17:20 +0100, Rémy Dernat wrote: > > > > > > > Ok, FYI, I think I have a working configuration with : > > > > > > > > ``` > > > > # example of new config file for setup-storage > > > > # > > > > # <type> <mountpoint> <size> <fs type> <mount options> <misc > > > > options> > > > > > > > > disk_config disk1 disklabel:gpt fstabkey:uuid bootable:1 > > > > primary - 2G - - > > > > primary /boot/efi 1G vfat rw createopts="-F32" > > > > primary swap 2G swap sw,pri=1 > > > > primary - 0- - - > > > > > > > > #disk_config disk2 sameas:disk1 > > > > disk_config disk2 disklabel:gpt fstabkey:uuid > > > > primary - 2G - - > > > > primary /boot/efi2 1G vfat rw createopts="-F32" > > > > primary swap 2G swap sw,pri=1 > > > > primary - 0- - - > > > > > > > > disk_config raid fstabkey:uuid > > > > #raid1 /boot disk1.1,disk2.1 - - mdcreateopts="--verbose > > > > --metadata=1.0" createopts="-L boot" > > > > raid1 /boot disk1.1,disk2.1 vfat rw,noatime mdcreateopts="--verbose > > > > --metadata=0.90" > > > > raid1 - disk1.4,disk2.4 - - > > > > > > > > disk_config lvm fstabkey:uuid > > > > vg vg_system md1 > > > > vg_system-root / 60G-200G ext4 rw,noatime > > > > vg_system-tmp /tmp 20G-50G ext2 rw,noatime,nosuid,nodev > > > > ##vg_system-scratch /scratch 30%- ext2 rw,noatime > > > > vg_system-home /home 30%- ext4 rw,noatime,nosuid,nodev > > > > ``` > > > > > > > > My previous message was crystal clear. md0 did not have any formatted > > > > filesystem on it. So I added `vfat` option on /boot. > > > > > > > > Best regards, > > > > Rémy > > > > > > > > > > > > Le jeu. 30 nov. 2023 à 16:53, Rémy Dernat > > > <[remy...@gmail.com](mailto:remy...@gmail.com)> a écrit : > > > > > > > > > > > > > > What is strange here, is that md1 is found, as well as md0, but > > > > > contrary to md0, md1 have an UUID ... > > > > > > > > > > However, UUID seems to be retrieve by blkid from /dev/disk/by-id, > > > > > where both md array have their own mdadm UUID, but only md1 have an > > > > > UUID from LVM (with pvdisplay / pvs ) ... > > > > > > > > > > > > > > > Le jeu. 30 nov. 2023 à 14:52, Rémy Dernat > > > <[remy...@gmail.com](mailto:remy...@gmail.com)> a écrit : > > > > > > > > > > > > > > > > > Hi Andrew, > > > > > > > > > > > > Indeed, this removed the syntax issue. However, I have a new error > > > now: > > > > > > > > > > > > ``` > > > > > > /sbin/blkid -c /dev/null -s UUID -o value /dev/md0 had exit code 2 > > > > > > Command had non-zero exit code > > > > > > ``` > > > > > > Indeed, blkid is not able to found md0 UUID. > > > > > > > > > > > > I tried to modify the disk_config file to avoid this error, but I am > > > > > > still stuck with it. Here is what look like my disk_config file now: > > > > > > > > > > > > ``` > > > > > > # example of new config file for setup-storage > > > > > > # > > > > > > # <type> <mountpoint> <size> <fs type> <mount options> <misc > > > options> > > > > > > > > > > > > disk_config disk1 > > > > > > primary - 2G - - > > > > > > primary /boot/efi 1G vfat rw createopts="-F32" > > > > > > primary swap 2G swap sw,pri=1 > > > > > > primary - 0- - - > > > > > > > > > > > > #disk_config disk2 sameas:disk1 > > > > > > disk_config disk2 disklabel:gpt fstabkey:uuid > > > > > > primary - 2G - - > > > > > > primary /boot/efi2 1G vfat rw createopts="-F32" > > > > > > primary swap 2G swap sw,pri=1 > > > > > > primary - 0- - - > > > > > > > > > > > > disk_config raid fstabkey:uuid > > > > > > raid1 /boot disk1.1,disk2.1 - - mdcreateopts="--verbose > > > > > > --metadata=0.90" createopts="-L boot" > > > > > > raid1 - disk1.4,disk2.4 - - > > > > > > > > > > > > disk_config lvm fstabkey:uuid > > > > > > vg vg_system md1 > > > > > > vg_system-root / 60G-200G ext4 rw,noatime > > > > > > vg_system-tmp /tmp 20G-50G ext2 rw,noatime,nosuid,nodev > > > > > > ##vg_system-scratch /scratch 30%- ext2 rw,noatime > > > > > > vg_system-home /home 30%- ext4 rw,noatime,nosuid,nodev > > > > > > ``` > > > > > > > > > > > > I also tried to change the raid metadata to 1.0, but that did not > > > > > > change anything. I wrote back ` fstabkey:uuid` behind disk_config > > > > > > raid > > > > > > because it does not fail with syntax error. > > > > > > > > > > > > Thanks for help > > > > > > > > > > > > Le jeu. 30 nov. 2023 à 11:55, Andrew Ruthven > > > <[and...@etc.gen.nz](mailto:and...@etc.gen.nz)> a écrit : > > > > > > > > > > > > > > > > > > > > Hey, > > > > > > > > > > > > > > On Thu, 2023-11-30 at 11:14 +0100, Rémy Dernat wrote: > > > > > > > > > > > > > > > Hi awesome Fai guys, > > > > > > > > > > > > > > > > I have a weird syntax issue with the following disk_config: > > > > > > > > > > > > > > > > > > > > > > [snip] > > > > > > > > > > > > > > > disk_config raid disklabel:gpt fstabkey:uuid bootable:1 > > > > > > > > > > > > > > > > > > > > > This line is your problem, remove everything after the word raid. > > > > > > > So > > > it > > > > > > > should just be: > > > > > > > > > > > > > > disk_config raid > > > > > > > > > > > > > > Cheers, > > > > > > > Andrew > > > > > > > -- > > > > > > > Andrew Ruthven, Wellington, New Zealand > > > > > > > [and...@etc.gen.nz](mailto:and...@etc.gen.nz) | > > > > > > > Catalyst Cloud: | This space intentionally left blank > > > > > > > [https://catalystcloud.nz](https://catalystcloud.nz) | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ``` > > > -- > > > Andrew Ruthven, Wellington, New Zealand > > > and...@etc.gen.nz | > > > Catalyst Cloud: | This space intentionally left blank > > > https://catalystcloud.nz | > > > > > > > > > ``` > > > > > >