Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Thomas Schmitt]

Hi,

Pete Batard wrote:
> Debian does not use an efi.img.

Oh it does with ISOs for i386 and amd64. There is a data file in the ISO
filesystem named
  /boot/grub/efi.img
advertised as MBR partition of type 0xEF and as El Torito boot image
for EFI:

  $ xorriso -indev debian-11.5.0-amd64-netinst.iso -report_system_area plain 
-report_el_torito plain
  ...
  MBR partition table:   N Status  TypeStart   Blocks
  MBR partition  :   1   0x80  0x000   782336
  MBR partition  :   2   0x00  0xef 4064 5184
  MBR partition path :   2  /boot/grub/efi.img
  ...
  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
  El Torito boot img :   1  BIOS  y   none  0x  0x00  42312
  El Torito boot img :   2  UEFI  y   none  0x  0x00   51841016
  El Torito img path :   1  /isolinux/isolinux.bin
  El Torito img opts :   1  boot-info-table isohybrid-suitable
  El Torito img path :   2  /boot/grub/efi.img

But Debian is nice by having in the ISO filesystem a copy of the file
tree in the FAT filesystem of the EFI partition.

  # mount debian-11.5.0-amd64-netinst.iso /mnt/iso
  ...
  # mount /mnt/iso/boot/grub/efi.img /mnt/fat
  $ (cd /mnt/fat ; find . -type f -exec ls -ld '{}' ';')
  -rwxr-xr-x 1 root root 934240 Sep 10  2022 ./efi/boot/bootx64.efi
  -rwxr-xr-x 1 root root 1684864 Sep 10  2022 ./efi/boot/grubx64.efi
  -rwxr-xr-x 1 root root 101 Sep 10  2022 ./efi/debian/grub.cfg
  $ (cd /mnt/iso ; find ./EFI -type f -exec ls -ld '{}' ';')
  -r--r--r-- 1 root root 934240 Sep 10  2022 ./EFI/boot/bootx64.efi
  -r--r--r-- 1 root root 1684864 Sep 10  2022 ./EFI/boot/grubx64.efi
  -r--r--r-- 1 root root 101 Sep 10  2022 ./EFI/debian/grub.cfg

The differences in the paths become insignificant when copying to FAT.
The tests results indicate that the lack of x-permissions with the ISO
files is not an issue either.

In the arm64 ISOs the /EFI tree is present three times. Once as appended
MBR partition 2, once as FAT image file /boot/grub/efi.img in the ISO
which serves as El Torito boot image, and again as unpacked /EFI tree in
the ISO.
(xorriso could point El Torito to the appended partition, thus eliminating
the need for the /boot/grub/efi.img file.)


> From what I can see, the maximum individual file name length when using
> Joliet extensions is 128 "characters"

It's less. A Joliet directory record has the same maximum size as an ISO
directory record: 255 bytes. Joliet encodes names in UCS-2 which uses
16 bit per character. The fixed header part of a directory record is 34
bytes large. So only 110 UCS-2 characters have room. For some reason
libisofs offers to write 103 characters at most. (It's long ago that this
decision was made not by me.)

In file
  /mnt/iso/.disk/mkisofs
i see the recorded -as mkisofs option:
  -joliet-long
which means according to man xorrisofs:
Allow  103  characters in Joliet file names rather than 64 as is
prescribed by the specification. Allow Joliet paths longer  than
the prescribed limit of 240 characters.


> if you are using Windows' native utilities, you won't be able
> format a partition that is larger than 64 GB to FAT32,

The largest official Debian ISOs are 50 GB.
But i have a script which can merge a complete set to a ~90 GB ISO. :))


Have a nice day :)

Thomas

Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Pete Batard]

Hi,

Just going to report that I have tested media creation from the 
2023-03-13 version of debian-testing-amd64-netinst.iso, using only the 
Windows native utilities (i.e. formatting a USB drive to FAT32 using 
Windows Disk manager and mounting the ISO and copying the files using 
Windows File Explorer) and I can report that everything looks good:


1. The .deb files in /firmware/ are now being listed by File Explorer, 
and therefore copied over.

2. The installer does successfully boot and detect the installation media.

Now, I don't have a platform where any of the /firmware/ files are 
relevant, so I haven't been able to validate that part, but seeing that 
the .deb are present and successfully copied over, I don't anticipate an 
issue.


Thus, as far as I am concerned, this bug can be closed, and I would like 
to thank everyone who devoted some of their time to ensure that a fix 
was applied in a very timely manner.



On 2023.03.15 07:48, Thomas Schmitt wrote:

Besides such user pitfalls with the produced ISO and the problem of
symbolic links there are other constraints which an ISO has to fulfill
for this use case.
At least:
- The file tree of the FAT filesystem in the EFI partition needs to be
   mirrored by a copy in the ISO filesystem.


Indeed. That used to be less of an issue when Linux installation media 
used to treat the whole media as the ESP, but the move towards using 
self-contained efi.img does indeed mean that FST will only work if the 
distro maintainers do take care of duplicating the efi.img content at 
the ISO-9660 file system level (or use a utility, like grub-mkrescue, 
that will do that for them).


As far as Debian is concerned, this is not currently an issue, as Debian 
does not use an efi.img.



- File names must be unique in respect to case-insensitive comparison.


True, though I have yet to encounter any installation media where 
someone had named two file/folders in the same directory with the same 
name but different capitalization. I'm pretty sure that we can count on 
people mastering an image to avoid that, as it would be very confusing 
to have different files/folders at the same level, that differ only from 
capitalization.


So I think we can discount this as a corner case that's unlikely to be 
an issue.



- (I am not sure whether file name length can be a problem.)


From what I can see, the maximum individual file name length when using 
Joliet extensions is 128 "characters" (I'm not going to go into Unicode 
glyphs vs characters here) and 255 for Rock Ridge. The latter is also 
the maximum maximum individual file name length for FAT32 with long 
names. So, even if Debian tends to have fairly long names for some .deb 
packages, I don't anticipate much of an issue. And case sensitivity 
isn't an issue either, meaning that looking for a mixed case file name 
on FAT32 will resolve properly, even as the underlying file system is 
not exactly one that could be qualified as case sensitive.



I guess Pete Batard can give a more comprehensive list.


Well, once you eliminate the search of installation media by label 
(which Debian doesn't do), the lack of symbolic links, which is what 
we've been dealing with here, is actually the biggest limitation of 
trying to work with FAT.


Otherwise, it's really the volume labelling constraints of FAT that's 
the most problematic, because, this time, you *must* use UPPERCASE and a 
few very limited subset of non alphanum characters for FAT labels, and 
you are also limited to 11 characters, which is very very short. The end 
result of this is that, pretty much any Linux distro that does a search 
of the installation media by label, and doesn't pay attention to file 
system transposition to FAT, is likely to use a regular mixed case label 
that is also longer than 11 characters, and you must alter the 
GRUB/Syslinux kernel options in the config files is you ever want that 
media to work with FST.


Finally, to be completely comprehensive, though this is not an actually 
constraint of the FAT file system, but one of Windows, I am going to 
point out that if you are using Windows' native utilities, you won't be 
able format a partition that is larger than 64 GB to FAT32, which can 
come as an issue for users who picked a large USB Flash Drive. However, 
you can either use non native utilities to format larger partitions to 
FAT32 (since this is not a limitation of the File System, just of the 
default Windows utilities) or you can always create a partition that is 
small enough to be formatted to FAT32 and leave the rest of the media free.


Regards,

/Pete

Re: 11.7 planning

[Laura Arjona Reina]

Hello

El 16 de marzo de 2023 13:21:52 CET, Donald Norwood  
escribió:
>29th is workable.
>

I can the 15th and 22nd of April.
For now I cannot say anything about my availability for May (bookworm release).

Kind regards

>On 3/15/23 18:36, Andy wrote:
>> On 15 March 2023 20:33:47 GMT, Jonathan Wiltshire  wrote:
>>> Hi,
>>> 
>>> We're overdue for 11.7 and need it done with a keyring update included
>>> before bookworm can be released. The wheels are turning on the keyring so
>>> how do dates in April look for everybody? Saturdays are 1st (probably too
>>> soon), 8th, 15th, 22nd and 29th.
>>> 
>>> Thanks,
>>> 
>> 
>> 
>> I can do 15th, 22nd or 29th
>> Isy is only available from the 29th (mocks)
>> 
>> Best wishes
>> /Andy
>

-- 
Laura Arjona Reina
https://wiki.debian.org/LauraArjona
Sent with K-9 mail

Re: 11.7 planning

[Donald Norwood]

29th is workable.

On 3/15/23 18:36, Andy wrote:

On 15 March 2023 20:33:47 GMT, Jonathan Wiltshire  wrote:

Hi,

We're overdue for 11.7 and need it done with a keyring update included
before bookworm can be released. The wheels are turning on the keyring so
how do dates in April look for everybody? Saturdays are 1st (probably too
soon), 8th, 15th, 22nd and 29th.

Thanks,




I can do 15th, 22nd or 29th
Isy is only available from the 29th (mocks)

Best wishes
/Andy


--



Be well,

-Donald

--
-
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Donald Norwood
⢿⡄⠘⠷⠚⠋⠀ B7A1 5F45 5B28 7F38 4174
⠈⠳⣄ D5E9 E5EC 4AC9 BD62 7B05

Re: 11.7 planning + bookworm planning

[Cyril Brulebois]

Hi,

Paul Gevers  (2023-03-16):
> From where I'm looking, bookworm is looking pretty good. Obviously
> we'll have to follow through on the flurry of unblock requests that
> came in after the hard freeze, but that should be manageable in a
> couple of weeks. kibi just told me on IRC that asking this question
> now is not crazy from a d-i point of view.

Asking questions is never crazy. :)

> That hopefully means that around the end of April, also the bookworm
> d-i should™ be ready for release (kibi, I'm sure you'll comment on
> this if I got you wrong or if you want to provide more details).

At this stage, I'd still like to have at least two releases out. The
first few weeks after the Alpha 2 release were awfully quiet
feedback-wise, that's much better now, and we've got a few fixes and
improvements queued up. There's also a brand new set of shim* packages,
which should definitely end up in some release at some point.

Without having checked with Steve yet, I certainly expect us to be able
to publish at least a release in March and another before say mid-April…
 
> So, shall we add availability for May too? 6th, 13th, 20th (Ascension
> weekend), and 27th (coincides with DebianReunionHamburg)?

… which should be all good for a tentative release in May, we would
still have time for a few extra tweaks with a final debian-installer
upload, should they be needed at that stage (rather than wait until
12.1).


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: 11.7 planning + bookworm planning

[Luna Jernberg]

Can the 6th and 20th in May

On Thu, Mar 16, 2023 at 11:26 AM Paul Gevers  wrote:
>
> Dear all,
>
> On 15-03-2023 21:33, Jonathan Wiltshire wrote:
> > We're overdue for 11.7 and need it done with a keyring update included
> > before bookworm can be released. The wheels are turning on the keyring so
> > how do dates in April look for everybody? Saturdays are 1st (probably too
> > soon), 8th, 15th, 22nd and 29th.
>
> I know some people will call me crazy, but can and should we combine
> this with finding a release date for bookworm?
>
>  From where I'm looking, bookworm is looking pretty good. Obviously
> we'll have to follow through on the flurry of unblock requests that came
> in after the hard freeze, but that should be manageable in a couple of
> weeks. kibi just told me on IRC that asking this question now is not
> crazy from a d-i point of view. That hopefully means that around the end
> of April, also the bookworm d-i should™ be ready for release (kibi, I'm
> sure you'll comment on this if I got you wrong or if you want to provide
> more details).
>
> So, shall we add availability for May too? 6th, 13th, 20th (Ascension
> weekend), and 27th (coincides with DebianReunionHamburg)?
>
> Paul

Re: 11.7 planning + bookworm planning

[Paul Gevers]

Dear all,

On 15-03-2023 21:33, Jonathan Wiltshire wrote:

We're overdue for 11.7 and need it done with a keyring update included
before bookworm can be released. The wheels are turning on the keyring so
how do dates in April look for everybody? Saturdays are 1st (probably too
soon), 8th, 15th, 22nd and 29th.


I know some people will call me crazy, but can and should we combine 
this with finding a release date for bookworm?


From where I'm looking, bookworm is looking pretty good. Obviously 
we'll have to follow through on the flurry of unblock requests that came 
in after the hard freeze, but that should be manageable in a couple of 
weeks. kibi just told me on IRC that asking this question now is not 
crazy from a d-i point of view. That hopefully means that around the end 
of April, also the bookworm d-i should™ be ready for release (kibi, I'm 
sure you'll comment on this if I got you wrong or if you want to provide 
more details).


So, shall we add availability for May too? 6th, 13th, 20th (Ascension 
weekend), and 27th (coincides with DebianReunionHamburg)?


Paul


OpenPGP_signature
Description: OpenPGP digital signature

Re: 11.7 planning

[Luna Jernberg]

If you want my help with testing i can do the later April dates 15th,
22nd and 29th

On 3/15/23, Jonathan Wiltshire  wrote:
> Hi,
>
> We're overdue for 11.7 and need it done with a keyring update included
> before bookworm can be released. The wheels are turning on the keyring so
> how do dates in April look for everybody? Saturdays are 1st (probably too
> soon), 8th, 15th, 22nd and 29th.
>
> Thanks,
>
> --
> Jonathan Wiltshire  j...@debian.org
> Debian Developer http://people.debian.org/~jmw
>
> 4096R: 0xD3524C51 / 0A55 B7C5 1223 3942 86EC  74C3 5394 479D D352 4C51
> ed25519/0x196418AAEB74C8A1: CA619D65A72A7BADFC96D280196418AAEB74C8A1
>
>