Bug#1067103: libisoburn1t64: should not depend on libburn4 nor libisofs6 after time_t transition
Hi, being the one who usually prepares the releases, i am currently standing in hands-off position until the time_t change is completed. The package tracker is still complaining bitterly about the current intermediate state. Consider to re-open https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1062380 "libisoburn: NMU diff for 64-bit time_t transition" and/or to contact its submitter. Have a nice day :) Thomas
Bug#1062380: libisoburn: NMU diff for 64-bit time_t transition
Hi, i'm the upstream and help with preparing the Debian packages. So i assume that any needed action in this case is up to my sponsor Dominique Dumont (Cc'd). But: Are you aware that the effort to switch to 64 bit time_t is not worth much in a ISO 9660 producing software, even on amd64, as long as https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h has int as return type of the ISO-9660-to-Unix-time function extern int iso_date(u8 *, int); (implemented in fs/isofs/util.c) ? I once made a successful attempt to get the attention of parts of the kernel community: https://lore.kernel.org/all/20221024135517.7d40b96a206020eca03e3...@linux-foundation.org/t/ But as it looks, it never made it into the torvalds repo. Have a nice day :) Thomas
Bug#1062381: libisofs: NMU diff for 64-bit time_t transition
Hi, i'm the upstream and help with preparing the Debian packages. So i assume that any needed action in this case is up to my sponsor Dominique Dumont (Cc'd). But: Are you aware that the effort to switch to 64 bit time_t is not worth much in a ISO 9660 producing software, even on amd64, as long as https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h has int as return type of the ISO-9660-to-Unix-time function extern int iso_date(u8 *, int); (implemented in fs/isofs/util.c) ? I once made a successful attempt to get the attention of parts of the kernel community: https://lore.kernel.org/all/20221024135517.7d40b96a206020eca03e3...@linux-foundation.org/t/ But as it looks, it never made it into the torvalds repo. Have a nice day :) Thomas
Bug#1056998: cdrom: Installation media changes after booting it
Hi,
i wrote:
> >Does any of the bystanders know from what package this message text stems ?
Steve McIntyre wrote:
> That's cdrom-checker:
So that would be indirectly put into the code at
https://sources.debian.org/src/cdrom-checker/1.62/main.c/?hl=152#L152
by "Template: cdrom-checker/mismatch" in:
https://sources.debian.org/src/cdrom-checker/1.62/debian/cdrom-checker.templates/?hl=43#L43
Using my micking instincts i guess my proposal would be to create a
new template like:
Template: cdrom-checker/mismatch-efi
Type: error
# :sl3:
_Description: Integrity test failed on EFI boot image
The EFI boot image file ${FILE} failed the MD5 checksum verification.
Possibly some software or firmware made changes in the EFI System
Partition. This may be considered harmless unless booting of this
Debian ISO encounters difficulties or other files show checksum
mismatches, too.
and to add a case in main.c, line 151, like (shown with reduced
indentation):
if(status != 0) {
if(strcmp(filename, "./boot/grub/efi.img") == 0) {
debconf_subst(debconf, "cdrom-checker/mismatch-efi", "FILE", filename);
debconf_input(debconf, "high", "cdrom-checker/mismatch-efi");
} else {
debconf_subst(debconf, "cdrom-checker/mismatch", "FILE", filename);
debconf_input(debconf, "high", "cdrom-checker/mismatch");
}
} else {
debconf_input(debconf, "critical", "cdrom-checker/passed");
}
A native speaker can probably find a better text and my guessing of the
inner code relations may be wrong, of course. Even the path
"./boot/grub/efi.img" is guessed from Ram Reddy's reported error message.
Have a nice day :)
Thomas
Bug#1056998: cdrom: Installation media changes after booting it
Hi,
trying to get our mails in sync again. We seem to have reached consensus
in our understanding of the problem.
i wrote:
> > might possibly be "WPSettings.dat", which causes questions in the
> > internet.
Ram Reddy wrote:
> I think that this could possibly be explained by flaws in my testing. Some
> of the computers were using Windows, and for those computers I plugged in
> the USB drive, shut down the computer, and booted the USB drive from the
> BIOS.
This explains the MS-Windows file.
> the ones where I shut down windows first had no modifications.
So the Thinkpad firmwares are exonerated.
> it seems to be that the only modifications of the installer caused by
> firmware is with the Legion 7i gen 5's self healing BIOS backup.
This suspect can be considered to have confessed.
--
As for a better message from Integrity Check:
I find in the ISO in file /install.amd/initrd.gz the lines:
"The file ${FILE} failed the MD5 checksum verification. Your installation"
"media or this file may have been corrupted."
But i failed to find these texts in debian-cd or debian-installer
Even searching for just "corrupted" yields only a remark in debian-cd.
Does any of the bystanders know from what package this message text stems ?
Have a nice day :)
Thomas
Bug#1056998: cdrom: Installation media changes after booting it
Hi, Ram Reddy wrote: > Thomas Schmitt wrote: > > FWIW check the BIOS L[123] cache settings [...] This was posted by Nicholas Geovanis , not by me. My suspicion is that the Lenovo Legion EFI BIOS intentionally creates directories and empty file "/efi/Lenovo/BIOS/SelfHealing.fd". I do not see much benefit in investigating this further. More riddling is the directory and file found after using the USB stick on the Lenovo Thinkpad machines. The internet says that "/System Volume Information/WPSettings.dat" is part of Microsoft services. So it might be that the USB stick got in contact with MS-Windows while or before it was plugged into the Thinkpads. If this suspicion can be excluded then i'd say that Thinkpad has some strange BIOS habits. What remains for Debian in this bug report is the observation that the integrity check of the Debian installation ISO says "Your installation media or this file may have been corrupted" when just a harmless change has been made in the filesystem of the EFI partition. Some special handling of /boot/grub/efi.img during the integrity check would cause less surprise to the user. Like: "Possibly some software or firmware made changes in the EFI System Partition. This may be considered harmless unless booting of this Debian ISO encounters difficulties or other files show checksum mismatches, too." Have a nice day :) Thomas
Bug#1056998: cdrom: Installation media changes after booting it
Hi, Ram Reddy wrote: > https://drive.google.com/file/d/1Zd6iufVRsfIu-qzC-tJx4FEvCOESOz4_/view?usp=sharing I downloaded the tarball and compared the original FAT filesystem with the various modified filesystem images. -- In Legion7iG5-*_modified.esp the suspect lost its ID card at the crime scene: At byte 39072 (0x98a0) the changes go from 0-bytes to the text "LENOVO". At byte 9711680 (0x943040) i see a change from 0-bytes to "BIOS". Diffing the result of "find" on the mounted unmodified.esp filesystem and Legion7iG5-*_modified.esp shows that a new branch of directoriies with a new file is in each of the modified filesystems: > ./efi/Lenovo > ./efi/Lenovo/BIOS > ./efi/Lenovo/BIOS/SelfHealing.fd The file is empty. -- In ThinkpadX1CarbonG5-0_modified.esp there is no company name to see in the changed bytes. I see UTF-16 strings "mation", "System", and "Volum\000me". ASCII texts "SYSTEM~1", "WPSETT~1DAT". The latter might possibly be "WPSettings.dat", which causes questions in the internet. Most plausible seems an answer in the course of https://answers.microsoft.com/en-us/insider/forum/all/whats-wpsettingsdat-generated-by/e11bca97-8c76-4662-8897-774ea3d5691a "The WPSettings.dat file is generated by the Storage Service (StorSvc). It seems that WPSettings.dat means the data files of Windows Phone's Store Settings saved on the drives, [...]" Diffing the result of "find" on the mounted unmodified.esp filesystem and ThinkpadX1CarbonG5-0_modified.esp shows that a new directory with a new file is in the modified filesystem: ./System Volume Information ./System Volume Information/WPSettings.dat The file has 12 bytes of binary salad: Hex: 0c 00 00 00 2e 42 6b 82 5d 88 0e c5 Char: . B k ] Dec: 12 0 0 0 46 66 107 130 93 136 14 197 -- While it makes some sense to me that Lenovo Legion BIOS adds some Lenovo stuff to the EFI System Partition, i really wonder why Lenovo Thinkpad BIOS adds a Microsoft directory and file. Whatever, i'd say that the software in the ISO and especially Debian Installer are not suspicious to create directories with such names. Have a nice day :) Thomas
Bug#1056998: cdrom: Installation media changes after booting it
Hi, Ram Reddy wrote: > I tested the installer on one of my laptops, and found that its contents > didn't change. Good to know that the software in the ISO still works from read-only DVDs. > On the Lenovo Legion 7i Gen 5*, Lenovo Thinkpad X1 Carbon Gen 2*, Lenovo > Thinkpad X1 Carbon Gen 5 Intel*, and one Lenovo Thinkpad E14 Gen 5 all > showed the corruption error. > However, on the Lenovo Yoga C740 and a different Lenovo Thinkpad E14 Gen 5 > Intel showed no corruption issues. This seems to indicate that the firmware has a stake in the problem ... > Both the Thinkpad E14 Gen 5s had the same specifications and type number, > differing only in that the one with corruption of the installer has 24GB of > memory (16GB installed in the slot, 8GB soldered) and the other only has 8GB > soldered. They both have the same BIOS version, R2AET32W(1.07). ... but the trigger would have to be very subtle. > This seems to be really interesting because the corruption only happened on > certain computers, and it would stay that way on repeated attempts. I ponder how to distinguish a change made by the firmware itself from a change made by the software in the ISO image. > Again, the > ones that had corruption all had it at the same location. (byte 2303211, > line 21165) Maybe we can learn more by comparing the files /boot/grub/efi.img of the original ISO and of an altered USB stick. Mount both and copy out the files, or use dd to extract them from the partition interval that is marked in the ISO (ask by: fdisk -l). With debian-12.2.0-amd64-netinst.iso that would be iso=debian-12.2.0-amd64-netinst.iso or (with the approriate letter instead of "X") iso=/dev/sdX and then dd bs=512 if="$iso" of="$iso".esp skip=4476 count=18976 Both images will have about 9.5 MiB of size. I would be interested in seeing the altered one, as i can extract the original on my own. So if you have a way to upload the altered EFI partition image to a publicly accessible place i would be happy to investigate it. Much smaller and thus more bugreport-friendly would be a binary diff. You would need something more detailed than plain cmp. cmp -l gives the differing byte values one by line (i.e. lengthy if many bytes differ). I made an own willful alteration (subtracting 4476 * 512 from 2303211 yields the new problem byte address 11499): cp debian-12.2.0-amd64-netinst.iso.esp debian-12.2.0-amd64-netinst.iso.esp.altered echo ABCDE | dd conv=notrunc of=debian-12.2.0-amd64-netinst.iso.esp.altered bs=1 seek=11499 count=5 Then cmp -l debian-12.2.0-amd64-netinst.iso.esp debian-12.2.0-amd64-netinst.iso.esp.altered yields 11500 0 101 11501 0 102 11502 0 103 11503 0 104 11504 0 105 (Odd enough: Addresses are decimal starting at 1, not at 0. Values are octal.) Less useful is cmp -b: debian-12.2.0-amd64-netinst.iso.esp debian-12.2.0-amd64-netinst.iso.esp.altered differ: byte 11500, line 278 is 0 ^@ 101 A (Only the "A" is from my change. There seem to be pairs of octal value and character meaning.) I have my own unpublished differ for analyzing ISO entrails. So only the big battleship "diffoscope" comes to my mind as public binary differ. Probably there are leaner programs for that job. Have a nice day :) Thomas
Bug#1056998: cdrom: Installation media changes after booting it
Hi, Ram Reddy wrote: > [...] usb drive [...] debian-12.2.0-amd64-netinst.iso [...] > [...] Integrity Test section [...] > ./boot/grub/efi.img failed the MD5 checksum verification. Your installation > media or this file may have been corrupted. > [...] > It was always byte 2303211, line 21165 which changed. I'd say that corruption of ISO image or USB is quite unlikely. It rather looks like some change was made to the FAT filesystem in the EFI partition by the parts of the installer which ran in your experiments. (I expect EFI firmware not to manipulate that filesystem ...) The EFI partition of this particular ISO begins at byte 2291712. The alteration thus happened 11499 bytes inside that partition and its filesystem. In the original ISO i see a sequence of 0-bytes there. It would be interesting to check whether any errors get reported if the ISO is presented on a read-only CD or DVD instead of a writable USB stick. (To be absolutely read-only it would have to be CD-R, unformatted CD-RW, DVD+R, DVD-R, or unformatted DVD-RW. The others can be overwritten without burn program.) Explanation why the file in the read-only ISO filesystem is reported as being altered: The partition shares its block range with the ISO file /boot/grub/efi.img which is mentioned in the ISO file /md5sum.txt . So the integrity check detects the difference between recorded MD5 and the MD5 of the altered FAT filesystem image file /boot/grub/efi.img. Have a nice day :) Thomas
Bug#1051445: syslinux.efi crashes on isohybrid boot as cdrom
Hi, congrats for solving this long standing problem. A few years ago this would have been great news for Knoppix, which staid for quite a while with SYSLINUX for EFI. But somewhere between versions 8.6 (2019) and 9.3 (2022) it switched to GRUB. Although the most recent commit on https://repo.or.cz/syslinux.git is 4.5 years old, consider to post your patches to sysli...@syslinux.org Maybe it helps some puzzled SYSLINUX user in the future. Have a nice day :) Thomas
Bug#1038440: debian-cd: debian-12.0.0-amd64-netinst.iso is too big for a CD
Hi, when comparing 12.0.0 with 11.5.0 i see three file trees which have grown substantially: du of mounted debian-11.5.0-amd64-netinst.iso : 5 /firmware 126611 /install.amd 308479 /pool du of mounted debian-12.0.0-amd64-netinst.iso : 221108 /firmware 156509 /install.amd 589416 /pool In pool of 12.0.0 i see another firmware tree 220570 /pool/non-free-firmware but this does not cost much media capacity, because the file content is shared with the files in the /firmware tree: $ xorriso -indev debian-12.0.0-amd64-netinst.iso \ -find / -sort_lba -exec report_lba -- ... Report layout: xt , Startlba , Blocks , Filesize , ISO image path ... File data lba: 0 ,18420 , 8217 , 16827872 , '/firmware/firmware-atheros_20230210-5_all.deb' File data lba: 0 ,18420 , 8217 , 16827872 , '/pool/non-free-firmware/f/firmware-nonfree/firmware-atheros_20230210-5_all.deb' ... Such deduplication of hard links also happens with initrd.gz and vmlinuz files in /install.amd. The real storage sizes as measured in above -find output are: 11.5.0 12.0.0 Growth /firmware 0 MiB 216 MiB 216 MiB /install.amd 67 MiB 138 MiB 71 MiB /pool 301 MiB 360 MiB 59 MiB To fit on a regular "700 MB" (~ 702 MiB) CD-R/CD-RW these would have to shrink by at least 36 MiB. Have a nice day :) Thomas
Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support
Hi,
Pete Batard wrote:
> Debian does not use an efi.img.
Oh it does with ISOs for i386 and amd64. There is a data file in the ISO
filesystem named
/boot/grub/efi.img
advertised as MBR partition of type 0xEF and as El Torito boot image
for EFI:
$ xorriso -indev debian-11.5.0-amd64-netinst.iso -report_system_area plain
-report_el_torito plain
...
MBR partition table: N Status TypeStart Blocks
MBR partition : 1 0x80 0x000 782336
MBR partition : 2 0x00 0xef 4064 5184
MBR partition path : 2 /boot/grub/efi.img
...
El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
El Torito boot img : 1 BIOS y none 0x 0x00 42312
El Torito boot img : 2 UEFI y none 0x 0x00 51841016
El Torito img path : 1 /isolinux/isolinux.bin
El Torito img opts : 1 boot-info-table isohybrid-suitable
El Torito img path : 2 /boot/grub/efi.img
But Debian is nice by having in the ISO filesystem a copy of the file
tree in the FAT filesystem of the EFI partition.
# mount debian-11.5.0-amd64-netinst.iso /mnt/iso
...
# mount /mnt/iso/boot/grub/efi.img /mnt/fat
$ (cd /mnt/fat ; find . -type f -exec ls -ld '{}' ';')
-rwxr-xr-x 1 root root 934240 Sep 10 2022 ./efi/boot/bootx64.efi
-rwxr-xr-x 1 root root 1684864 Sep 10 2022 ./efi/boot/grubx64.efi
-rwxr-xr-x 1 root root 101 Sep 10 2022 ./efi/debian/grub.cfg
$ (cd /mnt/iso ; find ./EFI -type f -exec ls -ld '{}' ';')
-r--r--r-- 1 root root 934240 Sep 10 2022 ./EFI/boot/bootx64.efi
-r--r--r-- 1 root root 1684864 Sep 10 2022 ./EFI/boot/grubx64.efi
-r--r--r-- 1 root root 101 Sep 10 2022 ./EFI/debian/grub.cfg
The differences in the paths become insignificant when copying to FAT.
The tests results indicate that the lack of x-permissions with the ISO
files is not an issue either.
In the arm64 ISOs the /EFI tree is present three times. Once as appended
MBR partition 2, once as FAT image file /boot/grub/efi.img in the ISO
which serves as El Torito boot image, and again as unpacked /EFI tree in
the ISO.
(xorriso could point El Torito to the appended partition, thus eliminating
the need for the /boot/grub/efi.img file.)
> From what I can see, the maximum individual file name length when using
> Joliet extensions is 128 "characters"
It's less. A Joliet directory record has the same maximum size as an ISO
directory record: 255 bytes. Joliet encodes names in UCS-2 which uses
16 bit per character. The fixed header part of a directory record is 34
bytes large. So only 110 UCS-2 characters have room. For some reason
libisofs offers to write 103 characters at most. (It's long ago that this
decision was made not by me.)
In file
/mnt/iso/.disk/mkisofs
i see the recorded -as mkisofs option:
-joliet-long
which means according to man xorrisofs:
Allow 103 characters in Joliet file names rather than 64 as is
prescribed by the specification. Allow Joliet paths longer than
the prescribed limit of 240 characters.
> if you are using Windows' native utilities, you won't be able
> format a partition that is larger than 64 GB to FAT32,
The largest official Debian ISOs are 50 GB.
But i have a script which can merge a complete set to a ~90 GB ISO. :))
Have a nice day :)
Thomas
Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support
Hi, James Addison wrote: > The problem, in both cases, was that I hadn't copied the '.disk' dotfile > directory from the install media ISO filesystem(s) in each case. Besides such user pitfalls with the produced ISO and the problem of symbolic links there are other constraints which an ISO has to fulfill for this use case. At least: - The file tree of the FAT filesystem in the EFI partition needs to be mirrored by a copy in the ISO filesystem. - File names must be unique in respect to case-insensitive comparison. - (I am not sure whether file name length can be a problem.) I guess Pete Batard can give a more comprehensive list. Have a nice day :) Thomas
Bug#1032941: Additional use of Rock Ridge symbolic links in bookwork ISO images
Hi, the use of xorrisofs/genisoimage option -f would convert those links into shared content data files, too. Side note: Differing from my statement in bug #1031696, the current xorriso version does not complain about dangling symbolic links when -f is given. It silently includes them as symbolic links in the ISO. I will change the behavior so that a SORRY message is emitted in this case. With default settings of xorriso this will cause a non-zero exit value of xorriso. But processing will go on as with the current version. genisoimage -f emits a warning and does not include the dangling link in the result. I.e. the file is missing in the ISO. Have a nice day :) Thomas
Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support
Hi, James Addison wrote: > Thinking aloud: as an alternative, would adding the '-f' flag to MKISOFS > achieve the desired result for both documentation and firmware files, > without requiring any other changes? Let's pack up some symbolic links with -f: $ sudo mount firmware-bookworm-DI-alpha1-amd64-netinst.iso /mnt/iso ... $ ls -l /mnt/iso/firmware total 4 lr-xr-xr-x 1 root root 73 Sep 20 14:30 amd64-microcode_3.20220411.1_amd64.deb -> ../pool/non-free/a/amd64-microcode/amd64-microcode_3.20220411.1_amd64.deb lr-xr-xr-x 1 root root 62 Sep 20 14:30 atmel-firmware_1.3-6_all.deb -> ../pool/non-free/a/atmel-firmware/atmel-firmware_1.3-6_all.deb lr-xr-xr-x 1 root root 62 Sep 20 14:30 bluez-firmware_1.2-7_all.deb -> ../pool/non-free/b/bluez-firmware/bluez-firmware_1.2-7_all.deb ... So Linux shows the symbolic links of the mounted ISO. I pick the directory with the links and a tree with most of their target files and put them into an ISO: $ xorriso -as mkisofs -f -R -o test.iso -graft-points /firmware=/mnt/iso/firmware /pool/non-free=/mnt/iso/pool/non-free ... Our goal is to have link and target path pointing to the same blocks. And indeed: $ xorriso -indev test.iso -find / -type f -sort_lba -exec report_lba -- ... Report layout: xt , Startlba , Blocks , Filesize , ISO image path File data lba: 0 , 49 , 57 , 115444 , '/firmware/amd64-microcode_3.20220411.1_amd64.deb' File data lba: 0 , 49 , 57 , 115444 , '/pool/non-free/a/amd64-microcode/amd64-microcode_3.20220411.1_amd64.deb' File data lba: 0 , 106 , 72 , 147088 , '/firmware/atmel-firmware_1.3-6_all.deb' File data lba: 0 , 106 , 72 , 147088 , '/pool/non-free/a/atmel-firmware/atmel-firmware_1.3-6_all.deb' File data lba: 0 , 178 , 89 , 181620 , '/firmware/bluez-firmware_1.2-7_all.deb' File data lba: 0 , 178 , 89 , 181620 , '/pool/non-free/b/bluez-fir ... With genisoimage i see the same. The only drawback seems to be that this prevents any symbolic links from being in the ISO. Whether the detection of dangling symbolic links at ISO production time is an advantage or a disadvantage is not so clear. Have a nice day :) Thomas
Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support
Hi,
James Addison wrote:
> Please find below a proposed update to the commit message under discussion:
Only one nitpick:
> multiple filenames ("records"). The ISO9660 generation tools
> genisoimage and xorriso can use this to de-duplicate multiple linked
> files from their input, saving space in the output filesystem image.
They do this only with hard links, not with symbolic links.
So i propose: "de-duplicate multiple hard linked files".
Have a nice day :)
Thomas
Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support
Hi, James Addison wrote: > I've opened a merge request at > https://salsa.debian.org/images-team/debian-cd/-/merge_requests/30 > [...] > The commit message contains an effort to explain what's going on; please > consider that message as reviewable and open to feedback too. I'm clumsy with web based development. So my comments by mail: -- Commit message: > When creating links to firmware files on the installation media in > the /firmware directory, want to retain the same content de-duplication > and resulting space-saving efficiencies as symlinks have provided. Shouldn't that be ", we want to" ? Since the symlinks are currently produced, shouldn't it be "as symlinks currently provide." ? (Disclaimer: I'm not a native speaker of english.) > We'd also like the file entries that are created on the disc image > to be compatible-with and copyable-from as many operating systems > as possible. It's rather about being copyable _to_ many filesystems by many operating systems. Actually it is mostly about FAT as target filesystem. It is the intended target for copying because EFI is willing to search for its boot program in about any partition with a FAT filesystem. (The specs prescribe special partition types but in practice EFI does not care.) Pete Batard's use case has two problems with symbolic links: - The FAT filesystem has no concept of links. - MS-Windows does not read Rock Ridge and thus ignores the symbolic link info anyways. (But this problem is not significant in the end.) So how about: "We'd also like the files of the ISO 9660 filesystem to be copyable to as many filesystems by as many operating systems as possible." > Symlinks have some drawbacks in that regard, since not all operating > systems fully support symbolic linking. I propose to write: "Links are not supported everywhere. The best substitute is to create independent copies of the link target file for every link which is encountered." > Fortunately, the ISO9660 format supports a standardized way to refer > to a common block of content (by referencing a starting block and a > block count on-disc) from multiple file paths. > Both genisofs and xorriso (CD image generation tools) support those kind > of shared content file references, and are able to replace multiple > hardlinks (POSIX links) that reference a single on-disk file with a > single shared content range in the output disc image. How about: "The free ISO 9660 specs ECMA-119 give in 6.5.1 permission to use the same file content blocks for multiple files. The ISO 9660 generation tools genisoimage and xorriso do this to represent hard link relations among their input files and to save space. Reading operating systems usually perceive the files as not hardlinked, though. So this representation of hard links causes creation of the desired independent data files when they get copied out of the ISO filesystem." man genisoimage says that this feature is enabled by default on "Unix-like operating systems" and can be disabled by option -no-cache-inodes . xorriso does not offer to disable it. Its mkisofs emulation ignores the option -no-cache-inodes . > Those tools won't do that by default for symlinks, however (for cautious > and sensible reasons related partly to the potential for > infinite-link-loops), and so The tools represent symbolic links in form of Rock Ridge SL entries. The current Debian ISOs mounted by Linux show the links and Linux follows them just fine. (Else it won't work for the Debian Installer.) Handling of infinite link loops is the job of the reader. So i propose to omit this part of the sentence. > this changeset switches from the Perl symlink function (that > creates symbolic links) to the more configurable debian-cd good_link > function that will create links based on the demands of the > environment it is configured in. Well, this aspect is out of my area of expertise. Have a nice day :) Thomas
Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support
Hi, James Addison wrote: > My interpretation of the commands and output in your comment is that both > genisoimage and xorriso can translate hardlinks from a source filesystem > into deduplicated file references in a written ISO filesystem With genisoimage we only know empirically. With libisofs under xorriso i can tell that a red-black-tree of device and inode numbers is used to match hard links. This is well tested by Debian ISOs because already now the Linux kernels are hard link siblings. E.g. in firmware-bookworm-DI-alpha1-amd64-netinst.iso 15 MB are won: Report layout: xt , Startlba , Blocks , Filesize , ISO image path File data lba: 0 ,35982 , 3761 , 7700896 , '/install.amd/gtk/vmlinuz' File data lba: 0 ,35982 , 3761 , 7700896 , '/install.amd/vmlinuz' File data lba: 0 ,35982 , 3761 , 7700896 , '/install.amd/xen/vmlinuz' If genisoimage would not deduplicate some hardlinks then the ISO would simply become larger but stay functional for the boot paths which debian-cd prepares and also for the file copying method which Pete Batard wants to get enabled. This method is intended for booting via EFI from USB stick. In general i support Pete Batard's goal to have bootable ISOs ready for it, regardless whether it is a wish or the fix of a regression. (I looked into firmware-11.6.0-amd64-netinst.iso and found its /firmare directory filled with symbolic links and one sub directory ./deb11.) The file extraction method is supposed to be a behavioral pattern of experienced users of MS-Windows. Let's be inviting to them or else they might install Debian on WSL. Have a nice day :) Thomas
Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support
Hi, James Addison wrote: > it looks like the selection of CD image > creation tool is configured per-architecture here: > https://salsa.debian.org/images-team/debian-cd/-/blob/5aebb6794a3b8b2393663fb643e35eb8e510c9a4/Makefile#L24 I wish i would understand the clause ifneq (,$(filter i386 amd64 arm64 hppa,$(ARCHES))) Surely i386, amd64, and arm64 get their published Debian ISOs made by xorriso. hppa seems to have switched to xorriso between Debian 10 and 11. sparc64 10.0 is genisoimage, ppc64el 10.8.0 is xorriso, armhf 10.1.0 is xorriso ... powerpc needs HFS and thus its ISO is made by genisoimage. So let's test hardlink handling in genisoimage: $ ls -l /u/test/hardlinks total 88 -rw-r--r-- 2 thomas thomas 42786 Nov 14 2005 hardlink_x -rw-r--r-- 2 thomas thomas 42786 Nov 14 2005 x $ genisoimage -o test2.iso -R /u/test/hardlinks ... 196 extents written (0 MB) We inspect the resulting ISO by xorriso: $ xorriso -indev test2.iso -find / -type f -exec report_lba -- ... Report layout: xt , Startlba , Blocks , Filesize , ISO image path File data lba: 0 , 25 , 21 ,42786 , '/hardlink_x' File data lba: 0 , 25 , 21 ,42786 , '/x' $ Both files in the ISO refer to the same block range starting at 2048-byte block 25 up to block 45. So in the ISO, they are deduplicated Now for a bit of kernel slapstick: $ sudo mount test2.iso /mnt/iso mount: /mnt/iso: WARNING: source write-protected, mounted read-only. $ ls -li /mnt/iso 1479 -rw-r--r-- 2 thomas thomas 42786 Nov 14 2005 hardlink_x 1483 -rw-r--r-- 2 thomas thomas 42786 Nov 14 2005 x Note the link count 2 in combination with the differing inode numbers. (The link count stems from Rock Ridge field PX. By mistake i mentoned it as "PN" in my previous mail.) The false link count does not hamper restoring of the files: $ cp -r /mnt/iso /u/test/hardlinks_restored $ ls -li /u/test/hardlinks_restored total 88 8913418 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:38 hardlink_x 8913419 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:38 x So the files are independent after being restored by Debian 11 to ext4. The same happens with an ISO made by xorriso: $ xorriso -as mkisofs -o test.iso -R /u/test/hardlinks ... $ xorriso -indev test.iso -find / -type f -exec report_lba ... Report layout: xt , Startlba , Blocks , Filesize , ISO image path File data lba: 0 , 33 , 21 ,42786 , '/hardlink_x' File data lba: 0 , 33 , 21 ,42786 , '/x' $ sudo mount test.iso /mnt/iso ... $ cp -r /mnt/iso /u/test/hardlinks_restored $ ls -li /u/test/hardlinks_restored total 88 8913418 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:47 hardlink_x 8913419 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:47 x Just to prove that the restored files are really not hardlinked: $ echo some_tail_bytes >> /u/test/hardlinks_restored/x $ ls -li /u/test/hardlinks_restored total 88 8913418 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:47 hardlink_x 8913419 -rw-r--r-- 1 thomas thomas 42802 Mar 11 17:48 x $ Have a nice day :) Thomas
Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support
Hi, James Addison wrote: > Something that symlinks can do is to allow image creators to save space by > by de-duplicating files. [...] This could be achieved by a bunch of hard links instead of the symbolic links. In /firmware of firmware-bookworm-DI-alpha1-amd64-netinst.iso i see only symbolic links to data files. So replacing symlinks by hardlinks should be technically possible. xorriso represents hard link siblings in ISO 9660 and Joliet as data files which use the same data content blocks. So these views of the Debian ISO would produce separate copies of target and link when both get extracted to disk. A Rock Ridge reader could do better by help of the PN entry. But Linux computes own inode numbers from the byte addresses of the ISO 9660 directory records. So hardlinks do not get represented as such when Linux mounts an ISO. Thus copying them out of the ISO causes duplication. This would be a disadvantage of hard links for those people who unpack Debian ISOs on Linux. Have a nice day :) Thomas
Bug#1022851: xorriso: -clone across sessions copies, not clones
Hi, i have pushed the libisofs bug fix upstream https://dev.lovelyhq.com/libburnia/libisofs/commit/acb4bd143c6374319e4f0fd91c23cd5bd93a3616 (Small but decisive.) If you or any bystanders prefer a GNU xorriso development tarball, i can upload one. For now i guess that you are prepared to build a new libisofs from git. Have a nice day :) Thomas
Bug#1022851: xorriso: -clone across sessions copies, not clones
Hi, the miracle of staying or moving is decided by the alphabetic sequence of imported original and freshly made clone. This lets the files keep their old data storage address: -clone /hello/hello.world /hello/hello.z This causes data storage in the new session: -clone /hello/hello.world /hello/hello.clone By a shortcomming of the static function iso_tree_clone_file() in libisofs/tree.c the cloned file is not marked as being imported from the loaded ISO, as is its original. This mark is important for the decision at image write time, whether a node can re-use its data storage or needs data storage in the new session. When the image production traverses the tree in alphabetic order it meets "hello.clone" before "hello.world" and decides the storage need of both by the missing import mark of "hello.clone". When it encounters "hello.world" before "hello.z", then it decides that no new storage is needed. Please verify this behavior with your local experiment. I plan to fix the bug in libisofs this evening. Have a nice day :) Thomas
Bug#1022851: xorriso: -clone across sessions copies, not clones
Hi, after quite some problems to reproduce your report with a simpler sequence of xorriso runs, i come to a riddling intermediate result. It has to do with the name of the file which gets cloned: test -e /dvdbuffer/g && rm /dvdbuffer/g test -e /dvdbuffer/h && rm /dvdbuffer/h printf %s\\n "Hello, world!" >/dvdbuffer/f printf %s\\n "X Hello, world! X" >/dvdbuffer/x xorriso \ -outdev /dvdbuffer/g \ -map /dvdbuffer/x /hello/x \ -map /dvdbuffer/f hello/hello.world xorriso \ -indev /dvdbuffer/g \ -grow-blindly 256 \ -outdev /dvdbuffer/h \ -clone /hello/hello.world /hello/hello.clone \ -clone /hello/x /hello/xxx dd bs=2048 seek=256 conv=notrunc of=/dvdbuffer/g < /dvdbuffer/h yields by xorriso -load sbsector 256 -indev /dvdbuffer/g -find / -exec report_lba -- this block address list Report layout: xt , Startlba , Blocks , Filesize , ISO image path File data lba: 0 , 280 ,1 , 14 , '/hello/hello.clone' File data lba: 0 , 280 ,1 , 14 , '/hello/hello.world' File data lba: 0 , 57 ,1 , 18 , '/hello/x' File data lba: 0 , 57 ,1 , 18 , '/hello/xxx' I.e. /hello/x and its clone kept their block address, /hello/hello.world and its clone got their content moved into the second session. I will have to dig deeper. But for now the experiments ate up my time. So i will have to go on later. In any case thanks for the bug report. Have a nice day :) Thomas
Bug#1019935: xorriso: Please make Multi-Arch: foreign
Hi, the following commits have been made to Salsa: -- libisoburn: commit 69ecfb2e52295798b52532bd8b1a5c693705fafe Author: Thomas Schmitt Date: Sat Sep 17 17:42:52 2022 +0200 Added "Multi-Arch: foreign" to the package descriptions of xorriso, xorriso-tcltk, xorriso-dd-target -- libburn: commit 5096eff7318f25f4c1714e3ff436c72369db46c7 Author: Thomas Schmitt Date: Sat Sep 17 18:16:42 2022 +0200 Added "Multi-Arch: foreign" to the package description of cdrskin -- I think that libisoburn-1.5.4-3 deserves an upload if it passes your review. It would close two bugs. libburn-1.5.4-2 seems less important. Have a nice day :) Thomas
Bug#1019935: xorriso: Please make Multi-Arch: foreign
Hi, I wrote: > > My cheat sheet says that i shall add new sections with "UNRELEASED" > > instead of "unstable" Dominique Dumont wrote: > Actually, you did set it to unstable in commit > 06434fcf33b824059a20f5c788cd399cf2fd4ee8, It is obvious that one should not let me do changes on Salsa. But i did notice this omission, committed commit 469750f1b47f6acd2c69032b38014f5f35feb5dc Author: Thomas Schmitt Date: Sat Jul 2 10:41:26 2022 +0200 and reported it in bug #1013313. So the only explanation is that i did not push it. Google ... "git log origin/master..master" ... argh. Well, my cheat sheet says that i shall clone a fresh repo by gbp. No need to find out what would happen if i would just do the usual git push --all && git push --tags > but I was not aware of (or I forgot :-/ ) the need to upload this package. Bug #1013313 ends by my request that you review the commits and decide about upload urgency. A week later, Alexandre Ghiti thanked us in advance. I will try to collect myself and to add "Multi-Arch: foreign" to libburn package "cdrskin" and libisoburn packages "xorriso", "xorriso-tcltk", and "xorriso-dd-target". You'll get a note when this is done. Have a nice day :) Thomas
Bug#1019935: xorriso: Please make Multi-Arch: foreign
Hi, Samuel Thibault wrote: > I was more thinking about architecture (i386, amd64, arm, etc.) > difference, not OS difference :) As long as it is Linux, libburn indeed behaves the same, modulo configuration options. Similarly with libisofs, where the presence of development headers or configuration options is significant. With Debian i would expect to find the same headers and options in all Linux arches. > > Is the described situation still suitable for Multi-Arch: foreign ? > I'm not sure which situation you are referring to? The fact that xorriso and the libraries differ in their capabilities on the various supported OSes. In case of Debian i thought mostly of Hurd. > Libraries are sually Multi-Arch: same, meaning they have to be the same > as the packages using them. That's actually already the case for > libburn, libisofs, and libisoburn. The current Multi-Arch headers of libisoburn, libisofs, and libburn source packages were introduced on request of Matthias Klose in 2016: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813020 http://launchpadlibrarian.net/235561045/libburn_1.4.2-1_1.4.2-1ubuntu1.diff.gz > Looking further: cdrskin, xorriso-tcltk, and xorriso-dd-target should > probably also be Multi-Arch: foreign. Seems plausible if xorriso gets it. Dominique: Do you agree ? - If so: I am a bit confused by the presence of a section "libisoburn (1.5.4-3) unstable" in https://salsa.debian.org/optical-media-team/libisoburn/-/raw/HEAD/debian/changelog while libisoburn is at 1.5.4-2 in https://tracker.debian.org/pkg/libisoburn My cheat sheet says that i shall add new sections with "UNRELEASED" instead of "unstable" and that you change this word when uploading. So i wonder why it is "unstable" but did not make it into Sid or Testing. Shall i add the new changelog entry to 1.5.4-3 and change it back to "UNRELEASED" or shall i make a new "(1.5.4-4) UNRELEASED" ? Have a nice day :) Thomas
Bug#1019935: xorriso: Please make Multi-Arch: foreign
Hi, such Debian specifics are more a matter of my sponsor's opinion than of my own. Cc-ing Dominique Dumont, therefore. Samuel Thibault wrote: > Since AIUI the xorriso package provides only a tool whose behavior is > exactly the same on all architectures, It depends on libburn which behaves different on Hurd compared to Linux. On Linux it can operate optical drive by SCSI passthrough (SG_IO). On Hurd it cannot (and i am not aware of a passthrough opportunity there). Another dependency, libisofs, shows differences when it comes to reading or writing ACL and xattr. The use case of creating bootable ISO images is normally not affected by these differences. > could you make it > Multi-Arch: foreign > so that the Debian Installer can use it as foreign dependency for > cross-builds? Is the described situation still suitable for Multi-Arch: foreign ? If so: Should libburn, libisofs, and libisoburn get the same change ? Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, i added to merge_debian_isos the capability to burn the resulting ISO 9660 filesystem directly to optical media and to write it directly to storage device files. This removes the need for having substantial free disk space beyond the capacity to store the original ISO images. While burning to optical media is quite a harmless operation for the normal user, writing to USB stick devices is not. Therefore merge_debian_iso can cooperate with xorriso-dd-target, my offer for safe copying of image files onto USB sticks on GNU/Linux. I created https://wiki.debian.org/MergeDebianIsos to describe installation, feeding, and usage of merge_debian_isos. Review and testing are welcome. (The same applies to the older https://wiki.debian.org/XorrisoDdTarget ) Have a nice day :) Thomas
Bug#1014400: libisofs: Add symbol file
Hi, i wrote: > > So if the Debian packaging software decides on its own that > > libisofs-1.5.2 is enough at package build time, then compilation > > [of libisoburn] will fail. Alexandre Ghiti wrote: > In an ideal world, the symbols should be backward compatible and then the > 'symbols' mechanism should be enough: I'm curious, why would it be different > here? The ABI's of libburn and libisofs are indeed backwards compatible. I.e. it should be no problem to link an old libisoburn, Xfburn or Brasero with new versions of libisofs. What i prevent as upstream is that new libisoburn links with old versions of libisofs. Normally there is the hard reason that new symbols of libisofs being used. But there is also the reason to keep old bugs from infesting newer versions of xorriso. > > Is that file format specific to Debian and its derivatives or > > does it play a role in more generic build facilities ? > I can't really say, let's wait for the package maintainer on this. Well, half of this maintainer role is fulfilled by me. I care for upstream related aspects and try to keep some of the Debian tools happy with the package parameters on salsa. Dominique Dumont coaches me in respect to Debian packaging skills. He also has the last say about what i do on salsa and eventually risks his reputation by uploading our combined work. We have the lintian warning no-symbols-control-file since a while. If it is desirable to silence it by a more or less dummy symbol file, then i would develop a script which generates it from the existing .ver file. Dominque would then hopefully tell me how to integrate that script into debian/rules (or wherever it belongs). As stated above, there would be the risk of compilation failure with libisoburn, if we would give dpkg the opportunity to decide against the inner version demands of libisoburn. So i see only the option to have that dummy .symbols file which just says what is available now in the current version, but gives no version history. If .symbols were a general concept outside of Debian's packaging world, then i would consider to maintain it upstream and to keep libisoburn's inner expectations in sync with the .symbols file. (Normally it would still want the newest libisofs because normally new symbols there are introduced to fulfill needs of libisoburn and xorriso.) As it appears now from googling, i think Dominique can decide either that we leave out .symbols, or that we create the dummy symbols file. In the latter case i'd need coaching about generating the dummy under control of debian/rules. Have a nice day :) Thomas
Bug#1014401: libburn: Add symbol file
Hi, like with bug #1014400 i see problems between the purpose of .symbols and the expectations of upstream code. Upstream code of libisoburn knows what it demands as minimum version. Any contrary decision by a Debian packaging program can only lead to intentional failure at compile time or at run time. See also: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1014400#10 (Additionally to the scruples expressed there, it comes to me that the goal of .symbols does not care for important bug fixes. A symbol may exist longer than the version where such bugs are fixed.) Have a nice day :) Thomas
Bug#1014400: libisofs: Add symbol file
Hi, do i get it right that the format is specified by https://wiki.debian.org/Projects/ImprovedDpkgShlibdeps ? If so, then the goal of having the file "In many cases, the dependency generated is too strict as the application doesn't necessarily use the newly-added symbols which justify the dependency bump in the shlibs file." collides with the upstream version check of libisoburn towards libisofs. libisoburn will not work with a libisofs version that is older than seen at compile time. The minimum version at compile time is defined in libisoburn.h (i.e. part of the API): /** The minimum version of libisofs to be used with this version of libisoburn at compile time. @since 0.1.0 */ #define isoburn_libisofs_req_major 1 #define isoburn_libisofs_req_minor 5 #define isoburn_libisofs_req_micro 4 So if the Debian packaging software decides on its own that libisofs-1.5.2 is enough at package build time, then compilation will fail. If the package manager decides that libisofs-1.5.2 is good enough at run time, then the start-up of libisoburn will fail. Both failures are intentional and make function isoburn_initialize() as ugly as paranoid. https://sources.debian.org/src/libisoburn/1.5.4-2/libisoburn/burn_wrap.c/#L74 In general i don't think it is worthwhile to offer old versions of libisofs to programs which get installed on a system where a newer version is available. - The .symbols file looks like maintainance effort, if it shall express more than the existing .ver file (i.e. the list of ABI symbols of the current version). Google is not overly helpful with search test ".symbols". Is that file format specific to Debian and its derivatives or does it play a role in more generic build facilities ? The API description in libisofs.h contains information about the version where a symbol appeared first (IIRC starting with 0.6.2 of 2008). So it would be possible to create a more differenciated .symbols file than attributing any symbol to version 1.5.4. But if it is only to tell the Debian package build software that really, really 1.5.4 is the version to use, then i'd propose to generate the .symbols file freshly in the course of package generation. (Bystanding DDs please tell me where in ./debian this can be done. I would use a shell script that creates .symbols from .ver. In this case it would be nice if that script could learn the upstream version automatically, so that it has not to be changed with each version. Of course, some dh_ magic would be welcome too.) Have a nice day :) Thomas
Bug#1013313: libisoburn: Add autopkgtest
Hi, i committed upstream the discussed changes to releng/run_all_auto. https://dev.lovelyhq.com/libburnia/libisoburn/commit/4ff9f8eedf49ec75f6066ae8f396cf65318f11b5 Next i cherrypicked the patch steps from my release cheat sheet and created a patch from the new upstream state. Further i added the new file debian/tests/control to the salsa git. Committing went in three steps, because i managed to forget adding the patch itself with the first commit, and then noticed that i did not change "unstable" to "UNRELEASED" in the new changelog section. (Pfft ... one should not let me do procedures which i practice only once per year.) commit 469750f1b47f6acd2c69032b38014f5f35feb5dc Author: Thomas Schmitt Date: Sat Jul 2 10:41:26 2022 +0200 Changed new changelog entry from unstable to UNRELEASED commit 543fcccb177644c7699a20bd4645f2d4b9fe518b Author: Thomas Schmitt Date: Sat Jul 2 10:11:14 2022 +0200 Added forgotten patch 02-Patch-releng-run_all_auto.patch commit 06434fcf33b824059a20f5c788cd399cf2fd4ee8 Author: Thomas Schmitt Date: Sat Jul 2 10:02:49 2022 +0200 Patched releng/run_all_auto to return non-zero exit value on failure and introduced debian/tests/control. (Closes: #1013313) Before pushing, i restored the old state of releng/run_all_auto and ran gbp buildpackage which looked good, mentioned dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 02-Patch-releng-run_all_auto.patch ... and ended by: Successfully signed dsc, buildinfo, changes files Strangely lintian complains: W: libisoburn1: improbable-bug-number-in-closes 1013313 I triple checked that 1013313 is indeed the number of this bug. So as far as i can tell, the new state of g...@salsa.debian.org:optical-media-team/libisoburn.git is now ready for review by Dominique Dumont and for creating of package libisoburn 1.5.4-3 at his discretion. Only changelog entry is * Patched releng/run_all_auto to return non-zero exit value on failure and introduced debian/tests/control (Closes: #1013313) Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi,
Zhang Boyang wrote:
> All OK! Thank you again :)
So as far as we two are concerned, it seems we have a candidate for
release.
Currently it only goes into the GNU xorriso tarball, which is not src of
any Debian package. I could put it into the libisoburn tarball. But as
it is specific to debian-cd ISOs (not even to debian-live ones) i think
it should go into debian-cd (with a copy in GNU xorriso, which already
has libjte as Debian specialty).
Nevertheless, most important would be to offer it as plain text download
for those who have a shell and downloaded ISOs. If it were only available
as Debian package then it would not help those who still have to become
Debian users.
A cool place would be the download pages
https://cdimage.debian.org/debian-cd/current/{amd64,...}/jigdo-{dvd,bd,dlbd}
and mentioning in their *SUMS files.
Have a nice day :)
Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, Zhang Boyang wrote: > Theoretically if both file is already sorted, we can use the `-m' option I like this idea. Just in case Debian grows to a million packages. But i understand that i would need two separate files for sorting. grep ' ./pool/' file1 fgrep -v ' ./pool/' file2 sort -k 2 -m file1 file2 >temp_file More temporary files means more need for pre-existence tests and more cleanup effort. Currently i just have to concatenate two stdout streams. > However I don't think O(n * log(n)) is a bottleneck so we may just keep it > simple and stupid. Seems to be the best decision for now. (Unless some shell wizzard shows a way to pipe both streams separately into sort -m while staying dash compatible and without the need for new persistent file objects which have to be cleaned up afterwards.) Actually we did not yet identify a use case where md5sum.txt needs to be sorted at all. I only sort it because the input files are sorted. New version: https://dev.lovelyhq.com/libburnia/libisoburn/commit/0e8227e76ae4c4f24097cfac2f415ef8e25ae4e7 Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, Zhang Boyang wrote: > These experiments all succeeded. Thank you very much! Good Job! :) Thank you for testing and challenging. I wrote: > > For now i decided to take the 50 seconds with dash. > if you really want to reduce runtime I would suggest using > `sort -s -u -k 2 merged_md5sum.txt' instead of processing each line > by hand. The task is to identify those which need newly computed MD5 because they might have changed. Mostly i know which directories are suspects, because they are on hard disk and get mapped back into the emerging ISO. Their MD5s get recomputed from the files on hard disk. Some other paths in the md5sum.txt may appear multiple times. In this case it is clear that the data of the file in the emerging ISO stem from iso1. But it is not clear which of the multiple lines in md5sum.txt stems from iso1. So the MD5 has to be recomputed from the file in mounted iso1. > I saw there are some other logic to process md5 records from > different group of files, so we can use `grep' and `grep -v' to split them, > process them separately, then merge them finally. That's a great idea. The majority of files is in ./pool and surely needs no recomputing, even if listed multiple times (due to overlapping ISO pools). This here ( fgrep ' ./pool/' temp_file needs 1.9 seconds instead of 7.2 seconds with the old polish_md5sum_txt >temp_file Times were measured by date '+%s.%N' around the polishing commands. polish_md5sum_txt and its subordinate were slightly modified for the new method to read from stdin and to not expect any ./pool file. The latter brought 0.9 seconds. The number of lines in md5sum.txt is then the same as with the old method. My test loop with md5sum -c on the mounted result ISO reports no mismatches. (It is annoying that gzip inserts a time stamp, so that the Packages.gz files differ although they bear the same uncompressed content. So the md5sum.txt file shows differences, too, from run to run.) > Unfortunately the option `-s' of `sort' is not standard I understand that it is needed to keep sort -k 2 from distinguishing lines with differences outside of -k 2 so that sort -u could throw out surplus lines with duplicate paths. But with above code sort -u is not needed. The pool lines have to be identical even if duplicate paths appear at all. (I only know of one old debian package which existed with different content but same name, long ago.) So uniq can do its job. The other lines are made unique by the shell function polish_md5sum_txt. Complexity-wise this replaces a slow O(n) algorithm by a faster O(n) and an additional O(n * log(n)) run. At some size of Debian the slow speed of the linear loop will be compensated by the sorting complexity. But there is still room: A sort of 11,000 lines lasts about 0.03 seconds. I will probably commit this change tomorrow. Now it needs cleaning and handling of the new dependency uniq. Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi,
i explored two ways to get a correct and complete md5sum.txt after
merging and sorting:
- Generating the whole md5sum.txt from the emerging ISO's file tree is
possible by help of xorriso, although there is no file tree yet where
all regular files of the ISO can be found for checksumming.
But this would work only for MD5 and not for SHA256, because xorriso
has no API function to compute SHA256. (libjte has SHA256 but not as
API. There are many xorrisos out there which are not linked to libjte.)
Run time would be mediocre: 36 seconds for DVD-1+2+3 on a 500MB/s SSD.
(find+md5sum on mounted result ISO needs 27 seconds. But as said there
is no result ISO to mount when the script makes md5sum.txt.)
- Removing duplicates from md5sum.txt and identifying those files which
possibly have a changed MD5 after the merging activities is more error
prone but signficantly faster.
Especially it can be easily modified when Debian decides to retire
md5sum.txt in favor of a sha256sum.txt.
Run time is still annoying: 8 seconds with dash, 12 seconds with bash.
As it is now it works with echo | grep.
I could reduce it to less than a second by using the bashism
${Var:Offset:Count} to obtain a substring of the file paths.
Run time for an All-in-one ISO is estimated about 6 to 7 times the time
of DVD-1+2+3.
So i expect ~230 seconds for full MD5 regeneration, ~ 50 seconds for
a loop that runs on dash, and ~6 seconds with a bashism.
For now i decided to take the 50 seconds with dash.
The merged md5sum.txt is not 100% complete. Files in ./firmware which
appear in more than one input ISO will not be listed, because it is not
100% clear from which ISO the one stems which survives the competition.
It appears that the check in
https://salsa.debian.org/installer-team/cdrom-checker/-/raw/master/main.c
does not insist in a complete list. It only demands that all listed files
exist and yield the listed MD5 when being checksummed.
I tested the correctness of the merged md5sum.txt of DVD-1+2+3 by mounting
the result as /mnt/iso and running what i deem equivalent to the MD5 check
in installer-team/cdrom-checker :
(cd /mnt/iso
cat md5sum.txt | while read line
do
if echo -n "$line" | md5sum -c 1>/dev/null 2>&1
then
dummy=dummy
else
echo "BAD: $line"
fi
done ) 2>&1 | wc
Result from wc was "0 0 0" (after 47 seconds).
Regrettably i cannot check this with my dummy DLBD-1+2 ISO, because its
data files nearly all have fake content.
Committed changes:
https://dev.lovelyhq.com/libburnia/libisoburn/commit/0bc397c02c0ea7c960b59ce92daa267bed23fc07
Have a nice day :)
Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, i wrote: > > > md5sum.txt Zhang Boyang wrote: > I think maybe we should just create the md5sum from scratch? Well, not from scratch, because the paths would come from the merged md5sum.txt. But indeed the merged file needs polishing: - Some files are listed multiple times because they appear in the md5sum.txt files of multiple input ISOs. - The files in ./dists are probably newly generated. The bulk of packages and support files is not supposed to have changed. So i will have to promote the md5sum program to hard depenceny and make a filtering run over the merged and sorted md5sum.txt file to care for above cases. I will give a note when this is ready for testing. Have a nice day :) Thomas
Bug#1013313: libisoburn: Add autopkgtest
Hi,
Dominique Dumont wrote:
> You may want to log this case which leads to a test failure
I understand that George Danchev considered the output of grep enough
of a message:
grep -n -E "${LIST_KWD}" "${CLOG}"
RET_GREP="$?"
ok=0
case ${RET_GREP} in
0) # found
;;
With two dummy failure lines appended to the test log the messages end by:
...
run_all_auto: Trivial log examination: releng_generated_data/log.run_all_auto
254:Dummy SORRY
255:Dummy FAIL
...
With a successful test run it looks like:
...
run_all_auto: Trivial log examination: releng_generated_data/log.run_all_auto
run_all_auto: Log file looks clear.
...
I think it would be better to add a final message by:
--
@@ -259,5 +269,13 @@ fi
ls -lth "${GEN_DATA_DIR}" # | tee -a ${CLOG}
# Fin
+ if test "$exit_value" = 0
+ then
+ echo "${SELF}: +++ Test run and its log look like success."
+ else
+ echo
+ echo "${SELF}: --- Test run detected some failures."
+ echo
+ fi
exit $exit_value
--
> https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst
Since you don't object i assume that you are ok with the proposed new file
debian/tests/control
> An alternative is to run these tests in debian/rules so they are run at
> package build time.
> This assumes that these tests are reliable.
As upstream, i don't think that releng/run_all_auto should be run at
package build time. I run them locally before each upstream release.
Whatever:
The following tests are made by releng/run_all_auto :
./auto_cxx
Exercises inclusion of xorriso/xorriso.h and libisoburn/libisoburn.h
in C++ programs and linking of the libraries.
./auto_isocontent
Tests whether xorriso is able to record and restore two families of
hard links.
./auto_printsize
Tests how long xorriso needs to compose a medium sized directory tree.
If programs mkisofs and/or genisomage are available, then the same test
is done with them.
See also
https://sources.debian.org/src/libisoburn/1.5.4-2/releng/README/#L141
They cost about 10 seconds on a 3.5 GHz Xeon CPU.
Currently the only test at Debian package build-time is a run of
xorriso/xorriso -no_rc -version -list_extras all
which is probably triggered as upstream makefile target "check-local".
(I fail to find the trigger of "make check" in ./debian.)
The reason for this run is mainly to get into the buildd logs a report
about the characteristics of the produced binary.
Have a nice day :)
Thomas
Bug#1013313: libisoburn: Add autopkgtest
Hi,
Alexandre Ghiti wrote:
> Please find attached a patch that enables the releng testsuite.
With my upstream hat on:
I can imagine why it is desirable that log file anomalies cause a non-zero
exit value. But simply aborting the script by "exit 1" seems not to match
its concept (which is by George Danchev).
How about using the delayed exit mechanism which gets into effect if one
of the single tests exits non-zero:
---
--- releng/run_all_auto 2020-07-08 12:46:04.314650652 +0200
+++ releng/run_all_auto 2022-06-21 19:26:18.501863413 +0200
# lines, perl regex, leading tabs
grep -n -E "${LIST_KWD}" "${CLOG}"
RET_GREP="$?"
+ ok=0
case ${RET_GREP} in
0) # found
;;
1) # not found
echo "${SELF}: Log file looks clear." # | tee -a ${CLOG}
+ok=1
;;
*) #
echo "${SELF}: grep returned EXIT CODE: ${RET_GREP}." # | tee
-a ${CLOG}
;;
esac
+ if test "$ok" = 0 && test "$exit_value" = 0
+ then
+ exit_value=1
+ fi
set -e
fi
echo "" | tee -a
${CLOG}
--
With my sponsored packaging helper's hat on, Cc-ing Dominique Dumont
to get an opinion from under the sponsor's hat:
> +++ libisoburn-1.5.4/debian/tests/control
I don't find this described in
https://www.debian.org/doc/manuals/maint-guide
Google finds me
https://people.debian.org/~eriberto/README.package-tests.html
> +Test-Command: cd ./releng && ./run_all_auto -x $(which xorriso)
> +Depends: xorriso, g++, libburn-dev, libisofs-dev
At which occasion will this be run ?
Does this have influence on the dependencies of the binary packages ?
Have a nice day :)
Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, Zhang Boyang wrote: > 2) Create my own version of my-DLBD1.iso and my-DLBD2.iso from my private > mirror. This explains why a firmware package was missing which is in the official DLBD-1 but obviously on your my-DLBD2. With the new script version which merges /firmware trees: > This time the difference in /firmware is: > Only in /groundtruth/firmware: > arm-trusted-firmware-tools_2.4+dfsg-2_amd64.deb > I don't think this is the merge script's fault. This .deb is not exist in > both my-DLBD1.iso and my-DLBD2.iso. Then it's fully ok for me. :)) > There are other differences in filesystem-tree. > [attachment diff.details.txt] > .disk/mkisofs This is just an unchanged copy of the file in your DLBD-1. (As stated previously i see few reason to overwrite it with the xorriso arguments which were used for merging.) > boot/grub/efi.img Could be about timestamps in the FAT filesystem. One could try whether exporting SOURCE_DATE_EPOCH=...seconds.since.1970... with the same value for DLBD and CUSTOM would create the same efi.img files. ( https://reproducible-builds.org/docs/source-date-epoch/ ) > dists/bullseye/Release This is a very central file. It would be interesting to see the full diff. > dists/bullseye/contrib/binary-amd64/Packages.gz Could be the sequence of packages in this unsorted list of multi-line records. Critical are the last package record of DLBD-1 and the first of DLBD-2. In the merged ISO's Packages.gz they have to be listed with a neat single empty line between them. (In my tests this was the case.) > dists/bullseye/contrib/i18n/Translation-cs.gz At least in the DVD set of 11.2.0 the i18n files which are mentioned in Release are incomplete counterparts of the respective Packages.gz files in various languages. Many packages are not listed in them. > dists/stable That's a symbolic link to dists/bullseye. So i skip to: > isolinux/boot.cat This file contains the block addresses of isolinux/isolinux.bin and of boot/grub/efi.img. These addresses are subject to many influences during ISO production. Somewhat random. > isolinux/f1.txt Seems to be a production timestamp of d-i. It would be interesting to see whether it obeys SOURCE_DATE_EPOCH. > isolinux/isolinux.bin The -boot-info-table patching at the begin of ISO production writes the block address of isolinux/isolinux.bin into isolinux/isolinux.bin. > md5sum.txt Ouch. My script sorts the merged lines by the MD5 fields rather than by the file paths. Further this sorting is subject to locale settings, which is hardly desirable, if the sequence of lines has a meaning at all. I committed a new version which fixes both problems in my local tests: https://dev.lovelyhq.com/libburnia/libisoburn/commit/87aab730582cb4268f50062361fec7f13e2b50ab Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, i tested merging of /firmware directories with barely sufficently complete debian-11.0.0-amd64-DLBD-[12].iso.tmp from aborted jigdo-lite runs. All files which are reported as being only in CUSTOM(all-in-one) by https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1011343#115 are listed by xorriso as present in the emerging ISO, of which i suppressed the actual production out of storage space reasons. So if the next diff between self-made CUSTOM(all-in-one) and merged.iso shows again missing files in /firmware, i have to ask for full listings of the /firmware trees in DLBD-1, DLBD-2, CUSTOM(all-in-one), and merged.iso . --- About my shortcuts with download and ISO non-production: I waited with aborting jigdo-lite just long enough until the first downloaded packages caused the inflation of the .template file to .iso.tmp. The directory tree and the management files are then already present, whereas the packages' content is mostly waiting for being filled in. Further i modified the xorriso run to end by -find /firmware -- -rollback_end so that no ISO emerges but only the content of the planned /firmware directory is shown before the program ends. (I would go further. But the production of CUSTOM(all-in-one) via debian-cd is out of my reach, given the section "ABOUT MIRROR" in message #115. So the effort to complete the download and to produce a merged.iso would bring no benefit for now.) Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, Zhang Boyang wrote: > I will definitely try it Meanwhile i got some insight into the riddle about diffs between merged.iso and CUSTOM-1.iso like Only in /groundtruth/firmware: arm-trusted-firmware-tools_2.4+dfsg-2_amd64.deb Only in /groundtruth/firmware: gnome-firmware_3.36.0-1_amd64.deb Indeed arm-trusted-firmware-tools_2.4+dfsg-2_amd64.deb does not exist in the /firmware directory of DLBD-1 and thus did not get into merged.iso. But /firmware/gnome-firmware_3.36.0-1_amd64.deb is in DLBD-1 as symbolic link and its target is in DLBD-1, too. Please verify that it is really not in merged.iso. If so, then please record the messages of the next experiment with the merge script and send them to me in private. I now uploaded a new version of the script which merges the /firmware directories. Just a guess, until i know what's up with /firmware. To those who are familiar with debian-cd, especially Steve McIntyre: /firmware is not mentioned in https://wiki.debian.org/DebianRepository/Format So i guess that it is specific to debian-cd or the installer along https://wiki.debian.org/Firmware#Firmware_during_the_installation and that i should merge the /firmware directories. Correct ? I am a bit confused by the fact that debian-11.2.0-amd64-DVD-2.iso has no firmware directory at all. How come ? Can this happen to a first ISO, too ? (DVD-1 has a /firmware tree with only inhabitant dep11/README.txt.) Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, i followed David's advise to consider the warnings of shellcheck, to drop the demand for bash, and to check early whether all needed programs are available. xorriso gets now checked for being young enough for the job of replaying the boot related commands as detected with input ISO 1. (1.3.2 in Debian 8 is too old. 1.4.6 in Debian 9 is ok.) The warnings yielded no real problem fixes but several improvements in regard to general shell code quality. The switch from /bin/bash to /bin/sh was not costly. A printf formatter had to be changed and a use of "type -p" for a message was dropped. An test installation with the resulting ISO succeeded and was able install further packages from all three merged DVD ISOs. (I still could need proposals for further checking the fitness of the ISO.) As before, the script is available at https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos with https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos.sig The changes are to see at https://dev.lovelyhq.com/libburnia/libisoburn/commit/34981b1278610d3f31da90f57fdf1378d6012074 Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, i wrote: > > https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos Steve McIntyre wrote: > That might be a useful thing to include in a package. What do you think? Best would be if debian-cd would take it, so that it can be adapted when the repository format in the ISOs gets changed. It could be renamed to e.g. "debian-cd-merge-isos" and become a separate "binary" package from the debian-cd source package. I would of course be willing to help with maintaining it. But first it needs more testing, especially whether the resulting ISO lacks anything that an ISO-1 from debian-cd with the same packages has. Zhang Boyang posted a comparison in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1011343#115 https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1011343;filename=diff.details.txt;msg=115 about which i ponder: - Shall the script create a new /.disk/mkisofs ? The used xorriso run has few similarity with a xorriso run from debian-cd. The pool content and package lists have no influence on the xorriso run of debian-cd. So it might be better to stay with the original. - There are some /firmware files missing in the merged ISO. Like /firmware/arm-trusted-firmware-tools_2.4+dfsg-2_amd64.deb ... /firmware/gnome-firmware_3.36.0-1_amd64.deb They are in the pool of DLBD-1. But how did they get into /firmware of Zhang Boyang's CUSTOM(all-in-one) ? (Or: Why aren't they in /firmware of DLBD-1 ?) I hope that the script is sufficiently stable against bad arguments and bad combinations of ISOs. At least i inserted a lot of error messages and took care to trigger each of them. Nevertheless, it should be tested independently of me whether it can be tricked into destroying existing data on disk or leaving temporary files on disk. Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, although it was not the final solution of this bug report, i beefed up my merger script for Debian ISOs so that it can combine an arbitrary number of ISOs (within the limits of /dev/loop* and mount(8)). Maybe it can serve as answer for the next time this wish comes up. The script is uploaded as https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos with GPG detached signature https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos.sig for checking by gpg --verify. After download, the user probably has to give x-permissions to the script. When run without arguments it gives this help text: --- usage: merge_debian_isos result_iso mount_template iso1 iso2 [... isoN] Mounts by sudo the ISO 9660 images iso1 to isoN at directories mount_template1 to mount_templateN, if not already mounted that way. Then the Debian pools and package lists get merged and a new ISO 9660 image result_iso is produced, which must not yet exist. If iso1 is bootable then the new image will be bootable by the same means. At least the parent directory of mount_template must already exist. All arguments must be single words without using quotation marks. None of the isoN must be equal to another isoM. This script creates and finally removes the following temporary tree and files which must not yet exist in the current working directory: ./merged_dists , ./merged_md5sum.txt , ./merged_REAMDE.txt ./temp_file Further it creates and finally removes directories mount_template* if they are needed and do not exist when the script starts. It depends on the following programs: awk, basename, bash, cat, chmod, cp, dirname, expr, fgrep, grep, gunzip, gzip, head, ls, mkdir, mount, mv, rm, rmdir, sha256sum, sed, sort, stat, sudo, umount, xorriso Recommended are: md5sum, sha1sum, sha512sum Exported non-empty variable MERGE_DATE enforces a particular date string in the text which gets prepended to /README.txt . Exported non-empty variable MERGE_FOR_DIST enforces the use of a particular directory in /dists of iso1. Normally only one such directory is found and thus no need to set MERGE_FOR_DIST. Exported non-empty variable XORRISO overrides command xorriso. This may be needed if installed xorriso is older than 1.4.2. Example using GNU xorriso-1.5.4 instead of /usr/bin/xorriso: export XORRISO=$HOME/xorriso-1.5.4/xorriso/xorriso mkdir merge_mount merge_debian_isos merged.iso merge_mount/iso \ debian-11.2.0-amd64-DVD-[12345].iso rmdir merge_mount --- The old script merge_2_debian_isos still exists but will refuse to run, unless the user removes an exit command near the start of the script. In its downloaded form it tells the user: --- THIS SCRIPT IS DEPRECATED ! USE ITS SUCCESSOR: merge_debian_isos The script merge_2_debian_isos still exists only because it was mentioned in Debian bug 1011343. The successor can merge more than two ISOs. So do not edit this script to remove this warning and the 'exit 7' line. --- I tested the new script by merging the first three ISOs of debian-11.2.0-amd64-DVD-*.iso and installing Debian from the result into a qemu-system-x86_64 VM with 4 GB RAM and 128 GB system disk. After the installation was completed, i successfuly installed packages which are in DVDs 1, 2, and 3 and were not installed yet: apt-get install apt-file apt-get install lame apt-get install xorriso To verify that this success does not come from an unwanted network connection i tried to install packages from DVD 4 apt-get install bash-doc apt-get install devede Both attempts failed with the message that the packages are not available. Have a nice day :) Thomas
Bug#1010098: xorriso: please allow -cut-out directly from block devices
Hi, Ivan Shmakov wrote: > An alternative would be to check if it’s possible to seek to > specified positions rather than to the end of file This should work now, by an attempt of lseek(wanted_size, SEEK_SET) if lseek(0, SEEK_END) yields failure or 0 size. (libisoburn commit 0c0d542, libisofs commit c6cb7df) Nevertheless, the attempt to read from /dev/zero still yields failure, because lseek() returns 0 when it is supposed to return the new offset position. This is already so in your example with strace and dd: > lseek(0, 2560, SEEK_CUR)= 0 Actually i think that the behavior of lseek() with /dev/zero is not conformant with man 2 lseek: RETURN VALUE Upon successful completion, lseek() returns the resulting offset loca‐ tion as measured in bytes from the beginning of the file. On error, the value (off_t) -1 is returned. The xorriso code does not ask for offset 0. So the lseek() call does not succeed if the resulting offset is 0. So the return value should be -1. If non-random-access file content is desired and can be produced with repeatable size, then consider to define a filter by -external_filter and attach it to a file by -set_filter. The file may be inserted with an arbitrary file as source. The filter program is free to ignore it and to do its own thing. --- > if the target file is created in a yet-nonexistent > directory, the directory created inherits the permissions > of the source file as well (only adding +x where there’s r.) This is an intentional feature introduced in 2008. SVN rev 1643, now git commit a44e7db9: Bug fix: Implicite directory attribute copying with -cut_out was wrong There are reasons why this makes sense if the disk file's permissions are restrictive and its parent's permissions are liberal. Embarrassingly, only a week later i introduced automatic file splitting, controlled by command -split_size, where this peculiar source of directory permissions would make even more sense. But function Xorriso_graft_split() inserts a new directory with the permissions of the parent directory in the ISO. I have now corrected this to the behavior of -cut_out and documented it with both commands in the man page: -cut_out disk_path byte_offset byte_count iso_rr_path ... E.g. ... -cut_out /my/disk/file 4094m 2047m \ /file/part_3_of_3_at_4094m_with_2047m_of_5753194821 If the directory /file does no yet exist, then its permissions are not taken from directory /my/disk but rather from /my/disk/file with additional x-permission for those who have r-permission. ... -split_size number["k"|"m"] ... The newly created ISO directory inherits its permissions from the data file with additional x-permission for those who have r-permission. (libisoburn commit 5aac3dd) --- Then there was the problem of -paste_in not working with devices which are suitable for -cut_out (and thus not being the reverse operation of -cut_out). I changed -paste_bin so that it is willing to work with the same files as target which -cut_out would accept as source (modulo permissions, of course). (libisoburn commit 865115f) Have a nice day :) Thomas
Bug#1010098: xorriso: please allow -cut-out directly from block devices
Hi, Ivan Shmakov wrote: > An alternative would be to check if it’s possible to seek to > specified positions rather than to the end of file; e. g.: I see at least the problem that the end position of -cut_out (byte_offset + byte_count + 1) is allowed to be unrealisticly high in order to simply address the end of the file. This means that xorriso and libisofs need to know this end. I will have to think whether a file is usable which does not return success on lseek(,,SEEK_END) but on lseek(,desired_size,SEEK_SET) ... and whether all size inquirers can know the desired_size ... > > Please give the code a thorough test, especially with weird > -cut_out arguments. > This looks like a job for a test suite. I gather there’s > none as of yet? There were visits by the American Fuzzy Lop. It is an art of its own to design tests which catch unexpected bugs. > Meanwhile, I’ve noticed that the files created via -cut-out > inherit the permissions and m-time of the original file. > The former might be reasonable, but the latter doesn’t seem > to make much sense, at least for device files (whose m-times > tend to have no relation to their content proper.) It's for backup fidelity. osirrox command -paste_in can be used to restore a file that was stored as several -cut_out pieces. (This raises the question whether -paste_in works with devices ... ... No, it insists in regular files. Another thing to think about.) If you want other timestamps, use command -alter_date (+0 = right now). > Worse still, if the target file is created in a yet-nonexistent > directory, the directory created inherits the permissions > of the source file as well (only adding +x where there’s r.) This looks like a bug. Actually it should copy the permissions from the parent directories in the -cut_out disk_path, like command -map does. But indeed the non-x permissions are taken from the data file itself. I will investigate (and try to remember what i did 14 years ago, as i suspect command -split_size to be the regular customer of this surprise feature). Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, for now it looks like the merged ISO works as fat DVD-1. I installed a Debian 11.2.0 system from the merged DVD-1+ DVD-2 ISO in a qemu VM via option -cdrom. Installation went smoothly. During reboot i aborted the VM to next deface the El Torito boot sector of the ISO. I removed the -net options from the qemu start command, hopefully simulating a pulled network adapter. Then i booted from the virtual -hda, having the -cdrom present too. I removed from sources.list the deb and deb-src entries with http. ping debian.org gets no replies. So the network is effectively down, although ip addr shows some "ens3" with "altname enp0s3", which looks much like the "nnp5s0" of the host machine except that the latter has no "altname". To make sure that i can install stuff, i did as superuser dpkg -s apt-file dpkg -s lame dpkg -s xorriso which all three indicated that the packages are not yet installed. Then i tried to install them: # From pool of DVD-1 apt-get install apt-file # From pool of DVD-2 apt-get install lame # Not in the two DVD images apt-get install xorriso The first two succeeded and behaved as i know from apt-get. The third did not succeed, as expected. -- But apt-file update fails with E: The repository 'cdrom://[Debian GNU/Linux 11.2.0 _Bullseye_ - Official amd64 DVD Binary-1 20211218-11:13] bullseye Release' does not have a Release file N: Update [...] can't be done securely [...] disabled by default. N: See apt-secure(8) [...] Suspecting a flaw with my merge procedure, i booted with the original DVD-1 ISO as -cdrom. I had to run apt-cdrom add before the repo was accepted (i'm riddling a lot) but then i could prove that it works by apt-get install needrestart which is on DVD-1. Nevertheless apt-file update tells the same "E:" and "N:" lines. So the shortcomming seems already to be in DVD-1. Anything known about it ? (I learned that it was not necessary to deface El torito on DVD-1+2. DVD-1 with El Torito at -cdrom does not keep -hda from booting.) -- What shall i further test to challenge the merged ISO whether it has any shorcommings compared to the original DVD-1 ? Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, i uploaded the first version of my merger script as https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_2_debian_isos Please download and give x-permission. The need for sudo can be avoided by already mounting the two ISOs at the chosen mount points before running the script. Review and test reports are appreciated. As motivation for tests by Debian installation experts: If this works, then Debian could replace the DLBD sized ISOs by the BD sized ISOs and instructions how to merge them to DLBD ISOs or a QLBD/All-in-one ISO. (In that case the script should move into the hands of debian-cd, i guess.) === $ ./merge_2_debian_isos usage: merge_2_debian_isos \ boot_iso boot_mount add_iso add_mount result_iso [for_dist] Mounts by sudo the boot_iso at directory boot_mount and add_iso at add_mount, if not already mounted that way. Then both Debian pools and package lists get merged and a new ISO 9660 image result_iso is produced, which must not yet exist. If boot_iso is bootable then the new image will be bootable by the same means. This script creates the following temporary tree and files which must not yet exist in the current working directory: ./merged_dists , ./merged_md5sum.txt , ./merged_REAMDE.txt ./temp_file The optional sixth argument for_dist should only be given if this script refuses to work and proposes to give this argument. Exported non-empty variable MERGE_DATE enforces a particular date string in the text which gets prepended to /README.txt . Exported non-empty variable XORRISO overrides command xorriso, which may be needed if installed xorriso is older than 1.4.2. Example using GNU xorriso-1.5.4 instead of /usr/bin/xorriso: export XORRISO=$HOME/xorriso-1.5.4/xorriso/xorriso merge_2_debian_isos debian-11.2.0-amd64-DVD-1.iso /mnt/iso1 \ debian-11.2.0-amd64-DVD-2.iso /mnt/iso2 merged.iso === The /README.txt of the result ISO gets a prefix text: === Result of a run of merge_2_debian_isos at 20220523-20:09 Package pools and Packages lists were merged. The other files stem from the first input ISO. Input ISO: debian-11.2.0-amd64-DVD-1.iso Debian GNU/Linux 11.2.0 "Bullseye" - Official amd64 DVD Binary-1 20211218-11:13 Input ISO: debian-11.2.0-amd64-DVD-2.iso Debian GNU/Linux 11.2.0 "Bullseye" - Official amd64 DVD Binary-2 20211218-11:13 -- [... text of boot_iso's README.txt ...] === The result of merging debian-11.2.0-amd64-DVD-[12].iso boots with qemu-system-x86_64 -enable-kvm -m 512 -hda merged.iso to a boot loader menu with Debian logo. I did not go further with installation yet, mainly because i still lack ideas and experience how i would verify that the ISO's main repo knows 6788 *.deb files instead of 4753 in DVD-1 and 2035 in DVD-2. To ease usage and to avoid quadratic behavior (actually triangular), i plan to beef up the script so that it can merge more than 2 ISOs in one run. E.g. for downloading all 19 11.3.0 amd64 DVDs and merging them. (19 /dev/loop* should be no problem, i hope.) Zhang Boyang wrote: > I came up with a idea. Maybe you can use 'debian-cd' to create a DLBD set, > say disc A1 and A2, then create another ALL-IN-ONE set, say disc B. Then > compare if A1+A2==B. There might be small differences, like the package > order in Packages.gz, but I think if the overall format is OK, then it will > be OK. I fear that several of the tasks in debian-cd are beyond my talents. (I am about the third worst sysadmin which i know of.) In the end it is about whether the merged ISO works fully or not. Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, i wrote: > > I began to ponder about a shortcut in libisofs which would trust the > > checksum file (-checksum-list , -md5-list) enough to omit the reading of > > all the package files' content. Steve McIntyre wrote: > Cute idea (grin!), but it's a non-started - we wouldn't be able to > generate the various checksums for the whole image. Yeah. Ain't it sad ? > > Would more than one empty line between paragraphs damage the readability ? > Not sure, to be honest. So i will have to let my script think more. (It's at 287 lines, meanwhile.) > Might just work, yeah! My script does meanwhile: - Take as arguments the paths of boot_iso boot_mount_directory add_iso add_mount_directory result_iso - Mount the bootable ISO (e.g. debian-11.3.0-amd64-DLBD-1.iso) and the add-on ISO (e.g. debian-11.3.0-amd64-DLBD-2.iso). - Determine the release name $dist of the bootable ISO (e.g. "bullseye"). - Copy /dists and /md5sum.txt from bootable ISO to local working directory. - Merge the /md5sum.txt of both ISOs. - Read the paths from first checksummed file lists of both /dists/$dist/Release files, sort and uniq them. (I don't bet that both have the same list, although DVD-1 and DVD-2 have the same.) - Merge the namesake .gz files from that list (if both exist) or copy .gz file from the add-on ISO (if not yet in bootable ISO). - Create a new file /dists/$dist/Release with the header lines from the file in the bootable ISO and freshly computed checksums of the merged files. - Produce the bootable result_iso, overwriting /dists and /md5sum.txt by the altered copies from the local directory. This can be repeated by using result_iso as new boot_iso and another add_iso and a new result_iso name. How can i verify that the resulting ISO properly announces all its packages ? (If i install it to a VM, what must i do to challenge its completeness ?) Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, Steve McIntyre wrote: > youre initial guess is correct. We don't generate the .iso files > at all for the larger images [1]. This means we also don't have > torrent files for them [2]. I began to ponder about a shortcut in libisofs which would trust the checksum file (-checksum-list , -md5-list) enough to omit the reading of all the package files' content. Size, ownership, permissions, et.al would still be taken from the package files on disk. No ISO image would emerge (because of no valid file content) but .jigdo and .template would be created. Probably libjte would need an API extension so that it knows that only the count parameter of a libjte_show_data_chunk() call is valid. Vice versa libisofs would have to ask libjte whether a particular file is covered by the checksum list. All tricky and probably not worth the risk of embarassing failure. So back to my idea of merging ISOs: > The debian-cd code in tools/make_disc_trees.pl is not documentation > **as such**, but it's exactly how we create disc trees: I am using it now for checking detail questions. > It's baiscally just making a self-contained apt repository on each medium. So https://wiki.debian.org/DebianRepository/Format looks like the specs to follow. Question (to everybody): The description of Packages[.gz] files talks of "paragraphs" but does not exactly define a paragraph's end delimiter. From Packages.gz in the ISO i'd guess it is an empty line or the "Package" field of the next paragraph. Is an empty line needed between paragraphs ? Would more than one empty line between paragraphs damage the readability ? Reason: I want to merge the Packages.gz files like (gunzip temp_file but am not sure that the found Packages.gz will always end by an empty line. So i could simply insert an echo between the gunzips, or simply trust that the empty line is not needed as separator, or begin to think ... Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, Zhang Boyang wrote: > I guess there is no ready tool can merge two ISOs together. Not directly. But xorriso can load the meta-data of an ISO, manipulate that loaded model, and write the result as new ISO with the same boot equipment as found in the loaded ISO. As said, that would be no problem. Only enough disk space is needed, of course, and a Debian version >= 9 or locally built GNU xorriso >= 1.4.2. > After a quick look on dists/, > I think these files are in same format with theses on online mirrors, so > referring apt's documentation might be useful. Searching "Debian Packages.gz" brought me to https://wiki.debian.org/DebianRepository/Format which gives some hope for enlightenment. I will try to merge DVD-1 and DVD-2 but then need a way to verify that the result is fully functional. It looks like i have to merge these files from DVD-1: ./dists/bullseye/contrib/binary-amd64/Packages.gz ./dists/bullseye/main/binary-amd64/Packages.gz ./dists/bullseye/main/debian-installer/binary-amd64/Packages.gz ./dists/bullseye/contrib/binary-amd64/Release ./dists/bullseye/main/binary-amd64/Release ./dists/bullseye/main/debian-installer/binary-amd64/Release ./dists/bullseye/Release with these from DVD-2: ./bullseye/contrib/binary-amd64/Packages.gz ./bullseye/main/binary-amd64/Packages.gz ./bullseye/contrib/binary-amd64/Release ./bullseye/main/binary-amd64/Release ./bullseye/main/debian-installer/binary-amd64/Release ./bullseye/Release Bystanders: Do i miss something yet ? -- To demonstrate the rest of my sketch i tried this with DVD instead of DLBD (xorriso is GNU xorriso-1.5.2, equivalent to xorriso in Debian 11): DLBD_1=debian-11.2.0-amd64-DVD-1.iso DLDB_1_MOUNT=/mnt/dlbd_1 DLDB_2=debian-11.2.0-amd64-DVD-2.iso DLDB_2_MOUNT=/mnt/dlbd_2 RESULT=ALL_IN_ONE.iso sudo mkdir "$DLDB_1_MOUNT" "$DLDB_2_MOUNT" sudo mount "$DLBD_1" "$DLDB_1_MOUNT" cp -a "$DLDB_1_MOUNT/dists" merged_dists sudo umount "$DLDB_1_MOUNT" chmod -R u+w merged_dists sudo mount "$DLBD_2" "$DLDB_2_MOUNT" # TODO: # Merge the files in merged_dists with those from "$DLDB_2_MOUNT" xorriso -indev "$DLBD_1" \ -outdev "$RESULT" \ -map "$DLDB_2_MOUNT"/pool /pool \ -map merged_dists /dists \ -chown_r 0 /dists -- \ -chgrp_r 0 /dists -- \ -chmod_r a-w /dists -- \ -boot_image any replay \ -blank as_needed \ -stdio_sync off \ -padding included \ -compliance no_emul_toc sudo umount "$DLDB_2_MOUNT" xorriso -indev "$RESULT" -report_el_torito plain -report_system_area plain yields the typical boot jackalope (still escapes more coyotes than any other ISO partition layout): El Torito catalog : 4301 1 El Torito cat path : /isolinux/boot.cat El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA El Torito boot img : 1 BIOS y none 0x 0x00 45598 El Torito boot img : 2 UEFI y none 0x 0x00 51844302 El Torito img path : 1 /isolinux/isolinux.bin El Torito img opts : 1 boot-info-table isohybrid-suitable El Torito img path : 2 /boot/grub/efi.img System area options: 0x0202 System area summary: MBR isohybrid cyl-align-off GPT APM ISO image size/512 : 16754652 Partition offset : 0 MBR heads per cyl : 0 MBR secs per head : 0 MBR partition table: N Status TypeStart Blocks MBR partition : 1 0x80 0x000 16754652 MBR partition : 2 0x00 0xef17208 5184 MBR partition path : 2 /boot/grub/efi.img GPT: N Info GPT disk GUID : 657a86211710b54c8e25101781372d9f GPT entry array: 12 208 overlapping GPT lba range : 64 16754598 16754651 GPT partition name : 1 490053004f00480079006200720069006400 GPT partname local : 1 ISOHybrid GPT partition GUID : 1 657a86211710b54c8e24101781372d9f GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7 GPT partition flags: 1 0x1001 GPT start and size : 1 0 16754596 GPT partition name : 2 490053004f004800790062007200690064003100 GPT partname local : 2 ISOHybrid1 GPT partition GUID : 2 657a86211710b54c8e27101781372d9f GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7 GPT partition flags: 2 0x1001 GPT start and size : 2 17208 5184 GPT partition path : 2 /boot/grub/efi.img APM: N Info APM block size : 2048 APM gap fillers: 0 APM partition name : 1 EFI APM partition type : 1 Apple_HFS APM start and size : 1 4302 1296 APM partition path : 1 /boot/grub/efi.img For details of the xorriso run, see man xorriso. For the report format, see xorriso -report_el_torito help -report_system_area help | less Have a nice
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, i reply to 1011...@bugs.debian.org and zhangboyang...@gmail.com because of my question and sketch for a procedure to merge DLBD-1 and -2 after download into the desired ALL-IN-ONE ISO. Andrew M.A. Cater wrote to debian...@lists.debian.org: > If the original poster wants one huge .iso as one file to download from > cdimage.debian.org - then 2 x double layer Blu-Ray (say) as one file > would be 100GB or so. 11.3.0 is at (48245475328+35798159360)/1073741824 = 78.272 GiB. $ gunzip Even on a good quality link, that's quite a time. Yes. Especially since at least with jigdo-lite the bottleneck is the latency of downloading the packages. A 78 GiB ISO might need a day to become complete. jigdo-lite is graceful about being interrupted and resumed the other day, though. Running several jigdo-lite downloads simultaneously might mitigate the latency problem so that bandwidth becomes the bottleneck. > I don't actually know how long it take to create the .jigdo and .template > files - I don't think it's as long as generating the full .iso files They are created by libjte under control of libisofs under control of xorriso as side effect of actual ISO production under control of debian-cd. (I'm developer of libisofs and xorriso, and co-funder of libjte, which is now back in the hands of Steve McIntyre from whose genisoimage code it got large parts of its entrails.) I was possibly wrong with guessing that the ISO is dumped into /dev/null, although debian-cd seems to be smart enough to read the various checksums from the .jigdo files, rather than calculating them from the .iso files. The checksum code https://sources.debian.org/src/debian-cd/3.1.35/tools/imagesums/#L57 reminded me that there are .torrent files made, which obviously need the .iso files at least at build time. - > it's always good to have otehr people to think round a problem with. Is there documentation from which i could learn how the stuff in (i guess) /dists of DLBD-1 and DLBD-2 could be merged so that it properly describes a merged pool tree ? My rough idea would be: - mount both ISOs - derive merged /dists files - run xorriso to let it - load DLBD-1 with its boot equipment - merge-in the pool tree of mounted DLBD-2 - overwrite the old /dists files by the newly derived ones - automatically replay the commands for the loaded boot equipment - store the result as new .iso file The second step is where i would need info or advise. Have a nice day :) Thomas
Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?
Hi, some technical nitpicking. Andrew M.A. Cater wrote: > Also, keeping large files around on disk for a long > time - there's some likelihood of data corruption. The .jigdo and .template files of the DLBD ISOs are together smaller than a netinst CD ISO. (Less than 90 MiB, see https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dlbd/ ) Building an ALL-IN-ONE would cost the computing time of another ISO set (*) and 50 % percent more virtual memory than building DLBD1 (**). I assume that the main workload with an additional set of jigdoized ISOs is the need to once more shovel 75 GB of package files through libisofs and libjte so that they can create .template and .jigdo files. The ISOs themselves get directly piped into /dev/null, i guess. > I'd hate a couple of > bit flips three quarters of the way through a 6TB file, say, to mean that > the whole thing isuseless. The vast majority of data is stored as packages on the worldwide mirrors. I'd expect some quality of filesystem and backup which keeps damage confined to a few packages. There are a lot of outdated mirrors around where one could dig for those which really got lost from all active mirrors. I remember the hunt for a package which once was overwritten by a newer version with the same .deb file name. It was needed for building an old ISO from .jigdo which was created when the new version did not yet exist. But even without finding the older version, the emerging ISO would have been usable for all purposes which don't touch that one rogue package. Every time this wish pops up, i begin to ponder what is theoretically needed to unite several pool trees from multiple ISOs into one, so that it works like an official ALL-IN-ONE ISO. Putting all files together and making the ISO bootable would be no problem. But what does a neat pool have to offer as merged lists or other meta-data so that it properly announces its content ? Footnotes: (*) Computing time: Maybe it needs a bit more than DLBD[12] together, because the insertion algorithm of libisofs has a quadratic aspect in its personality. But the branchy pool tree helps a lot to keep this problem under the cover. (**) Virtual memory consumption: The memory consumption of libisofs mainly depends on the number of files and the length of their names. *11.3.0-amd64-DLBD-[12].jigdo together list ~ 59,000 files with ~ 2.2 MiB of basename length. My daily BD backups have ~ 70,000 files with ~ 1.4 MiB of basename length and do not demand gigabytes of RAM. Have a nice day :) Thomas
Bug#1010098: xorriso: please allow -cut-out directly from block devices
Hi, i decided to regard a device with 0 lseekable size as not suitable for -cut_out. The test for suitable type rejects S_ISDIR, S_ISLNK, S_ISFIFO, and S_ISSOCK. So if an operating system offers non-POSIX types, it will be possible to test whether they are elsewise suitable. Please give the code a thorough test, especially with weird -cut_out arguments. --- Committed are: https://dev.lovelyhq.com/libburnia/libisofs.git commit 011e2e85e6dfe6e5d882d198c29fd15c02d38e5e Author: Thomas Schmitt Date: Tue Apr 26 12:12:15 2022 +0200 Allowed lseekable device files with iso_tree_add_new_cut_out_node(). Proof-of-concept by Ivan Shmakov. commit f457a4f8b9a59906cde1b6577e7116944a37d2d0 Author: Thomas Schmitt Date: Tue Apr 26 12:06:18 2022 +0200 Added missing stream type names to a diagnostic function commit 2af17490a08c29b033942da42d27b5bda076ad05 Author: Thomas Schmitt Date: Tue Apr 26 12:03:53 2022 +0200 Bug fix: The lseek methods of IsoFileSource for local filesystem and loaded ISO returned libisofs error codes as positive off_t numbers https://dev.lovelyhq.com/libburnia/libisoburn.git commit fc587966d3c770be8b71e57e924cb5cbf7121f76 Author: Thomas Schmitt Date: Tue Apr 26 12:17:22 2022 +0200 Allowed lseekable device files with -cut_out. Proof-of-concept by Ivan Shmakov. --- Mentioning in man xorriso: === Map a byte interval of a regular disk file or of a device file into a regular file in the ISO image. The file depicted by disk_path has to support random read access. ... Another use case is copying the content of a device file as interval or as a whole into the emerging ISO filesystem. The fact that the byte_count is allowed to be unreasonably high enables copying of a whole device: -cut_out /dev/sdd3 0 1000g /content_of_sdd3 --- Behavior with the various file types and situations: === Intentional failures: --- S_IFIFO: xorriso : ERRFILE : /home/test/fifo xorriso : FAILURE : -cut_out: File type (name_pipe) is not suitable for this command: '/home/test/fifo' --- S_IFLNK: xorriso : ERRFILE : /home/test/link xorriso : FAILURE : -cut_out: File type (symbolic_link) is not suitable for this command: '/home/test/link' --- S_IFSOCK: xorriso : ERRFILE : /run/user/1000/pulse/native xorriso : FAILURE : -cut_out: File type (unix_socket) is not suitable for this command: '/run/user/1000/pulse/native' --- S_IFDIR: xorriso : ERRFILE : /run/user/1000/pulse xorriso : FAILURE : -cut_out: File type (directory) is not suitable for this command: '/run/user/1000/pulse' --- S_IFREG not large enough: xorriso : ERRFILE : /home/x xorriso : SORRY : -cut_out: Byte offset 32768 larger than addressable file size 1209 : '/home/x' --- S_IFCHR (tests are made on Linux): xorriso : FAILURE : -cut_out: Special file with addressable size range of 0 encountered xorriso : ERRFILE : /dev/zero xorriso : FAILURE : -cut_out: File (char_device) does not support random read access: '/dev/zero' --- S_IFBLK without read permission: xorriso : FAILURE : Determination of random-access readable capacity failed: '/dev/sdb' : Permission denied xorriso : ERRFILE : /dev/sdb xorriso : FAILURE : -cut_out: File (block_device) does not support random read access: '/dev/sdb' --- S_IFBLK not large enough: xorriso : ERRFILE : /dev/sdc xorriso : FAILURE : -cut_out: Byte offset 4294967296 larger than addressable file size 2004877312 : '/dev/sdc' === Successes: --- S_IFBLK of sufficient size: (only a message of severity DEBUG appears, like with the others too: xorriso : DEBUG : -cut_out from /dev/sdc , byte 1024 to 3072, and graft as /sdc ) --- S_IFREG of sufficient size: (only a message of severity DEBUG appears, like with the others too: xorriso : DEBUG : -cut_out from /home/x , byte 1024 to 3072, and graft as /x
Bug#1010098: xorriso: please allow -cut-out directly from block devices
Hi, the plan to allow nearly all file types with -cut_out is dwindeling: - lseek(SEEK_END) on S_IFCHR /dev/zero returns 0, not -1. - open() on S_IFIFO blocks (and i don't want to know what it will do in libisofs). IIRC a successful open() has side effects on the fifo. - S_IFSOCK fails properly with the lseek test. But in this case open() fails with errno 6 "No such device or address" although it exists as file /run/user/...userid../pulse/native. So i don't know what might happen with other sockets. So if we exclude S_IFSOCK, S_IFLNK, S_IFDIR, S_IFIFO there remain only S_IFREG, S_IFBLK, S_IFCHR with the latter on Linux behaving like S_IFREG with 0 bytes of content. (As said on FreeBSD it could be a lseekable disk device.) Now i ponder whether i shall count S_IFCHR with size 0 as non-seekable or really as empty file. And whether i shall have a negative list of four types or a positive list of three types. Further it turned out that the lseek test in libisofs falls victim to a bug in fs_local.c and fs_image.c which trick iso_file_source_lseek() into returning libisofs error codes as positive off_t numbers. A bug from the very early days of libisofs. I am still testing but looking forward to committing the changes tomorrow. Have a nice day :) Thomas
Bug#1010098: xorriso: please allow -cut-out directly from block devices
Hi, i wrote: > > The newest easily compilable and then usable version would be > > http://www.gnu.org/software/xorriso/xorriso-1.5.5.tar.gz Ivan Shmakov wrote: > Not under http://ftp.gnu.org/gnu/xorriso/ , though? 1.5.5 is a development snapshot with potentially later changing new features of the APIs of the library code and with potentially later changing new xorriso commands or command parameters. Distros should not distribute it. In GNU ftp it has no place because its content changes in the course of the development cycle between the latest release 1.5.4 und the future release 1.5.6. > I’d frankly be surprised to learn that a proper release, > such as 1.5.5, has features which a recent revision from > Git master branch lacks. The development snapshot gets composed by libisoburn/xorriso/make_xorriso_standalone.sh from my local copy of the repos when it seems ready for public testing. It can happen that the repos are ahead of the snapshot tarball. But as of today, the tarball is at the same state as the repos. (It was uploaded on 23 Apr 2022.) > I’ll be checking the master branch from time to time, This is ok, although the tarball is easier to compile and needs no installation of any freshly produced dynamic libraries because its library code gets linked statically. Whatever, better wait until i announce that it is time for testing. > The patch was intended, first and foremost, as a proof of concept; As such it appears to be very good. My only scruples so far are about touching fsrc_open() which serves for the main use cases of libisofs and thus would need intense testing of those cases. I am now testing a variation which puts the changes into the code of the cut_out_*() functions in libisofs/stream.c rather than faking the value of struct stat.st_size . This will hopefully keep all other use cases safe. The new use case of copying the whole content of a device will then be a special case of -cut_out exploiting the rule in man xorriso: "It is permissible to request a higher byte_count than available." E.g. -cut_out /dev/XYZ 0 1000g /ABC > In the implementation, I’d rather see it not testing the > file type specifically, aside of S_ISREG, S_ISDIR and S_ISLNK, > but rather whether lseek (, 0, SEEK_END) gives a sane result, Yes. I first was sceptical about allowing nearly all file types although on Linux only S_IFREG and S_IFBLK give hope for seekability. But on FreeBSD all devices files identify as S_IFCHR. (Equivalent to Linux block devices are those which reply success of ioctl(DIOCGMEDIASIZE). At least i got no complaints that libburn would misclassify BSD devices as random-access pseudo-drives when they are actually just sequentially writable.) So overall it seems best to let just lseek(2) decide what goes and what doesn't. Have a nice day :) Thomas
Bug#1010098: xorriso: please allow -cut-out directly from block devices
Hi, Ivan Shmakov wrote: > I’m filing this bug against versions from oldstable and > stable, for that’s so far the only Debian packages’ versions > I’ve tested for this issue. As it is about an upstream wish: The newest easily compilable and then usable version would be http://www.gnu.org/software/xorriso/xorriso-1.5.5.tar.gz It can be compiled without much dependencies and then used without installation. See "Compilation, First Glimpse, Installation" in https://www.gnu.org/software/xorriso/README_xorriso_devel Said that, and now with my upstream hat on, i expect no difference to versions 1.5.0 or 1.5.4 in respect to command -cut_out. So there is no need to get xorriso-1.5.5.tar.gz unless you want to test a patch proposed by me. I downloaded your 114.xorriso-1.diff and will study it for the goal of enabling -cut_out for more file types. I already see that it makes changes about fsrc_open(), which is a central facility of libisofs. This means it will last several days of study and pondering until i would be convinced of any change. Also i need to check whether your diff interferes with the ability to backup a block device file as itself and not as bearer of its content: rm test.iso xorriso -outdev test.iso -map /dev/sr0 /sr0 -commit -lsdl /sr0 -- should produce even with a loaded BD_RE medium of 23 GiB a small ISO and report a block device file in it brw-rw1 024 11,0 Apr 24 14:15 '/sr0' So i guess that other changes are necessary to keep this capability while enabling -cut_out for block devices. I will report my findings. Have a nice day :) Thomas
Bug#1009803: xorriso : NOTE : -as mkisofs: Ignored option '-cache-inodes'
Hi, theoretically it should indeed be ok to remove this option. Background: The man page of genisoimage talks of "hard-linked [file] on the CD" (which obviously should rather be "in the ISO 9660 filesystem"). ISO 9660 (aka ECMA-119) does not specify hard links but rather permits to share data content (extents) among files. Technically this resembles hard links. This relation can be emphasized to the mounting operating by storing the same File Serial Number in the Rock Ridge PX entries of the hardlinked files. But Linux computes the inode numbers of mounted ISO 9660 from the byte addresses of the files' directory records divided by 32. So two different file names will always get attributed different inode numbers even if they share their file data blocks and the File Serial Number. libisofs does this unifying of file content unconditionally, if the dev_t and inode_t numbers are identical. (If it ever encounters an input filesystem with non-unique numbers for differing file content, then it will probably fail.) This should not be confused with an opportunity which libisofs offers to record real hardlink relations of the input files. These relations can be re-created when xorriso extracts files from ISO 9660 to the local filesystem. Linux ignores this extra info, which is stored in SUSP entries of my self-invented AAIP protocol beneath the Rock Ridge data. (xorriso command -hardlinks, xorrisofs option --hardlinks.) Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi,
Felix Zielcke wrote:
> > I just tried to test it with TZ set.
> > But I always get the same binaries. No matter to what I change TZ or
> > /etc/timezone between the builds.
Vagrant Cascadian wrote:
> I can reproduce the issue in a debian sid chroot with pcmemtest 1.5-3
> from debian
I don't have newest Debian in reach today and am too lazy to set up
pcmemtest package building anyways. :o)
So i rather tested near to the bottom, i.e. mcopy(1) and localtime(3).
Both react on TZ here.
If Felix Zielcke and Vagrant Cascadian get the same result, then i
wonder why TZ does not have an influence in the Makefile of pcmemtest.
---
Test of mcopy(1):
Using rather outdated libc (i.e. localtime(3)) and rather young locally
compiled mtools-4.0.37 (source of january 2022):
/sbin/mkdosfs -i 12345678 -n MEMTEST-ESP -F12 -C fat.img 1024
./mcopy -s -i fat.img x ::/x
TZ=UTF ./mcopy -s -i fat.img x ::/y
TZ=UTF+3 ./mcopy -s -i fat.img x ::/z
This yields after mounting fat.img :
-rwxr-xr-x 1 root root 9931 Mar 26 09:29 x
-rwxr-xr-x 1 root root 9931 Mar 26 08:29 y
-rwxr-xr-x 1 root root 9931 Mar 26 05:29 z
which is somewhat wrong because my local time (CET = UTF-1) was
$ date
Sat Mar 26 10:29:49 CET 2022
mdir from mtools does it better (but other than ls -l it does not
react on TZ):
$ mdir -i fat.img
...
x 9931 2022-03-26 10:29
y 9931 2022-03-26 9:29
z 9931 2022-03-26 6:29
Whatever, the mcopy command reacts on TZ.
---
Test of localtime(3):
In order to directly check the influence of TZ on localtime(3), which
does the time format conversion in mtools--4.0.37/directory.c function
mk_entry(), i wrote a test program:
--- File start
/*
cc -g -o localtime_test localtime_test.c
*/
#include
#include
int main()
{
time_t now;
struct tm *res;
now= time(NULL);
res= localtime();
printf("%d/%2.2d/%2.2d %2.2d:%2.2d:%2.2d (%d)\n",
1900 + res->tm_year, 1 + res->tm_mon, res->tm_mday,
res->tm_hour, res->tm_min, res->tm_sec,
res->tm_isdst);
return(0);
}
--- File end
After
cc -g -o localtime_test localtime_test.c
i get plausible results:
$ ./localtime_test
2022/03/26 10:43:05 (0)
$ TZ=UTC ./localtime_test
2022/03/26 09:43:14 (0)
$ TZ=UTC+3 ./localtime_test
2022/03/26 06:43:18 (0)
---
Have a nice day :)
Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Chris Lamb wrote: > More generally, I can't think of a scenario where the value of > SOURCE_DATE_EPOCH request a timezone implication, Alain Knaff mentioned mcopy option -m "Preserve the file modification time." which he wants to fulfill the user's expectation that the Y/M/D/h/m/s formatted timestamps show local time. (That's a similar problem as i perceived in xorriso with your wish to make the effects of SOURCE_DATE_EPOCH immutable in xorriso, although a design principle of xorriso is that any setting can be set and revoked by a xorriso command. My solution was to let SOURCE_DATE_EPOCH set the defaults of the various involved settings but to allow the commands to override those defaults, if really desired.) I plan to write a mail to rb-gene...@lists.reproducible-builds.org about my difficulties to justify the demand for a fixed time zone from the specs of https://reproducible-builds.org/specs/source-date-epoch/ I did not find a better motivation than by the specification "Formatting MUST be deferred until runtime if an end user should observe the value in their own locale or timezone." from which i argue: * FAT needs formatting of SOURCE_DATE_EPOCH. * One cannot do this at "runtime" of the FAT filesystem, which in the special case of pcmtest is when EFI reads the filesystem to find and run the \EFI\BOOT\BOOT*.EFI start program. => So one must not let the end user see their own locale or timezone. Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, i wrote to info-mtools about the problem that time zone influences the result of mcopy even if SOURCE_TIME_EPOCH is set. Alain Knaff then proposed to set the TZ variable to UTF when pcmemtest gets built. https://lists.gnu.org/archive/html/info-mtools/2022-02/msg2.html Would that be viable in pcmemtest ? Are any problems to foresee if TZ is set inside build*/Makefile ? Fritz Zielke: Can you give it a try in your patch ? Vagrant Cascadian: Can you then test whether this helped ? Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, my favorite suspect in mtools is to see in function mk_entry(): https://sources.debian.org/src/mtools/4.0.33-1+really4.0.32-1/directory.c/?hl=98#L107 now = localtime(); (I see no debian/patches directory for mtools. So this is probably really in effect with Debian's package.) I understand that this function gets the SOURCE_DATE_EPOCH value as time_t date2 = date; but then applies the local timezone when generating the timestamp components. From man localtime: The localtime() function converts the calendar time timep to broken- down time representation, expressed relative to the user's specified timezone. Unix time functions always confused me. So my idea of using gmtime(3) instead of localtime(3) has to be taken with a grain of salt. (Further it might be necessary to use gmtime(3) only when SOURCE_DATE_EPOCH is defined and not in normal operation mode.) Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Vagrant Cascadian wrote: > I haven't looked into it too deeply, but got the basically the same > results as currently showing on: > > > https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/pcmemtest.html Looks again like the FAT directory entries. The byte address is sufficiently above the 1.4 MiB of ISO 9660 payload to be in partition 2. We see the directory names "BOOT" and "EFI", and the file name "BOOTX64" with extension "EFI". The differences are at offsets 0x0f, 0x10, 0x12, 0x17, 0x18 after the name start bytes. According to https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#Directory_entry that would be timestamp fields. 0x0f : belongs to the 16 bit of "Create time" beginning at offset 0x0e. Values as little endian: 0x9a91 <-> 0xaa91 Hour:Minute:Second 19:20:34 <-> 21:20:34 0x10 : 16 bit of "Create date". Values as little endian: 0x544c <-> 0x544d Year/Month/Day 2022/02/12 <-> 2022/02/13 0x12 : 16 bit of "Last access date". 2022/02/12 <-> 2022/02/13 0x17 : 16 bit of "Last modified time" beginning at 0x16. 19:20:34 <-> 21:20:34 0x18 : 16 bit of "Last modified date". 2022/02/12 <-> 2022/02/13 So for some reasons the timestamps of files do not stay stable. Since the timestamps of the filesystem label (a pseudo file name) staid stable, i would now rather suspect mcopy and not mkdosfs as culprit. - So the task seems to be to find out how to force timestamps with this line from build*/Makefile: mcopy -s -i esp.img iso/EFI :: Google brings me to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837044 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900410 where Chris Lamb fought such effects and let mtools obey SOURCE_DATE_EPOCH. Was this change not enough ? Does tests.reproducible-builds.org use an older version of mcopy ? Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Vagrant Cascadian wrote: > I noticed 1.5-3 is still affected by timezone differences, Can you already say whether the differences are in the range of partition 1 or in the range of partition 2 of the .iso (as told by fdisk -l) ? > If I come up with an updated patch, I'll maybe submit a new bug... Please Cc me then. (At least i'm curious, if not being the culprit.) Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Chris Lamb wrote: > Did you try the --invariant flag? Well, i step aside and let this question reach Felix. :)) https://manpages.debian.org/unstable/dosfstools/mkdosfs.8.en.html says that this is what we need if my theory about the second difference is correct: --invariant Use constants for normally randomly generated or time based data such as volume ID and creation time. Multiple runs of mkfs.fat on the same device create identical results with this option. So the options to be added to the mkdosfs runs in the hope for a fully reproducible ISO are: -i 12345678 --invariant If a more more individual -i number is desired, one could use something like -i $(printf '%8.8x' "$SOURCE_DATE_EPOCH" | head -c 8) Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, First an observation from digging in pcmemtest/1.5-2/build64/Makefile: What about target grub-memtest.iso ? It has its own xorriso run and EFI System Partition image. I guess they need treatment for reproducibility, too. Felix Zielcke wrote: > Here's the output for the 64bit ISO: > [...] > │ │ │ 00179000: eb3c 906d 6b66 732e 6661 7400 0204 0100 .<.mkfs.fat. > │ │ │ 00179010: 0200 0200 20f8 0600 2000 0200 ... ... > │ │ │ -00179020: 8000 296c 13ae 4d4d 454d 5445 ..)l..MMEMTE > │ │ │ +00179020: 8000 29d6 8629 594d 454d 5445 ..)..)YMEMTE > │ │ │ 00179030: 5354 2d45 5350 4641 5431 3220 2020 0e1f ST-ESPFAT12 .. > [...] > │ │ │ -0017aa00: 4d45 4d54 4553 542d 4553 5008 a750 MEMTEST-ESPP > │ │ │ -0017aa10: 4b54 4b54 a750 4b54 KTKT...PKT.. > │ │ │ +0017aa00: 4d45 4d54 4553 542d 4553 5008 0951 MEMTEST-ESPQ > │ │ │ +0017aa10: 4b54 4b54 0951 4b54 KTKT...QKT.. > │ │ │ 0017aa20: 4546 4920 2020 2020 2020 2010 186a EFI j The clear text looks like EFI System Partition (ESP). The byte address 0x00179000 = 1,544,192 would roughly match the end of an ISO 9660 filesystem which only has the 1440 KiB of /boot/floppy.img as payload. So it would be roughly at the start of the partition in the ISO created by -append_partition 2 0xef ./esp.img A partition editor listing like /sbin/fdisk -l memtestx64.iso or a run of xorriso -indev memtestx64.iso -report_system_area plain would tell where partition 2 begins and whether the byte addresses are located in there. Assumed that the differences are indeed inside partition 2: The content of the appended partition image does not get altered by xorriso. So i would expect that differences stem from the creation of ./esp.img. Do i get it right from https://sources.debian.org/src/pcmemtest/1.5-2/build64/Makefile/#L121 that esp.img stems from /sbin/mkdosfs -n MEMTEST-ESP -F12 -C esp.img 4096 mcopy -s -i esp.img iso/EFI :: ? Assumed yes: We see in the first difference the string "MEMTEST-ESPFAT12" which would match the "Partition Volume Label" and "File system type" fields as of https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#Extended_BIOS_Parameter_Block The differing byte would then be a part of "Volume ID (serial number)". In man mkdosfs i read -i volume-id Sets the volume ID of the newly created filesystem; volume-id is a 32-bit hexadecimal number (for example, 2e24ec82). The default is a number which depends on the filesystem creation time. So consider to add an option -i to the mkdosfs run and give it either a constant number or a number which gets reproducibly derived from variable SOURCE_DATE_EPOCH. -- The other difference could be a neighbor of "directory volume label", which Wikipedia mentions as a kind of mirror of "Partition Volume Label". If i get it right then this is in the name and extension fields of a "Directory entry". The differing bytes would then be the timestamps "Create time" at offset 0x0E and "Last Modified time" at offset 0x16. Probably the date fields at 0x10 and 0x18 could show differences, too. But i am far off my usual playground now. Maybe Chris already knows a good trick to enforce values for those time and date fields in FAT filesystems. Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Felix Zielcke wrote: > I modifed the patch now to use --set_all_file_dates. I wonder whether the "file list" in "data.tar.xz" of https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffosc ope-results/pcmemtest.html is made from the files in the ISO or from the input files on hard disk. In the latter case, it would be better to use the solution of Chris which enforces dates on hard disk. But then this solution should also enforce user id and group id numbers on hard disk. > diffoscope still tells me some differences in the ISOs. Is there a chance to quote some of the found differences here ? If it's about meta data of the ISO then i would be the person to decode them and to make first theories from where the deviations might come. Have a nice day :) Thomas
Bug#1005197: pcmemtest: please make the build partly reproducible
Hi, Chris Lamb wrote: > Have you had a look at the diffoscope output after this > patch is applied? Where can a curious bystander get that look ? I only found a diff from february 8: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/pcmemtest.html where i see in the isoinfo output only differences of user id, group id, and timestamps of /boot and /boot/floppy.img. The patch in the original message of this bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1005197;filename=pcmemtest.diff.txt;msg=5 should fix both. > Perhaps looking through > the xorriso manual page would be helpful, as there are a number of header > checksums in such files. If SOURCE_DATE_EPOCH is set then all automaticly generated ids in the ISO meta data should be reproducibly derived from that time value. If all input is the same, then all checksums should be the same. If not, it would be a bug in xorriso. (I just tested a mock-up of the xorrisofs run in pcmemtest.diff.txt. Both resulting ISOs were identical.) -- Side note: The gesture find iso -print0 | xargs -0r touch --date="@$(SOURCE_DATE_EPOCH)" could be replaced by xorrisofs option --set_all_file_dates "=$SOURCE_DATE_EPOCH" which acts on the files in the emerging ISO filesystem rather than on the input files on hard disk. (The '=' is xorriso's equivalent of date's '@'. See man xorriso, command -alter_date .) Have a nice day :) Thomas
Bug#999906: After upgrade to Debian Bullseye, I am not able to burn DVDs.
Hi, > I am not such a purist, so I really appreciate the comfort of a desktop like > KDE or LXDE in this case. It's not so much about purism in my personal case, but rather about the unwillingness to repeatedly learn how to do the same things in frequently (*) changing ways. (*) Once every ten years is too much for my ageing brain. > So is there anything I can do to help? Not before somebody shows up and asks for tests or experiments. I don't see suspicious upstream commits at https://github.com/KDE/k3b/commits/master and multiple pages until version 18.08.1 which is in Debian 10. In Debian 11 the K3B version is 20.12.2. The Debian 11 changelog https://tracker.debian.org/media/packages/k/k3b/changelog-20.12.2-1 gives no hint that the Debian maintainers changed something related to burn speed or burning in general. The patches in https://sources.debian.org/src/k3b/20.12.2-1/debian/patches/ don't look suspicious either. So i am out of ideas about how to find the trigger for your problems in Debian 11. > PS: I am on vacation till December 14th. Stay safe. Best in a plague-free hut in a remote mountain valley ... Have a nice day :) Thomas
Bug#999906: After upgrade to Debian Bullseye, I am not able to burn DVDs.
Hi,
w_t...@t-online.de wrote:
> When I click the tiny icon beside "DVD-R sequential recording",
> the former grayed menus for speed and writing mode are now available.
> Then I can also
> start the writing process, and it does not go to max speed and completes
> successfully.
So - with the necessary black magic - it is possible to use Xfburn
for your purpose with your drive.
Insofar this is a success. {:)
> This is all very confusing.
I agree. Desktops like XFCE, GNOME, or KDE and their applications bring
my blood pressure up to unhealthy levels every time i have use them.
(As a good classic Linuxer i use fvwm2 as window manager with a config
file that essentially stems from a 20 year old SuSE installation.)
> I have no clue what's going on here, and I am so
> sorry for bothering you with that.
No need to apologize. If i would not be interested i could just have
staid away from this bug report.
Given the lack of any active GUI developers for optical disc burning,
i have to check out the user problems with those programs in order to
distinguish their own problems from potential problems in libburn or in
my command line programs.
Any normal desktop user is more qualified than me to operate Xfburn,
Brasero, or K3B. So i cannot help with talking one of them into doing
what the user wants. At best i can lookup error messages in their code
and follow the traces to a burner problem for which i have knowledge.
But especially with K3B's C++ spaghetti code i have big difficulties
to understand. Program execution tends to vanish in a fog of class
inheritance and function overloading. Often it is hard to find the code
part which does the actual work of talking to the drive or to the burn
program.
I will next dig out the instructions how to attribute this bug report
to the Debian package K3B, because you report a reproducible K3B problem
with the upgrade from Debian 10 to Debian 11.
(I don't hold a Debian rank. But any amateur is allowed to operate the
bug tracking system by mail messages. See:
https://www.debian.org/Bugs/server-control
)
Have a nice day :)
Thomas
Bug#999906: After upgrade to Debian Bullseye, I am not able to burn DVDs.
Hi, > I tested at first the live CD of Debian 10. > K3B also disrespects the speed setting. But the process seems to > adjust the speed automatically. 18x was starting speed and it was > reduced, so that the resulting speed was 4x. An initial utopic speed is not necessarily the fault of the burn program. The drive's buffer accepts data with high speed until it is full, so only after 2 or 4 MB the observed speed drops to the real speed of burning. DVD-R usually get burned with constant angular velocity (rounds per minute), so that the linear speed reaches the desired speed only at the end of burning. > K3B under Debian 11 still behaves as before. > Setting speed is totally ignored, the burner makes noise like a starting > jet, and finishes with error and overall speed of 18,4x. I really wonder what might have changed in K3B. You could try with Xfburn again. Its source code looks like it would ask libburn to set a speed. Regrettably i find no hints in the web how to increase Xfburn's verbosity so that you get to see the debug message Set speed to ... kb/s which is to see immediately before the call to burn_drive_set_speed(). See: https://sources.debian.org/src/xfburn/0.6.2-1/xfburn/xfburn-burn-audio-cd-composition-dialog.c/?hl=299#L299 > Unfortunately, I am not able to reproduce the error again with xorrecord. I count this as a piece of luck for me. :)) It is not normal that a drive produces bad results if it is asked for high speed or if it is not asked for low speed. So whatever K3B or growisofs (*) do wrong, your drive did its share to cause the failure. (*) It might be that K3B uses cdrskin instead of growisofs. But i have no idea how to find out which of both is in charge for DVD. (cdrskin is the cdrecord compatibility wrapper of libburn.) Have a nice day :) Thomas
Bug#999906: After upgrade to Debian Bullseye, I am not able to burn DVDs.
Hi, i am the developer of libburn, which serves underneath xfburn and xorriso. (No need to Cc: me, as i have now subscribed to this bug.) -- Quite obviously this report is attributed to the wrong package. But it is not obvious which one would actually be the right one. The overall Debian system and the Linux kernel have not much stake in speed setting of DVD burners or burn success. To blame would rather be the backends, growisofs under K3B and libburn under the others. growisofs did not change in a decade. The jump from libburn-1.5.0 in Debian 10 to libburn-1.5.2 in Debian 11 isn't much suspicious either. Other distros use it since more than a year earlier. Please show your xorriso command line and the messages which you get at the end of the run. (No need to show the many progress messages prefixed by UPDATE. Everything else might be of interest. Most the very start and the very end of the run.) If you can afford experiments with rebooting, then it would be interesting to see whether you get better results from a Debian 10 Live ISO. E.g. pick your favorite desktop version from: https://cdimage.debian.org/mirror/cdimage/archive/10.11.0-live/amd64/iso-hybrid/ put it onto a USB stick: https://www.debian.org/CD/faq/#write-usb and after booting install xorriso like sudo apt-get update sudo apt-get install xorriso (Disclaimer: I did not test whether the 10.11.0 ISOs have apt-get.) Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, i wrote: > > My only idea which does not need a code change is to employ the wodim > > plugin of Brasero and to hope that it burns by write type SAO. Mauro Sacchetto wrote: > Which way? Good question. I have wodim installed. But last time when i tried, the wodim item was greyed out in the plugin list window which i somehow managed to pop up. Google brings me to https://github.com/lmedinas/brasero which in "Notes on plugins for advanced users" proposes to remove libburn (bleh !) or to use something named Gconf for changing plugin priorities. Wikipedia says: "GConf was a system used by the GNOME desktop". The riddle remains, i fear. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, Mauro Sacchetto wrote: > sg_raw /dev/sr0 be 00 ff ff ff ff 00 00 01 10 00 00 --outfile=tdb_data.bin > After this, the drive is unavailable Well, then we have found: - The immediate cause of the problem: READ CD with LBA -1. - A good suspect in Brasero: Function brasero_medium_track_written_SAO() https://sources.debian.org/src/brasero/3.12.3-1/libbrasero-media/brasero-medium.c/?hl=1565#L1565 - The generally problematic expectation of Brasero that it can determine whether a CD track was written by write type SAO (and not by TAO). - We have not explored yet: - Why does Brasero believe to see 2 tracks on a closed CD-RW with a single TAO track ? BraseroMedia: (at brasero-medium.c:2062) 2 track(s) found This misperception triggers the call to brasero_medium_track_written_SAO(). - Why does brasero_medium_track_get_info() want to distinguish SAO from TAO only if its parameter multisession is set to "true". Its only caller has a strange comment before setting multisession: https://sources.debian.org/src/brasero/3.12.3-1/libbrasero-media/brasero-medium.c/?#L2052 num = (size - sizeof (BraseroScsiFormattedTocData)) / sizeof (BraseroScsiTocDesc); /* remove 1 for leadout */ multisession = !(priv->info & BRASERO_MEDIUM_BLANK) && num > 0; If the leadout is counted as surplus track, then why not "num > 1" ? - So the Brasero "Copy" task of TAO CDs is not possible with the ASUS drive. The code of Brasero would have to be changed for that. We have no workaround yet for the case of burning the netinst ISO by Brasero. I found no way to tell Brasero that its libburn plugin shall ask libburn for SAO. libburn would of course be able to avoid SAO if it is told to do so. It would propose SAO if being asked to find a suitable write type for track input of which the size is known in advance. My only idea which does not need a code change is to employ the wodim plugin of Brasero and to hope that it burns by write type SAO. If so, then the drive would not get lost after a successful burn run. Of course you would have to carefully avoid to show Brasero a TAO CD in the ASUS drive. Best would be to blank any CD-RW before giving it to Brasero. - Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, i made the experiments which i mentioned in the previous mail: - The Pioneer drive does not go mad from READ CD for LBA -1 or from READ CD MSF for sector 00:01:74. - The ASUS drive does not go mad from READ CD MSF for sector 00:01:74. - The TSSTcorp drive tolerates READ CD MSF for sector 00:01:74 without a message in dmesg about USB reset. (It did that reset with READ CD for LBA -1 but showed no lasting problem.) None of the drives reports success with the READ CD MSF command. So with this command the Brasero function brasero_medium_track_written_SAO() would return TRUE, which is wrong with the given CD. The desire to read the Lead-in sector before the start of the payload data simply seems not fulfillable here for the first track of a CD. Brasero will have to abandon this idea or come up with a better read command - which i currently fail to imagine. - The SCSI specs say that LBA -1 is bad: While exploring how to encode the READ CD MFS command for 00:01:74, i found this statement in the MMC-5 specs: "3.2.21 Method 1 Addressing [...] Method 1 logical sector numbering is not defined for sectors outside of the program area." This sector numbering is in effect with command READ CD. The "program area" is the payload part of the CD with a single track. So READ CD with logical block address -1 is not defined. - Experiments with the ASUS drive: (Anybody remembers BCD number encoding ? I can be glad that they did not choose to use Gray codes. BCD is easy to write in hex. :)) The READ CD MSF expects an inclusive start address (here 00:01:74) and an exclusive end address (here 00:02:00). Except the address fields it has the same byte layout as READ CD. So the command as replacement for the bad READ CD command be 00 ff ff ff ff 00 00 01 10 00 00 is b9 00 00 00 01 74 00 02 00 10 00 00 This does not yield a good result in form a of data, but also no bad consequences for the drive or complaints from the transport layer in dmesg. $ sg_raw /dev/sr0 b9 00 00 00 01 74 00 02 00 10 00 00 --request=2352 --outfile=tdb_data.bin SCSI Status: Check Condition Sense Information: Fixed format, current; Sense key: Illegal Request Additional sense: Invalid field in cdb Error 5 occurred, no data received (I meanwhile learned that i need to tell sg_raw to expect a certain number of bytes as reply. My choice 2352 is the size of CD-DA payload. The data CD is supposed to deliver only 2048 bytes of payload. I tried a request size of 2048, but the behavior of sg_raw did not change.) In order to prove that my SCSI command is correctly encoded, i asked for the first valid payload block at 00:02:00 = LBA 0 : $ sg_raw /dev/sr0 b9 00 00 00 02 00 00 02 01 10 00 00 --request=2352 --outfile=tdb_data.bin SCSI Status: Good Writing 2352 bytes of data to tdb_data.bin The file tdb_data.bin shows the MBR data of the Debian netinst ISO. The same result i get with a valid READ CD command for LBA 0: $ sg_raw /dev/sr0 be 00 00 00 00 00 00 00 01 10 00 00 --request=2352 --outfile=tdb_data.bin None of these experiments caused problems like with Brasero. - What does PIONEER BD-RW BDR-S09 say to such commands: $ sg_raw /dev/sr1 b9 00 00 00 01 74 00 02 00 10 00 00 --request=2352 --outfile=tdb_data.bin SCSI Status: Check Condition Sense Information: Fixed format, current; Sense key: Illegal Request Additional sense: Invalid field in cdb Error 5 occurred, no data received The drive is still usable and dmesg shows no new complaints. (Except blkid's graceful failures to read the Run-out blocks.) With CDB b9 00 00 00 02 00 00 02 01 10 00 00 i get the first data block from the CD: the isohybrid MBR, which xorriso inserted when it created the ISO for the debian-cd project. Now for the ugly test: $ sg_raw /dev/sr1 be 00 ff ff ff ff 00 00 01 10 00 00 --request=2352 --outfile=tdb_data.bin SCSI Status: Check Condition Sense Information: Fixed format, current; Sense key: Illegal Request Additional sense: Logical block address out of range Error 22 occurred, no data received It turns out that this drive and its transport layers in the kernel can stand the bad READ CD command. No dmesg complaints to see from this experiment. xorriso still works with the drive. This quite kills the theory that the kernel transport layer gets a hickup without the help of the drive's SATA interface. So the ASUS drive indeed has a stake in the problem. - And the TSSTcorp CDDVDW SH-S223B : $ sg_raw /dev/sr2 b9 00 00 00 01 74 00 02 00 10 00 00 --request=2352 --outfile=tdb_data.bin SCSI Status: Check Condition Sense Information: Fixed format, current; Sense key: Illegal Request Additional sense:
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, Mauro Sacchetto wrote: > samiel@darkstar:~$ sg_raw /dev/sr0 be 00 ff ff ff ff 00 00 01 10 00 00 > --outfile=tdb_data.bin > SCSI Status: Check Condition > > Sense Information: > Fixed format, current; Sense key: Illegal Request > Additional sense: Unaligned write command Now we know from where that error came, which is inappropriate for optical drives. (Maybe that the kernel created it, and not the drive.) > Is it my test right and enough? The main question is: Is the drive unusable afterwards ? (I.e. do i have identified the culprit for your drive problem ?) > In fact I am a little surprised (although I don't know how these things are > handled by the Debin team) > that so far there has been no intervention by the packagers of Brasero I understand that there is no upstream developer any more and no community which would continue the GUI part and build it with patch proposals. It looks like i was involved in the second youngest thread on the mailing list, back in 2019: https://mail.gnome.org/archives/brasero-list/2019-January/thread.html The youngest thread is a request for help with building Brasero. It got no answer. Possibly i will post a report about the problem there. Just in case some new developer shows up. Said that, and contrary to my statement in the previous mail, i have not yet used up my ideas for experiments: - Will the Pioneer drive go bad if i send the READ CD 0x command ? It looks like Brasero doesn't do this. - Will READ CD MSF 00:01:74 work better ? Maybe with all three drives without incident and dmesg complaint ? (I have to study the specs, because i never used that command before.) If the Pioneer drive goes bad too, then it might even be a kernel problem with that obviously disliked sector address, and not a problem only with the ASUS drive. This could become obvious if the Additional sense: Unaligned write command from your above experiment changes to my result >>> transport error: Host_status=0x03 [DID_TIME_OUT] if you use kernel 5.10. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi,
i now have clear evidence from a run of
sg_raw /dev/sr0 be 00 ff ff ff ff 00 00 01 10 00 00 --outfile=tdb_data.bin
that the read attempt for sector 0x (= -1) brings the
ASUS DRW-24D5MT into the unusable state.
@ Mauro Sacchetto:
Could you please try to reproduce this finding after installing sg3-utils.
The main suspect for emitting SCSI this command is Brasero function
brasero_medium_track_written_SAO()
in libbrasero-media/brasero-medium.c .
Code study of that function and the logs of Brasero "Copy" runs with
the Pioneer and the TSSTcorp show that bad things happen if the function
is called:
- The Pioneer drive does not trigger the log message from
brasero_medium_track_written_SAO()
and leaves no problem report lines in dmesg.
So i conclude that the function was not called.
- The TSSTcorp drive triggers this message:
BraseroMedia: (at brasero-medium.c:1573) Checking for TDBs in track pregap.
and dmesg reports a (successful / harmless ?) USB reset.
-
Possible remedies:
We would need a Brasero developer to find a workaround, because not
only the bad READ CD command needs to be avoided, but also a replacement
for the code gesture to distinguish TAO CDs from SAO CDs would have to
be developed.
Maybe this distinction is not really necessary. It seems not to work
properly on the TSSTcorp drive by making a false correction of the
track size. But that would be up to a Brasero developer to decide.
A cheaper workaround would be to identify the drive model from the
function parameter
BraseroDeviceHandle *handle
and to return before any read attempt if it is an ASUS DRW-24D5MT
and maybe others, which still need to have been found.
Again it would be up to a Brasero developer to decide whether TRUE or
FALSE would be the best reply in this case.
Maybe it would be possible to read that sector by SCSI command
READ CD MSF with address 00:01:74.
(The track begins at 00:02:00. Brasero wants one sector before that
start which to my computation would be 00:01:74.)
-
Experiments made:
I tried the "Copy" task of Brasero with the TSSTcorp drive in its USB box
at /dev/sr2.
Other than with the Pioneer at /dev/sr1, this run reports
BraseroMedia: (at brasero-medium.c:2062) 2 track(s) found
BraseroMedia: (at brasero-medium.c:1645) Retrieving track information for 1
BraseroMedia: (at brasero-medium.c:1715) Data track belongs to first session
of multisession CD. Checking for real size (193686 sectors currently).
BraseroMedia: (at brasero-medium.c:1573) Checking for TDBs in track pregap.
BraseroMedia: (at brasero-medium.c:1742) Correcting track size (now 193688)
which indicates that Brasero thinks this CD was written with write
type SAO.
But i burnt it with xorriso -as cdrecord -tao. libburn's demo program
telltoc reports that blocks 193687 and 193688 are not readable and thus
supposes the CD to be burnt by TAO.
Later the log confrims telltoc's findings:
BraseroBurn: (at burn-process.c:417) BraseroCdrdao stderr: Found L-EC error
at sector 193686 - ignored.
BraseroBurn: (at burn-job.c:1430) BraseroCdrdao called brasero_job_get_action
BraseroBurn: (at burn-process.c:417) BraseroCdrdao stderr: Found L-EC error
at sector 193687 - ignore
("L-EC error" would mean a medium problem. So the drive maybe contributes
to the confusion of Brasero. dmesg reports the same errors, probably from
blkid inspecting the medium for an UDF anchor.)
dmesg reports around the time when Brasero was inspecting the drive:
usb 1-13: reset high-speed USB device number 5 using xhci_hcd
and with another "Copy" task again:
usb 1-13: reset high-speed USB device number 5 using xhci_hcd
Whatever, the drive stays usable and no error of READ CD for sector
0x is reported in dmesg.
Poking manually into that drive:
# apt-get install sg3-utils
$ sg_raw /dev/sr2 be 00 ff ff ff ff 00 00 01 10 00 00 --outfile=tdb_data.bin
>>> transport error: Host_status=0x03 [DID_TIME_OUT]
Driver_status=0x00 [DRIVER_OK]
SCSI Status: Good
and another USB device reset is reported in dmesg
usb 1-13: reset high-speed USB device number 5 using xhci_hcd
Still the drive is usable afterwards.
Now the decisive test with /dev/sr0, the ASUS drive:
No Brasero is running.
After inserting the CD into /dev/sr0, i see in dmesg complaints about
i/o errors with the last two blocks. This time not "L-EC error" but
rather "Illegal Request", as i would expect from the specs.
The drive is usable for xorriso.
So:
$ sg_raw /dev/sr0 be 00 ff ff ff ff 00 00 01 10 00 00 --outfile=tdb_data.bin
SCSI Status: Check Condition
Sense Information:
Fixed format, current; Sense key: Illegal Request
Additional sense: Unaligned write command
dmesg reports
ata3.00: exception Emask 0x0 SAct
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, Mauro Sacchetto wrote: > brasero -g --brasero-media-debug > log 2>&1 (After knowing the magic words i find man pages in the web which have it.) In the log from a successful Copy job with the Pioneer drive (sr1) i do not see the message about "TDB". But in the log of an immediate failure with the TAO CD in the ASUS drive, i see: BraseroMedia: (at brasero-drive.c:983) No medium inserted BraseroMedia: (at brasero-medium.c:1738) Following two sectors are readable. BraseroMedia: (at brasero-medium.c:1573) Checking for TDBs in track pregap. A problem with this log seems to be that the messages for both drives are intermixed without an indication which drive is meant. I assume that the "No medium inserted" is for empty sr1. So i cannot really tell from comparing both logs what triggers the TDB check on the ASUS or what prevents it on the Pioneer. But the Debian code search for "Following two sectors are readable" yields a suspect: https://sources.debian.org/src/brasero/3.12.3-1/libbrasero-media/brasero-medium.c/?hl=1738#L1738 The call brasero_medium_track_written_SAO() is made only if /* Test the last block, the before last and the one before before last */ result = brasero_mmc1_read_block (handle, ... yields result == BRASERO_SCSI_OK This seems to be true with the ASUS but not with the Pioneer. The log message immediately before "Following two sectors ..." is to see in both logs and seems to be falsely issued: BraseroMedia: (at brasero-medium.c:1715) Data track belongs to first session of multisession CD. Checking for real size (193536 sectors currently). Both logs have BraseroMedia: (at brasero-medium.c:2062) 2 track(s) found but the CD-RW is closed and has only one session with one track. If not the parameter "multisession" was set with the call of brasero_medium_track_get_info() then we possibly would not have this lengthy bug discussion. Well, i will know for sure only after i hacked libburn to issue such a READ CD command for sector number -1 with exactly the same CDB bytes as reported by dmesg: [20205.014237] ata3.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 5 Volume set (in), Read cd be 00 ff ff ff ff 00 00 01 10 00 00 res 40/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout) If this experiment makes the drive stuck, then we have the culprit. Hmm ... i dimly remember a project to send SCSI commands from the shell. Google brings me to sg_raw of sg3-utils. Looks like we have it. https://manpages.debian.org/testing/sg3-utils/sg_raw.8.en.html Probably: sg_raw /dev/sr0 be 00 ff ff ff ff 00 00 01 10 00 00 \ --outfile=tdb_data.bin I will explore this later this weekend. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, am i too stupid or is there really no user option to set write type SAO ? Neither clicking around in Brasero's GUI nor searching for BRASERO_BURN_FLAG_DAO in the source brings any insight. I want to burn a blank CD with SAO in order to check whether this avoids the problem with the ASUS drive at the end of burning. (SAO would be set by libburn API call burn_write_opts_auto_write_type() if the struct burn_disc, which will be used for burning, already has its burn_track objects attached and all track sources have a predictable size. The libburn plugin of Brasero would call burn_write_opts_set_write_type() with BURN_WRITE_SAO if flags from brasero_job_get_flags() would have the BRASERO_BURN_FLAG_DAO bit set. https://sources.debian.org/src/brasero/3.12.3-1/plugins/libburnia/burn-libburn.c/?hl=526#L526 ) It looks like the wodim plugin uses SAO by default. But i fail to activate this plugin. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi,
the "Copy" task yields very interesting results:
A closed CD-R and a closed CD-RW which were written by write type SAO
don't spoil the drive.
Inserting a closed TAO CD-RW (burnt by Brasero) spoils the drive.
This spoiling is in effect already when the dialog window of Brasero asks
me to submit a medium or to choose a drive for copying.
dmesg shows:
[ 715.623542] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 715.623564] ata3.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 13
Volume set (in), Read cd be 00 ff ff ff ff 00 00 01 10
00 00res 40/00:02:00:00:02/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 715.623570] ata3.00: status: { DRDY }
[ 715.623577] ata3: hard resetting link
[ 715.938617] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 715.944589] ata3.00: failed to IDENTIFY (device reports invalid type,
err_mask=0x0)
[ 715.944594] ata3.00: revalidation failed (errno=-22)
[ 720.963384] ata3: hard resetting link
[ 721.278510] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 726.339201] ata3.00: qc timeout (cmd 0xa1)
[ 726.339213] ata3.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[ 726.339217] ata3.00: revalidation failed (errno=-5)
[ 726.339223] ata3.00: disabled
[ 726.654802] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 726.654833] ata3: EH complete
[ 726.660786] sr 2:0:0:0: [sr0] tag#16 FAILED Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK cmd_age=0s
[ 726.660788] sr 2:0:0:0: [sr0] tag#16 CDB: Test Unit Ready 00 00 00 00 00 00
(As one can see, the kernel's hard resetting the link does not help.)
The line [715.623564] tells that the bad command was BEh READ CD for
sector -1.
This only looks wrong, but isn't by itself, because CDs have a Lead-in
which usually starts at sector -150.
Before that experiment i already learned something new about CDs
from reading brasero source code about media inspection:
The function brasero_medium_track_written_SAO() looks for an indication
that the CD was written with TAO or Packet
https://sources.debian.org/src/brasero/3.12.3-1/libbrasero-media/brasero-medium.c/#L1576
MMC-5 4.2.3.12 "The Track Descriptor Block" specifies this indication
for non-SAO CD to be in each block of the "pre-gap", which preceeds the
payload of the track. I.e. sector -1 is such a Track Descriptor Block.
This is a valuable information for possibly avoiding the readahead bug
with TAO CDs.
I was tempted to explore this trick myself for libburn. But now i think
that it could be the gesture which does not play well with the ASUS
DRW-24D5MT.
Now i will probably need some help by the GNOME community:
How to build Brasero from source ?
ldd /usr/bin/brasero | wc -l
yields a library count of 115. Ewww.
Although i know how to prepare the .deb packages of libburn on salsa and
to test-build them on a virtual Sid, i expect that Brasero is more effort.
In genral i am not trustworthy as operator beyond apt-get "update",
"dist-upgrade", and "install".
My plan for an experiment would be to let function
brasero_medium_track_written_SAO()
immediately and unconditionally return FALSE without any read experiment.
(This might confuse Brasero at the end checkreading. But i know that
the errors from reading the Run-out blocks on the ASUS do not yield a
spoiled drive.)
Some instructions how to achieve such a change in Brasero would be highly
welcome.
Next problem of me unskilled Brasero user:
I was not able to talk Brasero into showing me its session log for the
failed "Copy" run. If my new theory is right there should be a log line
"Checking for TDBs in track pregap."
before nothing works any more.
It would be interesting to see the log of a successful medium inspection
on one of the two other drives whether the "TDBs" got really checked.
But i don't know how to get the log of a successful run.
Best would be to let Brasero log into its start terminal from the
very beginning.
Does somebody have instructions for me how to achieve such a visible log ?
Have a nice day :)
Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, Mauro Sacchetto wrote: > that is (more or less): "Impossible to block unity". "unit", maybe ? The device. > After Brasero fails, it's impossible to erase the CD with K3, Either Brasero did its evil deed before trying to lock the unità or the failure to do so did not prevent further access to the drive. Do i get it right that you ordered Brasero to copy the CD to an image file, not a second CD/DVD/BD drive ? Something in Brasero's own activities for drive and medium inspection must be to blame. I will try the "Copy" task with fvwm tomorrow, just to be sure that it's not about the desktops. I wrote: > > (How stupid can Brasero and udisks become ? Is there any limit ?) Simon McVittie wrote: > My understanding is that udisks does not automatically mount anything on > its own Indeed it looks like that. With fvwm instead of XFCE the pop-up windows and the automounting don't happen. My apologies towards udisks. > If you're using a purely XFCE system, thunar-volman is probably the > component responsible for mounting removable media. So curses towards thunar-volman ... and an apology because this is off-topic now that i have seen the problem with fvwm, too. Two of my questions in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=998718#110 are answered: - Would it happen without desktop ? = Yes. - Is the XFCE (?) window involved which offers a file manger even for blank CDs when they get inserted ? = No There are no indications that automats outside of Brasero have a stake in the problem. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, i managed to start fvwm2 (by Ctrl+Alt+F2 in the XFCE login window and startx /usr/bin/fvwm2) and tried whether Brasero and ASUS drive have a better relationship then. They don't. It's as bad as with XFCE. The Brasero run was started with a blank CD-RW in the drive and came, with two pauses of a minute, up the "Checkreading" stage. Then it complained that the drive was in use and the ASUS drive went unusable as with the XFCE tests. After a power cycle i verified that the CD was burned completely and tried whether readom is to blame: readom dev=/dev/sr0 f=cdimage.raw It isn't. The drive stays usable after aborting readom with Ctrl+C because it was stuck with futile retries to read the two TAO Run-out sectors at the end of the track. The MD5 of cdimage.raw then matches the ISO image. The CD does not get automounted with fvwm2. So this aspect of XFCE is not a suspect any more. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, what i believe to have learned so far: - The problem is a co-operation of ASUS DRW-24D5MT and Brasero or some software which Brasero employs. This other software (if involved at all) does not strike with K3B (tested by Mauro Sacchetto) or non-GUI burn programs like wodim or xorriso. - The problem happens only with CD media. This media family is the most complicated among the optical media, because it not only can record data blocks of 2048 bytes per sector but also other sector types like CD-DA with 2352 bytes. - Reading data CD by audio CD commands is supposed to fail on the first read attempt, but the ASUS DRW-24D5MT tolerates this against the SCSI specs. The drive announces the obsolete feature 103h "CD Audio External Play Feature". (PIONEER BDR-S09 does not. TSSTcorp SH-S223B does, but throws the specified error when reading data CD as audio CD.) Weak theory: This tolerance might lure Brasero or other software into sending SCSI commands which are inappropriate with data CD, like the obsolete SCAN command or the obsolete STOP PLAY/SCAN command. These commands were obsoleted in 2006 by MMC-5 together with the PLAY AUDIO and other commands which were specific to standalone playing of audio CD. (All drives still support READ CD which can be used to read audio CD into the computer and let its media players create sound.) - There seems to be involved a difference between Cinnamon of Debian Testing and XFCE of Debian 10. While Mauro Sacchetto does not get an unusable drive with Brasero and a blank CD-RW on Cinnamon, i get to that problem after writing succeeded but before the checksum gets verified by Brasero. (The CD verifies good in another drive by comparing the MD5s of ISO image and CD.) With non-blank CD-RW i experienced sometimes the problem already after inserting it into the ASUS drive while Brasero is running. Mauro Sacchetto did not report such an early problem. Open questions: - Would it happen without desktop ? E.g. with fvwm2 as window manager. - Is udisks involved ? It seems to be very eager to automount as soon as the CD contains data. When i kill its process, they come back quickly. - Is the XFCE (?) window involved which offers a file manger even for blank CDs when they get inserted ? - Is the way of attachment of the drive to the computer involved ? My TSSTcorp drive offers obsolete CD audio commands but is in a USB box, whereas the ASUS is directly at SATA. - Why does only Brasero invite the problem ? Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, the spoiled ASUS drive does not seem to be a direct result of an attempt to read audio CD sectors from a data CD. The drive is rather too tolerant in that aspect. The MMC-5 specs say that it should throw error 5,64,00 "ILLEGAL MODE FOR THIS TRACK", but the drive doesn't. It does not go bad either. So my nice theory about READ CD causing the SCSI error about unaligned writing is officially dead. And i have an unexpected casualty to bemoan ... - Experiments made: I removed the security barrier in cdrskin which prevents reading of data CDs as audio CDs and told it to read a CD with data as audio: mkdir audio_test cdrskin -V -v dev=/dev/sr0 extract_audio_to=audio_test \ >audio_test/scsi_log 2>&1 Normally it says: cdrskin: WARNING : Track 01 is not an audio track. cdrskin: SORRY : Medium in drive is not an audio CD. but now it reads to my big surprise the whole track into file audio_test/01.wav . The scsi_log shows READ CD commands which demand CD-DA sectors and get data back without error indication: READ CD be 04 00 00 00 00 00 00 18 10 00 00 841123 us [ 4002657 ] The resulting file is too large, because CD-ROM checksums and other non-data parts of the CD sectors where read together with the CD-ROM payload data. Divided by 2352 the number of read bytes gives the number of sectors of the track as announced by the CD table-of-content. To prove that the specs are normally obeyed, i insert the same CD into the TSSTcorp drive at /dev/sr2 and execute the cdrskin command (without option -V because i need no SCSI log). The drive takes duely offense: cdrskin: Writing audio track file: audio_test/01.wav cdrskin: SORRY : SCSI error on read_cd(0,0): [5 64 00] Illegal request. Illegal mode for this track. cdrskin: SORRY : SCSI error on read_cd(0,0): [5 64 00] Illegal request. Illegal mode for this track. but next the device file vanishes cdrskin: FAILURE : Failure to read audio sectors cdrskin: FATAL : Failed to transfer command to drive cdrskin: ( Most recent system error: 19 'No such device' ) cdrskin: FATAL : --- SG_IO: return= -1 , errno= 19 , host_status= 0x0 , driver_status= 0x0 cdrskin: FATAL : Attempted command: TEST UNIT READY : 00 00 00 00 00 00 cdrskin: FATAL : Lost connection to drive dmesg reports of repeated USB disconnect and connect going on. No power cycle helps. It seems that the USB hub went mad. Unplugging and replugging it into another mainboard USB port does not help. A very unexpected victim of a drive and media experiment. The drive comes back to life after i removed the hub and plugged the drive's USB box cable directly into the mainboard USB port. Violating SCSI specs brings bad luck. I won't ever repeat this test. (I am not superstitious, because superstition brings even more bad luck.) - Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, Mauro Sacchetto wrote: > I actually managed to burn a blank CD-RW with brasero as well. > The problem arises when the CD-RW already has content. Not for me. Blank CDs behave better with the automounter (but not with the prompt window which offers me to start the file manager). But when they are written, Brasero gets stuck after the checkreading stage and the drive is then unusable until power cycle. I think we have different desktop systems: You tested with GNOME or KDE, i tested with XFCE (which i plan to replace by fvwm2 to get rid of the file manager prompt). Normally this should make no difference. But if it is about unwanted disc groping, the desktop is indeed a suspect. > Do you have any idea what software might be doing this anomalous attempt, > which is to try to read the data CD as if it were audio? Not yet. I normally don't use desktops but rather a window manager and i hate the kind of perky automats which i see on Debian 10 XFCE when i insert a medium or Brasero is done with writing a medium. Actually i use that machine mainly headless via ssh from an older machine which fits me like an old shoe. When i managed to get rid of XFCE i will try whether Brasero alone is able to spoil the drive. (I'd expect so, because the desktop's automats have enough opportunity to grope the medium after a xorriso run, but don't spoil the ASUS DVD burner.) > Given their modest cost, ... which might be part of the problem. What costs nothing is nothing. My ASUS was the second attempt to get a new DVD burner after a HL-DT-ST (LG) GH24NSC0 became unrealiable after only 2.5 years. The first new one was a Lite-On which could not read what it had written. I sent it back and got the ASUS as replacement. It works well for me, as long as i don't let Brasero act on CD media. All three burners were at sale for less than 20 dollars. (I pay in EUR.) Half of my burners are from LG. But this is only due to what was offered when i wanted a new drive. I have a very good ASUS BW-16D1HT, a noisy Optiarc BD-5300S (company has vanished), 4 LGs, a Samsung SH-S223B, and a Pioneer BDR-S09. Most of them never had to suffer a Brasero burn run. :)) (The Pioneer grips so hard and rotates so fast that modern Verbatim BD-RE media get cracks at the inner hole after having been read 20 times by it. Old Verbatims and modern Primeon BD-RE survive undamaged. I blame it on engraved letters on the modern Verbatim media which are very near to the inner rim, i.e. the place where the drive grips the medium. The Pioneer does not react on read speed settings but delivers data at maximum speed as long as the reader consumes that fast. I had to enhance libburn so that it can enforce a lower read speed by waiting between READ commands. At 6x BD speed = 27 MB/s the Verbatim BD-RE survive without cracks. This speed makes much fewer noise than the drive's favored 10x.) > Have you encountered any models that offer greater guarantees than the ASUS > in my possession? Normally they are ill only individually. The fact that our two burners of the same model fail in a very similar way is really unusual. It seems that Blu-ray drives are of better quality than drives which can only do CD and DVD. The BD drives cost about four times what DVD drives cost, although BD differs from DVD only by the presence of the third (blue) laser. The mechanics are mostly the same. DVD drives have only two lasers: infrared for CD and red for DVD. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, the drive and Brasero work with DVD+RW, unformatted DVD-RW, and formatted DVD-RW. No bad symptoms to see, except that i get molested by the automounter and that i often have to repeatedly double click the drive field and the "Burn" button before burning begins. These successes give me the idea that the culprit is some software which wants to play the CD as audio CD. This needs a different SCSI command for reading. It's name is READ CD and it can be told to require CD-DA sectors of 2352 bytes rather than the 2048 bytes of data CD sectors. This would somehow explain the SCSI error "unaligned write", which then actually would rather mean "unaligned read". (This imposes the riddle why the TSSTcorp drive in the USB box does not cause such an SCSI error message.) Tests made: First i put the DVD+RW into the ASUS and start Brasero. The DVD gets found and after a few double clicks on the drive field, the Burn button finally works and "Burning" begins. "Finalizing", "SUCCESS", "Checksum", loud reading, "Ejecting medium". No SCSI error. xorriso -toc shows the written session. dd if=/dev/sr0 bs=2048 count=193536 | md5sum yields the correct MD5. (That @#$%^ automounter mounted it without asking me. Some other software pops up the window offering to start the file manager.) Next try: Brasero starts up with empty tray. Then i put in the DVD+RW from which i removed the ISO 9660 superblock by xorriso -blank. All works fine. No SCSI error message. Now with unformatted DVD-RW, fast blanked by xorriso -outdev /dev/sr0 -blank deformat_quickest This yields a medium state that is only suitable for write type DAO, which can only take a single session. The DVD-RW is an old 2x speed type. Full blanking would last about 40 minutes. Starting Brasero while the DVD is in the ASUS drive. All is well. Just half as fast as with the DVD+RW. Now after formatting the DVD-RW by unmounting (g) and this run: xorriso -outdev /dev/sr0 -format as_needed which lasts 40 minutes, i can burn and the drive stays usable. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, Mauro Sacchetto wrote: > I don't have the option of using more than one burner. I could try with more USB burners. But we already know that it is a problem between Brasero or its helper software on one side and the firmware of ASUS DRW-24D5MT on the other side. There may be other burner firmwares with the same vulnerability. But finding them would not bring more insight. (Of course i will keep this possibility in mind for occasions when other Brasero users report similar problems.) > It is difficult to understand how we can get out of it. I lack of a plausible theory how Brasero or its helpers can spoil the drive when Brasero is up and waiting for a disk to appear in the drive. I meanwhile have ideas for more experiments: - Does it happen only with CD but not with DVD+RW or DVD-RW ? I will probably test this today. - Does it work if i manage to permanently disable udisks ? (I once did this on Debian 8. But memories are dim and udisks quite surely evolved a lot since then.) I will have to find out about udisks configuration. Simply killing its two processes does not help. I tried and they came back. Any hints are welcome. - Does it happen when the ASUS burner is in a USB box ? If no: Does it depend on the maximum speed of the SATA port ? (See below the quote from the Manjaro thread.) This will mean that i have to break out my screw driver and the docs about the mainboard. (Me and a screw driver is a dangerous combination.) Update: The docs say that my mainboard ASUS Pro WS C246-ACE has 4 SATA 6.0 Gb/s ports and no slower SATA ports. So a port change looks not very promising. -- If Brasero had a maintainer and that maintainer has no better idea about the particular cause of this quite catastrophic behavior, then i would propose that Brasero shall try early to identify the drive model name, and refrain from doing anything more with the drive if it is an ASUS DRW-24D5MT (and maybe other types which might be found by users). It should first try to learn the drive model name from udev, so that it does not have to touch the drive by itself. If udev does not tell, then it would have to issue an SCSI INQUIRY command, which will tell the drive type. --- The software side of the problem seems old enough to have left traces in the web. But there are not so many found by Google: I now learn that i participated in a german thread without noticing that Brasero is at the core of the problem: https://forum.ubuntuusers.de/topic/problem-mit-dvd-rw-laufwerk-sata/ The thread starter reports at some stage that he succeeded with Brasero and CD. But we have to take such statements with a grain of salt. In the end he bought another drive and got happy. In https://forum.manjaro.org/t/solved-asus-cd-writer-installation-issues/37659 i see an interesting but riddling success report: "I moved the device on the MB from SATA 5 to SATA 6 and everything run smoothly" I assume the poster means the numbering of the SATA ports on the board. It is fewly plausible that Mauro Sacchetto and i both have bad SATA ports or cables. But what about SATA generations ? Maybe that board has ports for 3 Gbps and for 6 Gbps which makes a difference. --- Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, i booted kernel 4.19.0-17 which stems from a Debian 10 upgrade. Same as with 5.10-rc3: I started up brasero with empty tray of all drives. After inserting the CD-RW into the ASUS, brasero does not see it. xorriso shows that the drive is already in the bad state. So i did a power cycle with empty ASUS tray. Before starting Brasero i insert the CD-RW into the ASUS drive. Brasero sees it when starting up. So i can start the burn run. It gets to "Success", throws SCSI error 5,21,04, 1 minute of no action, then "Creating image Checksum", 1 minute of no action, then a new window pops up: "Error while burning. The drive is busy" and the drive is spoiled. So it is quite surely not about the kernel version. (But i meanwhile suspect that the SCSI error 5,21,04 is generated by the kernel and not by the drive. It does not happen with USB, it is outruled by the specs for optical drives, and i cannot imagine what command could be considered unaligned by the drive.) Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi i have to correct my previous report: > The obscure SCSI error 5,21,04 "Unaligned write" appears only with the ASUS. It appears with the PIONEER too, but not with the TSSTcorp drive. (And Brasero spoils the ASUS even if the message does not appear.) Nevertheless, i riddle what SCSI command can cause this error. WRITE commands tell block address and block count of their payload. They have no means to address single bytes. CDs can take single blocks as payload of a WRITE command. So i fail to see how such a command can be unaligned. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, Mauro Sacchetto wrote: > I installed on testing (kernel 5.14...) the kernel 5.10 from stable > (backports), but I received the same error So i fired up my experimental machine which has Debian 10 but kernel 5.10.0-rc3 cloned from git branch "linux-block" and modified to fix a few bugs in modules sr, cdrom and isofs. (linux-scsi is not interested in my patches. Shrug.) Three drives are attached: 0 -dev '/dev/sr0' rwrw-- : 'ASUS' 'DRW-24D5MT' 1 -dev '/dev/sr1' rwrw-- : 'PIONEER ' 'BD-RW BDR-S09' 2 -dev '/dev/sr2' rwrw-- : 'TSSTcorp' 'CDDVDW SH-S223B' sr0 and sr1 are connected via SATA. sr2 is in a USB box. sr0 is the same model as yours. Best chances for reproducing the burn disaster. Brasero and udisks on XFCE seem to be much at odds. Windows pop up which offer mounting and file managering. I decline. - Experiment reports about the ASUS: First run with /dev/sr0, the ASUS: Brasero burns but stalls with its "Success" stage. The SCSI error reply is reported in Brasero's start terminal: Sense key: 0x70 0x00 0x05 0x00 0x00 0x00 0x00 0x0a 0x00 0x40 0xa0 0x03 0x21 0x04 0x00 0x00 0x00 0x00 0x00 but no crash happens. The drive is now unusable but the CD yields the correct MD5 for debian-11.1.0-amd64-netinst.iso when read by /dev/sr2. I power cycle the machine to get my drive back. Next try, still with sr0 ASUS: Brasero says that no supported disk is in the drive. I click "Cancel" and again "Burn image". It finds the CD in sr0 but then immediately crashes with "Segmentation fault". Drive is unusable afterwards. Power cycle and next try with sr0 ASUS: I start Brasero and put the CD into sr0. Immediately the drive becomes unusable. No further action is needed. Power cycle and next try with sr0 ASUS: I blank the CD-RW by xorriso to keep udisks as far away as possible. I start Brasero and a window pops up which says it cannot mount the CD. (How stupid can Brasero and udisks become ? Is there any limit ?) I confirm that i noticed and let Brasero burn. It burns up to "Finalizing" and "Success" but then stalls for about a minute while the drive is blinking. Then comes the SCSI error message and after another minute a window pops up: "Error while burning. The drive is busy." I click "Close" and get the big Brasero window. I quit Brasero and the drive is unusable. Whatever i do, i don't get a new /dev/sr3 with the ASUS drive. --- Experiment reports about PIONEER and TSSTcorp (Samsung): Power cycle and CD into /dev/sr2, the TSSTcorp in a USB box. First i let md5sum compute the MD5 of the previously burnt CD. It matches the ISO's MD5. I blank the CD-RW by xorriso, eject, and start Brasero. CD back into sr2 Complaint about being not mountable. Whatever, Brasero begins to burn, finalizes, says "success", waits a minute while the drive is blinking, shows "Creating image checksum", begins to read at high and noisy speed, and finally ejects the medium. I insert again. Some software lets the drive blink and then complains that it cannot mount. md5sum /dev/sr2 yields the correct MD5. Blanked again by xorriso. I insert the CD into /dev/sr1, the PIONEER at SATA. This run is the smoothest of all: "Burning image to CD", "Finalizing", "Success", SCSI error message in start terminal, "Creating image checksum", "Ejecting medium", "Success". No minute of inactivity to see. Afterwards the drive is still functional with xorriso. md5sum yields the correct checksum. -- Summary: Brasero (maybe by help of udisks) is absolutely ill with the ASUS drive. With the other two drives it is still behaving confused but in the end can burn and does _not_ spoil the connection between drive and kernel. The obscure SCSI error 5,21,04 "Unaligned write" appears only with the ASUS. So the ASUS firmware quite probably has a stake in the problem. But it seems to need some abuse by Brasero (or udisks) to throw the SCSI error and then to become unusable. -- Outlook: I will try later with kernel 4.19.0-17 which is currently the official one for Debian 10 (upgraded in october due to the certificate blooper around "DST Root CA X3"). For now i would bet that the problem with the ASUS drive persists. I assume that it is a cooperation of buggy userland tools and buggy drive firmware. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, Mauro Sacchetto wrote: > samiel@darkstar:~$ xorriso -as cdrecord -v dev=/dev/sr0 blank=as_needed > $isoimage > [...] > xorriso : UPDATE : 378 of 378 MB written (fifo 0%) [buf 100%] 0.0x. > xorriso : UPDATE : Thank you for being patient. Working since 728 seconds. > Writing to '/dev/sr0' completed successfully. > [...] > Media current: CD-RW > Media status : is written , is closed > Media summary: 1 session, 193686 data blocks, 378m data, 0 free > It's all ok! Phew ! It would have been a hell of a job to debug a broken relationship of libburn and the kernel of Debian Testing, as i'd need it on a real machine. (I have a Sid in a virtual machine, but there the ioctl(SG_IO) takes a different path through the device driver jungle.) > I eject the CD-RW, inser it again and try to mount it: > root@darkstar:~# mount /home/samiel/debian-11.1.0-amd64-netinst.iso > /media/cdrom > mount: /media/cdrom0: WARNING: source write-protected, mounted read-only. So the drive itself is still operational after the mishap. > In Nemo -- Devices now I see twice the CD-RW, the old one (not inspectable) > and the new one (inspectable). Looks like the drive was re-connected by the kernel without completely dropping the old connection. Possibly udev adjusted the symbolic link /dev/cdrom to the newer device file (/dev/sr1 ?). The only plausible theory which i have is that Brasero now hits a new problem in the kernel (5.14 ?) which may or may not involve the drive's firmware but does not show up with older kernels. (I have a kernel 5.10 which works well with 3 drives.) I will ask at debian-user mailing list whether there are users of Testing who have burners and can test them with Brasero. Maybe significant, maybe not: While riddling what Brasero can do to the drive-kernel connection i see: > Sense key: 0x70 0x00 0x05 0x00 0x00 0x00 0x00 0x0a 0x00 0x40 0xa0 0x03 0x21 > 0x04 0x00 0x00 0x00 0x00 0x00 That's an SCSI error reply: Key 5, ASC 21, ASCQ 04. Key 5 means "Illegal Request". https://www.t10.org/lists/asc-num.htm#ASC_21 shows a list of ASC 21 errors up to ASCQ 09. According to that it is 21h/04h DZ UNALIGNED WRITE COMMAND where a missing "R" between "DZ" and "UNALIGNED" indicates that this is not supposed to happen with optical drives. But i fail to imagine what other SCSI device would have been addressed by Brasero directly. The Linux block device layer does not forward such SCSI error codes to user space. A program gets to see them only if it sends SCSI commands via the lower level sg driver. The form of above "Sense key:" indicates that it does not stem from libburn, which works via the sg driver. Debian's code search https://codesearch.debian.net/search?q=package%3Abrasero+Sense+key finds the origin of the message in brasero_sense_data_print() https://sources.debian.org/src/brasero/3.12.3-1/libbrasero-media/scsi-sense-data.c/?hl=93#L93 Another search finds the only caller in brasero_sense_data_unknown() which has more customers. Among them is brasero_sense_data_illegal_request() called by brasero_sense_data_process(), called by brasero_scsi_command_issue_sync(). But then the trace becomes fuzzy. https://codesearch.debian.net/search?q=package%3Abrasero+brasero_scsi_command_issue_sync=0 yields 35 matches. (Brasero surely does too much SCSI on its own.) The source file names seem to tell what SCSI commands shall be performed. No write command among them, and only one setter named brasero_spc1_mode_select(), which is not a good suspect for the "unaligned write command error". It is not clear whether that error is involved in spoiling the relationship between kernel and drive. Have a nice day :) Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi,
(No need to Cc: me, i am subscribed to the bug report 998718.
If you are subscribed too and don't want to be Cc:ed, then tell me.)
> thanx for your quick reply
That's because new udevadm said "XORRISO". I found the bug report by my
evening patrol with Google.
---
Diagnosis:
The unusable drive state looks like a bad relationship between drive
and kernel. Regrettably the dmesg log does not tell how Brasero brought
this relationship in its bad shape.
What to do next:
Did you already try what happens if you eject the CD and insert it again
after spoiling the drive state with Brasero and before using the drive
for mounting or for xorriso -toc ?
If not yet, then please do.
> xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
I guess this means you are running Debian "testing".
("stable" has version 1.5.2.)
To outrule the possibility that libburn underneath Brasero is to blame,
please use xorriso to burn an image by this drive:
isoimage=...path.to.your.ISO.image.file...
xorriso -as cdrecord -v dev=/dev/sr0 blank=as_needed $isoimage
Then try whether the drive is still usable.
(Crossing fingers now.)
-
Just in case somebody is interested in details of my diagnosis:
Mauro Sacchetto wrote:
> dmesg tell me:
> [23315.697087] sr 5:0:0:0: [sr0] tag#16 FAILED Result:
> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
> [23315.697092] sr 5:0:0:0: [sr0] tag#16 CDB: Prevent/Allow Medium Removal 1e
> 00 00 00 01 00
The drive did not respond to a SCSI command from the kernel's SCSI
driver. Brasero obviously tried to lock the drive tray so that pressing
the eject button would have no effect. (At least with motorized trays.)
> [23315.697128] brasero[9154]: segfault at 2f400 ip 7fb6350a5231 sp
> 7fff0e805918 error 4 in libc-2.32.so[7fb634f6c000+149000]
This might be due to a bug in Brasero, which would not react properly
on the failure of the tray locking attempt.
> xorriso tells me only
> TEST UNIT READY
> 00 00 00 00 00 00
> 19 us [ 1091 ]
The drive obviously responded to this SCSI command (indicating that it
is ready for work).
> INQUIRY
> 12 00 00 00 24 00
> From drive: 36b
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 10 us [ 1207 ]
> --- SG_IO: host_status= 0x4 SG_ERR_DID_BAD_TARGET (Bad target, device not
> responding ?)
> --- SG_IO: Gave up connection to drive
Here the drive did not respond to the kernel, which thus reported the
DID_BAD_TARGET to libburn. The Linux SG_IO adapter of libburn then gave
up the connection.
> libburn : FATAL : Lost connection to drive
> xorriso : FAILURE : Cannot acquire drive '/dev/sr0'
> xorriso : aborting : -abort_on 'FAILURE' encountered 'FATAL'
But xorriso at least found a way out without producing a segmentation
fault in libc or elsewhere.
> Better than my previous report:
> xorriso -scsi_log on -outdev /dev/sr0 -toc
> [... longer list of SCSI transactions ...]
> Media current: CD-RW
> [...]
> Media product: 97m26s65f/79m59s74f , CMC Magnetics Corporation
> Media status : is written , is closed
> [...]
> TOC layout : Idx , sbsector , Size , Volume Id
> ISO session : 1 , 0 , 193536s , Debian 11.1.0 amd64 n
> Media summary: 1 session, 193536 data blocks, 378m data, 0 free
All seems well. The drive responds to commands in reasonable time,
the medium is recognized as CD-RW, and the filesystem is recognized
as ISO 9660.
Have a nice day :)
Thomas
Bug#998718: Brasero fails (on Testing) burning an ISO image on a CD-RW
Hi, i am upstream programmer of libburn, which might be in charge of burning underneath Brasero. Regrettably i cannot interpret Brasero's messages unless they come from libburn or libisofs. So i am not sure whether i can help with fixing Brasero. The statement by other software that no medium is found in /dev/sr0 surprises me. Such a status assessment is usually done by the drive and i am not aware of any regular means to let a drive ignore or deny its loaded medium. Do you see anything in dmesg that occured when Brasero failed and looks related to "sr" or "cdrom" ? Can i talk you into inquring the drive by xorriso and logging its SCSI transactions when the drive has gone mad from Brasero and a medium is still inserted ? xorriso -scsi_log on -outdev /dev/sr0 -toc 2>&1 | tee -i /tmp/xorriso.log This would give insight into what the drive tells about its status without the programs-in-the-middle. The log is quite verbose and will have several hundred lines. Please post /tmp/xorriso.log as attachment or send it to me by private mail. (I wonder why udevadm monitor of Testing is so eager to insert blanks everywhere. But that would be a different bug, if it is not a copy+paste artefact.) Have a nice day :) Thomas
Bug#993646: xorriso-dd-target: unclear in long description if general-purpose
Hi, the commits 69b1290 and 68e0410 of g...@salsa.debian.org:optical-media-team/libisoburn.git should solve the issue with the next upload. See for the new state of the description https://salsa.debian.org/optical-media-team/libisoburn/-/raw/68e041060496b0f4804cdee12059df971eec25f2/debian/control I will ponder about a home page for xorriso-dd-target. For now it seems more important to get a link to https://wiki.debian.org/XorrisoDdTarget into https://www.debian.org/CD/faq/#write-usb I will ask Steve McIntyre, once the package is in testing and the wiki is updated to reflect this. (At least i can change the current download link from libburnia website to a https://sources.debian.org URI.) Have a nice day :) Thomas
Bug#993646: xorriso-dd-target: unclear in long description if general-purpose
Hi, it is usable as a standalone tool, which shall make safer the procedure of copying a Debian ISO image onto a USB stick. As a shell script it is hopefully widely usable on all GNU/Linux systems which offer lsblk(8). Nevertheless, it already has one customer which uses it as component: https://help.ubuntu.com/community/mkusb/xscl I have difficulties to find a short statement which addresses your question in that light. I could simply state: This is a standalone tool but also usable as part of other tools which aim for safer copying of disk images. But in my eyes this would raise the question why the statement is there. (Actually i don't see why a reader would suspect that it is not standalone. Wouldn't this suspicion apply to the xorriso binary, too ?) Do you have a better proposal for the wording ? Is it permissible to add a link to https://wiki.debian.org/XorrisoDdTarget instead ? (I Cc: Dominique Dumont who is the package's sponsor and keeps me from spoiling Debian with my activities. I myself am subscribed to this bug now.) Have a nice day :) Thomas
Bug#800627: linux: fs/isofs/util.c iso_date() will map years >= 2028 to 1970
Hi,
Salvatore Bonaccorso wrote:
> This has been fixed upstream with 34be4dbf87fc ("isofs: fix timestamps
> beyond 2027") which was backported to various table series.
> [committed 2017]
Regrettably it does not suffice, because of the remaining int bottleneck
iso_date(), which is still to see in line 19 of
https://github.com/torvalds/linux/blob/master/fs/isofs/util.c
For the records and with me duely acknowledging that Debian is not
in charge of fixing this, here is my unposted patch set for the kernel
branch of Jens Axboe as of september 2020.
git://git.kernel.dk/linux-block
---
From 154a68527351db091e5de60388ba4cfb1fe779fd Mon Sep 17 00:00:00 2001
From: Thomas Schmitt
Date: Mon, 21 Sep 2020 18:20:14 +0200
Subject: [PATCH 0/1] isofs: prevent file time rollover after year 2038
The time values in struct inode of isofs result from calls to function
iso_date() in isofs/util.c, which returns seconds in the range of signed
int. This will rollover in 2038.
ISO 9660 directory record timestamps are good for up to year 2155.
(ECMA-119 9.1.5: 1900 + 255)
The only callers of iso_date() are in isofs/inode.c and isofs/rock.c
and put the result into struct inode.i_{a,c,m}time.tv_sec which is
of type time64_t.
The time value of iso_date() essentially stems from mktime64().
So return type time64_t is appropriate for iso_date().
--
Demonstration of the problem:
Create an ISO 9660 filesystem with file date in 2040, using file /bin/true
as victim payload:
xorriso -outdev /tmp/test_date.iso \
-blank as_needed \
-map /bin/true /victim \
-alter_date m 'Oct 01 22:06:12 2040' /victim --
Inspect the current representation by isofs:
mount /tmp/test_date.iso /mnt/iso
ls -l /mnt/iso/victim
This yields with int iso_date():
... Aug 26 1904 /mnt/iso/victim
After changing the type of iso_date() to time64_t:
... Oct 1 2040 /mnt/iso/victim
For completeness i tested the last possible second:
xorriso ... -alter_date m 'Dec 31 23:59:59 2155' /victim --
and got properly:
... Dec 31 2155 /mnt/iso/victim
(When reproducing this it might be to wise to use December 30, to avoid
any potential timezone problems.)
--
Have a nice day :)
Thomas
Thomas Schmitt (1):
isofs: prevent file time rollover after year 2038
fs/isofs/isofs.h | 3 ++-
fs/isofs/util.c | 4 ++--
2 files changed, 4 insertions(+), 3 deletions(-)
--
2.20.1
---
From 154a68527351db091e5de60388ba4cfb1fe779fd Mon Sep 17 00:00:00 2001
From: Thomas Schmitt
Date: Mon, 21 Sep 2020 18:20:06 +0200
Subject: [PATCH 1/1] isofs: prevent file time rollover after year 2038
Change the return type of function iso_date() from int to time64_t.
Signed-off-by: Thomas Schmitt
---
fs/isofs/isofs.h | 3 ++-
fs/isofs/util.c | 4 ++--
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/fs/isofs/isofs.h b/fs/isofs/isofs.h
index 055ec6c586f7..527c0db72ff9 100644
--- a/fs/isofs/isofs.h
+++ b/fs/isofs/isofs.h
@@ -107,7 +107,8 @@ static inline unsigned int isonum_733(u8 *p)
/* Ignore bigendian datum due to broken mastering programs */
return get_unaligned_le32(p);
}
-extern int iso_date(u8 *, int);
+
+time64_t iso_date(u8 *, int);
struct inode; /* To make gcc happy */
diff --git a/fs/isofs/util.c b/fs/isofs/util.c
index e88dba721661..348af786a8a4 100644
--- a/fs/isofs/util.c
+++ b/fs/isofs/util.c
@@ -16,10 +16,10 @@
* to GMT. Thus we should always be correct.
*/
-int iso_date(u8 *p, int flag)
+time64_t iso_date(u8 *p, int flag)
{
int year, month, day, hour, minute, second, tz;
- int crtime;
+ time64_t crtime;
year = p[0];
month = p[1];
--
2.20.1
---
Have a nice day :)
Thomas
Bug#798300: linux: fs/isofs/rock.c coarsely truncates file names of 254 or 255 bytes length
Hi,
> if this still should be trackled,
It's still in kernel 5.10, at least.
> please report it upstream
I agree that this bug is not Debain's business.
But i'd need a kernel list member who would be ready to review a patch.
Warning:
From here on it's just whining of a userlander about kernel cruelty.
There is no maintainer for isofs. Nearest is linux-scsi because of the
semi-proximity between cdrom and isofs.
Having had a real test machine last year, i tried to submit patches for
both subsystems [1] [2], but did not get any reaction.
After this experience i did not submit my patch for the name length issues.
It is based on 5.10:
--
From 3d484405f0ad8d10ef490281da157bfdd7450cb6 Mon Sep 17 00:00:00 2001
From: Thomas Schmitt
Date: Tue, 22 Sep 2020 12:34:50 +0200
Subject: [PATCH 1/1] isofs: truncate oversized Rock Ridge names to 255 bytes
Enlarge the limit for name bytes from 253 to 255.
Do not discard all bytes of the NM field where the overflow occurs, but
rather append them to the accumulated name before truncating it to exactly
255 bytes.
Map trailing incomplete UTF-8 bytes to '_'.
Signed-off-by: Thomas Schmitt
---
fs/isofs/rock.c | 73 ++---
fs/isofs/rock.h | 2 ++
2 files changed, 71 insertions(+), 4 deletions(-)
diff --git a/fs/isofs/rock.c b/fs/isofs/rock.c
index 94ef92fe806c..e1db8776b67e 100644
--- a/fs/isofs/rock.c
+++ b/fs/isofs/rock.c
@@ -192,6 +192,63 @@ static int rock_check_overflow(struct rock_state *rs, int
sig)
return 0;
}
+/*
+ * Find backward from idx the start byte of a possible UTF-8 character.
+ * https://en.wikipedia.org/wiki/UTF-8#Description
+ * Return -1 if no incomplete UTF-8 sequence is found at the name end.
+ */
+static int rock_find_utf8_start(char *name, int idx)
+{
+ unsigned char *uname, uch;
+ int i;
+
+ uname = (unsigned char *)name;
+ /* Up to 4-byte codes */
+ for (i = 0; i < 4 && idx - i >= 0; i++) {
+ uch = uname[idx - i];
+ if ((uch & 0xe0) == 0xc0) {
+ /* UTF-8 start byte for 2-byte codes */
+ if (i >= 1)
+ return -1; /* tail is complete */
+ else
+ return (idx - i); /* tail not complete */
+ } else if ((uch & 0xf0) == 0xe0) {
+ /* UTF-8 start byte for 3-byte codes */
+ if (i >= 2)
+ return -1;
+ else
+ return (idx - i);
+ } else if ((uch & 0xf8) == 0xf0) {
+ /* UTF-8 start byte for 4-byte codes */
+ if (i >= 3)
+ return -1;
+ else
+ return (idx - i);
+ } else if ((uch & 0xc0) != 0x80) {
+ /* not an UTF-8 tail byte, so no UTF-8 */
+ return -1;
+ }
+ }
+ /* That would be an oversized UTF-8 code or no UTF-8 at all */
+ return -1;
+}
+
+/*
+ * Replace trailing incomplete UTF-8 sequence by '_' characters.
+ */
+static void rock_erase_incomplete_utf8(char *name, int len)
+{
+ int i;
+
+ if (len <= 0)
+ return;
+ i = rock_find_utf8_start(name, len - 1);
+ if (i < 0)
+ return;
+ for (; i < len; i++)
+ name[i] = '_';
+}
+
/*
* return length of name field; 0: not found, -1: to be ignored
*/
@@ -271,16 +328,24 @@ int get_rock_ridge_filename(struct iso_directory_record
*de,
break;
}
len = rr->len - 5;
- if (retnamlen + len >= 254) {
- truncate = 1;
- break;
- }
p = memchr(rr->u.NM.name, '\0', len);
if (unlikely(p))
len = p - rr->u.NM.name;
+ if (retnamlen + len > RR_NAME_LEN) {
+ truncate = 1;
+ /* Take as many characters as possible */
+ len = RR_NAME_LEN - retnamlen;
+ if (len <= 0) {
+ rock_erase_incomplete_utf8(retname,
+ retnamlen);
+ break;
+ }
+ }
memcpy(retname + retnamlen, rr->u.NM.name, len);
retnamlen += len;
Bug#982746: live-build: autopkgtest failure on arm64, armhf and ppc64el: Failed to prepare session write run
Hi, this could be a bug in live-build. It seems to use "xorriso -as mkisofs" option -isohybrid-mbr on systems which have no PC-BIOS firmware and thus get no ISOLINUX BIOS bot equipment in the ISO. Line 69 in https://sources.debian.org/src/live-build/1:20210122/scripts/build/binary_iso/ adds -isohybrid-mbr to the options without checking that LB_BOOTLOADER_BIOS is set to "syslinux". But only with this setting, the option -b isolinux/isolinux.bin is used, which is a precondition for using -isohybrid-mbr. (Option -partition_offset 16 would be ok without "syslinux".) Long story: I am the developer of libisofs. The error message is sparse, i have to confess. Code study shows three occasions where libisofs: MISHAP : Cannot patch isolinux boot image can be emitted without more explanation. 1: The size of the El Torito boot image file isolinux.bin is less than 64 bytes. So it cannot take a Boot Info Table. 2: There was no isolinux.bin submitted at all. (option -b in mkisofs emulation, or command -boot_image isolinux dir=/isolinux in native command mode.) 3: is like 2 in a different code path. I can provoke the sparse message by omitting -b and its subordinates: xorriso -as mkisofs -o /dev/null -isohybrid-mbr "$HOME"/x "$HOME"/xx with "$HOME"/x and "$HOME"/xx being some small dummy files: ... xorriso : NOTE : Copying to System Area: 36 bytes from file '[...]/x' libisofs: MISHAP : Cannot patch isolinux boot image xorriso : FAILURE : Failed to prepare session write run ... That would be occasion 3. Now why should there be no -b isolinux/isolinux.bin ? Do i get it right, that the xorriso run is started by https://sources.debian.org/src/live-build/1:20210122/scripts/build/binary_iso/ ? (Would it be possible to let the script report the actual xorriso arguments ?) My best guess is that variable LB_BOOTLOADER_BIOS is not set to "syslinux". It controls in line 110 ff. which BIOS boot equipment is to be submitted by option -b. The option -isohybrid-mbr is used if LB_IMAGE_TYPE is "iso-hybrid" (line 66 ff.). This seems wrong, because it is used even if no ISOLINUX is selected by LB_BOOTLOADER_BIOS. So if amd64 and i386 succeed, it is most probably the inappropriate option -isohybrid-mbr which spoils the runs for the others. Have a nice day :) Thomas
Bug#982221: k3b: please stop using cdrkit
Hi, i am the developer of libburnia, including cdrskin and xorriso. Feel free to ask for advise about how to use their features to achieve what K3B needs. I have to warn, though, that UDF is not supported by libisofs. Besides the fact that some smaller operating systems cannot deal with files of isoze >= 4 GiB in ISO 9660, there is the bigger obstacle that DVD video mastering needs UDF, as provided by genisoimage and mkisofs. (Another reason to keep genisiomage alive is HFS production for booting old powerpc machines.) wodim is inferior to libburn in respect to data recording on DVD and BD media. But on CD it is able to burn CD-XA and some more exotic modes like "RAW". CD-XA is a format similar to CD-ROM with some extra bytes settable by the user. Although Joliet specifies it for multi-session (and the kernel shows traces of an old disabled demand for it), i am not aware of any use case where it cannot be replaced by CD-ROM, the usual data format of CD and of all DVD and BD media. "RAW is needed for cloning CDs with strange sectors, e.g. of copy-protected disks. libburn will quite surely not support CD-XA and RAW in future. So decommissioning wodim is ok if we do not strive for cloning. But decommissioning genisoimage will leave a noticable gap. Where is the discussion happening about cdrkit's demise ? Have a nice day :) Thomas
