Re: Help! secure boot is preventing boot of debian

[Thomas Schmitt]

Hi,

Richmond wrote:
> OK I got it booted and re-installed grub from debian. But I don't know
> why it happened, I haven't changed any keys or done anything except an
> opensuse update. I will ask the opensuse list

I remember to have seen discussions about newly installed shim adding
names of older shims or bootloaders to something called SBAT.
I find in my mailbox a mail with a link to
  https://bugzilla.opensuse.org/show_bug.cgi?id=1209985

About SBAT i found in the web:
  
https://www.gnu.org/software/grub/manual/grub/html_node/Secure-Boot-Advanced-Targeting.html
  https://github.com/rhboot/shim/blob/main/SBAT.md


Have a nice day :)

Thomas

[solved] Re: No login with Debian 12 ssh client, ssh-rsa key, Debian 8 sshd

[Thomas Schmitt]

Hi,

Jeffrey Walton wrote:
> If I am not mistaken, the problem you are experiencing is due to using
> RSA/SHA-1 on the old machine.

Max Nikulin wrote:
> My reading of /usr/share/doc/openssh-client/NEWS.Debian.gz is that ssh-rsa
> means SHA1 while clients offers SHA256 for the same id_rsa key.

Indeed NEWS.Debian.gz links
  PubkeyAcceptedAlgorithms +ssh-rsa
to RSA/SHA1.
This is the explanation why the message does not say that ssh-rsa is
disabled and why the web is so unclear about the ssh-rsa hash algorithm.

So the Debian 12 client really offered the RSA key but not in a way the
Debian 8 server could handle.
The ssh -v messages have a line

  debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 
Debian-5

(I wonder what the string "Debian-5" may mean. The Debian 12 machine has
   debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2
 So "-5" is not the Debian version.
)
NEWS.Debian.gz says

  OpenSSH has supported RFC8332 RSA/SHA-256/512
  signatures since release 7.2 and existing ssh-rsa keys will
  automatically use the stronger algorithm where possible.

So the Debian 8 sshd is too old for a better ssh-rsa handshake and the
connection might have been highjacked since 2022 "for 

Re: After upgrade, what do you do about "removed" and "obsolete" packages ?

[Thomas Schmitt]

Hi,

Florent Rougon wrote:
> AFAIK, these are not wildcards; each star appended to a package name
> indicates that the package is going to be purged

At least it is a good way to catch the attention of the apt operator. :))


> tl;dr: aptitude praise

Thanks for this description of a real world procedure.
Now i know at least that i am not the only one who cares about the
post-upgrade steps in the manual. I already began to think that everybody
lets the surplus packages rot in the dark.


Have a nice day :)

Thomas

[solved] Re: No login with Debian 12 ssh client, ssh-rsa key, Debian 8 sshd

[Thomas Schmitt]

Hi,

the following line in ~/.ssh/config did the trick:

  PubkeyAcceptedAlgorithms +ssh-rsa

This lets ssh -v report:

  debug1: Offering public key: /home/.../.ssh/id_rsa RSA SHA256:...
  debug1: Server accepts key: /home/.../.ssh/id_rsa RSA SHA256:...
  Authenticated to ... ([...]:22) using "publickey".

and leads to a shell session on the Debian 8 machine.

So the mere message
  debug1: Offering public key: /home/.../.ssh/id_rsa RSA SHA256:...
does not mean that RSA would be acceptable on the client side.
It would be nice if the refusal message would be somewhat clearer than
  debug1: send_pubkey_test: no mutual signature algorithm


I wrote:
> > The ssh-rsa key was generated by Debian 10. man ssh-keygen of buster
> > says the default of option -b with RSA was 2048.
> > (Does anybody know how to analyze a key file in regard to such
> > parameters ?)

Michael Kjörling wrote:
> $ ssh-keygen -l -f $pubkeyfile

Says "2048 SHA256:... ...@... (RSA)".
(Now that i know the right option, i can suddenly see it in the man page.)


Have a nice day :)

Thomas

No login with Debian 12 ssh client, ssh-rsa key, Debian 8 sshd

[Thomas Schmitt]

Hi,

i still have network access to a Debian 8 system, to which i logged in
from Debian 11 via ssh and a ssh-rsa key. After the upgrade to Debian 12
ssh fails with this public key authentication.
The probably relevant messages from a run of ssh -vvv are:

  debug1: Offering public key: /home/.../.ssh/id_rsa RSA SHA256:...
  debug1: send_pubkey_test: no mutual signature algorithm

To my luck, the old sshd already supports ssh-ed25519 and i was able to
add the content of the Debian 12 id_ed25519.pub to the Debian 8 file
.ssh/authorized_keys2 . Now ssh to the Debian 8 machine works again.


But i find this error message "no mutual signature algorithm" strange.
The Debian 12 ssh client is obviously willing to try ssh-rsa.
The Debian 8 sshd accepted that key from Debian 11. Why not from 12 ?


In
  https://www.openssh.com/releasenotes.html
i find for 9.2 or older only a RequiredRSASize directive of which
man sshd_config says the default is 1024.
The ssh-rsa key was generated by Debian 10. man ssh-keygen of buster
says the default of option -b with RSA was 2048.
(Does anybody know how to analyze a key file in regard to such
parameters ?)

In the web i find the reverse problem, i.e. older machine cannot ssh to
Debian 12, because ssh-rsa would now be disabled by default.


Have a nice day :)

Thomas

Re: After upgrade, what do you do about "removed" and "obsolete" packages ?

[Thomas Schmitt]

Hi,

just for the archive:

I think i found the source code which emits the "[...]" strings of
apt-list:

  https://sources.debian.org/src/apt/2.9.4/apt-private/private-output.cc/#L292

The possible status strings are:

  [installed,upgradable to: ...]
  [installed,local]
  [installed,auto-removable]
  [installed,automatic]
  [installed]
  [upgradable from: ...]"
  [residual-config]

More seem not to exist.

Their meaning has to be guessed from the names of methods, variables, and
constants in the code. (I believe "pkgCache::Flag::Auto" is a constant and
not an overloaded monster car. With C++ one never knows what's behind a
name.)

   if (P->CurrentVer != 0)
   {
  if (P.CurrentVer() == V)
  {
 if (state.Upgradable() && state.CandidateVer != NULL)
strprintf(StatusStr, _("[installed,upgradable to: %s]"),
 else if (V.Downloadable() == false)
StatusStr = _("[installed,local]");
 else if(V.Automatic() == true && state.Garbage == true)
StatusStr = _("[installed,auto-removable]");
 else if ((state.Flags & pkgCache::Flag::Auto) == pkgCache::Flag::Auto)
StatusStr = _("[installed,automatic]");
 else
StatusStr = _("[installed]");
  }
  else if (state.CandidateVer == V && state.Upgradable())
 strprintf(StatusStr, _("[upgradable from: %s]"),
   InstalledVerStr.c_str());
   }
   else if (V.ParentPkg()->CurrentState == pkgCache::State::ConfigFiles)
  StatusStr = _("[residual-config]");

-
About my post-upgrade activities:

I ran the command that is proposed in
  
https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.html#purge-removed-packages
  apt purge '~c'

It flooded me with package names under the headline

  The following packages were automatically installed and are no longer 
required:
  ...
  Use 'apt autoremove' to remove them.

Then it offered me a list with slightly frightening wildcards:

  The following packages will be REMOVED:
fuse* libreoffice-avmedia-backend-gstreamer* linux-image-4.19.0-17-amd64*
linux-image-4.19.0-20-amd64* linux-image-4.19.0-9-amd64* python*
python-twisted-core* wicd-daemon* wicd-gtk*

After my confirmation it purged the 9 configurations which were reported
by "apt list '~c'"
  Purging configuration files for fuse (2.9.9-5) ...
  ...
  Purging configuration files for python (2.7.16-1) ...
and as last line said
  Processing triggers for dbus (1.14.10-1~deb12u1) ...

Somewhat mistrusting about the removal of "python" i asked apt-file from
where my current /usr/bin/python stems. Answer: python-is-python3 .
(As side result i now wonder how the result of /usr/bin/python3-pasteurize
might taste and how long it stays fresh.)


I will probably run "apt autoremove" after verifying that the few
worthy local packages are not in the list proposed for autoremoval.


Have a nice day :)

Thomas

Re: After upgrade, what do you do about "removed" and "obsolete" packages ?

[Thomas Schmitt]

Hi,

i wrote:
> > What kind of programming language can have inspired the developers
> > to define such a syntax ?

Max Nikulin:
> https://blog.jak-linux.org/2019/08/15/apt-patterns/

This points to aptitude. The package description of aptitude says
"mutt-like syntax for matching packages". Indeed
  https://www.sendmail.org/~ca/email/mutt/manual-4.html
has in its second half some lines which resemble apt-patterns.


> As to obsolete vs. local packages, my guess is that apt may label some
> version as obsolete if another version of the same package is still
> available from some repository. Otherwise it is local.

To me it seems that apt-patterns simply calls "obsolete" what apt-list
then marks in its output as "[... local]".
Obviously these terms refer to different reasons why a package is not
found in the official repos. But these reasons seem to be
indistinguishable. So in the end both terms depict the same status.


Have a nice day :)

Thomas

Re: After upgrade, what do you do about "removed" and "obsolete" packages ?

[Thomas Schmitt]

Hi,

i wrote:
> > Next documenation riddle is what the word "local" means in output lines
> > like
> >   linux-image-5.10.0-rc2-ts/now 5.10.0-rc2-ts-37 amd64 [installed,local]

Florent Rougon wrote:
> I don't use this but guess it is as in aptitude, where “obsolete/local
> packages” are packages that are installed (from dpkg's POV) but not
> available from any of the repositories scanned in the last 'apt update'
> run.

So "local" would be just another word for "obsolete" ?

Indeed, the output of

  apt list '?installed !?obsolete' | grep local

shows only some packages with "locale" in their name, but none with
"local" in the []-brackets. On the other hand

  apt list '?installed ?obsolete' | grep -v 'local]$'

shows no packages, i.e. all lines of obsolete packages end by "local]".


Nevertheless it would be nice to find documentation about this kind of
info in the output of "apt list".


Have a nice day :)

Thomas

Re: After upgrade, what do you do about "removed" and "obsolete" packages ?

[Thomas Schmitt]

Hi,

i wrote:
> > But i am not sure whether the commercial package which i have to keep
> > will be preserved with "apt autoremove".
> > Is there a way to do a dry run which only tells what would happen if i
> > were more courageous ?

Mike Kupfer wrote:
> When I use "apt autoremove", I am given a list of proposed removals and
> a prompt about whether I want to proceed.

Good to know that there are safeguards when i finally remove some of the
"obsolete" packages.


I wrote:
> > How could i get a list of only the automatically installed obsolete
> > packages ?
> > (I still did not find any documentation about the '~c' or '~o' with
> > "apt list".)

Max Nikulin wrote:
> apt-patterns(7)

Wow. What kind of programming language can have inspired the developers
to define such a syntax ?
But hey, at least there is logic provided. \o/

So i try

  apt list '?installed ?obsolete ?automatic'

This narrows the list from 220 to 192 packages.
Even better, i don't have to diff the lists but can see the 28 other
obsolete packages by

  apt list '?installed ?obsolete !?automatic'

Among them are "hfsprogs", the self-made kernels, and the commercial
package which i need to keep.


Next documenation riddle is what the word "local" means in output lines
like

  linux-image-5.10.0-rc2-ts/now 5.10.0-rc2-ts-37 amd64 [installed,local]

(I may have missed something in the man pages of dpkg and dpkg-query, but
their occurences of the word "local" do not look like related to the info
from "apt list".)


Have a nice day :)

Thomas

Re: After upgrade, what do you do about "removed" and "obsolete" packages ?

[Thomas Schmitt]

Hi,

i wonder why none of the electricians on this list has an anecdote to
share about dealing with "obsolete" packages after upgrade.
No triumphs, defeats, or global catastrophes ?


I wrote:
> > https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.h
tml#purge-removed-packages
> > What does "[residual-config]" mean ?

Marco Moock wrote:
> Packages include system-wide configuration files. If packages are
> removed, this configuration will not be deleted. You need to purge such
> packages to remove it.

So the smaller list of packages can be dealt with what the upgrade
instructions propose:
  apt purge '~c'


There remains the list of 220 "obsolete" packages.

> > https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.html#obsolete

> Packages have dependencies. Those will be marked as automatically
> installed. They can be removed if no other package depends on them.

But several of those packages were surely installed manually by me via
dpkg -i after being made made as descibed in
  
https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-building
The predicate "obsolete" is not the same as "automatically installed".
I understand that obsolete means having no successor package in the
upgraded Debian release.

How could i get a list of only the automatically installed obsolete
packages ?
(I still did not find any documentation about the '~c' or '~o' with
"apt list".)


> Be aware: If you install software beyond apt/dpkg that depends on files
> in installed packages, you need to mark them as manually installed to
> avoid being removed by autoremove.

Google leads me to apt-mark for that purpose.
But i am not sure whether the commercial package which i have to keep
will be preserved with "apt autoremove".
Is there a way to do a dry run which only tells what would happen if i
were more courageous ?


Have a nice day :)

Thomas

After upgrade, what do you do about "removed" and "obsolete" packages ?

[Thomas Schmitt]

Hi,

today i upgraded a Debian 11 system to 12 and am now scratching my head
over the final steps as described in
  
https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.html#purge-removed-packages
  
https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.html#obsolete

The command
  apt list '~c'
shows 9 "removed packages":
  fuse/stable 2.9.9-6+b1 amd64 [residual-config]
  libreoffice-avmedia-backend-gstreamer/now 1:7.0.4-4+deb11u9 amd64 
[residual-config]
  linux-image-4.19.0-17-amd64/now 4.19.194-3 amd64 [residual-config]
  linux-image-4.19.0-20-amd64/now 4.19.235-1 amd64 [residual-config]
  linux-image-4.19.0-9-amd64/now 4.19.118-2+deb10u1 amd64 [residual-config]
  python-twisted-core/now 18.9.0-3 all [residual-config]
  python/now 2.7.16-1 amd64 [residual-config]
  wicd-daemon/now 1.7.4+tb2-6 all [residual-config]
  wicd-gtk/now 1.7.4+tb2-6 all [residual-config]
What does "[residual-config]" mean ?
The man page of apt is quite sparse. Is there something more detailed
available online ?

  apt list '~o'
shows 220 "obsolete packages", of which at least one is a commercial
non-Debian package which is on the machine for commercial reasons.
Others are gcc-{8,9,10}, hfsprogs, linux-image-* from Debian 10 and from
my own kernel experiments, and lots of stuff of which i have no clue.

I wonder how others sift through such a list and decide what to do.


Have a nice day :)

Thomas

Re: live-build: what do live-task-* do?

[Thomas Schmitt]

Hi,

Hans wrote:
> I am playing around with live-build.

This package has its own specialized mailing list:
  debian-l...@lists.debian.org


Have a nice day :)

Thomas

Re: Will te UUID or blkid of a device change?

[Thomas Schmitt]

Hi,

Hans wrote:
> > I want to make sure, that the correct USB-stick is used.
> > Thus I can do by using the UUID of the target stick like
> > dd if=/path/to/myfile.iso of=UUID="123456-abcd-"

David Wright wrote:
>   # dd of=/dev/disk/by-id/JetFlash_Transcend_4GB_JKNB2FYG-0:0  …  …

This is indeed a good solution if the device ID is known and systemd does
not change its way of composing the "by-id" name.
But if you first have to find out the right "by-id" name, then there is
again the risk of user error, especially when in a hurry.


There is a different, interactive approach which depends on the fact that
the Linux kernel creates a new device file when a USB stick is plugged in:

  https://packages.debian.org/stable/xorriso-dd-target
  https://wiki.debian.org/XorrisoDdTarget

The man page of xorriso-dd-target demonstrates more use cases, like:
  - List all devices with reasoning
  - Evaluate particular given devices


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

in the second part of this mail i discuss a possible problem with script
04_create_iso.sh about bootability on Legacy BIOS from USB stick.
(I apologize already now for being off topic by talking about an Ubuntu
ISO.)


Roland Clobus wrote:
> Thanks for pointing to live-build :-)

Just connecting my users. :))


> > > [1] https://github.com/t2linux/T2-Ubuntu/tree/LTS
I wrote:
> > Just out of pure curiosity: From what script in particular do you get
> > this impression ?

Roland Clobus wrote:
> The scripts with the sequence numbers 01-04 mirror the steps that are done
> in live-build,

01_build_file_system.sh looks quite debianish. (Not my turf.)
02_build_image.sh mentions squashfs, which is indeed typical for Live ISOs.
But as said, with Ubuntu desktop there is no non-Live ISO.
03_prepare_iso.sh prepares bootloader files.
(I know the topics of 02 and 03 only as bystander.)

--
Now for the announced discussion of a possible problem in the production
of derived Ubuntu ISOs:

04_create_iso.sh looks on the first glimpse like producing a usual amd64
hybrid ISO for {Legacy,EFI} x {DVD,HDD}, but in detail it is somewhat
riddling:

  20   --grub2-mbr "/usr/lib/grub/i386-pc/boot_hybrid.img" \
  ...
  24   -isohybrid-mbr "${ROOT_PATH}/files/isohdpfx.bin" \

First xorriso is instructed to use boot_hybrid.img as MBR with some extra
info for GRUB.
But then this is overridden by the instruction to use isohdpfx.bin as MBR
with some extra info for ISOLINUX/SYSLINUX.

In my local experiments this leads to dysfunctional MBR code because
-isohybrid-mbr works its magic only if a ISOLINUX boot image for El Torito
is present. But i understand from 03_prepare_iso.sh that the image file
given by option -b is despite its name "isolinux/bios.img" concatenated
from GRUB files cdboot.img and core.img.

I wonder whether "T2 Macs" would be able to boot via MBR code at all.
That would be EFI in CSM mode and ISO on USB stick.
Does this work with the currently produced ISOs ?

Obviously i am not smart enough to find one of the resulting ISOs in the
web for inspection. Shrug.

Can it be that -isohybrid-mbr was used because else
  25   -isohybrid-gpt-basdat -isohybrid-apm-hfsplus \
would cause an error ?
If so, then option

  -part_like_isohybrid

might be the better alternative because it does not override the
preparation for using boot_hybrid.img as --grub2-mbr.
If i replace in my experiment the line
  -isohybrid-mbr "${ROOT_PATH}/files/isohdpfx.bin" \
by -part_like_isohybrid, then i get an ISO where xorriso can recognize
an MBR with the patched-in extra info caused by option --grub2-mbr.
The partition table is more sparse than with a real ISOLINUX MBR. But
it marks the EFI partition in MBR and (pseudo-)GPT.


Whatever, current Ubuntu desktop ISOs do not use -isohybrid-gpt-basdat
and -isohybrid-apm-hfsplus any more.
One may run e.g.:

  xorriso -indev ubuntu-22.04.3-desktop-amd64.iso -report_el_torito as_mkisofs

to see -append_partition , -appended_part_as_gpt ,
and -e '--interval:appended_partition_2:all::' serving instead of
-part_like_isohybrid (or -isohybrid-mbr), -e "EFI/efiboot.img", and
-isohybrid-gpt-basdat .


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

since Aditya Garg gets a reply here at debian-live, i add a link to the
thread on debian-user, beginning with Marvin Renich's proposal to
continue the thread there:

  https://lists.debian.org/debian-user/2024/05/msg00149.html

(I proposed for Debian Live
  https://live-team.pages.debian.net/live-manual/html/live-manual/index.en.html
and for Debian installation ISOs
  https://wiki.debian.org/RepackBootableISO
)


Roland Clobus wrote:
> I had a quick peek at the scripts you use [1].
> [1] https://github.com/t2linux/T2-Ubuntu/tree/LTS
> It appears to me that you want to create a custom Debian Live ISO (not a
> netinst image).

Just out of pure curiosity: From what script in particular do you get
this impression ?

AFAIK, Ubuntu desktop ISOs are always Live systems.
  https://releases.ubuntu.com/noble/
says
  "The desktop image allows you to try Ubuntu without changing your
   computer at all, and at your option to install it permanently later."


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

Aditya Garg wrote:
> This one is gonna be interesting.
> Wish me luck.

Fingers are crossed ...

(But everything in the procedure is supposed to be deterministic. So there
is few room for luck, good or bad. We rather have to navigate the chaos.)


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Not the OP, but thanks, Thomas.

Well, ISO 9660 is known to be my hobby. So i can hardly resist trying
to acquire new users for xorriso.


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

Aditya Garg wrote:
> I would prefer making the ISO as similar to the official Debian ISO and just
> replace the Debian kernel with the customised kernel.

In that case, i'd go along
  https://wiki.debian.org/RepackBootableISO
Either by using the xorrisofs options in /.disk/mkisofs of the ISO :

  
https://wiki.debian.org/RepackBootableISO#Learn_about_the_actually_used_ISO_production_command

or by relying on the capability of xorriso to determine the commands
which will reproduce the boot equipmemt of the loaded ISO :

  
https://wiki.debian.org/RepackBootableISO#In_xorriso_load_ISO_tree_and_write_modified_new_ISO

If you need help with finding the appropriate xorriso commands, ask me
in private or in public at bug-xorr...@gnu.org .


What remains is to find out whether this works out of the box or whether
the kernel has to be announced in some files of the ISO or even
cryptographically signed in some way.

--

Just in case your adventure goes beyond replacing the kernel and possibly
the boot loader menu files, i warn of a bug in xorriso-1.5.6 and older:
Don't overwrite the El Torito boot image files in a xorriso run that uses
  -boot_image "any" "replay"
The boot image files in Debian amd64 ISOs are /isolinux/isolinux.bin
and /boot/grub/efi.img .
If you need to replace them, then we have to talk.


Have a nice day :)

Thomas

[off topic] High Sierra, was: Cindex

[Thomas Schmitt]

Hi,

Brad Rogers quoted:
> macOS 10.13 (High Sierra)

Hah ! Do they think that ISO 9660 is dead enough so they can highjack
its birth name ?

  https://en.wikipedia.org/wiki/ISO_9660
  "ISO 9660 traces its roots to the High Sierra Format, [...]
   In November 1985, representatives of computer hardware manufacturers
   gathered at the High Sierra Hotel and Casino [...] in Stateline, Nevada.
   This group became known as the High Sierra Group (HSG).
   Present at the meeting were representatives from Apple Computer [...]"

A company now run by cultureless bleeps suffering from historical amnesia.


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

Aditya Garg wrote to debian-devel:
> > I wanted to create a custom ISO of Debian, with the following 
> > customisations:
> > 1. I want to add a custom kernel that supports my Hardware.
> > 2. I want to add my own Apt repo which hosts various software packages to
support my hardware.
> > I am not able to get any good documentation for the same. Please help.

Marvin Renich wrote:
> The package live-build from the Debian Live project might help you do
> what you want.

Indeed the live-build package seems to be in use outside Debian's own
ISO production. Mailing list is
  debian-l...@lists.debian.org
There exists a manual
  https://live-team.pages.debian.net/live-manual/html/live-manual/index.en.html

Installation ISOs are made by package debian-cd, of which i am not aware
that it would have have users outside the official ISO production.i
Mailing list is
  debian...@lists.debian.org
Your impression about lack of documentation is not wrong as far as this
project is concerned. :))


Nevertheless the production step of packing up the ISO from a prepared
file tree is documented together with methods to use a Debian installation
ISO as base for the preparation:
  https://wiki.debian.org/RepackBootableISO

Packages may probably be added at the appropriate place in the directory
tree under /pool. (Managing a Debian repo is not my turf. Sorry for
being vague here.)

Changing the content of a Debian ISO might need some follow-up work in
administrative files of the ISO.
When merging Debian ISOs, my script
  
https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos
manipulates:
  /README.txt
  /dists/*/Release
and merges the files listed in /dists/*/Release.
You would have to explore whether these files are affected by your
changes.


Have a nice day :)

Thomas

Re: Adding package to Debian Distro

[Thomas Schmitt]

Hi,

kiruthikaanbusuresh wrote:
> Hi Debian Team,

Standard disclaimer: We are the users. A team only by coincidence.

(And you seem not to be subscribed to the mailing list.
Thus i CC: your mail address.)


> There is a package by name rsct which is specific to IBM. I
> would like to know the process to get this added to the Debian Distro.

If it complies to
  https://wiki.debian.org/DebianFreeSoftwareGuidelines
and is of some general use, then have a look at
  https://wiki.debian.org/RFP


> Should I have to get sponsorship for getting it added to Debian ?

If you want to contribute own work to this packaging endeavor: Yes.
It will probably increase your chances for success.

See
  https://mentors.debian.net/
  https://wiki.debian.org/ITP


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Max Nikulin wrote:
> I was not aware that partition type might be an issue.

Thanks to the normative power of the facts a "may" in the specs becomes
a reason to return a mainboard with an EFI that chooses to join the
"may not" side.


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

i wrote:
> > It is disputed, whether the specs say that the partitions must be marked
> > by 0xEF in legacy MBR tables and by C12A7328-F81F-11D2-BA4B-00A0C93EC93B
> > in GPT.

Max Nikulin wrote:
> It happened so that I had locally a file with UEFI spec Version 2.3.1,
> Errata C June 27, 2012.
> [...]
> "12.3.3 Number and Location of System Partitions
> ... Further, UEFI implementations may allow the use of conforming FAT
> partitions which do not use the ESP GUID."
> [...]
> From my point of view it is opposed to "must be" for strict partition type
> checks.

It is not forbidden, indeed. But it is not guaranteed that it works.
So if the boot success were not covered by tradition, it would be a
case of "your mileage may vary".

Another problem with the statement is that it only talks of GUID and
thus of GPT partitioning, while the specs allow MBR partitioning too.
It needs a bit of semantical stretching to let it cover the whole specs.


> Later versions may have some updates.

The statement is still in UEFI 2.8 as 13.3.3.


> Some details are in 12.3.4.2 Diskette
> USB pen drive is not a diskette, but it increases probability that
> superfloppy formatting style is supported. Of course, singe FAT partition is
> more portable.

If that's a quote from the specs, then it isn't in UEFI 2.8 any more.
If it's a statement by you, then i agree that there is a chance for
unpartitioned USB sticks to work.
You may also see implementations which boot via the El Torito boot
catalog from USB stick and via a partition table from optical media.


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Max Nikulin wrote:
> Out of curiosity, does the requirement of specific GUID exist for removable
> drives?

It is disputed, whether the specs say that the partitions must be marked
by 0xEF in legacy MBR tables and by C12A7328-F81F-11D2-BA4B-00A0C93EC93B
in GPT.
In practice there seems to be no such demand. A zillion Microsoft users
would complain if the specs were interpreted narrowly.


> A USB drive may be formatted without partition table.

The specs only talk of partitions. Whether the real implementations would
look into the FAT filesystem of an unpartitioned device would have to be
tested.

(Boot firmware is a bitch. The hunchbacked partition layout of Debian ISOs
is still the one which boots on most machines. Other distros decided to
clean up at the cost of losing some old laptops.)


> 7z and bsdtar can extract content of ISO files without mounting images.

But mounting needs no special software and gives you the opportunity
to use many different ways of copying, which may be decisive when the
target is a heavily restricted filesystem like FAT.

(I still wonder which software in the Debian ISO needs the symbolic link
"/debian -> ." and which parts of the file tree are accessed via this
link. Probably one can avoid to duplicate the whole tree under /debian.)


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Luis Muñoz Fuente wrote:
> I assume the problem is the debian link, which points to the same directory:
> $ ls -l tmp/debian
> lrwxrwxrwx 1 user user 1 Apr 22 20:47 tmp/debian -> .
> and creates a loop,

That's not a link loop, because "." is not a symbolic link.
But if a tree traversal is instructed to follow symbolic links, then the
traversal becomes endless recursion which is quite equivalent to an
endless loop.

I guess it would work better with zip option --symlinks.
But FAT cannot represent symbolic links. So in the FAT filesystem you will
possibly need at least one layer of duplication to emulate the link.
I.e. a copy of the whole file tree which is accessible via /debian. In
that copy there will probably be no need for another tree under
/debian/debian.


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Luis Muñoz Fuente wrote:
> why does extracting the files from the debian iso increase the
> size so much?

Hard to say if you do not show what you do in particular.


In general an increase of about 120 MB is to be expected because of
expansion of hardlinks:

  $ du debian-12.2.0-amd64-netinst.iso
  643076  debian-12.2.0-amd64-netinst.iso
  $ sudo mount debian-12.2.0-amd64-netinst.iso /mnt/iso
  mount: /dev/loop0 is write-protected, mounting read-only
  $ du -s /mnt/iso
  762497  /mnt/iso

The bulk of duplication is with directory trees /firmware and
/pool/non-free-firmware . Some is with kernels and initrds.


> When I take a folder that occupies 5 GiB and with mkisofs I create an iso
> file, it still occupies 5 GiB.

Not if your disk filesystem supports sparse files and your files contain
substantial areas of unwritten bytes. In that case the ISO will be larger.

> And if I later extract the files it takes up 5 GiB again,

Not if there was a substantial amount of hardlink siblings among your
input files. mkisofs will store them with shared content but Linux will
represent them as independent files.

But an increase of an amd64 netinst ISO from 659 to 1500 MB cannot be
explained by hardlinks alone. Maybe you put it into the ZIP archive
twice ?


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Luis Muñoz Fuente wrote:
> I recently used clonezilla and followed these instructions:
> https://clonezilla.org/liveusb.php#linux-setup

The variation for "uEFI", i assume.

This aims at an undocumented habit of EFI implementations to look in
any FAT filesystem for a \EFI\BOOT directory with a suitable BOOT*.EFI
file and to start it, if found.
(Officially documented is to look in FAT filesystems of partitions with
MBR type 0xEF or GPT type GUID C12A7328-F81F-11D2-BA4B-00A0C93EC93B.)


> I have tried to transform the debian-12.5.0-amd64-netinst.iso file into zip
> but the size has gone from 659 MiB to 1.5 GiB, [...]
> Does anyone know why this happens.

To get an answer you will have to show what you did and where you
measured the resulting size (as ZIP archive file or on the USB stick ?).


But i don't think that intermediate storage as ZIP is needed at all.

The make-bootable-by-copy trick depends on /EFI/BOOT and BOOT*.EFI files
being in the ISO or ZIP archive. A Debian netinst ISO filesystem for amd64
contains an unpacked copy of its EFI boot partition files.
(Others do too, thanks to the relentless proselytization of Pete Batard,
 the developer of program Rufus.)

So just mount the ISO:

  $ sudo mount debian-12.5.0-amd64-netinst.iso /mnt/iso
  mount: /dev/loop0 is write-protected, mounting read-only

and copy its content to the mounted USB stick.
You will perceive about 100 MB increase in size, because Linux does not
represent the hardlinks in the ISO which save some space.

The result is supposed to boot where a Clonezilla stick boots after it
was made by unpacking the ZIP archive.
Another question is how far the programs in a Debian netinst ISO are
prepared to run from a FAT filesystem and to find their files in it.

Your mileage may vary.


Illustration: The two copies of the \EFI\BOOT directory in a netinst ISO

  $ sudo mount debian-12.2.0-amd64-netinst.iso /mnt/iso
  $ find /mnt/iso/EFI/boot
  /mnt/iso/EFI/boot
  /mnt/iso/EFI/boot/bootia32.efi
  /mnt/iso/EFI/boot/bootx64.efi
  /mnt/iso/EFI/boot/grubia32.efi
  /mnt/iso/EFI/boot/grubx64.efi
  $ mount /mnt/iso/boot/grub/efi.img /mnt/fat
  $ find /mnt/fat/efi/boot | sort
  /mnt/fat/efi/boot
  /mnt/fat/efi/boot/bootia32.efi
  /mnt/fat/efi/boot/bootx64.efi
  /mnt/fat/efi/boot/grubia32.efi
  /mnt/fat/efi/boot/grubx64.efi
  $


Have a nice day :)

Thomas

Re: LibreOffice removed from Debian

[Thomas Schmitt]

Hi,

Vincent Lefevre wrote:
> Is there any reason why LibreOffice has been removed from Debian???

  https://tracker.debian.org/pkg/libreoffice

says

  The dependencies of libuno-cppuhelpergcc3-3=4:24.2.0-1 cannot be
  satisfied in unstable on arm64, s390x, i386, ppc64el, armel, amd64,
  and armhf
  [...]
  Depends on packages which need a new maintainer
  [...]
  This package has been requested to be removed. [...]
  Please see bug number #1069123 for more information.

This bug looks somewhat like a misunderstanding between the maintainer
and the Debian FTP Master.
The maintainer declares a lot of packages to be "cruft"
(https://en.wikipedia.org/wiki/File:Harvard_Cruft_Hall.png)
referring to various arches, but not to amd64.
The Masters seem to react by marking the source package "libreoffice"
for removal. (It is in the maintainer's cruft list, indeed.)


Have a nice day :)

Thomas

Re: [Libcdio-devel] Vulnerable use of strcpy in iso9660_fs.c

[Thomas Schmitt via Libcdio-devel]

Hi,

Pete Batard wrote:
> Or maybe there's a mathematical proof that
> a UTF-8 glyph byte encoding can never be larger than 1.5 the UTF-16 glyph
> byte encoding

I thought to have given one. Let me try again:

  https://datatracker.ietf.org/doc/html/rfc3629
  "In UTF-8, characters from the U+..U+10 range (the UTF-16
   accessible range) are encoded using sequences of 1 to 4 octets."
The table after this statement shows that it can encode 21 bits that
way.
The older FSS-UTF proposal of 1992 had up to 6 octets for up to 31 bits
but was restricted in 2003 to 21 bits by above RFC. This is also defined
in ISO/IEC 10646:2014 to ISO/IEC 10646:2020.

My proof is that UCS-2 encodes the Unicode points U+ to U+
in 2 bytes which is in UTF-8 encoded in at most 3 bytes.

If the producer of the ISO uses UTF-16 instead of the older UCS-2,
then the input Unicode range is like with UTF-8: U+..U+10.
Characters which do not fit into 2 bytes (and thus possibly not into
3 UTF-8 bytes) get represented as 4 bytes. Given that UTF-8 cannot
exceed 4 bytes, the number of bytes cannot grow during conversion.

(My proposal would accomodate up to 6 UTF-8 bytes for 4 UTF-16 bytes
and thus even suffice for FSS-UTF.)


> So I'm going to stick to i_fname for length, with the expectation that we're
> unlikely to see realistic truncations outside of images designed to trigger
> one,

I try to obey specs and to avoid speculations about what of their
provisions would possibly not happen in practice.
To my experience this pays off on the long run.


> I'm not
> sure I like the idea of trying to be too smart about or expecting specs not
> to change the deal.

My proposal with name allocation of 3*if_name/2 and a result size
parameter of _iso9660_recname_to_cstring() would be as safe against
result overflow as would be yours.
It would additionally guarantee that all valid UCS-2 names lead to valid
and untruncated UTF-8 names.

(One would separately have to check what the character conversion in
libcdio makes out of invalid UTF-16 byte sequences. Whatever the
proposed size check would avoid memory corruption in
_iso9660_recname_to_cstring().)


Have a nice day :)

Thomas

Re: PPC64 install && Firefox

[Thomas Schmitt]

Hi,

Mike Hosken wrote:
> When I burned it to a cd it didn’t boot. When I tried a dvd it did boot.

That's strange. Do you still have that CD with the ISO on it ?
If so, what do you get from a run of

  xorriso -indev /dev/sr0 -toc -report_system_area plain -check_media --

assumed that /dev/sr0 is the drive by which you tried to boot.

It will show what the drive perceives as table of content and it will
try to read all blocks that are covered by the ISO.
(No need to show all of the many pacifier lines like
   xorriso : UPDATE : .. blocks read in ... seconds , ...xC
but any other text output might be of interest.)

The same run with the DVD might show significant differences.


Have a nice day :)

Thomas

Re: [Libcdio-devel] Vulnerable use of strcpy in iso9660_fs.c

[Thomas Schmitt via Libcdio-devel]

Hi,

Pete Batard wrote:
>   strncpy(cpy_result, p_psz_out, i_inlen);

Known as nitpicker i want to to point out that this would avoid a memory
corruption in case of overflow but would also truncate the name,
potentially to an incomplete UTF-8 byte sequence at the end.

I add the technical part of my private mail to Rocky of yesterday with
assessment and proposal which is in part combinable with Pete's.

--
My assessment for now:

The  strcpy() gesture in function _iso9660_recname_to_cstring() is part
of the conversion from UCS-2 to UTF-8.
I don't see the need for strncpy(), because the result of
cdio_charset_to_utf8() seems to be 0-terminated. (At least it does not
reply a length which would be needed if no terminating 0 is appended.)

But the memory safety depends on the caller having allocated cpy_result
with a sufficient size.
And there i see a potential problem with some languages.

The allocation of cpy_result happens in _iso9660_dir_to_statbuf(),
line 858 ff.:

  if (!dir_len) return NULL;

  i_fname = from_711(p_iso9660_dir->filename.len);

  /* .. string in statbuf is one longer than in p_iso9660_dir's listing '\1' */
  stat_len = sizeof(iso9660_stat_t) + i_fname + 2;

  /* Reuse multiextent p_stat if not NULL */
  if (!p_stat) {
p_stat = calloc(1, stat_len);
first_extent = true;

The size of p_stat->filename is then (i_fname + 2), i.e.  what is needed
for the UCS-2 representation of the name plus trailing 16-bit 0.
p_stat->filename becomes parameter cpy_result:

Line 956:
if (!_iso9660_recname_to_cstring(_iso9660_dir->filename.str[1],
 i_inlen, NULL, p_stat->filename,
 u_joliet_level))
Line 965:
if (!_iso9660_recname_to_cstring(_iso9660_dir->filename.str[1],
 i_inlen, NULL, p_stat->filename,
 u_joliet_level))

Normally text shrinks during the conversion from UCS-2 to UTF-8. But
there are UCS-2/UTF-16 characters which need more than 2 bytes. E.g.:
  https://www.compart.com/en/unicode/U+0800
  Samaritan Letter Alaf
  UTF-8 Encoding:   0xE0 0xA0 0x80
  UTF-16 Encoding:  0x0800

So if the UCS-2 name contains more than one of these UTF-8 3-byters the
surplus space for the 16-bit 0 can be used up and the cpy_result can
overflow, regardless of strcpy() versus strncpy().
(Each 1-byte UTF-8 character in the conversion result adds another byte
 of room for 3-byters. But i doubt that a string with samaritan letters
 will contain many letters from 7-bit US-ASCII.)

It looks like UTF-8 4-byters are not possible in UCS-2.
If we encounter UTF-16 instead, the Joliet characters which yield 4-byte
UTF-8 bytes are 4-byters themselves. (Conversion might yield poor results
if really UCS-2 is expected. But memory should be safe.)

--
Proposal in addition to Pete's:

In case of HAVE_JOLIET and u_joliet_level allocate in
   _iso9660_dir_to_statbuf()
3/2 of the UCS-2 name length (i_fname of a good UCS-2 string is divisibe
by 2):

  stat_len = sizeof(iso9660_stat_t) + 3 * i_fname / 2 + 2;

Rather an alternative to Pete's proposal:

Further i propose to add a parameter
  size_t cpy_result_size
to
  _iso9660_recname_to_cstring()
and to feed it with (3 * i_fname / 2 + 2)  from _iso9660_dir_to_statbuf().
In case of overflow, a message about programming error should be emitted.


Have a nice day :)

Thomas

Re: NextGov: Linux XZ Utils Backdoor Was Long Con, Possibly With Support

[Thomas Schmitt]

Hi,

Nicholas Geovanis wrote:
> But what if next time the back-doored software _does_ build without error?

The initial build problems did not cause suspicion.
It was the CPU load of sshd and an obscure complaint by valgrind which
caused the discovery.
  https://boehs.org/node/everything-i-know-about-the-xz-backdoor
quotes the discoverer Andres Freund:
  "I was doing some micro-benchmarking at the time, needed to quiesce
   the system to reduce noise. Saw sshd processes were using a surprising
   amount of CPU, despite immediately failing because of wrong usernames
   etc. Profiled sshd, showing lots of cpu time in liblzma, with perf
   unable to attribute it to a symbol. Got suspicious. Recalled that I had
   seen an odd valgrind complaint in automated testing of postgres, a few
   weeks earlier, after package updates.
   Really required a lot of coincidences."


gene heskett wrote:
> In light of that its worth noting that an M$ employee was the first to
> spot it.

Indeed.
Thus we should also praise the peace between Microsoft and free software
which broke out a few years ago.


There remains the question, whom a good citizen should contact when
spotting something that could be a backdoor (or a subtenant ?) of
Debian's content or infrastructure.

It seems unwise for a non-expert to do this in public, unless one wants
to accuse the innocent or to warn the hoodlums.


Have a nice day :)

Thomas

Re: Debian ISOs on USB stick

[Thomas Schmitt]

Hi,

i read from bytes 2085412 to 2085479:
  "Info rrmation Syste rm VolumeSYSTEM~"
which is similar to the alterations of one of the USB sticks shown in
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998#35

The web knows about a Microsoft folder named "System Volume Information".
  
https://answers.microsoft.com/en-us/windows/forum/all/system-volume-information-what-is-it-and-what-is/3bc81844-0baa-46bd-9949-4efb4678b677
  "whenever I put my flash-drive or my micro sd adapter and sd card into
   my windows 8.1 something called "System Volume Information" is always
   getting added on."

So did you perhaps show this USB stick to a running MS-Windows system ?


Have a nice day :)

Thomas

Re: Debian ISOs on USB stick

[Thomas Schmitt]

Hi,

David Christensen wrote:
> # cmp --verbose debian-11.3.0-amd64-netinst.iso /dev/sdb

I got my copy from
  
https://get.debian.org/images/archive/11.3.0/amd64/iso-cd/debian-11.3.0-amd64-netinst.iso
SHA256 matches:
  7892981e1da216e79fb3a1536ce5ebab157afdd20048fe458f2ae34fbc26c19b

In a further mail:
> https://cdimage.debian.org/cdimage/archive/11.3.0/amd64/iso-cd/

Same SHA256 there.


>   2083201   0 377

Byte counting of cmp is decimal and starts at 1. xorriso can search for
files which have their data in a block range. 2083201 / 2048 = block 1017.
Range size in this case is just 1 block:

  $ xorriso -indev debian-11.3.0-amd64-netinst.iso -find / -lba_range 1017 1 
-exec report_lba --
  ...
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  File data lba:  0 , 1016 , 1296 ,  2654208 , '/boot/grub/efi.img'

So it's indeed occupied by the FAT filesystem image which contains the
EFI-specific boot equipment.

>   4719105   0  56

Byte 4719105 is in block 2304, i.e. still in /boot/grub/efi.img, which
has bytes up to the end of block 2311.

I guess the bytes with the 2xx numbers are the directory change and
the 4xx numbers are content of new files.


You could mount both ISOs (e.g. at /mnt/iso1 and /mnt/iso2) and then the
two FAT image files (e.g. /mnt/iso1/boot/grub/efi.img and
/mnt/iso2/boot/grub/efi.img) in order to learn which files have emerged
or changed in the USB stick's mounted FAT filesystem.

Maybe we find a new ESP groper additionaly to Lenovo and Microsoft.
Usually they leave traces for which one can search in the web.


Have a nice day :)

Thomas

Debian ISOs on USB stick, was: SOLVED

[Thomas Schmitt]

Hi,

David Christensen wrote:
> It's a relatively simple experiment to confirm that a USB flash drive with
> d-i changes after the first boot.

This could still be
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
where Lenovo BIOS and/or MS-Windows altered the USB stick.


> Same for finding which bytes change.

I fail to find this particular info in
  Date: Tue, 2 Apr 2024 14:46:42 -0700
  From: David Christensen 
  Message-ID: 

If we have the exact ISO name (i.e. URL from where it stems) and the
byte address of the alteration, xorriso can find the affected file, if
any.

In case of bug #1056998 it was the EFI partition image /boot/grub/efi.img.
Mounting the altered and unaltered image files showed changes in the
FAT filesystem which point to the culprits Lenovo and Microsoft.


The other plausible way of altering the ISO image on the stick would be
adding a new partition.
The MBR partition table is part of the Debian ISO and thus part of the
checksummed area. Even if all other alterations happen after the end
of the checksummed ISO image, the changed partition table will cause the
Debian checksum to become invalid.

(I am not aware that Debian installer changes the table. If it does indeed
then this might be worth a new bug discussion.)


Have a nice day :)

Thomas

Re: Debian 12.5: pigz 2.6-1 fails with error message (Upstream issue 111)

[Thomas Schmitt]

Hi,

Chung  Jonathan wrote:
> Yes, I think the local fix is the way to go.

I wrote:
> > (You forgot to Cc: debian-user@lists.debian.org.
> > Consider to send your mail to the list address, too. I too would then
> > resend my following reply to the list.)

Since my "following reply" is quoted in Jonathan Chung's reply to the list
i don't have to resend it. (I gave my opinion that the problem is not a
bug in the context of Debian 12 or 13 and pointed to
https://wiki.debian.org/BuildingTutorial for a private fix of the problem.)


Jeffrey Walton wrote:
> Your problem is one that plagues Linux. You compile and link against
> one version of a library, and then you runtime link against another
> version.

This should not be a problem with a well maintained library which cares
to stay ABI compatible with its older releases.
In the present case it was a bug in the loading program pigz which
prevented zlib from being usable.


> I consider it a
> security bug since essentially random libraries are being loaded at
> runtime.
> To fix the problem yourself, add an RPATH to your LDFLAGS when
> building your program:
> -Wl,-rpath=/path/to/expected/libz -Wl,--enable-new-dtags

Well, this is nearly as unflexible as static compilation but does not
seem to prevent the use of a replaced library at the given path.

Using .so files has its advantages and disadvantages. For a distro the
advantage (without the pigz bug) is that customers of different versions
of a library can be consolidated to using the newest available version.
An advantage for the user is that bugs in a library can be fixed without
the need for re-building all its customers.


Have a nice day :)

Thomas

Debian ISOs on USB stick, was: SOLVED

[Thomas Schmitt]

Hi,

David Christensen wrote:
> > the Debian installer modifies the contents of the USB flash drive when
> > it runs.

Do you mean inside the range of the ISO image or outside by creating a
new partition ?


songbird wrote:
> if it is an iso image copied to the USB stick it should not
> be modified if you haven't somehow told the installer to
> install the system to that USB stick (somehow).

There are other parties which feel entitled to operate on the EFI System
Partition of a USB stick.
In
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
we found that Lenovo Thinkpad firmware created directories for storing
an empty file named "/efi/Lenovo/BIOS/SelfHealing.fd" and that MS-Windows
created a 12-byte file named "/System Volume Information/WPSettings.dat"
when it had contact with the USB stick.


> i guess if you wanted to be really sure you could mount it read-only.

I think it's the installer which mounts the ISO 9660 filesystem.
Whatever, the Linux kernel has no regular means to alter an ISO 9660
filesystem. Neither kernel nor Debain installer will be so daring to
operate with byte level commands on that filesystem.

But the FAT filesystem in file /boot/grub/efi.img of the ISO 9660
filesystem in debian-12.*-amd64-netinst.iso is advertised by the partition
table of the image and thus attracts vermin.


Have a nice day :)

Thomas

Re: Debian 12.5: pigz 2.6-1 fails with error message (Upstream issue 111)

[Thomas Schmitt]

Hi,

Jonathan Chung wrote:
> > pigz 2.6-1 on Debian 12.5 fails to execute due to a fixed bug on
> > upstream https://github.com/madler/pigz/issues/111
> > Installing the version from sid resolves the issue which is clearly not
> > optimal. I think the fix should be backported.
> > Can someone help me to file a bug report?

Michael Kjörling wrote:
> https://www.debian.org/Bugs/Reporting

This covers the question "how", but not "what".
So my two cents:

"Backported" is probably the wrong wish.
  https://backports.debian.org/Contribute/
says:
  "Backports are about additional features that are only offered in a
   new version, not a replacement for getting fixes into stable - use
   stable-updates for that."

That would probably mean to patch the existing pigz package in Debian 12
by the upstream remedy
  "Make pigz compatible with two-component zlib version numbers. "
  https://github.com/madler/pigz/commit/907ca0763be4547a9b0cce8c105721748814974

pigz.c:
@@ -1333,7 +1333,7 @@ local long zlib_vernum(void) {
}
ver++;
} while (left);
-   return left < 2 ? num << (left << 2) : -1;
+   return left < 3 ? num << (left << 2) : -1;
 }

 // -- check value combination routines for parallel calculation --


The new state can already be seen in
  https://sources.debian.org/src/pigz/2.8-1/pigz.c/#L1339
The old state is in
  https://sources.debian.org/src/pigz/2.6-1/pigz.c/#L1317
But there the "// -- check ..." comment is missing in favor of a line
  #ifndef NOTHREAD
So a patch proposal would need some minor hand work.


There remains the question, though, why you run into zlib-1.3 on a
Debian "stable" system where the zlib version is "1:1.2.13.dfsg-1":
  https://tracker.debian.org/pkg/zlib


Have a nice day :)

Thomas

[k3b] [Bug 429584] growisofs crashes @ 99%

[Thomas Schmitt]

https://bugs.kde.org/show_bug.cgi?id=429584

--- Comment #9 from Thomas Schmitt  ---
Hi,

Leslie Zhai wrote:
> Lng time no see :)

Hehe. I'm still around, trying to uphold ISO 9660 and burning on free
operating systems. (Just no talent for working on GUIs.)

Stay well and ...

Have a nice day :)

Thomas

-- 
You are receiving this mail because:
You are watching all bug changes.

Re: Where to download cd iso?

[Thomas Schmitt]

Hi,

Tatsu Takamaro wrote:
> I've received my own message to you. But I don't see your answer...

Well, i have't seen that message and the mailing list archive doesn't
show it either:
  https://lists.debian.org/debian-cd/2024/03/threads.html
(The one i reply to is your follow-up:
  https://lists.debian.org/debian-cd/2024/03/msg00022.html
)


> > I don't see the cd iso on the iso-cd page.
> > I mean a usual one, not a netinst.

I guess they have been discontinued because of the insane number of ISO
images with 650 MB which would emerge. (I estimate ~ 130 pieces.)

The smallest granularity seems to be the ~4.5 GB sized DVD images which
are available via Jigdo download:
  https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/
21 pieces.


Have a nice day :)

Thomas

Re: printing QR-codes on labels with 300dpi label printers with LaTeX

[Thomas Schmitt]

Hi,

Max Nikulin wrote:
> I admit "dithering" may be incorrect term, [...]
> Consider 2 squares having size of 2.5×2.5 pixels. Non-even sizes and fuzzy
> lines variants:
> █████
> ██████
> ████ ██
>██ ██
>█████
> Second variant might have sense if an image is treated as a photo unlikely
> having regular patterns with horizontal and vertical lines.

I see ...
The second rendering style is probably best described as "Bad Distortion".


Have a nice day :)

Thomas

Re: How does the 64bits time_t transition work?

[Thomas Schmitt]

Hi,

Marco Moock wrote:
> The libs will have a suffix of t64

I wonder whether those suffixes will go away at some stage of this effort.

(Further i wonder when the package tracker appearance of libisoburn
 will become less ugly than currently:
   https://tracker.debian.org/pkg/libisoburn
 and how i shall deal with a bug report which complains about the
 inconsistent state of the control file in respect to libburn and
 libisofs:
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067103
)


Have a nice day :)

Thomas

Re: printing QR-codes on labels with 300dpi label printers with LaTeX

[Thomas Schmitt]

Hi,

Max Nikulin wrote:
> When vector graphics, that does not match device resolution, is rasterized,
> the result is either non-even sizes of similar elements or fuzzy lines due
> to dithering.

Nitpicking:

"Dithering" in raster graphics is emulation of color resolution at the
expense of space resolution. Multiple coarsly colored pixels together
create the impression of a finer color tone. A certain random aspect is
added to prevent unwanted patters.
  https://en.wikipedia.org/wiki/Dither

The fuzzy lines are rather the opposite. They use surplus color
resolution to emulate ibetter spacial resolution. Over here the usual
term is "Anti-aliasing".
  https://en.wikipedia.org/wiki/Line_drawing_algorithm
  https://www.geeksforgeeks.org/antialiasing/
I get a suspiciously high share of german language results when
searching this term in the web. But "Computer Grapics" by Foley, Van Dam,
Feiner, Hughes of 1990 has a dozen occurences in its Index.


Have a nice day :)

Thomas

Bug#1067103: libisoburn1t64: should not depend on libburn4 nor libisofs6 after time_t transition

[Thomas Schmitt]

Hi,

being the one who usually prepares the releases, i am currently standing
in hands-off position until the time_t change is completed.
The package tracker is still complaining bitterly about the current
intermediate state.

Consider to re-open
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1062380
  "libisoburn: NMU diff for 64-bit time_t transition"
and/or to contact its submitter.


Have a nice day :)

Thomas

Bug#1067103: libisoburn1t64: should not depend on libburn4 nor libisofs6 after time_t transition

[Thomas Schmitt]

Hi,

being the one who usually prepares the releases, i am currently standing
in hands-off position until the time_t change is completed.
The package tracker is still complaining bitterly about the current
intermediate state.

Consider to re-open
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1062380
  "libisoburn: NMU diff for 64-bit time_t transition"
and/or to contact its submitter.


Have a nice day :)

Thomas

Re: OT: End the Phone-Based Childhood Now

[Thomas Schmitt]

Hi,

Curt wrote:
> as I believe Paul Valéry once noted, even the past isn't what it used
> to be.

That's why i want everything back exactly the way as it never was.


Have a nice day :)

Thomas

Re: OT: End the Phone-Based Childhood Now

[Thomas Schmitt]

Hi,

when i was a young Babyboomer in the late 1970s we were accused of
destroying society by a "twin culture of sexual license and cannabis".
The big Decline of the West was a sure thing, accelerated by excessive
tv consumption.
Other future threats were drying up oil wells, a comming ice age,
and the unstoppable fertility of the asian peoples.

Meanwhile the future is past and the pundits of the 70s are dead and
ridiculed.
Predictions are difficult - especially when it's about the future.


Have a nice day :)

Thomas

Re: Spam from the list?

[Thomas Schmitt]

Hi,

Andy Smith wrote:
> [...] I argue that at present it
> isn't a good idea to just reject all DKIM failures like OP's mailbox
> provider appears to be doing.

Just for the records:
The mails in question don't get rejected but rather marked as spam
and then get delivered.

The currently best theory is that megamailservers.eu adds a header
  X-Spam-Flag: YES
if it perceives DKIM problems, and that the local anti-spam software
of the receiver takes this header as reason to alter the subject by
the prefix "*SPAM*".

Whether it is a good idea to map DKIM failure to a spam marking header
is another interesting topic.

Original post of this thread with an example of all headers of a mail:
  https://lists.debian.org/msgid-search/3371640.PXJkl210th@protheus2


Have a nice day :)

Thomas

Re: Spam from the list?

[Thomas Schmitt]

Hi,

Hans wrote:
> Re: *SPAM* Re: Spam from the list?
> In-Reply-To: <20240306112253.55e25...@earth.stargate.org.uk>

referring the mail

> > Date: Wed, 6 Mar 2024 11:22:53 +
> > From: Brad Rogers 
> > Message-ID: <20240306112253.55e25...@earth.stargate.org.uk>

I assume that this mail appeared with the "*SPAM*" marker in
your mailbox.
(The currently most plausible theory is that megamailservers.eu adds
"X-Spam-Flag: YES" and your local mail processing takes this header as
reason to change the subject.)

So what does the mail which you received from Brad via debian-user
say about "Authentication-Results:" ?
Your initial post quoted three such headers. Expect more than one.


Have a nice day :)

Thomas

Re: Spam from the list?

[Thomas Schmitt]

Hi,

Hans wrote:
> I changed nothing and suddenly many mails from debian-user
> (but not all, only some) are recognized as spam.

But the one you posted here did not come from debian-user.

So maybe what changed is an inadverted subscription to one of
  debian-bugs-d...@lists.debian.org
  debian-de...@lists.debian.org
  debian-pyt...@lists.debian.org,
  w...@debian.org
This might have broadened the set of mail senders and thus gives your
mail provider opportunities to complain like spotted by Dan Ritter:

> Authentication-Results: mail104c50.megamailservers.eu;
>   dkim=fail reason="signature verification failed" (4096-bit key)
> header.d=4angle.com header.i=@4angle.com header.b="bS+3bWmq"

"4angle.com" matches the mail address of the bug submitter
"Edward Betts ".


The shown message headers offer unsubscription from debian-devel:

> List-Unsubscribe: 
> 

I.e. to send a mail to debian-devel-requ...@lists.debian.org with the
subject line
  unsubscribe


Have a nice day :)

Thomas

Re: Spam from the list?

[Thomas Schmitt]

Hi,

Hans wrote:
> during the last moonths I get more mails from the debian-user list marked as
> spam than before.
> [...]
> Below I send the header of an example of such a mail, maybe you can see the
> reason?

The message does not look like it came to you via debian-user:

> X-Original-To: lists-debian-de...@bendel.debian.org
> Delivered-To: lists-debian-de...@bendel.debian.org
> Received: from localhost (localhost [127.0.0.1])
> by bendel.debian.org (Postfix) with ESMTP id B720220598
> for ; Wed,  6 Mar 2024
> [...]
> Resent-To: debian-bugs-d...@lists.debian.org
> Resent-CC: debian-de...@lists.debian.org, debian-pyt...@lists.debian.org,
> w...@debian.org

Are you perhaps subscribed to one of the "Resent-*" lists ?


> Subject: *SPAM* Bug#1065537: ITP: bleak-retry-connector -- Connector
> for Bleak Clients that handles transient connection failures

The mark "*SPAM*" does not appear in the archive

  https://lists.debian.org/debian-devel/2024/03/msg00076.html

All in all it looks like a legit message, not like spam.
So the suspect would sit after Debian's mail servers.


The only Received header i see between Debian and you is:

> Received: from bendel.debian.org (bendel.debian.org [82.195.75.100])
> by mail104c50.megamailservers.eu (8.14.9/8.13.1) with ESMTP
> id 4269vZOl098298
> for ; Wed, 6 Mar 2024 09:57:37 +

It looks like either megamailservers.eu or your own processing added
the spam mark to the subject.


Have a nice day :)

Thomas

[k3b] [Bug 429584] growisofs crashes @ 99%

[Thomas Schmitt]

https://bugs.kde.org/show_bug.cgi?id=429584

Thomas Schmitt  changed:

   What|Removed |Added

 CC||scdbac...@gmx.net

--- Comment #7 from Thomas Schmitt  ---
Hi,

medium type DVD-R, write type DAO, not aligned to full 16 blocks,
and failure within the last 16 blocks of the run.

This matches
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=794868
  "dvd+rw-tools: Burn failure of growisofs on DVD-R[W] with write type DAO"

Consider to check whether a patch like this is applied:
---
--- growisofs_mmc-7.1-11.cpp2015-08-07 13:07:52.0 +0200
+++ growisofs_mmc.cpp   2015-08-07 14:06:31.375597960 +0200
@@ -540,7 +540,7 @@ ssize_t poor_mans_pwrite64 (int fd,const
// own higher HZ value and disrespects the user-land one.
// Sending them down as milliseconds is just safer...
//
-   if (!(errcode=cmd.transport (WRITE,(void *)buff,size)))
+   if (!(errcode=cmd.transport (WRITE,(void *)buff,nbl*2048)))
break;

//--- WRITE failed ---//
---

(The write error with DVD+R indicates a bad relationship between drive
and medium.)

Have a nice day :)

Thomas

-- 
You are receiving this mail because:
You are watching all bug changes.

Re: partition reporting full, but not

[Thomas Schmitt]

Hi,

> when cfdisk reports:
> Device  Start   End  Sectors   Size  Type
> /dev/sda2   1785522176  1786245119 722944  353M  EFI System
> /dev/sda3   1786245120  1933045759  146800640  70G   EFI System
> I don't understand the 'EFI System' note /dev/sda3 is /

The partition type does not necessarily match the type of filesystem
which is currently in that partition. It's just a field in the
partition table which must have some value.


Have a nice day :)

Thomas

Re: Contact Name...

[Thomas Schmitt]

Hi,

Clayton Penn wrote:
> I have attempted to register for the Debian Forums, but have not received a
> verification email

Did you try wether your new account is already working ?
(Sorry, i'm not familiar with the current registration procedure.)

If not:
There seems to be some problem with GMail:

  https://forums.debian.net/viewtopic.php?t=158230

Are you perhaps directly or indirectly using GMail ?


> Do you happen to have another email address contact?

I have an account at forums.debian.net, although not used in years.
So if you want to add information to above topic, i could try to do so
on your behalf.

But first consider to try registering with a completely different mail
provider which is surely not using GMail's software.


Have a nice day :)

Thomas

Re: "I: update-initramfs is disabled (live system is running on read-only media)" ...

[Thomas Schmitt]

Hi,

Albretch Mueller wrote:
> > How can you update the initramfs on read-only media?

to...@tuxteam.de wrote:
> You can't. Initramfs resides in the boot medium. To update it,
> you have to write to said medium.

One will have to create a new read-only medium.


In case the original is a Debian Live ISO:

One would have to extract the initramfs file out of the ISO. If its name
is not known, then the boot loader configuration file should tell. Like
in /boot/grub/grub.cfg of debian-live-12.0.0-amd64-standard.iso:
   initrd  /live/initrd.img-6.1.0-9-amd64
or in its /isolinux/live.cfg:
   initrd /live/initrd.img

Next one would modify the extracted initramfs.
(This is an adventure on its own. Other will know more about it than me.)

Finally one would pack up a new ISO, taking all files from the old ISO but
replacing the initramfs file by the modified one from hard disk.
Roughly like in
  
https://wiki.debian.org/RepackBootableISO#In_xorriso_load_ISO_tree_and_write_modified_new_ISO

Details could be determined when the name of ISO and initramfs file is
known. If it's about DVD media, it would be interesting to learn about
the DVD drives at the computer which shall do the modification.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Gene Heskett wrote:
> Next experiment is a pair of 4T Silicon Power SSD's

When f3 has (hopefully) given its OK, the topic of a full write-and-read
test will come up again. I'm looking forward to all the spin-off topics.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> Heh.  Don't forget your own attempts to use a shredder as a PRNG stream.

My original idea was to watch a minimal shred run by teeing its work into
a checksummer.

But then topic drift came in. So we got a farm show of random generators
and a discussion about what exactly is a bug in shred's documentation.
Plus the shell programming webinar. And a diagnosis about a rightously
failed attempt to change the partition table type from MBR/DOS to GPT.

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> Let me write out the example again, but with the bug fixed, and then
> explain what each line does, [... lecture about advanced shell
> programming ...]

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk. :))


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

"info shred" says:
> > >   i=$(mktemp)
> > >   exec 3<>"$i"
> > >   rm -- "$i"
> > >   echo "Hello, world" >&3
> > >   shred - >&3
> > >   exec 3>-

Greg Wooledge wrote:
> In fact, that last line is
> written incorrectly.  It should say "exec 3>&-" and what that does
> is close file descriptor 3, which was previously opened on line 2.
> [...]
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#36
of 22 Dec 2005 states:

  "I'll assume that this is now adequately explained in the info page
   (below).  If not then please reopen.  // Thomas Hood
   [...]
   exec 3>-"

The bug report is from 02 Aug 2002 and states that the info page contains
the short and broad promise which we can still see in "man shred".

So we can assume a bug age of 18 to 22 years.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

> https://en.wikipedia.org/wiki/Everything_is_a_file
> But, there is more than one kind of file.

"All files are equal.
 But some files are more equal than others."

(George Orwell in his dystopic novel "Server Farm".)


Have a nice day :)

Thomas

Re: Combining Distro DVD's

[Thomas Schmitt]

Hi,

Steve Matzura wrote:
> I thought it'd be a nice idea to combine any and all distribution media for
> a release into a single medium--a USB drive, of course.

The initial situation will depend much on the distro ...
But given that Debian is about the last one i know with all its packages
in DVD images, i assume you mean the Debian installer ISO sets. Like
  https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/


> I'd start by
> creating my USB drive by extracting the first DVD to it, thereby ensuring
> the boot block and boot material is where it should be.

There is the dilemma that unpacking the first ISO into the USB stick's
filesystem will not enable the copied boot stuff, and that putting the
first ISO plainly onto the stick (e.g. by dd) will leave you with a
read-only filesystem on the stick.

So you'd need to invest some extra expertise.
Either for making a bootable USB stick with read-write filesystem from
the first ISO, or for merging the ISOs into a single ISO. In the former
case you need knowledge about booting Debian (by GRUB, i guess). In the
latter case you need xorriso.


> But then what do I
> do with the additional media? Surely there will be some files with the same
> name among the individual pieces of media, and some will probably contain
> the same info while others probably will not.

They should be either either merged or discarded, depending on their
meaning and on their importance for your final result. I.e. whether you
merged the ISOs or you extracted to a writable filesystem and installed
a bootloader yourself.

The way of merging multiple Debian DVD ISOs of the same release and
CPU architecture to a single ISO by xorriso is described in

  https://wiki.debian.org/MergeDebianIsos

(The script merge_debian_isos would also be the source to learn about
the proper handling of files with expected name collision.)


I leave it to others to elaborate the ways of creating a bootable USB
stick with writable filesystem and the content of Debian DVD 1 with
work-ready installer software. Then it's an adventure on its own to
add all the packages from the other DVD images and to properly care
for their non-package files.


Have a nice day :)

Thomas

Re: grub-pc error when upgrading from buster to bullseye

[Thomas Schmitt]

Hi,

John Boxall wrote:
> I am aware that the label and uuid (drive and partition) are replicated on
> the cloned drive, but I can't find the model number (in text format) stored
> anywhere on the drive.

Maybe the grub-pc package takes its configuration from a different drive
which is attached to the system ?

Somewhat wayward idea:
Does the initrd contain the inappropirate address ?
(I don't see much connection between initrd and grub-pc. But initrd is a
classic hideout for obsolete paths after modification of boot procedures.)


Have a nice day :)

Thomas

Re: grub-pc error when upgrading from buster to bullseye

[Thomas Schmitt]

Hi,

John Boxall wrote:
>   Setting up grub-pc (2.06-3~deb11u6) ...
>   /dev/disk/by-id/ata-WDC_WDS100T2B0A-00SM50_21185R801540 does not
> exist, so cannot grub-install to it!
> What is confusing to me is that the error indicates the source SDD even
> though I have updated the boot images and installed grub on the cloned HDD.

The disk/by-id file names are made up from hardware properties.
I believe to see in the name at least: Manufacturer, Model, Serial Number.

So you will have to find the configuration file which knows that
/dev/disk/by-id address and change it either to the new hardware id or
to a /dev/disk/by-uuid address, which refers to the cloned disk content.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Still there's the discrepancy between doc and behaviour.

Depends at which documentation you look. Obviously stemming from
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#36
i read in
  https://www.gnu.org/software/coreutils/manual/html_node/shred-invocation.html

 "A file of ‘-’ denotes standard output. The intended use of this is to
  shred a removed temporary file. For example: [shell wizzardry]"

It works as long as stdout is connected to a data file, or block device,
or directory, or symbolic link, or to a character device that is not a
terminal.
(Maybe it refuses later on some of these types, but not at the location
with the message "invalid file type". I wonder if i can connect stdout
to a symbolic link instead of its target.)

The bug would thus have to be filed against the man page
  https://sources.debian.org/src/coreutils/9.4-3/man/shred.1/
which says only

  "If FILE is \-, shred standard output."

The info empire of coreutils says what above web manual says.
  https://sources.debian.org/src/coreutils/9.4-3/doc/coreutils.texi/#L10705


Have a nice day :)

Thomas

Re: shred bug?

[Thomas Schmitt]

Hi,

debian-u...@howorth.org.uk wrote:
> Maybe it is unstated but mandatory to use -n 1 as well?
> And optionally -s N?

Naw. It just doesn't want to work pipes.

Initially i tried with these options:

  shred -n 1 -s 1K -v - | sha256sum

as preparation for a proposal to Gene Heskett, like:
  shred -n 1 -s 204768K -v - | tee /dev/sdm1 | sha256sum


> I expect reading the code would tell.

My code analysis is in
  
https://lists.debian.org/msgid-search/1162291656137153...@scdbackup.webframe.org

to...@tuxteam.de found bug 155175 from 2002, which explains why. See
  https://lists.debian.org/msgid-search/zcdxzya0ayrmp...@tuxteam.de


Have a nice day :)

Thomas

Re: Fast Random Data Generation

[Thomas Schmitt]

Hi,

Linux-Fan wrote:
> I wrote a program to automatically generate random bytes in multiple
> threads:
> https://masysma.net/32/big4.xhtml
> ...
> || Wrote 102400 MiB in 13 s @ 7812.023 MiB/s

That's impressive.


> Secure Random can be obtained from OpenSSL:
>
> | $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 1024 
> * 1024)); done
> |
> | real  0m49.288s
> | user  0m44.710s
> | sys   0m4.579s
> Effectively 2078 MiB/s (quite OK for single-threaded operation).

Probably the best choice for sceptics who critizise a reproducible random
stream or mistrust random people's aptness to produce random.
(If the number of desired loops gets larger, then one could do arithmetik
 in a "while" loop instead of "for" and "seq".)


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

David Christensen wrote:
> Concurrency:
> threads throughput
> 8   205+198+180+195+205+184+184+189=1,540 MB/s

There remains the question how to join these streams without losing speed
in order to produce a single checksum. (Or one would have to divide the
target into 8 areas which get checked separately.)

Does this 8 thread generator cause any problems with the usability of
the rest of the system ? Sluggish program behavior or so ?

The main reason to have my own low-quality implementation of a random
stream was that /dev/urandom was too slow for 12x speed (= 1.8 MB/s) CD-RW
media and that higher random quality still put too much load on a
single-core 600 MHz Pentium system. That was nearly 25 years ago.


I wrote:
> > Last time i tested /dev/urandom it was much slower on comparable machines
> > and also became slower as the amount grew.

> Did you figure out why the Linux random number subsystem slowed, and at what
> amount?

No. I cannot even remember when and why i had reason to compare it with
my own stream generator. Maybe 5 or 10 years ago. The throughput was
more than 100 times better.


I have to correct my previous measurement on the 4 GHz Xeon, which was
made with a debuggable version of the program that produced the stream.
The production binary which is compiled with -O2 can write 2500 MB/s into
a pipe with a pacifier program which counts the data:

  $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | $(scdbackup -where bin)/raedchen -step 100m -no_output 
-print_count
  100.0g bytes

  real0m39.884s
  user0m30.629s
  sys 0m41.013s

(One would have to install scdbackup to reproduce this and to see raedchen
 count the bytes while spinning the classic SunOS boot wheel: |/-\|/-\|/-\
   http://scdbackup.webframe.org/main_eng.html
   http://scdbackup.webframe.org/examples.html
 Oh nostalgy ...
)

Totally useless but yielding nearly 4000 MB/s:

  $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni >/dev/null

  real0m27.064s
  user0m23.433s
  sys 0m3.646s

The main bottleneck in my proposal would be the checksummer:

  $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | md5sum
  5a6ba41c2c18423fa33355005445c183  -

  real2m8.160s
  user2m25.599s
  sys 0m22.663s

That's quite exactly 800 MiB/s ~= 6.7 Gbps.
Still good enough for vanilla USB-3 with a fast SSD, i'd say.


> TIMTOWTDI.  :-)

Looks like another example of a weak random stream. :))


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

David Christensen wrote:
> $ time dd if=/dev/urandom bs=8K count=128K | wc -c
> [...]
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.30652 s, 249 MB/s

This looks good enough for practical use on spinning rust and slow SSD.

Maybe the "wc" pipe slows it down ?
... not much on 4 GHz Xeon with Debian 11:

  $ time dd if=/dev/urandom bs=8K count=128K | wc -c
  ...
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.13074 s, 260 MB/s
  $ time dd if=/dev/urandom bs=8K count=128K of=/dev/null
  ...
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.95569 s, 271 MB/s

Last time i tested /dev/urandom it was much slower on comparable machines
and also became slower as the amount grew.

Therefore i still have my amateur RNG which works with a little bit of
MD5 and a lot of EXOR. It produces about 1100 MiB/s on the 4 GHz machine.
No cryptographical strength, but chaotic enough to avoid any systematic
pattern which could be recognized by a cheater and represented with
some high compression factor.
The original purpose was to avoid any systematic interference with the
encoding of data blocks on optical media.

I am sure there are faster RNGs around with better random quality.


> $ time perl -MMath::Random::ISAAC::XS -e 
> '$i=Math::Random::ISAAC::XS->new(12345678); print pack 'L',$i->irand while 1' 
> | dd bs=8K count=128K | wc -c
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 82.6523 s, 13.0 MB/s

Now that's merely sufficient for shredding the content of a BD-RE medium
or a slow USB stick.


> I suggest using /dev/urandom and tee(1) to write the same CSPRN
> stream to all of the devices and using cmp(1) to validate.

I'd propose to use a checksummer like md5sum or sha256sum instead of cmp:

 $random_generator | tee $target | $checksummer
 dd if=$target bs=... count=... | $checksummer

This way one can use unreproducible random streams and does not have to
store the whole stream on a reliable device for comparison.


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

i wrote:
> > In the other thread about the /dev/sdm test:
Gene Heskett wrote:
> > > Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
> > > [...]
> > > $ sudo f3probe --destructive --time-ops /dev/sdm
> > > Bad news: The device `/dev/sdm' is a counterfeit of type limbo
> > > Device geometry:
> > >  *Usable* size: 59.15 GB (124050944 blocks)
> > > Announced size: 1.91 TB (409600 blocks)

David Christensen wrote:
> Which other thread?  Please provide a URL to archived post.

https://lists.debian.org/msgid-search/e7a0c1a1-f973-4007-a86d-8d91d8d91...@shentel.net
https://lists.debian.org/msgid-search/b566504b-5677-4d84-b5b3-b0de63230...@shentel.net


> So, the 2 TB USB SSD's are really scam 64 GB devices?

The manufacturer would probably state that it's no intentional scam but
just poor product quality. (Exploiting Hanlon's Razor.)

It might be intentional that the write speed gets slower and slower as
the end of the usable area is approached. This avoids the need for
confessing the failure to the operating system.
But it might also be an honest attempt of the disk's firmware to find
some blocks which can take and give back the arriving data.


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

gene heskett wrote:
> my fading eyesight couldn't see
> the diffs between () and {} in a 6 point font.  I need a bigger, more
> legible font in t-bird.

That's why i propose to copy+paste problematic command lines.

Your mouse can read it, your mail client can send it, and we have
youngsters here of merely 60 years who will be glad to tell our
most senior regular why the lines don't work.

With some luck this creates nostalgic tangents about how the used features
evolved since the early 1980s.


In the other thread about the /dev/sdm test:
> Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
> but is taking a few bytes now and then.
> [...]
> $ ls -l
> total 40627044
> [...]
> $ sudo f3probe --destructive --time-ops /dev/sdm
> Bad news: The device `/dev/sdm' is a counterfeit of type limbo
> Device geometry:
> *Usable* size: 59.15 GB (124050944 blocks)
>Announced size: 1.91 TB (409600 blocks)

That's really barefaced.
How can the seller believe to get away with a problem which will show
already after a few dozen GB were written ?


> Probe time: 2.07s

That's indeed a quick diagnosis. Congrats to the developers of f3probe.


Have a nice day :)

Thomas

Re: shred bug?

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Ah, it seems to be this one, from 2002:
>  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175

So it's not a bug but a feature. :(

I'm riddling over the code about the connection to an old graphics
algorithm (Bresenham's Algorithm) and how shred produces a random pattern
at all.


Have a nice day :)

Thomas

Re: shred bug?

[Thomas Schmitt]

Hi,

i wrote:
> >   shred: -: invalid file type

to...@tuxteam.de wrote:
> Hmm. This looks like a genuine bug: the man page mentions it.

Even the help text in
  https://sources.debian.org/src/coreutils/9.4-3/src/shred.c/
says
  If FILE is -, shred standard output.

The name "-" is recognized in line 1257 and leads to a call of wipefd()
in line 958. The error messages there look different from above.
So i hop from line 973 to do_wipefd() and find the message in line 845.
fd is supposed to be 1 (= stdout). fstat(2) was successful but now this
condition snaps:

  if ((S_ISCHR (st.st_mode) && isatty (fd))
  || S_ISFIFO (st.st_mode)
  || S_ISSOCK (st.st_mode))

The problem seems to be in the S_ISFIFO part.
In a little test program fstat() reports about fd==1:
  st.st_mode= 010600 , S_ISFIFO(st.st_mode)= 1
(st_mode shown in octal as in man 2 fstat.)

It looks like the test expects a pipe(2) file descriptor to be
classified as S_ISCHR and !isatty().
Without redirection through a pipe, fd 1 has st.st_mode 20620, S_ISCHR,
and isatty()==1. The isatty() result is indeed a good reason, not to
flood stdout by a zillion random bytes.


Does anybody have an old GNU/Linux system where a file descriptor from
pipe(2) is classified as character device (S_IFCHR, S_ISCHR) ?

Does anybody have an idea why shred would want to exclude fifos ?
(What ciould shred do with a non-tty character device that cannot be done
 with a fifo ?)


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

Gene Heskett wrote:
> Is bash not actually bash these days? It is not doing for loops for me.

Come on Gene, be no sophie. Copy+paste your failing line here. :))


IIRC the for-loop in question writes several copies of the same file.
( https://lists.debian.org/debian-user/2024/02/msg00318.html )
Others already pointed out that this invites for firmware scams like
deduplication or silent overwriting of older data.

I would vote for a filesystem killer like
  shred -n 1 -s 204768K -v - | tee /dev/sdm1 | sha256sum
  dd if=/dev/sdm1 bs=32K count=6399 skip=32 | sha256sum

But shred(1) on Debian 11 refuses on "-" contrary to its documentation:
  shred: -: invalid file type
A non-existing file path causes "No such file or directory".


The filesystem killer aspect could be removed by creating large data files
in the readily partitioned and formatted filesystems of the disk.
(Replace "/dev/sdm1" by "/where/mounted/random_test_file" and reduce the
 numbers 204768K and bs=32K count=6399 to what you expect to
 fit into the filesystem. Like 20K and bs=32K count=6250.
 Ask df for storage capacities.)

But there needs to be a fast pseudo-random byte stream (which shred would
provide if i could talk it into writing to stdout) which you can split
for the device and for sha256sum.
I have an own weak-random generator, but shred beats it by a factor of 10
when writing to /dev/null.


Have a nice day :)

Thomas

Re: Generic Linux / clib question

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> > The readline library is released under the full GPL, not the LGPL.  If
> > you dynamically link it with a program, then you can only release that
> > program under terms compatible with the GPL.  This is an intentional
> > choice.

to...@tuxteam.de wrote:
> Practically it doesn't change much, but for some lawyer at Intel it might.

A substantial problem can be that libreadline became "GPLv3 or later" a
few years ago and that GPLv3 is not compatible with GPLv2-only (i.e. not
"or later"). The main reason is in the patent-fighting restrictions in
GPLv3, which would be forbidden extra restrictions under GPLv2.

So at best this causes "GPLv2 or later" programs to become "GPLv3 or later",
like the Debian binary of xorriso. At worst you would have a license clash
and would have to resort to e.g. libedit. (Not what the FSF would like.)


> The gist is that, if the user is the one doing the last linking step,
> all is fine. She's allowed anything.

She's allowed to use it but not to give it to others without obeying the
demands and restrictions of GPL.

Personally i would be ok with a more liberal use of my GPL'ed software.
But the official stance of the FSF is that the license applies on first
contact. In case of libreadline this would the line
  #include 
and as next occasion the preparations of the compiler and linker to employ
libreadline.so at run time.


Have a nice day :)

Thomas

Re: Generic Linux / clib question

[Thomas Schmitt]

Hi,

Van Snyder wrote:
> Years ago, I knew the name of the routines one could use to have some stdin
> history and be able to edit it, like you can do in XTerm or gnuplot or 

Sounds like readline:
  https://en.wikipedia.org/wiki/GNU_Readline
  https://tracker.debian.org/pkg/readline

An alternative with BSD license is libedit:
  https://tracker.debian.org/pkg/libedit


Have a nice day :)

Thomas

Re: Things I don't touch with a 3.048m barge pole

[Thomas Schmitt]

Hi,

gene heskett wrote:
> GPT fdisk (gdisk) version 1.0.9
>
> Partition table scan:
>   MBR: MBR only
> [...]
> Found invalid GPT and valid MBR; converting MBR to GPT format
> in memory.
> [...]
> Warning! Secondary partition table overlaps the last partition by 33 blocks!
> You will need to delete this partition or resize it in another utility.
> [...]
> Disk /dev/sdm: 409600 sectors, 1.9 TiB
> Number  Start (sector)End (sector)  Size   Code  Name
>1  64  409599   1.9 TiB 0700  Microsoft basic data

It wants to say "found no GPT at all and cannot easily convert the MBR
partition table".


> What do we make of that?  Some sort of NTFS?

NTFS would be a possible candidate for the filesystem inside the
partition. What does this command report:

  file -s /dev/sdm1


But the complaint from gdisk is just about an MBR partitioned stick with
no unclaimed room at the end for storing the GPT backup table.
Indeed if you want to convert the MBR partition table to GPT you will have
to delete or shrink the single partition number 1. (Shrinking would
of course depend on what kind of filesystem is in the partition and
whether Debian has shrink software for it.)
The same will be necessary if you want more than one partition.

Whatever, there is nothing suspicious about the gdisk behavior with that
particular disk.


Have a nice day :)

Thomas

Re: How can we change the keyboard layout?

[Thomas Schmitt]

Hi,

Loris Bennett wrote:
> As many have pointed out, it is short for 'Steuerung', but I have met
> many Germans who refer to this key as 'String'.  I am not sure why

BASIC ?

Or the popular bundle theory:
[Strg] (= [Ctrl]) means "String",
[AltGr] (= right side [Alt]) means "Altgriechisch" (= ancient greek),
[Entf] (= [Delete]) means "Entfetten" (= degrease).


Have a nice day :)

Thomas

Re: about 64bits time_t transition and deborphan

[Thomas Schmitt]

Hi,

Patrice Duroux wrote:
> Out of curiosity, I started this transition on some packages from
> experimental and I observed that deborphan is not without
> «disruption».
> [...]
> Is this something to be reported to deborphan as it could also be in
> some other cases than this time_t transition? or wait and see... ;-)

I guess "wait and see" for now.

The whole operation seems to be still in progress and leaves traces
in the package tracker. See e.g. section "testing migrations" in
  https://tracker.debian.org/pkg/libisoburn

The associated mailing list got notifications as bug reports 1062380 for
libisoburn and 1062381 for libisofs on february 1st. Yesterday came
bug report 1063123 for libburn.
I am curious how this will unfold further. Especially whether the new
binary package name "libisoburn1t64" will persist for future releases.


Have a nice day :)

Thomas

Re: install Kernel and GRUB in chroot.

[Thomas Schmitt]

Hi,

Dmitry wrote:
> Yep. `dd` copy partitions table. Amazing.

Not so amazing after you realize that a partition table is just data on
the storage medium and not some special property of the storage device.
dd copies data. If these data contain a partition table and get copied to
the right place on the storage medium, the partition table will be
recognized by EFI and Linux.


> applies no 'intelligence' to the operation.

This describes it very well. Sometimes dumb is good. Sometimes not.


Initially you stated in
  https://lists.debian.org/debian-user/2024/02/msg8.html
>...> I need to prepare that system for booting.
>...> 1. Install Kernel.
>...> 2. Install GRUB and Configure.
>...> 3. Add changes to UEFI to start booting.

dd-ing a bootable Debian ISO will not do what you describe.
Assumed the ISO is prepared for booting from USB stick, you will get a
bootable Live or an installer system. At least ISOs for i386, amd64, and
arm64 should be prepared for that.

If it is not ready for booting from USB stick, it will be just a storage
with a mountable filesystem and Debian files in it.


Max Nikulin wrote:
> > Just copy files from LiveCD (it should have EFI/Boot/bootx64.efi)
> > to the ESP partition on the USB stick.

The /EFI/boot directory of a bootable Debian ISO usually does not contain
the full GRUB equipment for EFI. Important parts of an amd64 Live ISO are
in /boot/grub.
The programs in /EFI/boot are specialized on convincing Secure Boot and
on finding the ISO filesystem with /boot/grub in it. (Actually those are
copies of the EFI boot partition files. The boot partition is a FAT
filesystem image file inside the ISO named /boot/grub/efi.img .)


Tim Woodall wrote:
> > I'm not exactly sure what you're doing.

I join this statement. :))

Do you want a normal changeable Debian system installation or do you want
a Live system with its immutable core and maybe some partition where you
can store files ?
(Just curiosity of mine. Possibly i could not help much with chroot
questions anyway.)


Have a nice day :)

Thomas

Bug#1062380: libisoburn: NMU diff for 64-bit time_t transition

[Thomas Schmitt]

Hi,

i'm the upstream and help with preparing the Debian packages.
So i assume that any needed action in this case is up to my sponsor
Dominique Dumont (Cc'd).

But:

Are you aware that the effort to switch to 64 bit time_t is not worth much
in a ISO 9660 producing software, even on amd64, as long as

  https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h

has int as return type of the ISO-9660-to-Unix-time function

  extern int iso_date(u8 *, int);

(implemented in fs/isofs/util.c) ?

I once made a successful attempt to get the attention of parts of the
kernel community:
  
https://lore.kernel.org/all/20221024135517.7d40b96a206020eca03e3...@linux-foundation.org/t/
But as it looks, it never made it into the torvalds repo.


Have a nice day :)

Thomas

Bug#1062380: libisoburn: NMU diff for 64-bit time_t transition

[Thomas Schmitt]

Hi,

i'm the upstream and help with preparing the Debian packages.
So i assume that any needed action in this case is up to my sponsor
Dominique Dumont (Cc'd).

But:

Are you aware that the effort to switch to 64 bit time_t is not worth much
in a ISO 9660 producing software, even on amd64, as long as

  https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h

has int as return type of the ISO-9660-to-Unix-time function

  extern int iso_date(u8 *, int);

(implemented in fs/isofs/util.c) ?

I once made a successful attempt to get the attention of parts of the
kernel community:
  
https://lore.kernel.org/all/20221024135517.7d40b96a206020eca03e3...@linux-foundation.org/t/
But as it looks, it never made it into the torvalds repo.


Have a nice day :)

Thomas

Bug#1062381: libisofs: NMU diff for 64-bit time_t transition

[Thomas Schmitt]

Hi,

i'm the upstream and help with preparing the Debian packages.
So i assume that any needed action in this case is up to my sponsor
Dominique Dumont (Cc'd).

But:

Are you aware that the effort to switch to 64 bit time_t is not worth much
in a ISO 9660 producing software, even on amd64, as long as

  https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h

has int as return type of the ISO-9660-to-Unix-time function

  extern int iso_date(u8 *, int);

(implemented in fs/isofs/util.c) ?

I once made a successful attempt to get the attention of parts of the
kernel community:
  
https://lore.kernel.org/all/20221024135517.7d40b96a206020eca03e3...@linux-foundation.org/t/
But as it looks, it never made it into the torvalds repo.


Have a nice day :)

Thomas

Bug#1062381: libisofs: NMU diff for 64-bit time_t transition

[Thomas Schmitt]

Hi,

i'm the upstream and help with preparing the Debian packages.
So i assume that any needed action in this case is up to my sponsor
Dominique Dumont (Cc'd).

But:

Are you aware that the effort to switch to 64 bit time_t is not worth much
in a ISO 9660 producing software, even on amd64, as long as

  https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h

has int as return type of the ISO-9660-to-Unix-time function

  extern int iso_date(u8 *, int);

(implemented in fs/isofs/util.c) ?

I once made a successful attempt to get the attention of parts of the
kernel community:
  
https://lore.kernel.org/all/20221024135517.7d40b96a206020eca03e3...@linux-foundation.org/t/
But as it looks, it never made it into the torvalds repo.


Have a nice day :)

Thomas

Re: [Libcdio-devel] [PATCH v3 0/4] Add El Torito virtual image support

[Thomas Schmitt via Libcdio-devel]

Hi,

the v3 patches apply, but with number 4 there are still "git am"
complaints:

  Applying: Add El Torito tests
  /home/stacker/libcdio_dir/libcdio.rocky/.git/rebase-apply/patch:69: trailing 
whitespace.
  # Tests El Torito (via iso-info and iso-read).
  /home/stacker/libcdio_dir/libcdio.rocky/.git/rebase-apply/patch:158: new 
blank line at EOF.
  +
  /home/stacker/libcdio_dir/libcdio.rocky/.git/rebase-apply/patch:172: new 
blank line at EOF.
  +
  warning: 3 lines add whitespace errors.

I have no big-endian system at hand, but the provisions for them
look good to me (assuming the functions do indeed what their names say):

  + p_iso->boot_img[k].lsn = uint32_from_le(br[j].image_lsn);
  + p_iso->boot_img[k++].num_sectors = 
uint16_from_le(br[j].num_sectors);


To make some realistic test i tried ./src/iso-info with the three ISO
images which came today with the DELUG DVD of german Linux Magazin.
(All three were made by xorriso, while the DVD ISO was made by DELUG by
genisoimage):

alpine-standard-3.19.0-x86_64.iso:
  /[BOOT]/:
- [LSN 56]  2048 Dec 04 2023 21:01:51  0-Boot-NoEmul.img
- [LSN 77]   1474560 Dec 04 2023 21:01:51  1-Boot-NoEmul.img

kali-linux-2023.4-live-amd64.iso:
  /[BOOT]/:
- [LSN   2519]  2048 Nov 30 2023 11:21:39  0-Boot-NoEmul.img
- [LSN   2055]950272 Nov 30 2023 11:21:39  1-Boot-NoEmul.img

proxmox-backup-server_3.1-1.iso:
  /[boot]/:
- [LSN   5768]  2048 Nov 30 2023 11:19:33  0-boot-noemul.img
- [LSN108]   8388608 Nov 30 2023 11:19:33  1-boot-noemul.img

The latter shows lowercase names although the ISO contains Rock Ridge
information. Does its lack of Joliet cause the lowercase ?

Further i looked into the OpenSUSE ISO, which bears unlimited boot image
size:

openSUSE-13.1-NET-x86_64.iso:
  /[BOOT]/:
- [LSN   4238]  2048 Nov 06 2013 16:27:21  0-Boot-NoEmul.img
- [LSN 70]   512 Nov 06 2013 16:27:21  1-Boot-NoEmul.img

The size of 512 bytes is unfortunate. But it is what Pete announced to
implement.
The ISO stems from mkisofs or genisoimage: Rock Ridge NM comes before PX.
(The load size 1 was not caused by oversize but rather looks deliberate.)


So modulo my known objections, this patch set seems to implement what was
announced as goals.


Have a nice day :)

Thomas

Re: [Libcdio-devel] [PATCH v2 0/4] Add El Torito virtual image support

[Thomas Schmitt via Libcdio-devel]

Hi,

Pete Batard wrote:
> If you are
> going to set a EFI partition to a size of 0 or 1, then the image will occupy
> all sectors from the LSN to the end of the volume.

I understand that this was specified as a last resort because
- UEFI did not want to invent a new competitor to El Torito,
- the El Torito boot catalog only has 16 bits for the field which
  describes the load size,
- and UEFI found no other field to express the size of the EFI System
  Partition.


> Therefore, on systems that comply with the UEFI specs, that
> 1-boot-noemul.img image at sector 34 will actually include the data from the
> 0-boot-noemul.img at LSN 20514, and the size of the image that the system
> will process will be further extended by 2048 bytes... which makes me think
> that xorriso is somewhat abusing the UEFI specs.

The specs do not prescribe a particular sequence of boot images or their
position within the overall body of an ISO 9660 filesystem.

Sure, it would be better to have the oversized EFI image near the end of
the filesystem data range. This would reduce the trailing backpack that is
caused by the fallback prescription in UEFI specs.
But achieving that position would be up to the ISO producer. The reader
must not flatly assume it.


> Especially, as opposed to what Thomas stated in his other reply, nowhere
> does it states that: "The real size is the size of the FAT filesystem in
> such an image". Instead, UEFI is quite explicit that the real size is the
> size from the LSN to the end of the volume.

The size that matters is indeed the size of the FAT filesystem which
starts at the LBA which is given in the boot catalog.
The size which is spedified in UEFI for values 0 or 1 is just a coarse
workaround for a situation where the catalog format is insufficient and
where any EFI firmware can get a better size by looking into the FAT
filesystem.

Actually the exact partition size does not matter for a filesystem driver
as long as the partition is not smaller than the file system.


> So I'm going to ask Thomas, though I suspect the answer is "ISOHybrid
> tweaking", is: Why is xorriso is placing the images that way?

No isohybrid is involved in the proposed xorriso run.

The immediate reason for the sequence of file data content is in the
alphabetic order of the paths in the ISO:
/efi.img comes before /isolinux.bin .

(Reason is that ISO 9660 prescribes alphabetically sorted file names
directories. Putting the content in the same sequence gives fewer moves
of the CD read head when copying out the files in the sequence given in
the directory. Fewer moves = fewer clonks and better copy throughput.)

It is possible to influence this content sequence by the -as mkisofs
option --sort-weight. If you attribute a low weight to /efi.img then its
content will swim up to the end of the resulting filesystem.

  $ xorriso ...options.used.in.40.MB.example... \
--sort-weight -10 /efi.img
  ...
  $ xorriso -indev test.iso -report_el_torito plain
  ...
  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
  El Torito boot img :   1  BIOS  y   none  0x  0x00  4  34
  El Torito boot img :   2  UEFI  y   none  0x  0x00  0  54
  El Torito img path :   1  /isolinux.bin
  El Torito img opts :   1  boot-info-table
  El Torito img path :   2  /efi.img

But it is tradition among Linux installation ISO producers to apply a
high weight to boot images in order to get their content to low block
addresses.
So expect the whole bulk of data file content to come after the content
of the boot images.

(Reason is probably an old bug of ISOLINUX which cleared a processor
register by 16 bit before it gets used as 32 bit. Previously the register
held the address of the legacy boot image. If that address is lower than
65536, then the high bits of the register are already 0 and nothing bad
happens.
Thus grow our urban legends and will influence the properties of ISO
filesystems long after the bug in ISOLINUX was fixed.)


> also have to state that modifying the current proposal to handle xorriso's
> way of placing the ESP is going to be problematic because for one thing,

The sequence of data files is by default the same as in mkisofs or
genisoimage:

  $ genisoimage -v -o test.iso isolinux.bin efi.img
  ...
  $ xorriso -indev test.iso -find / -exec report_lba --
  ...
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  File data lba:  0 ,   24 ,20480 , 41943040 , '/EFI.IMG'
  File data lba:  0 ,20504 ,   20 ,40960 , '/ISOLINUX.BIN'

Note the address numbers under "Startlba".


> so we're going to have to craft a new codepath
> for the non-compliant xorriso layout,

You just have to drop the assumption about the convenience of the position
of the EFI System Partition.


> I am very much trying to
> avoid having to sink time into full blown processing of platform id

I wonder why you are so opposed to expanding the if-construct

+   

Re: [Libcdio-devel] [PATCH v2 0/4] Add El Torito virtual image support

[Thomas Schmitt via Libcdio-devel]

Hi,

Pete Batard wrote:
> I'll just add that I didn't formally test this, since I haven't had a chance
> to locate an ISO with a > 32MB El-Torito image yet.

You can build yourself one:

- Get legacy PC-BIOS boot image isolinux.bin from e.g. a Debian netinst
  ISO:

xorriso -osirrox on -indev debian-12.2.0-amd64-netinst.iso \
-extract /isolinux/isolinux.bin isolinux.bin

  Or just make a fake file:

dd if=/dev/zero bs=1K count=40 of=isolinux.bin


- Make yourself a FAT filesystem image >= 32 MiB or just some large dummy
  file named efi.img :

dd if=/dev/zero bs=1M count=40 of=efi.img


- Pack up as bootable ISO for legacy and for EFI:

xorriso -as mkisofs -v -o test.iso \
-c /boot.cat \
-b /isolinux.bin \
   -no-emul-boot -boot-info-table -boot-load-size 4 \
-eltorito-alt-boot
-e /efi.img -no-emul-boot \
./isolinux.bin ./efi.img

  With contemporary xorriso expect messages like

  xorriso : WARNING : Boot image load size exceeds 65535 blocks of 512 bytes. 
Will record 0 in El Torito to extend ESP to end-of-medium.
  xorriso : WARNING : EFI boot equipment is provided but no directory /EFI/BOOT
  xorriso : WARNING : will emerge in the ISO filesystem. A popular method to
  xorriso : WARNING : prepare a USB stick on MS-Windows relies on having in the
  xorriso : WARNING : ISO filesystem a copy of the EFI System Partition tree.
  ISO image produced: 20683 sectors

  If you do not see the warning about "size exceeds 65535 blocks" then
  your xorriso might be too old for the purpose.
  (https://www.gnu.org/software/xorriso/xorriso-1.5.6.pl02.tar.gz
   is the current version, easy to compile, and works without system-wide
   installation.)

---

The result is supposed to be reported by:

$ xorriso -indev test.iso -report_el_torito plain

as:

El Torito cat path : /boot.cat
El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
El Torito boot img :   1  BIOS  y   none  0x  0x00  4   20514
El Torito boot img :   2  UEFI  y   none  0x  0x00  0  34
El Torito img path :   1  /isolinux.bin
El Torito img opts :   1  boot-info-table isohybrid-suitable
El Torito img path :   2  /efi.img


Have a nice day :)

Thomas

Re: [Libcdio-devel] [PATCH 1/4] Add case insensitive _cdio_stricmp and _cdio_strnicmp function calls

[Thomas Schmitt via Libcdio-devel]

Hi,

Rocky Bernstein wrote:
> Overall, I'd like to get those patches without any controversy into the
> code soon. I am not 100% certain which ones have disputes in them. Is it
> just Patch 2?

Summary of everything that comes to my mind with v1. I read over v2 and
find not much changes in regard to my observations.

Pick what you deem important:

--
[PATCH 1/4] Add case insensitive _cdio_stricmp and _cdio_strnicmp function
calls:

- I have no objections. (But did not test explicitely beyond "git am"
  and "make")


--
[PATCH 2/4] Add El Torito virtual boot image support:

- git complaint v1:
$ git am  if Thomas you have the
> inclination and time to make more perfect, that can be done as a follow on.

Some of my objections are about design decisions. Pete strives for
compatibility with 7z rather than for exposing all El Torito aspects.
Further the patch knowingly risks wrong results.
So we would first have to find some more common ground before it makes
sense that i do more than proof-reading.


Have a nice day :)

Thomas

Re: AW: AW: su su- sudo dont work

[Thomas Schmitt]

Hi,

Andy Smith wrote:
> It is hard to understand how what Michael/Sophie/Tobias does can in
> any way be "fun" for them, though maybe that is just our lack of
> understanding.

I expressed my suspicion of a "Hurz" performance in
  https://lists.debian.org/debian-user/2023/05/msg00100.html


Have a nice day :)

Thomas

Re: Automatically installing GRUB on multiple drives

[Thomas Schmitt]

Hi,

Nicolas George wrote:
> You seem to be assuming that the system will first check sector 0 to
> parse the MBR and then, if the MBR declares a GPT sector try to use the
> GPT.

That's what the UEFI specs prescribe. GPT is defined by UEFI-2.8 in
chapter 5 "GUID Partition Table (GPT) Disk Layout". Especially:

  5.2.3 Protective MBR
  For a bootable disk, a Protective MBR must be located at LBA 0 (i.e.,
  the first logical block) of the disk if it is using the GPT disk layout.
  The Protective MBR precedes the GUID Partition Table Header to maintain
  compatibility with existing tools that do not understand GPT partition
  structures.


> I think it is the other way around on modern systems: it will first
> check sector 1 for a GPT header, and only if it fails check sector 0.

Given the creativity of firmware programmers, this is not impossible.
At least the programmers of older versions of OVMF took the presence of
a GPT header as reason to boot, whereas without it did not boot.
Meanwhile this demand for GPT debris has vanished and OVMF boots from
media with only MBR partitions, too.


I wrote:
> > This layout [used by Debian installation ISOs] was invented by Matthew
> > J. Garrett for Fedora

> I think I invented independently something similar.
> https://www.normalesup.org/~george/comp/live_iso_usb/grub_hybrid.html

Not to forget Vladimir Serbinenko who specified how a grub-mkrescue ISO
shall present its lures for BIOS and EFI on optical media and USB stick.
The ISO has a pure GPT partition table, where the ISO filesystem is not
mountable as partition but only by the base device (like /dev/sdc) or
possibly by its HFS+ directory tree via the Apple Partition Map, if
present.

(To create such an ISO, install grub-common, grub-efi-amd64, grub-efi-ia32,
and grub-pc. Then run grub-mkrescue with some dummy directory as payload.
The ISO will boot to a GRUB prompt.)


Have a nice day :)

Thomas

Re: Automatically installing GRUB on multiple drives

[Thomas Schmitt]

Hi,

i hate to put in question the benefit of my proposal, but:

Nicolas George wrote:
> The firmware would never write in parts of the
> drive that might contain data.

See

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
  "cdrom: Installation media changes after booting it"

Two occasions were shown in this bug where the EFI system partition of
a Debian installation ISO on USB stick changed. One was caused by a
Microsoft operating system, writing a file named WPSettings.dat. But the
other was from Lenovo firmware writing /efi/Lenovo/BIOS/SelfHealing.fd .

One may doubt that the success of these operations is desirable at all.
The ISO was also tested with a not-anymore-writable DVD. In that case the
Lenovo firmware did not raise protest over the fact that it was not
possible to write to the EFI partition.


Have a nice day :)

Thomas

Re: [Libcdio-devel] [PATCH 2/4] Add El Torito virtual boot image support

[Thomas Schmitt via Libcdio-devel]

Hi,

i wrote
> > [...] UEFI-2.8:
> >13.3.2.1 ISO-9660 and El Torito
> >[...]
> >If the value of Sector Count is set to 0 or 1, EFI will assume the
> >system partition consumes the space from the beginning of the
> >“no emulation” image to the end of the CD-ROM.

Peter Batard wrote:
> Do you know if the El Torito specs were ever updated to reflect this?

Not that i would have noticed. I still use
  “El Torito” Bootable CD-ROM Format Specification
  Version 1.0 January 25, 1995
  Phoenix (R) IBM

It wasn't updated for DVD or Blu-ray either. Everybody just extrapolated
the specs and was in sync with every other involved party.


> Hmmm. So we're going to decide that the chances of someone creating a 32MB -
> 512 image, because they read the El Torito specs and went for the maximum
> allowed value are lower than the chances of someone abusing the specs?

If you first verify that the boot image is for EFI, i would propose to
look into the FAT filesystem whenever the El Torito Load Size is smaller
than the smallest meaningful FAT filesystem or larger than 65532
(= 65536 - 4, because it is the smallest possible load size which is
divisible by 4).


> Or do we interpret the UEFI specs to mean that this only ever applies if the
> platform ID is set to 0xefi

This special meaning of 0 and 1 is only defined in UEFI. I am not aware
that it was ever interpreted that way in legacy BIOS.


>  and that, if platform ID is set to anything else,
> then 1 or 65535 are to be interpreted verbatim.

The problem is that the field in the El Torito catalog was defined in
1995 as:

  Sector Count. This is the number of virtual/emulated sectors the system
  will store at Load Segment during the initial boot procedure.

This definition does not describe the size of the boot image but only the
size of the data to be transferred from CD-ROM to RAM.
So BIOS programmers felt entitled to implement only the capability for a
small read, which then caused boot loader programmers to understate the
size of their no-emulation images. This understatement to size 4 usually
is accompanied by a boot info table.


Anecdote. Not a proposal:

In xorriso i make a different approach by estimating the boot image size
from other information in the ISO. Best is of course if the image is
also represented as data file in the ISO. Then i can take its file size.

Strange enough:
The EFI boot image of openSUSE-13.1-NET-x86_64.iso is also represented
as file /boot/x86_64/efi with just 4,194,304 bytes of size. So it would
not have been necessary to set the Load Size to unlimited.
The EFI image in slint64-current-goofiboot-1.4.0.iso is 64 MiB of size,
i.e. too big for the 16 bit field in the boot catalog.

More guesswork is needed with hidden boot images, where i look at the
start addresses of all known objects in the ISO (file or not) and assume
that the size reaches from the start LBA of the boot image to the start
of the next known object (e.g. the next boot image, the start of the
directory tree).

Hiding boot image files is not usual in GNU/Linux. But for example
Fedora-Server-dvd-x86_64-37-1.7.iso does so:

  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
  El Torito boot img :   1  BIOS  y   none  0x  0x00  4 562
  El Torito boot img :   2  UEFI  y   none  0x  0x00  20352 1195692
  El Torito img blks :   1  18
  El Torito img opts :   1  grub2-boot-info
  El Torito img blks :   2  5088

(The estimation and load size of the UEFI image matches: 4 * 5088 = 20352)

A prolific image hider is FreeBSD,
e.g. FreeBSD-12.0-RELEASE-amd64-bootonly.iso :

  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
  El Torito boot img :   1  BIOS  y   none  0x  0x00  4 420
  El Torito boot img :   2  UEFI  y   none  0x  0x00   1600  20
  El Torito img blks :   1  1204
  El Torito img blks :   2  400

(I suspect that 1204 * 2048 for the BIOS image is overestimated. But too
long is better than too short when you want to repack the ISO.)

End of anecdote.


> There was no need to throw a 1 in there, outside of trying to appease people
> who were already abusing the specs.

Well, probably some participants in the standardization process were for
size 0 and others for size 1. So they made a compromise.


> As far as I'm concerned, I think we need to handle 2 branches: verbatim for
> platform ID != 0xef and till the end of the media for platform ID == 0xef

So your read loop needs to recognize byte 0 with values 0x01, 0x90, 0x91
in the 32-byte catalog records. If found, the next byte tells the platform
id for the next boot image catalog records with byte 0 value 0x88. This
platform id stays in effect until a new platform id is encountered.

Do not assume that each section of the catalog has only one boot image
entry. The 

Re: [Libcdio-devel] [PATCH 2/4] Add El Torito virtual boot image support

[Thomas Schmitt via Libcdio-devel]

Hi,

something i forgot to mention yesterday:

  $ xorriso -indev openSUSE-13.1-NET-x86_64.iso -report_el_torito plain
  ...
  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
  El Torito boot img :   1  BIOS  y   none  0x  0x00  44238
  El Torito boot img :   2  UEFI  y   none  0x  0x00  1  70
  El Torito img path :   1  /boot/x86_64/loader/isolinux.bin
  El Torito img opts :   1  boot-info-table isohybrid-suitable
  El Torito img path :   2  /boot/x86_64/efi
  ...

The size of the EFI boot image is recorded in the boot catalog as 1, which
complies to UEFI-2.8:

  13.3.2.1 ISO-9660 and El Torito
  [...]
  If the value of Sector Count is set to 0 or 1, EFI will assume the
  system partition consumes the space from the beginning of the
  “no emulation” image to the end of the CD-ROM.

This is necessary if the EFI partition is larger than 65535 * 512 bytes
= 32 MiB - 512.

Some misguided ISO producers use 65535 instead of 0 or 1:

  slint64-current-goofiboot-1.4.0.iso

  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
  El Torito boot img :   1  BIOS  y   none  0x  0x00  4   32807
  El Torito boot img :   2  UEFI  y   none  0x  0x00  65535  39
  El Torito img path :   1  /isolinux/isolinux.bin
  El Torito img opts :   1  boot-info-table isohybrid-suitable
  El Torito img path :   2  /efi.img

(I take these examples out of the log of my regression test which repacks
about 200 GB of bootable ISOs whenever i change the boot related code in
libisofs.)


> so, on real-life El-Torito media, you will find:
> - [BOOT]/0-Boot-NoEmul.img (the initial 2048 byte loader)
> - [BOOT]/1-Boot-NoEmul.img (the rest of the image)

Urm, in the usual Linux distro installation ISO this will be the BIOS
boot image (shortened) and the EFI system partition image.

The short size of the BIOS image was chosen by the boot loader programmers
to circumvent a bug in old BIOSes, which refused to load any other size.
The rest of the BIOS image is directly following the first 2048 bytes
and is not announced by the boot catalog.
Its presence and size can only be told if the boot info table is read.


I understand your patch is only striving for EFI boot images.
That would be ok for me, if it would exclude other images from being
mapped to the pseudo-directory and if it would get the EFI image size
rules correctly.
(For being able to tell the intented firmware system, you'd need to
interpret the Validation Entry and the Section Header Entries for
platform ids.)


> Gotta love how on one hand I'm kind of being chastised for taking shortcuts
> with the specs, but, when I follow the specs exactly, that make absolutely
> zero mention about a lower limit for num_sectors, I'm being told about
> "tradition".

Hey, this is a technical discussion about El Torito and your patch.
No offense intended.

The existing habits around boot loaders are of importance. The world of
legacy BIOS nearly entirely consists of habits.


Have a nice day :)

Thomas

Re: Automatically installing GRUB on multiple drives

[Thomas Schmitt]

Hi,

Nicolas George wrote:
> Interesting. Indeed, “table-length: 4” causes sfdisk to only write 3
> sectors at the beginning and 2 at the end. I checked it really does not
> write elsewhere.
> That makes it possible to use full-disk RAID on a UEFI boot drive. Very
> good news.

\o/

(Nearly as good as Stefan Monnier's crystal ball. And that without
understanding the dirty details which cause the need for a small partition
table.)


> More and more firmwares will only boot with GPT. I think I met
> only once a firmware that booted UEFI, 32 bits, with a MBR

The Debian installation and live ISOs have MBR partitions with only a
flimsy echo of GPT. There is a GPT header block and an entries array.
But it does not get announced by a Protective MBR. Rather they have two
partitions of which one is meant to be invisible to EFI ("Empty") and
one is advertised as EFI partition:

$ /sbin/fdisk -l debian-12.2.0-amd64-netinst.iso
...
Disklabel type: dos
...
Device   Boot Start End Sectors  Size Id Type
debian-12.2.0-amd64-netinst.iso1 *0 1286143 1286144  628M  0 Empty
debian-12.2.0-amd64-netinst.iso2   4476   23451   18976  9.3M ef EFI (FAT-12

So any system which boots this ISO from USB stick does not rely on
the presence of a valid GPT.
(The only particular example of GPT addiction i know of are old versions
of OVMF, the EFI used by qemu, which wanted to see the GPT header block,
even without Protective MBR.)

This layout was invented by Matthew J. Garrett for Fedora and is still
the most bootable of all possible weird ways to present boot stuff for
legacy BIOS and EFI on USB stick in the same image. (There are mad legacy
BIOSes which hate EFI's demand for no MBR boot flag. Several distros
abandoned above layout in favor of plain MBR or plain GPT. The price
is that they have to leave behind some of the existing machines.)


> GPT
>  ├─EFI
>  └─RAID
> └─LVM (of course)
>
> Now, thanks to you, I know I can do:
>
> GPT
>  ┊  RAID
>  └───┤
>  ├─EFI
>  └─LVM

Ah. Now i understand how accidentially useful my technical nitpicking was.
(A consequence of me playing Dr. Pol with the arm in the ISO 9660 cow
up to my shoulder.)


Have a nice day :)

Thomas

Re: [Libcdio-devel] [PATCH 2/4] Add El Torito virtual boot image support

[Thomas Schmitt via Libcdio-devel]

Hi,

Pete Batard wrote:
> Enables the El Torito boot images to be listed and extracted from a virtual
> "[BOOT]/" root directory.

In [PATCH 3/4]:
> +"  --no-el-toritoDon't use El-Torito extension information\n"

Maybe you should not make it default but enable it only on demand ?
Then people would not be surprised by files which do not show up in the
mounted filesystem.


Some nitpicking about [PATCH 2/4]:

> +  \brief ISO-9660 Boot Record Volume Descriptor.
> + */
> +struct iso9660_br_s {
> +uint8_t  boot_id;  /**< Boot indicator - 
> 0x88 */

This looks like the Initial/Default Entry of the boot catalog, not
like a "ISO-9660 Boot Record Volume Descriptor".

(I wonder how you represent the other record types of the boot catalog:
Validation Entry, Section Header, Section Entry, Section Entry Extension.)


> +   iso9660_br_t br[ISO_BLOCKSIZE / sizeof(iso9660_br_t)];
> +   if (iso9660_iso_seek_read(p_iso, , p_brvd->boot_catalog_sector, 1) 
> == ISO_BLOCKSIZE) {
> + for (j = 0, k = 0;
> +  j < (ISO_BLOCKSIZE / sizeof(iso9660_br_t)) && k < 
> MAX_BOOT_IMAGES;
> +  j++) {
> +   if (br[j].boot_id == 0x88 && br[j].media_type == 0) {
> + p_iso->boot_img[k].lsn = br[j].image_lsn;
> + p_iso->boot_img[k++].num_sectors = br[j].num_sectors;
> +   }
> + }

This works only by the fact that the El torito specs prescribe the
values 0x90 and 0x91 for byte 0 of Section Header Entries, value 1 for
byte 0 of the Validation Entry, value 0x44 for Section Entry Extension,
and that Section Entry has nearly the same layout as Initial/Default
Entry. All have 32 bytes of size, to your luck.

The loop reads possibly into an undefined remainder of the first catalog
block and relies on no finding no trailing garbage there.

You should at least install a warning sign, that this reading procedure
only vaguely reflects the specs.


In the light of the reading loop, i revisit these previous lines:
> +  struct {
> +uint32_t lsn;  /**< Start LSN of an El-Torito bootable image */
> +uint32_t num_sectors;   /**< Number of virtual sectors occupied by the
> +   bootable image */
> +  } boot_img[MAX_BOOT_IMAGES];

Be aware that especially with no-emulation boot images for legacy BIOS
it is tradition to set num_sectors to 4 (= 1 CD-ROM sector) and to let
these first 2048 bytes of code load the rest of the boot image as code.
The boot info table tells this first code stage where to find itself and
the rest of the boot program.

The ISO producing program writes this table into the boot image file.
ISOLINUX and GRUB boot images expect it.
Regrettably there is no defined indication for the presence of a boot info
table. One could take matching file_lba iand checksum as indication (but
should be careful not to read too much due to a bogus file_len if it is
not a boot info table).
>From libisofs/boot_sectors.txt:

The Boot Info Table begins at byte 8 of the boot image content.

Byte Range | Value  | Meaning
-- | -- | 
   8 -  11 |pvd_lba | Block address of the Primary Volume Descriptor.
   || This is the session start LBA + 16.
   ||
  12 -  15 |   file_lba | Block address of the start of the boot image file
   || content. Block size is 2048.
   ||
  16 -  19 |   file_len | Number of bytes in boot image file content.
   ||
  20 -  23 |   checksum | The sum of all 32-bit words of the file content
   || from byte 64 to file end.
   ||
  24 -  63 |  0 | Reserved
-- | -- | 
All numbers are stored little-endian.



> +if ((_cdio_strnicmp(path, "[BOOT]/", 7) == 0) &&
> +   (_cdio_stricmp([8], "-Boot-NoEmul.img") == 0)) {
> +  int index = path[7] - '0';

I think it would be valuable if the file name would reflect the platform
id from byte 1 of Validation Entry (byte0 == 1) and Section Header Entry
(byte0 == 0x90 or byte0 == 0x91):
0= x86 legacy PC-BIOS (no emulation = plain x86 machine code)
1= PowerPC
2= Mac
 0xef= EFI (always no emulation, FAT filesytem image)

Recognizing Section Header Entries would be valuable also because they
tell the number of following Section Entries in bytes 2 and 3 and they
tell whether another Section Header Entry will follow (byte0 = 0x90)
or whether this is the last one (byte0 = 0x91).
So you won't read into undefined territory.

The specs prescribe this catalog structure:
- 1 Validation Entry (byte0 == 1, byte14=0x55 , byte15= 0xAA),
  tells platform id
- 1 Initial/Default Entry (byte0 == 

Re: Automatically installing GRUB on multiple drives

[Thomas Schmitt]

Hi,

i cannot make qualified proposals for the GRUB question, but stumble over
your technical statements.

Nicolas George wrote:
> Since mdadm can only put its superblock at the end of the device (1.0),
> at the beginning of the device (1.1) and 4 Ko from the beginning (1.2),
> but they still have not invented 1.3 to have the metadata 17 Ko from the
> beginning or the end, which would be necessary to be compatible with
> GPT,

Although it would be unusually small, it is possible to have a GPT of
only 4 KiB of size:
- 512 bytes for Protective MBR (the magic number of GPT)
- 512 bytes for the GPT header block
- 3 KiB for an array of 24 partition entries.

Question is of course, whether any partition editor is willing to create
such a small GPT. The internet says that sfdisk has "table-length" among
its input "Header lines". So it would be a matter of learning and
experimenting.
(Possibly i would be faster with writing the first header blocks by
hand, following UEFI specs or my cheat sheet in libisofs.)


> we have to partition them and put the EFI system partition outside
> them.

Do you mean you partition them DOS-style ?
If so, then a partition of type 0xEF could be used as system partition.
Probably any partition type will do, because EFI is very eager to look
into any partition with FAT filesystem.


Have a nice day :)

Thomas

Re: Looking for archive management system for backups burned to optical discs

[Thomas Schmitt]

Hi,

David Christensen wrote:
> I have a SOHO file server with ~1 TB of data.  I would like archive the data
> by burning it to a series of optical discs organized by time (e.g. mtime).
> I expect to periodically burn additional discs in the future, each covering
> a span of time from the previous last disc to the then-current time.

I use my own software for making incremental multi-volume backups, based
on file timestamps (m and c), inode numbers, and content checksums.
  http://scdbackup.webframe.org/main_eng.html
  http://scdbackup.webframe.org/examples.html#incremental

The software and the texts are quite old. The proposed backup scheme
is not in use here any more.
Instead i have four independent backup families, each comprised of
level 0 to N with no repetitions below the current level N.
Further i have backups of the configuration and memorized file lists
on 4 CDs.

Level 0 fills dozens of BD-RE discs. The other levels fill at most one
BD-RE. Level N of each family exists in three copies which get larger
with each backup run of that level. Whenever this level BD threatens to
overflow, i archive the latest BD of that level and start level N+1.
That step is a bit bumpy, because i have to restore the file lists of
level N from CD after a backup has been planned but was not performed.
When overflow is foreseeable, i make a copy of the file lists on disk
before i start the planning run, or i simply start level N+1 without
waiting for the overflow.

I use scdbackup for the slowly growing bulk of my file collection.
The agile parts of my hard disk are only about 5 GB and get covered by
incremental multi-session backups of xorriso (which learned a lot about
incrementality from scdbackup). With zisofs compression i can put about
30 incremental backups of one DVD+RW or 250 backups on one BD-RE.
Day by day.


> The term "archive management system" comes to mind.

I would not attribute this title to scdbackup. It was created to scratch
my itch when hard disks grew much faster in capacity than the backup
media. (Also it was the motivation to start programming on ISO 9660
producers and burn programs.)

So it might be that you are better off with a more professional backup
system. :))

(Else we will probably have to read together
  http://scdbackup.webframe.org/cd_backup_planer_help
and my backup configurations to compose configurations for you.)

--
About timestamps and incremental backup:

If you only go for mtime, them you miss changes of file attributes
which are indicated by ctime.
Even more, timestamps alone are not a reliable way to determine which
files are new at their current location in the directory tree.
If you move a file from one directory to the other, then the timestamps
of the file do _not_ get updated. Only the two involved directories get
new timestamps.
So when the backup tool encounters directories with young timestamps
it has to use other means to determine whether their data files were
moved. scdbackup uses recorded device and inode numbers, and as last
resort recorded MD5 sums for that purpose.

(Of course, content MD5 comparison is slow and causes high disk load,
compared to simple directory tree traversal with timestamps and inode
numbers. So scdbackup tries to avoid this when possible and allowed
by the -changetest_options in the backup configuration file.)


Have a nice day :)

Thomas

Re: OT: Is there any size limit for ISO's?

[Thomas Schmitt]

Hi,

Hans wrote:
> does anyone know, if there is a limit of the size an iso may have?

With xorriso: 4 TiB = 4096 GiB.

An ISO 9660 filesystem may have 2 exp 32 data blocks. The usual block
size is 2 exp 11 = 2048 bytes. That would be 2 exp 43 = 8 TiB in total.
But xorriso uses libburn for writing and a poor design decision of
about 20 years ago lets it use signed int for block addresses.
So only 2 exp 31 blocks = 4 TiB are possible with libburn and xorriso.


> Background: I have a selfmade lifefile iso image, which was created by
> bootcdwrite. The size of the imagefile is about 32GB.

Looks like bootcd uses genisoimage for producing the ISO,
(https://tracker.debian.org/media/packages/b/bootcd/control-6.7)

Anyways, 32 GB of size should not be a problem.


> When the stick is booting, first appear the usual text messages,

Can you determine the origin of the last visible messages ?
BIOS/EFI ?
Boot loader (GRUB or ISOLINUX/SYSLINUX) ?
Linux ?
Debian installation software ?


> but then the screen goes blank and the further boot hangs.
> However, when I am booting this file (this iso-image) with Virtualbox,
> everything is working well.

Is Virtualbox using the same BIOS type as the real machine ?
I mean EFI versus Legacy BIOS aka CSM mode.


> Note: In earlier times I had a smaller image (about 8 GB), which booted
> perfectly.

Is the USB stick large enough for the image ?

Can dd read the whole image back from the USB stick ?

  # Set variable blocks to image file size divided by 2048
  blocks=...
  # Set variable stick to device file address of the USB stick
  stick=...

  dd if="$stick" bs=2048 count="$blocks" | md5sum

If no i/o error is reported by dd, compute the MD5 sum of the image file
and compare both.


> So I believe, the size of the image is, what matters. But maybe I am wrong.

It's not impossible. Especially if the stick hardware has problems.
But depending on the origin of the last visible message there is also
a chance that boot loader or Linux derail because of buggy file content
in the elsewise healthy ISO.


Have a nice day :)

Thomas

Re: counting commas

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> unicorn:~$ string="apple,banana,cherry,date"
> unicorn:~$ commas=${string//[!,]/}
> unicorn:~$ echo "${#commas}"
> 3

Always astonishing what a good bashism can do.


> But at this point, we have to wonder what the *actual* goal is.

Up to now we only know about the astonishment of fxkl4...@protonmail.com
that grep -c does not count characters.

For a more complicated use case i would write a little C program where
i'd be in control of every single bit of throughput.
(Ok, C causes scars on the programmer's self esteem. But what does not
kill me makes me just stronger. I'm a vim user.)


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

Anssi Saari
> It does seem strange to me, even in MS-DOS era I was able to set a
> terminal scrollback to 5000 lines without issue, when RAM was maybe 4 MB
> and a DOS terminal program probably had access to way less than that.

I have no problems with 130 xterms of 10,000 lines each.


> So does rsync really generate gigabytes of verbose output?

rsync can be extremely verbose when the number of transferred files is
very high.


> Or is xfce-terminal storing the scrollback in a very inefficient way?

I would not be astonished to learn that the luxury ornamented terminals
of the various desktops waste many extra bytes when memorizing plain text.
But the real bug is the fact that the scroll back memory is unlimited and
can summon the OOM killer. (I imagine it like the Discworld Death of Rats.)

If i were a user of Xfce i would report this as bug to its Debian
maintainer. Bug title "xfce-terminal: A landmine on the kids' playground".


Have a nice day :)

Thomas

Re: normally start new xterms

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> > > I coopted the otherwise useless "Windows" key (aka "Left Super" for
> > > WM things: Super-L makes an xterm:
> > >  # Terminal
> > >  Key   "t"   A   4  Exec exec xterm

i wrote:
> > For me the Flying Windows keys pop up or push down the affected window:
> >   Key Super_L A N RaiseLower
> >   Key Super_R A N RaiseLower
> > What i don't understand in your example is the Keyname "t".

to...@tuxteam.de wrote:
> It's Modifier (that Flying Window thing), aka "4" and the regular "t"
> (my mnemonics: "terminal").

Ah. You use it as modifier for other keys, i use it as plain key.
The quotation marks around "t" are only ornamental ?


Have a nice day :)

Thomas