Re: Sorry for the noise... WAS: Re: Switching debian-installer CD images to GRUB

[John Paul Adrian Glaubitz]

On 5/1/19 9:50 PM, Linux User #330250 wrote:
> Sorry for my stupid comment, but I wanted to know why you don't consider
> using a simple CHRP boot script as a wrapper?

My primary intention is to simplify and unify the bootloader configuration
for debian-installer across all architectures and therefore I'm adopting
the current configuration that is used for ppc64el which is based on xorriso
and grub-ieee1275.

I have already successfully done that for sparc/sparc64 and am working on
powerpc/ppc64 now. Once the changes are committed, we can get rid of
Yaboot and at some point also cdrkit which are both unmaintained.

The current design is simple enough that anyone who reads the code for
debian-installer and debian-cd can understand what's going on.

> According to this FreeBSD document, a CHRP script must be used for the
> "c" key to work and let OF find the optical drive (such as CD-ROM) to be
> bootable in the first place.
> https://www.netbsd.org/docs/bootcd.html

I think booting with "c" works with the current Debian CD images and
will work in the future. But we'll see.
 
> Such CHRP-BOOT script could also help to boot the correct installer
> (i.e. install on powerpc64 and install32 on powerpc).

I don't really think this is a real problem as the user can just select
the desired installation kernel using GRUB.

> What's currently missing is how to find the actual boot device from
> within this script, since a hardcoded "cd:" doesn't make sense
> considering USB pen drive or USB optical drive boot solutions. Maybe
> there is some similar logic like the cpus find-package stuff for
> boot-device.

This sounds PowerMac-specific.

> Especially the bootpath property of the chosen node seems to be the
> right place to start looking:
> https://flylib.com/books/en/3.126.1.47/1/
>> The default name of the kernel file is mach_kernel. BootX refers to
> several pieces of information while constructing the path to the kernel
> file. It first attempts to use the path contained in the bootpath
> property of the chosen node. If that fails, it looks at the boot-device
> property of the options node.
> 
> If this logic could be used to find out the actual real boot path, like,
> "/ht/pci@7/k2-sata-root/k2-sata@0/disk@0:3" (seems to be from a G5),
> then this information could be used by the CHRP script to load the
> correct a) kernel directly, or b) bootloader such as yaboot or grub2,
> favorably c) with passing along the information if the system is running
> a 32-Bit or 64-Bit PowerPC processor so that i.e. grub2 could default to
> either install or install32.

Yaboot is going to be removed from the archives, so there is no need for
the bootloader to be selectable. As for the kernel, I think it's okay for
users to choose the kernel they want. A 32-bit kernel will be default for
the 32-bit image, the 64-bit image will just have a 64-bit kernel.

> In addition to this maybe it is possible to generate a CHRP script that
> is suitable for other CHRP machines as well, not only Power Macs.
> Likewise the CHRP script could be used to load the Quik bootloader on
> OldWorld Power Macs.

I assume for OldWorld PowerMacs we would need to be able to write HFS
filesystems which we currently can't when we switch to xorriso. Either
way, I want to focus on NewWorld Macs first.

> And last but not least, the most important impact: When started from the
> internal optical drive (or any other supported boot device by the Open
> Firmware OS Picker, i.e. holding the option key at the chime to see the
> directly supported boot options), you'd get to see a nice Debian
> GNU/Linux logo for the installer. This is something that seems not less
> important to me as well... ;-)

The GRUB rescue ISO that I created actually showed the GNU logo on my
iBook G4 and the ISO also contains "ppc/bootinfo.txt" which seems to
be a CHRP-BOOT script containing the bitmap logo.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Sorry for the noise... WAS: Re: Switching debian-installer CD images to GRUB

[Linux User #330250]

Hi!

Sorry for my stupid comment, but I wanted to know why you don't consider
using a simple CHRP boot script as a wrapper?

According to this FreeBSD document, a CHRP script must be used for the
"c" key to work and let OF find the optical drive (such as CD-ROM) to be
bootable in the first place.
https://www.netbsd.org/docs/bootcd.html

Looking at the source of yaboot
(http://yaboot.ozlabs.org/releases/yaboot-1.3.17.tar.gz) /doc/examples,
there are a couple of simple CHRP boot scripts that could be used to
boot any bootloader we choose from the CD-ROM.

Such CHRP-BOOT script could also help to boot the correct installer
(i.e. install on powerpc64 and install32 on powerpc).
https://groups.google.com/forum/#!topic/linux.debian.ports.powerpc/MfDT7fVbq80
> 
> " screen" output
> load-base release-load-area
> " /cpus/@0" find-package if
>   " 64-bit" rot get-package-property 0= if
>    2drop
>    " boot cd:,\install\yaboot conf=cd:,\install\yaboot.conf" eval
>   else
>    " boot cd:,\install\yaboot conf=cd:,\install\mac32.conf" eval
>   then
> then
> 

What's currently missing is how to find the actual boot device from
within this script, since a hardcoded "cd:" doesn't make sense
considering USB pen drive or USB optical drive boot solutions. Maybe
there is some similar logic like the cpus find-package stuff for
boot-device.

Especially the bootpath property of the chosen node seems to be the
right place to start looking:
https://flylib.com/books/en/3.126.1.47/1/
> The default name of the kernel file is mach_kernel. BootX refers to
several pieces of information while constructing the path to the kernel
file. It first attempts to use the path contained in the bootpath
property of the chosen node. If that fails, it looks at the boot-device
property of the options node.

If this logic could be used to find out the actual real boot path, like,
"/ht/pci@7/k2-sata-root/k2-sata@0/disk@0:3" (seems to be from a G5),
then this information could be used by the CHRP script to load the
correct a) kernel directly, or b) bootloader such as yaboot or grub2,
favorably c) with passing along the information if the system is running
a 32-Bit or 64-Bit PowerPC processor so that i.e. grub2 could default to
either install or install32.

In addition to this maybe it is possible to generate a CHRP script that
is suitable for other CHRP machines as well, not only Power Macs.
Likewise the CHRP script could be used to load the Quik bootloader on
OldWorld Power Macs.

And last but not least, the most important impact: When started from the
internal optical drive (or any other supported boot device by the Open
Firmware OS Picker, i.e. holding the option key at the chime to see the
directly supported boot options), you'd get to see a nice Debian
GNU/Linux logo for the installer. This is something that seems not less
important to me as well... ;-)


Cheers,
Linux User #330250

Re: Switching debian-installer CD images to GRUB

[Thomas Schmitt]

Hi,

John Paul Adrian Glaubitz wrote:
> Looking at the grub-mkrescue.c source, I figured the proper linking would
> be:
>   xorriso_link ("/System/Library/CoreServices/grub.elf",
> "/boot/grub/powerpc-ieee1275/core.elf");

In the end it produces this pathspec

  /System/Library/CoreServices/grub.elf=/boot/grub/powerpc-ieee1275/core.elf

"pathspec" is a mkisofs term. About as old as the Macs with PowerPC.
/System/Library/CoreServices/grub.elf will be the path of the data file
in the ISO. /boot/grub/powerpc-ieee1275/core.elf is the path on hard disk
from which to read the content for the data file in the ISO.

xorriso_link() is kindof a misnamer. It should rather be something like
xorriso_make_pathspec() or xorriso_graft_in().
The variable names in its prototype
  xorriso_link (const char *from, const char *to)
are misleading by "from" becomming the path in the ISO whereas "to" is
the path on the hard disk with the prepared files. (It makes only sense
if time is running backward.)


I wrote:
> > Blessing "p" was not tested yet. You should use some Mac tool to
> > verify that it gets correcty applied by libisofs.

> Isn't the fact that my iBook G4 booted the grub-mkrescue image just
> find proof enough?

The GRUB_INSTALL_PLATFORM_POWERPC_IEEE1275 ISOs don't get blessing at all.
But i assume that they booted in some tests by Vladimir Serbinenko, back
in 2012.
We should not take anything for true before it was tested.


> > An ISO emerges.

> I will give it a try, however with the matching that for the elf files as
> mkrescue uses it. That's what works, after all.

Yeah. For now i refrained from trying to build a ppc64el grub-mkrescue ISO
on an amd64 machine.


Have a nice day :)

Thomas

Re: Switching debian-installer CD images to GRUB

[John Paul Adrian Glaubitz]

On 5/1/19 7:43 PM, Thomas Schmitt wrote:
> You need to enable HFS+ production (or maybe future HFS production)
> 
>   xorriso_push ("-hfsplus");
>   xorriso_push ("-apm-block-size");
>   xorriso_push ("2048");

Gotcha.

> The block size might be adjustable to 512, if a reason arises.
> File data start address granularity will be 2048, because of ISO 9660.

Ok.

> Type "tbxi" has not been used by grub-mkrescue yet. Googling ...
> It seems to be appropriate for BootX.
> 
> Did we have a link to this text already ?
>   https://opensource.apple.com/source/bless/bless-11/README.BOOTING
> It has a section  "== Old World booting from HFS+ ==".
> 
> Blessing "p" was not tested yet. You should use some Mac tool to
> verify that it gets correcty applied by libisofs.

Isn't the fact that my iBook G4 booted the grub-mkrescue image just
find proof enough?

> Is option
>   -sysid PPC
> really needed ?

No idea. But when I add it, xorriso actually complains.

> The old powerpc 6.0.5 and 8.0.0 netinst ISOs do not contain a BootX
> file or a /System/Library/CoreServices directory. 9.4.0-ppc64el does not
> either.
> 
> So i have to make it somewhat artificial on a x86 grub-mkrescue ISO,
> which has HFS+ for x86 Macs. I simulate the missing file BootX by a copy
> of the ISO's EFI image:
> 
>   mount /dvdbuffer/grub-mkrescue-original.iso /mnt/iso
> 
>   xorriso -as mkisofs \
>  -o test.iso \
>  -hfsplus \
>  -apm-block-size 2048 \
>  -hfsplus-file-creator-type chrp tbxi \
> /System/Library/CoreServices/BootX \
>  -hfs-bless-by p /System/Library/CoreServices \
>  -sysid PPC \
>  -chrp-boot-part \
>  -graft-points \
>  /mnt/iso \
>  /System/Library/CoreServices/BootX=/mnt/iso/efi.img

Looking at the grub-mkrescue.c source, I figured the proper linking would be:

  xorriso_link ("/System/Library/CoreServices/grub.elf", 
"/boot/grub/powerpc-ieee1275/core.elf");
  xorriso_link ("/boot/grub/powerpc.elf", 
"/boot/grub/powerpc-ieee1275/core.elf");

> An ISO emerges.
I will give it a try, however with the matching that for the elf files as 
mkrescue
uses it. That's what works, after all.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Switching debian-installer CD images to GRUB

[Thomas Schmitt]

Hi,

John Paul Adrian Glaubitz wrote:
> On the other hand, having HFS support in libisofs is still desirable
> for OldWorld Macs and m68k Macs,

(We should better not tell the Fridays-For-Future kids. The MIPS/Watt
 ratio must be subterran.)


> Looking at grub-mkrescue [1], I assume the options we need are:
>
>   if (system_area == SYS_AREA_COMMON)
> {
>   xorriso_push ("-hfsplus-file-creator-type");
>   xorriso_push ("chrp");
>   xorriso_push ("tbxi");
>   xorriso_push ("/System/Library/CoreServices/BootX");
>   xorriso_push ("-hfs-bless-by");
>   xorriso_push ("p");
>   xorriso_push ("/System/Library/CoreServices");
> }
>   xorriso_push ("-sysid");
>   xorriso_push ("PPC");
>
> Thomas, can you comment on the options for xorrisofs?

You need to enable HFS+ production (or maybe future HFS production)

  xorriso_push ("-hfsplus");
  xorriso_push ("-apm-block-size");
  xorriso_push ("2048");

The block size might be adjustable to 512, if a reason arises.
File data start address granularity will be 2048, because of ISO 9660.


Type "tbxi" has not been used by grub-mkrescue yet. Googling ...
It seems to be appropriate for BootX.

Did we have a link to this text already ?
  https://opensource.apple.com/source/bless/bless-11/README.BOOTING
It has a section  "== Old World booting from HFS+ ==".

Blessing "p" was not tested yet. You should use some Mac tool to
verify that it gets correcty applied by libisofs.

Is option
  -sysid PPC
really needed ?
ECMA-119 (aka ISO 9660)  says:
  "This field shall specify an identification of a system which can
   recognize and act upon the content of the Logical Sectors with
   logical Sector Numbers 0 to 15 of the volume."
That would be the firmware, i guess.
Whatver, it should not do harm.


> The question is whether we can create an image that boots both on IBM CHRP
> machines and PowerMacs.

There is an option -chrp-boot-part of which man xorrisofs says:

  Mark the block range of the whole  emerging  ISO  image  as  MBR
  partition  of  type  0x96. This is not compatible with any other
  feature that  produces  MBR  partition  entries.  It  makes  GPT
  unrecognizable.
  CHRP is often used in conjunction with HFS. It is not yet tested
  whether HFS+ filesystems produced  with  option  -hfsplus  would
  boot  on  any  CHRP capable machine which does not boot pure ISO
  9660 as well.

---
Let's try with not more dummy ingredients than necessary:

The old powerpc 6.0.5 and 8.0.0 netinst ISOs do not contain a BootX
file or a /System/Library/CoreServices directory. 9.4.0-ppc64el does not
either.

So i have to make it somewhat artificial on a x86 grub-mkrescue ISO,
which has HFS+ for x86 Macs. I simulate the missing file BootX by a copy
of the ISO's EFI image:

  mount /dvdbuffer/grub-mkrescue-original.iso /mnt/iso

  xorriso -as mkisofs \
 -o test.iso \
 -hfsplus \
 -apm-block-size 2048 \
 -hfsplus-file-creator-type chrp tbxi \
/System/Library/CoreServices/BootX \
 -hfs-bless-by p /System/Library/CoreServices \
 -sysid PPC \
 -chrp-boot-part \
 -graft-points \
 /mnt/iso \
 /System/Library/CoreServices/BootX=/mnt/iso/efi.img

An ISO emerges.
As far as xorriso can tell, it looks ok:

  $ xorriso -hfsplus on \
-indev test.iso \
-report_system_area plain \
-pvd_info \
-find / -has_hfs_bless any -exec get_hfs_bless -- \
-find / -has_hfs_crtp - - -exec get_hfs_crtp --
  ...
  System area summary: MBR CHRP cyl-align-off APM
  ISO image size/512 : 30796
  Partition offset   : 0
  MBR heads per cyl  : 64
  MBR secs per head  : 32
  MBR partition table:   N Status  TypeStart   Blocks
  MBR partition  :   1   0x80  0x96030796
  APM:   N  Info
  APM block size :  2048
  APM gap fillers:  2
  APM partition name :   1  Gap0
  APM partition type :   1  ISO9660_data
  APM start and size :   1  16  67
  APM partition name :   2  HFSPLUS_Hybrid
  APM partition type :   2  Apple_HFS
  APM start and size :   2  83  7466
  APM partition name :   3  Gap1
  APM partition type :   3  ISO9660_data
  APM start and size :   3  7549  150
  ...
  System Id: PPC
  ...
  ppc_bootdir  '/System/Library/CoreServices'
  chrp tbxi '/System/Library/CoreServices/BootX'

The report about blessing and creator/type stems from ISO extended
attributes which preserve the info independently of the HFS+ filesystem.
So it has to be checked by other tools whether these HFS+ attributes
are recognizable.

---

Have a nice day :)

Thomas

Re: Switching debian-installer CD images to GRUB

[John Paul Adrian Glaubitz]

On 5/1/19 4:13 PM, John Paul Adrian Glaubitz wrote:
> So, I think we currently have all the means to switch Debian powerpc
> and Debian ppc64 over from Yaboot to GRUB, so let's work on that.

Looking at grub-mkrescue [1], I assume the options we need are:

  if (system_area == SYS_AREA_COMMON)
{
  xorriso_push ("-hfsplus-file-creator-type");
  xorriso_push ("chrp");
  xorriso_push ("tbxi");
  xorriso_push ("/System/Library/CoreServices/BootX");
  xorriso_push ("-hfs-bless-by");
  xorriso_push ("p");
  xorriso_push ("/System/Library/CoreServices");
}
  xorriso_push ("-sysid");
  xorriso_push ("PPC");

The question is whether we can create an image that boots both on IBM CHRP 
machines
and PowerMacs.

Thomas, can you comment on the options for xorrisofs?

Adrian

> [1] http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Switching debian-installer CD images to GRUB

[John Paul Adrian Glaubitz]

On 4/29/19 2:05 PM, Thomas Schmitt wrote:
> But first you should explore whether ppc64 is served by grub-mkrescue's
> way of HFS+. Vladimir must have had some architecture in mind, when he
> contributed that code to libisofs and asked for the xorrisofs options
> to control it.
> 
> What happens if you install
>   https://packages.debian.org/sid/ppc64/grub-ieee1275-bin/filelist
> uninstall other GRUB targets and then make a minimal grub-mkrescue ISO:
> 
>   mkdir minimal
>   touch minimal/empty-file.txt
>   grub-mkrescue -o output.iso minimal
> 
> Does output.iso boot to a GRUB console prompt when offered on CD or
> hard-disk-like medium ?
I just tested this and yes, it does actually work on my iBook G4.

So, I think we currently have all the means to switch Debian powerpc
and Debian ppc64 over from Yaboot to GRUB, so let's work on that.

On the other hand, having HFS support in libisofs is still desirable
for OldWorld Macs and m68k Macs, so I will create a bounty source
for that.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Switching debian-installer CD images to GRUB

[Thomas Schmitt]

Hi,

i notice that i sent this to the wrong thread of debian-powerpc
and did not Cc the intended other receivers. So again.



John Paul Adrian Glaubitz wrote:
> > What about a funding? I would actually be willing to shell out some money

Mark Cave-Ayland wrote:
> Certainly if there were some funding available, this is something that I
> could look at if required.

If you do, i am ready for giving advise about libisofs architecture
and for reviewing changeset proposals.
As said, i am reluctant to accept large crowds of copyright holders,
but this can be solved by wrapping the new code in an extra library,
which then gets linked to libisofs optionally. (libjte is an example.)

But first we should explore the bootability of grub-mkrescue HFS+.


I wrote:
> > George Danchev tried whether it could substitute for genisoimage's HFS
> > on a virtual machine which booted the genisoimage ISO.
> > A user rported failure with
> >   $ qemu-system-ppc -boot d -cdrom repacked.iso -hda linux.img

> Do you have any more information on this?

It was in summer 2012. I see in the archives
  "ISO9660/HFS hybrid powerpc testers needed"
  https://lists.debian.org/debian-powerpc/2012/06/msg00030.html
but with no valid replies to see.
George revoked his call for testers after first negative results.
  https://lists.debian.org/debian-powerpc/2012/06/msg00031.html
Probably he got the failure reports in private or did own experiments.


> FWIW this is one of the ideal use cases for QEMU

I cannot judge Sparc emulation.
With x86 EFI firmware OVMF, you have to expect success even with clearly
inappropriate boot sectors. Partition table based images boot from virtual
CD-ROM and El Torito boots from virtual hard disk.
So success on OVMF does not predict success on real iron.


> bless the folder on the filesystem containing the bootloader and
> then set the type to TBXI (ToolBoXImage).

Hm. Can it be that there is a successor with "J" instead of "I":
grub-mkrescue asks xorriso to set creator and type of .disk_label:

  xorriso_push ("-hfsplus-file-creator-type");
  xorriso_push ("chrp");
  xorriso_push ("tbxj");
  xorriso_push ("/System/Library/CoreServices/.disk_label");

I don't think that "txbj" is a typo.


Have a nice day :)

Thomas

Re: Switching debian-installer CD images to GRUB

[John Paul Adrian Glaubitz]

On 5/1/19 5:52 AM, Mark Cave-Ayland wrote:
> Certainly if there were some funding available, this is something that I 
> could look
> at if required. Currently my free time is limited (and already fairly busy) 
> but with
> sponsorship there is the potential to allocate time within the working week.
> 
> I must say I'm surprised that HFS+ doesn't boot - certainly I see images in 
> QEMU that
> require both, and OpenBIOS has booted them successfully for many years.

I just noticed that HFS support is required either way as the boot CDs for m68k 
Macs
also come with an HFS partition. At least debian-cd currently uses that not sure
whether it's actually necessary after all. I will look at m68k later though, 
let's
focus powerpc/ppc64 first.

As for the funding, we could maybe use Bountysource [1]. I would guess we would
certainly get enough money together with some advertisements on reddit and 
similar
social media. After all, this would mean we can finally get rid of cdrkit in
Debian and other distributions.

Adrian

> [1] https://www.bountysource.com/

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Switching debian-installer CD images to GRUB

[Mark Cave-Ayland]

On 29/04/2019 13:05, Thomas Schmitt wrote:

> Hi,
> 
> i wrote:
>>> [...] Fedora HFS image [...]
>>> I think Fedora uses hfsutils to create and populate it.
> 
>> My colleague Marcus Schaefer at SUSE told me
>> about this solution. He's the maintainer of SUSE's KIWI image tool (but
>> I think you know that ;)).
> 
> I had contact with him last year.
> 
>> We could give indeed this solution a try, with EFI replaced with IEEE1275.
> 
> But first you should explore whether ppc64 is served by grub-mkrescue's
> way of HFS+. Vladimir must have had some architecture in mind, when he
> contributed that code to libisofs and asked for the xorrisofs options
> to control it.
> 
> What happens if you install
>   https://packages.debian.org/sid/ppc64/grub-ieee1275-bin/filelist
> uninstall other GRUB targets and then make a minimal grub-mkrescue ISO:
> 
>   mkdir minimal
>   touch minimal/empty-file.txt
>   grub-mkrescue -o output.iso minimal
> 
> Does output.iso boot to a GRUB console prompt when offered on CD or
> hard-disk-like medium ?
> (Beware: The grub-ieee1275-bin filelists differ from host arch to host
>   arch.
>https://packages.debian.org/sid/amd64/grub-ieee1275-bin/filelist
>  has files for i386-ieee1275 rather than powerpc-ieee1275.
>  But ppc64 and ppc64el look the same.
> )
> 
> The mini HFS image approach is probably needed with older powerpc.
> When the HFS+ code in libisofs was young, its then Debian maintainer
> George Danchev tried whether it could substitute for genisoimage's HFS
> on a virtual machine which booted the genisoimage ISO.
> A user rported failure with
>   $ qemu-system-ppc -boot d -cdrom repacked.iso -hda linux.img

Do you have any more information on this? As the current maintainer of OpenBIOS 
and
the QEMU Mac machines I'm not aware of any outstanding issues in this area.

FWIW this is one of the ideal use cases for QEMU since you can prototype the 
basic
boot process without going anywhere near physical media.

>> as far as I know, 64-bit PowerMacs aren't any different
>> than 32-bit PowerMacs when it comes to the firmware as long as the 32-bit
>> Macs are using NewWorld ROMs.
> 
> Hard to say what firmware was used by qemu-system-ppc at the end of 2012.

I would have expected it to be OpenBIOS around that time, but certainly its
predecessor OHW (OpenHackWare) was quite buggy.

>> What about a funding? I would actually be willing to shell out some money
>> for that if someone worked on implementing HFS support.
> 
> One could implement it in libisofs as alternative to HFS+.
> This would simplify the inner architectural effort by re-using the
> existing plug-in points of HFS+. (Calls of hfsplus_*() in
>   https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/ecma119.c
> )
> One would try to generalize the following files or to write new
> counterparts:
>   https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus.h
>   https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus.c
>   
> https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus_case.c
>   
> https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus_classes.c
>   
> https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus_decompose.c
> 
> It must look to me like safe code and it must be free of unfriendly
> licenses or copyright holders. (The license must enable derived LGPLv2.
> Foreign copyrights could be isolated in a separate library, like
> was done with libjte.)
> 
> If new options are needed, then several more files in libisofs and
> libisoburn will be affected. I would be willing to do this part which
> mainly is about pulling wires through the architecture layers.

As per my last email, if sponsorship is available then possibly this is 
something we
could collaborate on. But again I would expect HFS+ to work here...

> Maybe a more feasable job would be the Fedora image file approach:
> 
> I assume that files from the genisoimage ISO content would have to be
> migrated into a hfsutils made HFS image, which xorriso puts into the
> ISO as data file or appends after the ISO as partition. Then xorriso
> marks it by Apple Partition Map.
> 
> Knowledge about the boot procedure of powerpc firmware will surely be
> of help.
> 
> One would need to explore grub-mkrescue ISOs and existing powerpc
> ISOs, whether there are machines where genisoimage -hfs ISO succeeds but
> grub-mkrescue's ISO fails.
> 
> Then one would try to find out what miracles are done by genisoimage
> options
>   --netatalk
>   -hfs
>   -probe
>   -map /home/debian-cd/build/debian-cd.squeeze/data/hfs.map
>   -hfs-parms MAX_XTCSIZE=2656248
>   -part
>   -no-desktop
>   -hfs-bless CD1/install
> 
> I can make some sense out of the genisoimage man page:
> 
> -part causes Apple Partition Map, which would be the job of xorriso if
> a HFS image file is put into the ISO. Like isohybrid --mac for Fedora.
> 
> -hfs-bless blesses a directory. grub-mkrescue only blesses 

Re: Switching debian-installer CD images to GRUB

[Mark Cave-Ayland]

On 29/04/2019 11:08, John Paul Adrian Glaubitz wrote:

> Hi Thomas!
> 
> On 4/29/19 9:41 AM, Thomas Schmitt wrote:
>> John Paul Adrian Glaubitz wrote:
>>> Btw, I was wondering how much it would take to add HFS support to
>>> xorriso but I guess the necessary changes are so extensive which
>>> is why it's not been done yet.
>>
>> I obtained (hopefully complete) HFS specs and had to learn that it is
>> not easily implementable as downgraded version of HFS+.
> 
> That's a pity, indeed.
> 
>> libisofs got HFS+ with nearly no own effort from Validmir Serbinenko
>> because he needed it for grub-mkrescue. But he had no motivation for HFS.
> 
> Ah, too bad. But I guess he was primarily interested in x86 Macs.
> 
>> (Does GRUB2 support old 32-bit powerpc at all ?)
> 
> Yes, it does. But as far as I know, 64-bit PowerMacs aren't any different
> than 32-bit PowerMacs when it comes to the firmware as long as the 32-bit
> Macs are using NewWorld ROMs.
> 
>> So it is about the ratio of my motivation and the expectable pain to
>> squeeze another filesystem tree into libisofs and to find test cases.
> 
> What about a funding? I would actually be willing to shell out some money
> for that if someone worked on implementing HFS support.

Certainly if there were some funding available, this is something that I could 
look
at if required. Currently my free time is limited (and already fairly busy) but 
with
sponsorship there is the potential to allocate time within the working week.

I must say I'm surprised that HFS+ doesn't boot - certainly I see images in 
QEMU that
require both, and OpenBIOS has booted them successfully for many years.


ATB,

Mark.

Re: Switching debian-installer CD images to GRUB

[Linux User #330250]

On 28.04.2019, 23:15, Frank Scheiner wrote:
>
> I just checked the OF on my 11,2 type Power Mac G5 and it - in addition
> to `fat-files` (I assume this contains support for FAT file systems)
> `mac-files` (I assume this contains support for HFS) - has
> `hfs-plus-files` and `iso-9660-files` in `/packages`. So maybe we don't
> need an HFS image on the ISO and could do with an HFS+ image or maybe
> don't need an HFS(+) image at all.

Just to confirm, on my iBook G3 (original Clamshell, 1999) I also have
those packages in OF. I can read my HFS+ volume with it ("dir
ultra1:10,\" – intrestingly "dir hd:\" didn't work, neither did "dir
hd:10,\").

Cheers,
Linux User #330250

Re: Switching debian-installer CD images to GRUB

[Thomas Schmitt]

Hi,

i wrote:
> > [...] Fedora HFS image [...]
> > I think Fedora uses hfsutils to create and populate it.

> My colleague Marcus Schaefer at SUSE told me
> about this solution. He's the maintainer of SUSE's KIWI image tool (but
> I think you know that ;)).

I had contact with him last year.

> We could give indeed this solution a try, with EFI replaced with IEEE1275.

But first you should explore whether ppc64 is served by grub-mkrescue's
way of HFS+. Vladimir must have had some architecture in mind, when he
contributed that code to libisofs and asked for the xorrisofs options
to control it.

What happens if you install
  https://packages.debian.org/sid/ppc64/grub-ieee1275-bin/filelist
uninstall other GRUB targets and then make a minimal grub-mkrescue ISO:

  mkdir minimal
  touch minimal/empty-file.txt
  grub-mkrescue -o output.iso minimal

Does output.iso boot to a GRUB console prompt when offered on CD or
hard-disk-like medium ?
(Beware: The grub-ieee1275-bin filelists differ from host arch to host
  arch.
   https://packages.debian.org/sid/amd64/grub-ieee1275-bin/filelist
 has files for i386-ieee1275 rather than powerpc-ieee1275.
 But ppc64 and ppc64el look the same.
)

The mini HFS image approach is probably needed with older powerpc.
When the HFS+ code in libisofs was young, its then Debian maintainer
George Danchev tried whether it could substitute for genisoimage's HFS
on a virtual machine which booted the genisoimage ISO.
A user rported failure with
  $ qemu-system-ppc -boot d -cdrom repacked.iso -hda linux.img


> as far as I know, 64-bit PowerMacs aren't any different
> than 32-bit PowerMacs when it comes to the firmware as long as the 32-bit
> Macs are using NewWorld ROMs.

Hard to say what firmware was used by qemu-system-ppc at the end of 2012.


> What about a funding? I would actually be willing to shell out some money
> for that if someone worked on implementing HFS support.

One could implement it in libisofs as alternative to HFS+.
This would simplify the inner architectural effort by re-using the
existing plug-in points of HFS+. (Calls of hfsplus_*() in
  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/ecma119.c
)
One would try to generalize the following files or to write new
counterparts:
  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus.h
  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus.c
  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus_case.c
  
https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus_classes.c
  
https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus_decompose.c

It must look to me like safe code and it must be free of unfriendly
licenses or copyright holders. (The license must enable derived LGPLv2.
Foreign copyrights could be isolated in a separate library, like
was done with libjte.)

If new options are needed, then several more files in libisofs and
libisoburn will be affected. I would be willing to do this part which
mainly is about pulling wires through the architecture layers.

--

Maybe a more feasable job would be the Fedora image file approach:

I assume that files from the genisoimage ISO content would have to be
migrated into a hfsutils made HFS image, which xorriso puts into the
ISO as data file or appends after the ISO as partition. Then xorriso
marks it by Apple Partition Map.

Knowledge about the boot procedure of powerpc firmware will surely be
of help.

One would need to explore grub-mkrescue ISOs and existing powerpc
ISOs, whether there are machines where genisoimage -hfs ISO succeeds but
grub-mkrescue's ISO fails.

Then one would try to find out what miracles are done by genisoimage
options
  --netatalk
  -hfs
  -probe
  -map /home/debian-cd/build/debian-cd.squeeze/data/hfs.map
  -hfs-parms MAX_XTCSIZE=2656248
  -part
  -no-desktop
  -hfs-bless CD1/install

I can make some sense out of the genisoimage man page:

-part causes Apple Partition Map, which would be the job of xorriso if
a HFS image file is put into the ISO. Like isohybrid --mac for Fedora.

-hfs-bless blesses a directory. grub-mkrescue only blesses the Intel
Boot file, but not a directory for GRUB_INSTALL_PLATFORM_POWERPC_IEEE1275.
I assume that genisoimage -hfs-bless applies blessing PPC Bootdir to that
directory. hfsutils program hattrib option -b looks like that.

-map is probably the other job of hattrib: setting Type and Creator.
One would have to look for old hfs.map files or use hfsutils program hls
option -l to learn about attributes in existing genisoimage ISOs.



> cdrkit which Debian wants to get rid of as it's unmaintained upstream.

I am sure that Steve McIntyre lights a candle for every architecture ISO
production which you switch away from genisoimage.
But chances are that genisoimage will live as 

Re: Switching debian-installer CD images to GRUB

[John Paul Adrian Glaubitz]

Hi Thomas!

On 4/29/19 9:41 AM, Thomas Schmitt wrote:
> John Paul Adrian Glaubitz wrote:
>> Btw, I was wondering how much it would take to add HFS support to
>> xorriso but I guess the necessary changes are so extensive which
>> is why it's not been done yet.
> 
> I obtained (hopefully complete) HFS specs and had to learn that it is
> not easily implementable as downgraded version of HFS+.

That's a pity, indeed.

> libisofs got HFS+ with nearly no own effort from Validmir Serbinenko
> because he needed it for grub-mkrescue. But he had no motivation for HFS.

Ah, too bad. But I guess he was primarily interested in x86 Macs.

> (Does GRUB2 support old 32-bit powerpc at all ?)

Yes, it does. But as far as I know, 64-bit PowerMacs aren't any different
than 32-bit PowerMacs when it comes to the firmware as long as the 32-bit
Macs are using NewWorld ROMs.

> So it is about the ratio of my motivation and the expectable pain to
> squeeze another filesystem tree into libisofs and to find test cases.

What about a funding? I would actually be willing to shell out some money
for that if someone worked on implementing HFS support.

> Given the fact that genisoimage still offers -hfs and that Fedora showed
> a strategy how to please such firmware by a small, externally created
> filesystem image, i am not motivated enough yet.
That's understandable. However, genisoimage comes from cdrkit which Debian
wants to get rid of as it's unmaintained upstream. xorriso is the tool of
choice these days and if we could use it for all targets, that would be
very much desirable.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Switching debian-installer CD images to GRUB

[John Paul Adrian Glaubitz]

Hi Thomas!

On 4/28/19 9:49 PM, Thomas Schmitt wrote:
> John Paul Adrian Glaubitz wrote:
>> So, the first question is whether xorriso supports HFS volumes
> 
> No. Only HFS+, contributed by Vladimir Serbinenko for x86 Macs and
> used in grub-mkrescue ISOs for x86 EFI.

Okay.

> The xorrisofs options used by grub-mkrescue for HFS+ are:
>   -hfsplus
>   -apm-block-size 2048
>   -hfsplus-file-creator-type chrp tbxj \
>  /System/Library/CoreServices/.disk_label
>   -hfs-bless-by i /System/Library/CoreServices/boot.efi
> 
> Is ppc64 the same as GRUB_INSTALL_PLATFORM_POWERPC_IEEE1275 in
> grub-mkrescue ?

Yes, that's correct:

glaubitz@plummer:~$ uname -a
Linux plummer 4.9.0-9-powerpc64le #1 SMP Debian 4.9.168-1 (2019-04-12) ppc64le 
GNU/Linux
glaubitz@plummer:~$ ls -dl /usr/lib/grub/powerpc-ieee1275/
drwxr-xr-x 2 root root 28672 Nov 10 14:45 /usr/lib/grub/powerpc-ieee1275/
glaubitz@plummer:~$

> This target obviously gets the same HFS+ options as
> GRUB_INSTALL_PLATFORM_I386_EFI and GRUB_INSTALL_PLATFORM_X86_64_EFI.

Okay.

> One could do it with HFS like Fedora ISO does with HFS+.
> A small HFS+ filesystem image is marked in APM as Apple_HFS. In the ISO
> it has the name /images/macboot.img.
> (Could be easily added to current amd64 or i386 debian-cd production.
>  With arm64-like layout i'd need to make experiments and maybe add
>  some APM production capability to xorriso.)

I have heard about that. My colleague Marcus Schaefer at SUSE told me
about this solution. He's the maintainer of SUSE's KIWI image tool (but
I think you know that ;)).

> The Fedora HFS image contains GRUB stuff. Mostly .cfg and .efi. Obviously
> program execution breaks out from it like others do from EFI partition.
> I think Fedora uses hfsutils to create and populate it.

We could give indeed this solution a try, with EFI replaced with IEEE1275.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Switching debian-installer CD images to GRUB

[Thomas Schmitt]

Hi,

John Paul Adrian Glaubitz wrote:
> Btw, I was wondering how much it would take to add HFS support to
> xorriso but I guess the necessary changes are so extensive which
> is why it's not been done yet.

I obtained (hopefully complete) HFS specs and had to learn that it is
not easily implementable as downgraded version of HFS+.
libisofs got HFS+ with nearly no own effort from Validmir Serbinenko
because he needed it for grub-mkrescue. But he had no motivation for HFS.
(Does GRUB2 support old 32-bit powerpc at all ?)

So it is about the ratio of my motivation and the expectable pain to
squeeze another filesystem tree into libisofs and to find test cases.
Given the fact that genisoimage still offers -hfs and that Fedora showed
a strategy how to please such firmware by a small, externally created
filesystem image, i am not motivated enough yet.


Have a nice day :)

Thomas

Re: Switching debian-installer CD images to GRUB

[John Paul Adrian Glaubitz]

On 4/28/19 11:15 PM, Frank Scheiner wrote:
> Probably answering my second question:
> 
> I just checked the OF on my 11,2 type Power Mac G5 and it - in addition
> to `fat-files` (I assume this contains support for FAT file systems)
> `mac-files` (I assume this contains support for HFS) - has
> `hfs-plus-files` and `iso-9660-files` in `/packages`. So maybe we don't
> need an HFS image on the ISO and could do with an HFS+ image or maybe
> don't need an HFS(+) image at all.

Right, you mentioned at one point that booting from a FAT partition
would work as well.

Btw, I was wondering how much it would take to add HFS support to
xorriso but I guess the necessary changes are so extensive which
is why it's not been done yet.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Switching debian-installer CD images to GRUB

[Frank Scheiner]

Hi again,

On 4/28/19 21:56, Frank Scheiner wrote:

Other question: Does anybody know if NewWorld Power Macs could maybe
also boot from HFS+ instead of HFS or FAT(16)?


Probably answering my second question:

I just checked the OF on my 11,2 type Power Mac G5 and it - in addition
to `fat-files` (I assume this contains support for FAT file systems)
`mac-files` (I assume this contains support for HFS) - has
`hfs-plus-files` and `iso-9660-files` in `/packages`. So maybe we don't
need an HFS image on the ISO and could do with an HFS+ image or maybe
don't need an HFS(+) image at all.

E.g. I can list the contents of the installer ISO from 2019-04-20 burned
to a CDRW from the OF.

Cheers,
Frank

Re: Switching debian-installer CD images to GRUB

[Thomas Schmitt]

Hi,
i
wrote:
> > One could do it with HFS like Fedora ISO does with HFS+.
> > A small HFS+ filesystem image is marked in APM as Apple_HFS. In the ISO
> > it has the name /images/macboot.img.

Frank Scheiner wrote:
> Is that the same method as what `mkfs.hfs` can do with option:
> -w  Adds an HFS wrapper around the HFS Plus file system.

Probably not.

  $ file /mnt/iso/images/macboot.img
  /mnt/iso/images/macboot.img: Macintosh HFS Extended version 4 data
  last mounted by: 'H+Lx', created: Wed Nov  1 01:37:23 2147,
  last modified: Wed Nov  1 02:37:23 2147, block size: 4096,
  number of blocks: 5396, free blocks: 3457

Content is

  ./.VolumeIcon.icns
  ./EFI
  ./EFI/BOOT
  ./EFI/BOOT/.disk_label
  ./EFI/BOOT/BOOTX64.EFI
  ./EFI/BOOT/fonts
  ./EFI/BOOT/fonts/unicode.pf2
  ./EFI/BOOT/grub.cfg
  ./EFI/BOOT/grubx64.efi
  ./EFI/BOOT/MokManager.efi
  ./mach_kernel
  ./System
  ./System/Library
  ./System/Library/CoreServices
  ./System/Library/CoreServices/boot.efi
  ./System/Library/CoreServices/grub.cfg
  ./System/Library/CoreServices/grubx64.efi
  ./System/Library/CoreServices/SystemVersion.plist

I don't know whether this gesture would work for powerpc too, i.e.
how much brain it needs provided in HFS before it can deal with the
ISO 9660 filesystem for further progress.
But it's obviously the way how Matthew J. Garrett worked around the
lack of HFS+ support in genisoimage, back in 2012.

See
  http://mjg59.dreamwidth.org/11285.html
where he explained the layout produced by his modifications to program
isohybrid which added options --uefi and --mac.
Debian amd64 and i386 adopted the --uefi part of this layout.


Have a nice day :)

Thomas

Re: Switching debian-installer CD images to GRUB

[Frank Scheiner]

Hi,

On 4/28/19 21:49, Thomas Schmitt wrote:

One could do it with HFS like Fedora ISO does with HFS+.
A small HFS+ filesystem image is marked in APM as Apple_HFS. In the ISO
it has the name /images/macboot.img.


Is that the same method as what `mkfs.hfs` can do with option:
```
-w  Adds an HFS wrapper around the HFS Plus file system. This
wrapper is required if the file system will be used to boot natively
into Mac OS 9.
```

...?

Other question: Does anybody know if NewWorld Power Macs could maybe
also boot from HFS+ instead of HFS or FAT(16)?

Cheers,
Frank

Re: Switching debian-installer CD images to GRUB

[Thomas Schmitt]

Hi,

John Paul Adrian Glaubitz wrote:
> So, the first question is whether xorriso supports HFS volumes

No. Only HFS+, contributed by Vladimir Serbinenko for x86 Macs and
used in grub-mkrescue ISOs for x86 EFI.
The xorrisofs options used by grub-mkrescue for HFS+ are:
  -hfsplus
  -apm-block-size 2048
  -hfsplus-file-creator-type chrp tbxj \
 /System/Library/CoreServices/.disk_label
  -hfs-bless-by i /System/Library/CoreServices/boot.efi

Is ppc64 the same as GRUB_INSTALL_PLATFORM_POWERPC_IEEE1275 in
grub-mkrescue ?
This target obviously gets the same HFS+ options as
GRUB_INSTALL_PLATFORM_I386_EFI and GRUB_INSTALL_PLATFORM_X86_64_EFI.


One could do it with HFS like Fedora ISO does with HFS+.
A small HFS+ filesystem image is marked in APM as Apple_HFS. In the ISO
it has the name /images/macboot.img.
(Could be easily added to current amd64 or i386 debian-cd production.
 With arm64-like layout i'd need to make experiments and maybe add
 some APM production capability to xorriso.)

The Fedora HFS image contains GRUB stuff. Mostly .cfg and .efi. Obviously
program execution breaks out from it like others do from EFI partition.
I think Fedora uses hfsutils to create and populate it.


Have a nice day :)

Thomas