Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
Hi, Andrew Cater wrote: > It does seem that > we may be able to stop routine production of as many images. Netinst, > something DVD-ish sized (so smaller than 8G) and some (larger file size ??) > may do it. How about being storage-neutral and drop amd64-DVD-2.iso and amd64-DVD-3.iso in favor of a new amd64-STICK8GB-1.iso ? That would mean the dropped DVDs 2 and 3 have to be downloaded by jigdo like DVDs 4 to 16 (or like DVD 2 and 3 of arm64). Question is what is on DVD 3 that will be missing for a "normal" Debian installation from STICK8GB. Really generous with more than all DVD.iso packages would be a readily downloadable amd64-STICK16GB-1.iso (currently only available via jigdo). > Jigdo does work well to build .iso images > - it might need more documentation. Especially the documentation needs to be where the .jigdo files are. I meanwhile learned that SuSE has a package "jigdo" which provides jigdo-lite. (It's not easy to get a file list of their packages online.) Fedora too has jigdo-lite in package "jigdo". (Well, when debian-cd switches to SHA256 for jigdo, those will need a nudge.) Have a nice day :) Thomas
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
Hi, Stefan Monnier wrote: > I was thinking of a scheme by which the ISO is constructed and streamed > at the same time, so the complete ISO images aren't ever stored whole > anywhere on the server. Although there are no fundamental obstacles for stream production, the current software uncompresses the .template file to a file on disk and then fills the marked holes by the packages which it downloads from the chosen mirror server. I wrote: > > The production of jigdo files is a linear effort only if the producer > > knows where the filesystem stores file content data. Stefan Monnier wrote: > I'm not sure I understand what this means, Indeed "linear" is not the right term. I apologize for being misleading. Given the original description http://atterer.org/jigdo/debian-jigdo-mini-howto#PREPARINGTHEISOFORDOWNLOAD of the production process it is more about "scientific" versus "magic": "Through magic, jigdo-file finds out which of the loose files are contained in the ISO image and their offsets within the ISO file. It outputs two files: a ".template" file and a ".jigdo" file." All tools for creating .jigdo and .template get a list of MD5 sums and package file paths (the "loose" files). The original tool, jigdo-file, scans the readily prepared ISO image for the content of the data files without knowing where their content starts. This implies a large amount of try-and-error, even with a smart algorithm. The mismatching efforts are wasted. genisoimage and libjte (under libisofs and xorriso) get to see the content data of each data file, compute the MD5 and look up this MD5 in the list. They don't have to guess where the files are stored in the ISO. On the other hand, jigdo-file can apply its magic to any filesystem which stores file content unchanged in one byte string, whereas the non-guessing approach needs help from the filesystem producer. > it does sound like it > implies that streaming production of ISOs is technically possible. I actually meant production of .template and .jigdo, not re-assembly of the .iso from them and Debian packages. But yes, streaming re-assembly of .iso from .jigdo only needs structural changes in the download software. (The problem is how to motivate the change given that there hardly will be an official ISO-from-Jigdo server.) - Tixy wrote: > > Debian's 'DVD' images are isohybrid and work for booting from USB flash > > drives, Stefan Monnier wrote: > The discussion was about the creation of special ISOs for old Mac > hardware whose firmware isn't able to boot from those hybrid ISOs. The "mac" ISOs for i386 and amd64 are hybrid ISOs too. Just not for EFI. They have legacy MBR boot code which hops on the boot image for CD booting after it was started by legacy BIOS from USB stick. But they don't have a second boot image for EFI from CD (which also would serve as EFI System Partition on USB stick). Have a nice day :) Thomas
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
Streaming production of .iso files _is_ technically possible. Jigdo effectively builds the iso file from chunks of ten or so files until the disk is complete and checksummed. As mentioned, this query was about stopping production of the .iso files specifically meant for the oldestMac mini. That certainly sounds possible for Bullseye when it gets here since that particular model will be about 15 years old and there's likely to be very few of them still running. I'm collecting what people are sending me on and off list. It does seem that we may be able to stop routine production of as many images. Netinst, something DVD-ish sized (so smaller than 8G) and some (larger file size ??) may do it. The problem of using http to download large files is that it's fairly difficult on a slow link or one which is losing packets. Jigdo does work well to build .iso images - it might need more documentation. Hope this helps On Mon, Jul 20, 2020 at 2:25 PM Stefan Monnier wrote: > >> An alternative is to have "virtual ISO images", i.e. images which are > >> constructed on the fly (presumably by jigdo) on the web-server side. > > Assuming that a complete set of ISOs for whatever medium occupies > > at most 100 GB, it seems better to have the images ready rather than to > > assemble them on demand, even if the mirror latency and bandwidth are > > no problem. > > This would spare the nightmare of managing the life cycle of temporary > ISOs > > on the server. I assume that all images would fit on a single modern HDD. > > I was thinking of a scheme by which the ISO is constructed and streamed > at the same time, so the complete ISO images aren't ever stored whole > anywhere on the server. > > > But the reason for letting the user perform jigdo download is that the > network > > load vanishes in the normal traffic of the package servers. If download > gets > > interrupted, one just has to start it again to get the remaining work > > completed. > > Very good point. I did not consider the interrupt&restart issue. > > > The production of jigdo files is a linear effort only if the producer > > knows where the filesystem stores file content data. This knowledge is in > > the ISO 9660 producers genisoimage and xorriso. For other filesystems > > the matching jigdo producer software is not available yet. > > I'm not sure I understand what this means, but it does sound like it > implies that streaming production of ISOs is technically possible. > > > Stefan > >
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
>> An alternative is to have "virtual ISO images", i.e. images which are >> constructed on the fly (presumably by jigdo) on the web-server side. > Assuming that a complete set of ISOs for whatever medium occupies > at most 100 GB, it seems better to have the images ready rather than to > assemble them on demand, even if the mirror latency and bandwidth are > no problem. > This would spare the nightmare of managing the life cycle of temporary ISOs > on the server. I assume that all images would fit on a single modern HDD. I was thinking of a scheme by which the ISO is constructed and streamed at the same time, so the complete ISO images aren't ever stored whole anywhere on the server. > But the reason for letting the user perform jigdo download is that the network > load vanishes in the normal traffic of the package servers. If download gets > interrupted, one just has to start it again to get the remaining work > completed. Very good point. I did not consider the interrupt&restart issue. > The production of jigdo files is a linear effort only if the producer > knows where the filesystem stores file content data. This knowledge is in > the ISO 9660 producers genisoimage and xorriso. For other filesystems > the matching jigdo producer software is not available yet. I'm not sure I understand what this means, but it does sound like it implies that streaming production of ISOs is technically possible. Stefan
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
Hi, (The mail headers indicate that the OP, Andrew Cater, is subscribed to debian-cd but not to debian-user. So replies to debian-user only might not get to his attention.) Stefan Monnier wrote: > An alternative is to have "virtual ISO images", i.e. images which are > constructed on the fly (presumably by jigdo) on the web-server side. Assuming that a complete set of ISOs for whatever medium occupies at most 100 GB, it seems better to have the images ready rather than to assemble them on demand, even if the mirror latency and bandwidth are no problem. This would spare the nightmare of managing the life cycle of temporary ISOs on the server. I assume that all images would fit on a single modern HDD. But the reason for letting the user perform jigdo download is that the network load vanishes in the normal traffic of the package servers. If download gets interrupted, one just has to start it again to get the remaining work completed. These adavantages for Debian can hardly be uphold by a jigdo assembling server on Debian side. > I'm not sure why the discussion is always around DVD images. That's mainly about the image size factor of a 4+ GB DVD. There is one size factor that is not defined by optical media sizes https://cdimage.debian.org/debian-cd/current/amd64/jigdo-16G/ > Would you be equally happy with an image only for a USB flash drive, or > is it important for it to be an image for optical media? The ISOs work from optical media and from hard-disk-like media. The latter class includes USB sticks and memory cards. The production of jigdo files is a linear effort only if the producer knows where the filesystem stores file content data. This knowledge is in the ISO 9660 producers genisoimage and xorriso. For other filesystems the matching jigdo producer software is not available yet. Further the filesystem must store the content of each Debian package as one consequtive sequence of bytes. Fragmentation is not supported by jigdo. For some reason even fancy container systems use ISOs for deployment. https://nanikjava.github.io/post/insideminikubeiso/ The example i found is BIOS-only and (virtual) CD-only https://storage.googleapis.com/minikube/iso/minikube-v1.0.0.iso Maybe it's because the user would need to start an ISO 9660 manipulating program in order to alter them. This hardly happens by mistake. Have a nice day :) Thomas
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
> Thank you Thomas. Yes, that's obviously why it was done.So - a quick look > on Wikipedia suggests that this was a current machine in 2006 and was > replaced in 2007 / 2008. So - if anyone has one running anywhere, it's > somewhere between 12-15 years old. If anybody knows of any that they really > must keep running, speak now or forever hold your peace. It's probably time > for this to be dropped for Bullseye. I have such a macmini-1,1 which I upgraded to macmini-2,1 (it's the same hardware and the firmware upgrade was needed for one of the other upgrades (can't remember if it was to bring up the RAM to 3GB or to upgrade the CPU to a Core 2 Duo)). [ It's my office desktop, happily driving two 1600x1200 monitors (with a DVI-I => DVI-D + VGA splitter), something which Apple's own OS never bothered to support. ] But IIRC I install Debian on it by cloning some existing Debian root filesystem directly onto the drive, rather than doing the "normal Debian install". So don't make those extra sets for me. Stefan
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
I'm just a user, but: * I disagree with dropping the first 4GB "DVD" install image, I think it's quite useful (although you might just name it "4GB" and drop the DVD naming). Users shouldn't need Internet to have a mostly functional system, and I cannot imagine they produce too much extra overhead to maintain compared to the netinst images for example. * I agree that non-install images (-*DVD-2, etc.) should be dropped. Users can use tools such as debmirror to easily create a local repository if they wish to. * I know this is somewhat controversial, but images with non-free firmware are also quite important in practice. I suggest to also produce a 16GB image with non-free firmware. * 16GB is a common size for cheap USB drives nowadays, so I think 4GB and 16GB would be good sizes for these offline images. Thank you. Às 23:48 de 18/07/20, Dan Ritter escreveu: Andrew Cater wrote: Separately, we also had a quick think about the numbers of iso images in general. A suggestion: For the future, we should produce physical media for the netinst.iso, the first DVD image in any set and one larger image to be written to a USB stick if wanted - and corresponding source for each size. All other .iso files to be distributed as .jigdo and .template files.In most instances, the netinst is enough, if you have network connectivity and bandwidth. The DVD is enough to install the basis of any of the graphical environments readily. This does not mean that you couldn't produce every other image - but very few people are on a desert island and need every piece of software Debian has produced on physical media. At this point in 2020, I think it would be reasonable to only produce netinst images and jigdo (and live, but that's a different-ish project). Drop the DVD images. All of my installs in the last six years have either been via a USB netinst image or PXE. New laptops and desktops are generally shipping without optical drives at all. My oldest non-toy hardware can boot via USB. -dsr-
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
Andrew Cater wrote: > Separately, we also had a quick think about the numbers of iso images in > general. A suggestion: For the future, we should produce physical media for > the netinst.iso, the first DVD image in any set and one larger image to be > written to a USB stick if wanted - and corresponding source for each size. > All other .iso files to be distributed as .jigdo and .template files.In > most instances, the netinst is enough, if you have network connectivity and > bandwidth. The DVD is enough to install the basis of any of the graphical > environments readily. This does not mean that you couldn't produce every > other image - but very few people are on a desert island and need every > piece of software Debian has produced on physical media. At this point in 2020, I think it would be reasonable to only produce netinst images and jigdo (and live, but that's a different-ish project). Drop the DVD images. All of my installs in the last six years have either been via a USB netinst image or PXE. New laptops and desktops are generally shipping without optical drives at all. My oldest non-toy hardware can boot via USB. -dsr-
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
Hi, Andrew Cater wrote: > If anybody knows of any [Mac Mini] that they really must keep > running, speak now or forever hold your peace. It is possible to create an ISO without EFI boot lures from a normal i386 or amd64 ISO. Follow https://wiki.debian.org/RepackBootableISO but leave out the options beginning with -eltorito-alt-boot Others seem to have solved their mini mac problems similarly https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface#Remove_UEFI_boot_support_from_optical_media > For the future, we should produce physical media for > the netinst.iso, the first DVD image in any set and one larger image to be > written to a USB stick if wanted - and corresponding source for each size. A large image for USB stick or 25 GB Blu-ray would help those with insufficient network access at installation time. > All other .iso files to be distributed as .jigdo and .template files. Most DVD images and images for all larger media are already only available via .jigdo. Jigdo's software status is healthy. But we need better documentation and more advertising for it. How about a jigdo help file in the download directories with .jigdo files, which tells how to get the download software and how to use it. Steve McIntyre offers binaries for MS-Windows and source at https://www.einval.com/~steve/software/jigdo/download/ I documented the endeavor of getting and booting a Debian Live for the purpose of downloading jigdo ISOs in https://wiki.debian.org/JigdoOnLive (Still lacking knowledge about how to mount foreign OS disks for writing.) Then there is https://www.debian.org/CD/jigdo-cd/ We could need a list of distros and names of their packages which contain jigdo-lite. This all should be mashed up and condensed to a tangible description what to do with the presented .jigdo and .template files. There is also the fact that due to network and server latency the download on a fast internet connection via jigdo is much slower than direct download via HTTPS. (Netinst installation might have the same drawbacks. How much faster is installation from a 32 GB USB stick in comparison to netinst ?) Have a nice day :) Thomas
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
Thank you Thomas. Yes, that's obviously why it was done.So - a quick look on Wikipedia suggests that this was a current machine in 2006 and was replaced in 2007 / 2008. So - if anyone has one running anywhere, it's somewhere between 12-15 years old. If anybody knows of any that they really must keep running, speak now or forever hold your peace. It's probably time for this to be dropped for Bullseye. Separately, we also had a quick think about the numbers of iso images in general. A suggestion: For the future, we should produce physical media for the netinst.iso, the first DVD image in any set and one larger image to be written to a USB stick if wanted - and corresponding source for each size. All other .iso files to be distributed as .jigdo and .template files.In most instances, the netinst is enough, if you have network connectivity and bandwidth. The DVD is enough to install the basis of any of the graphical environments readily. This does not mean that you couldn't produce every other image - but very few people are on a desert island and need every piece of software Debian has produced on physical media. Producing every disk in every size would still be possble but using jigdo as an intermediate step. This would also mean dropping the single CD image that installs the XFCE graphical environment but that's one tasksel selection away anyway. This would ease the pressure on mirrors and the main cdimage.debian.org server and make the task of testing significantly simpler. New users find it difficult to work out why there should be so many DVDs and whether they need to download them all even though only one is bootable. Anything we can do to make our collective lives easier is a bonus, given the size of the team. Thoughts? On Sat, Jul 18, 2020 at 5:55 PM Thomas Schmitt wrote: > Hi, > > Andrew Cater wrote: > > [...] here are > > a couple of architectures that are built but not tested because the > testers > > have no hardware. > > One set is an old Mac on Intel media, > > Do you mean these ? > > https://cdimage.debian.org/debian-cd/current/i386/iso-cd/debian-mac-10.4.0-i386-netinst.iso > > https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-mac-10.4.0-amd64-netinst.iso > > They are related to > https://wiki.debian.org/MacMiniIntel#Macmini_1.2C1 > and following sections. > > - > > If it is not about above "mac" ISOs, which then ? I am curious. > > > Have a nice day :) > > Thomas > >
Re: Final CDs being written for Stretch - 9.13 release - prior to LTS
Hi, Andrew Cater wrote: > [...] here are > a couple of architectures that are built but not tested because the testers > have no hardware. > One set is an old Mac on Intel media, Do you mean these ? https://cdimage.debian.org/debian-cd/current/i386/iso-cd/debian-mac-10.4.0-i386-netinst.iso https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-mac-10.4.0-amd64-netinst.iso They are related to https://wiki.debian.org/MacMiniIntel#Macmini_1.2C1 and following sections. - If it is not about above "mac" ISOs, which then ? I am curious. Have a nice day :) Thomas
Final CDs being written for Stretch - 9.13 release - prior to LTS
As folk will be aware: we're building and releasing the final release of Stretch before transition to LTS support. Testing is going well - there are a couple of architectures that are built but not tested because the testers have no hardware. One set is an old Mac on Intel media, the other is boot media for S390X. Nobody has reported problems with these recently but it may be that nobody is using them at all. If this is the case, these are prime candidates for removal before Bullseye - Debian 11 - media preparation Please shout: if we don't build media for obsolete hardware or machines that nobody has, it makes the complex task of preparing media that much simpler.
