Re: [Freedos-user] ISO repack reduces size by 10%
Hi Darik and Jerome, > * The 1.44 MB floppy set is reduced from eight to three diskettes. > * The 1.20 MB floppy set is reduced from ten to four diskettes. > * The 720 KB floppy set is reduced from fifteen to six diskettes. That is a huge difference and RAR is not overly exotic. I also would like to point out that infozip also comes with zipsplit, so having archives spanning multiple disks is nothing strange for the well-established packers, including ZIP. > Creating 360 KB floppies or smaller would require a big extension of > the installation program. To be honest, people who have only 360k drives are unlikely to have any harddisk where you could install to. The one workstation type PC which had both (floppy limited to 360k and a harddisk) that I know about was too luxurious to bother to be fully PC compatible, so it would not run FreeDOS, although it could run some of our apps. So for THAT tiny disk size of 360k, I would rather focus on having a good way to work directly from floppy without installing. Preferably optimized for having very little RAM. Probably something MS DOS 3 style, with some of the modern features stripped away. > 1a. Replace FREEDOS.SAF with a FREEDOS.TGZ tarball. > 1b. Use coreutils to "SPLIT FREEDOS.TGZ" across floppy media. > 1c. Unpack it with a "COPY /B PART1+PART2" method and some batch logic. As mentioned, that would require extra temp space for the merged tarball before you unpack it, as DOS has no in-memory pipelines, but of course for our DISKETTE distro, the idea is to include only a limited selection of apps, so the tarball size is limited as well and could be okay. I myself once made a three floppy proof of concept distro with all BASE packages pre-installed, which had one archive file with various less important files on the third. Regards, Eric ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] ISO repack reduces size by 10%
Hi Darik, > On Nov 15, 2021, at 5:08 AM, Darik Horn wrote: > > Jerome, > >> If you can figure out how to get it to work, then it will be worth >> considering. > > It works. Updated builds are posted here: > > https://drive.google.com/drive/folders/1b7IE-EsK5R1ROmXTaEvYNi9-kfVvH5wi?usp=sharing I’ll try to look at it later today. > > * The 1.44 MB floppy set is reduced from eight to three diskettes. > * The 1.20 MB floppy set is reduced from ten to four diskettes. > * The 720 KB floppy set is reduced from fifteen to six diskettes. > > Creating 360 KB floppies or smaller would require a big extension of > the installation program. Yes. The installer needs updated to support smaller disk sizes. Eventually, I will add it. It wont be difficult. But, 360s aren’t common compared to the larger capacities. Therefore, it isn’t a priority. > > This update also includes forward error correction for bad media. > > The nearby BOOT.diff file describes what changed to make the demo happen. > > >> 1) Wether or not it’s License is using an acceptable open source license. >> 2) Usage is approved. Not up to me. But if licenses are ok, it should not be >> a problem. > > The encoder is non-free and zero-cost, but the demo otherwise > implements all features. Not being open source will probably be an issue. > > >> 3) Can be told to create spans of a specific size (so can be used on all >> floppy variations, one size fits all). > > Yes. > > >> 4) Can replace SLICER with little to no overhead added to the installer >> and/or RBE (Release Build Environment) >> ... >> https://github.com/shidel/FDI-x86/tree/master/SETTINGS > > Thanks for posting this Github link, which I will review later. I > looked for a FreeDOS build system earlier but didn't find it. Your welcome. The RBE is at https://gitlab.com/FreeDOS/OS/builder There are also mirrors for FDI & FDI-x86 there as well as some other stuff. If your interested in building a FreeDOS release using the RBE, it’s rather easy to setup and do. Just follow the wiki for it on GitLab. The RBE itself is rather complicated and is a work-in-progress. But it works. :-) On a side note, here is just something to ponder… A typical end user is installing FreeDOS into VirtualBox on Windows… So… I sit here running a Mac. Using VirtualBox, to run a Linux VM for the RBE. The RBE downloads and processes the raw installers and packages. It then dynamically builds some custom boot images and runs QEMU instances with FreeDOS. Those instances run batch files generated by the RBE to create the different install media. That media then is loaded into a different machine by the end user. They then run another virtual machine and run the installer for FreeDOS. The installer then installed the OS and creates custom config files for that system. And that was an over-simplification of a release build to install chain. Trying to do it by hand would be crazy making and prone to mistakes. > > >> 5) Can provide the same level of flexibility without the need to manually >> modify code when there are package/tag changes. > > In the updated demo, a @LISTFILE substitutes each possible "SLICER /G > %TPU%" invocation. > > >> Can it display general text during the install process? > > Yes, this can be scripted through the IDOS.SFX module. The given demo > displays progress per-file, which is the default. > > >> Does it support NLS and provide text in the users language for prompts? > > Dunno about NLS, but it can do ANSI.SYS style text in if/then/else blocks. > > >> On systems that support change-line, can it auto-resume without being told >> to continue? > > Dunno what change-line support is... > > But the extractor won't prompt for a media change if it can already > see the next volume, so floppy contents can be copied to a ZipDrive or > HDD and the installer will behave properly. > > >> After completion, can it just return an errorlevel to the installer and not >> stop with an “I’m done, Press OK” message? > > The available %ERRORLEVEL% returns are sensible and actionable. > > >> Those and other settings are available in the projects SETTINGS directory. >> Specifically, the X86_ALL.LST defines what gets what slicer tags for >> assembling the archive. > > This RAR demo is somewhat an accident and I appreciate how licensing > matters. Two alternatives to using RAR or finishing SLICER come to > mind now that I better understand the solution: > > 1a. Replace FREEDOS.SAF with a FREEDOS.TGZ tarball. > 1b. Use coreutils to "SPLIT FREEDOS.TGZ" across floppy media. > 1c. Unpack it with a "COPY /B PART1+PART2" method and some batch logic. > > This is how Linux distributions did installation before CD-R media was > invented. Zero new compiled code is required to implement it for > FreeDOS, and it will cost 10 MB of temporary hard disk space at > install time. > > 2. Use the GPL licensing option in unrarlib to create a RAR2 encoder. > Make a "grar" u
Re: [Freedos-user] ISO repack reduces size by 10%
Hi Darik, I appreciate the effort you put into creating a PowerShell script to repack the repository. However, I do not know if we are going to do that. First, the new versions would need to be checked to ensure compatibility. They would probably be fine. But, testing would still need done. I honestly just done have the time for that at present. Second, there is no way you could’ve known this. But, none of the machines (virtual or real) run Windows. Mostly Linux, some DOS and macOS (unix/BSD). There are numerous utilities and scripts we use that operate on all packages in the repo and elsewhere. So, whipping up something to run through them and recompress, is not too difficult. But, I think it is to late to consider such a large unplanned change this close to the release 1.3-RC5. Once RC5 is released, I will look into it some and possibly add some recompression into the tool chain. :-) Jerome ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] ISO repack reduces size by 10%
Jerome, > If you can figure out how to get it to work, then it will be worth > considering. It works. Updated builds are posted here: https://drive.google.com/drive/folders/1b7IE-EsK5R1ROmXTaEvYNi9-kfVvH5wi?usp=sharing * The 1.44 MB floppy set is reduced from eight to three diskettes. * The 1.20 MB floppy set is reduced from ten to four diskettes. * The 720 KB floppy set is reduced from fifteen to six diskettes. Creating 360 KB floppies or smaller would require a big extension of the installation program. This update also includes forward error correction for bad media. The nearby BOOT.diff file describes what changed to make the demo happen. > 1) Wether or not it’s License is using an acceptable open source license. > 2) Usage is approved. Not up to me. But if licenses are ok, it should not be > a problem. The encoder is non-free and zero-cost, but the demo otherwise implements all features. > 3) Can be told to create spans of a specific size (so can be used on all > floppy variations, one size fits all). Yes. > 4) Can replace SLICER with little to no overhead added to the installer > and/or RBE (Release Build Environment) > ... > https://github.com/shidel/FDI-x86/tree/master/SETTINGS Thanks for posting this Github link, which I will review later. I looked for a FreeDOS build system earlier but didn't find it. > 5) Can provide the same level of flexibility without the need to manually > modify code when there are package/tag changes. In the updated demo, a @LISTFILE substitutes each possible "SLICER /G %TPU%" invocation. > Can it display general text during the install process? Yes, this can be scripted through the IDOS.SFX module. The given demo displays progress per-file, which is the default. > Does it support NLS and provide text in the users language for prompts? Dunno about NLS, but it can do ANSI.SYS style text in if/then/else blocks. > On systems that support change-line, can it auto-resume without being told to > continue? Dunno what change-line support is... But the extractor won't prompt for a media change if it can already see the next volume, so floppy contents can be copied to a ZipDrive or HDD and the installer will behave properly. > After completion, can it just return an errorlevel to the installer and not > stop with an “I’m done, Press OK” message? The available %ERRORLEVEL% returns are sensible and actionable. > Those and other settings are available in the projects SETTINGS directory. > Specifically, the X86_ALL.LST defines what gets what slicer tags for > assembling the archive. This RAR demo is somewhat an accident and I appreciate how licensing matters. Two alternatives to using RAR or finishing SLICER come to mind now that I better understand the solution: 1a. Replace FREEDOS.SAF with a FREEDOS.TGZ tarball. 1b. Use coreutils to "SPLIT FREEDOS.TGZ" across floppy media. 1c. Unpack it with a "COPY /B PART1+PART2" method and some batch logic. This is how Linux distributions did installation before CD-R media was invented. Zero new compiled code is required to implement it for FreeDOS, and it will cost 10 MB of temporary hard disk space at install time. 2. Use the GPL licensing option in unrarlib to create a RAR2 encoder. Make a "grar" utility like there is "gzip", which would have general application, especially in the retro computing. ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] ISO repack reduces size by 10%
Jerome,
This PowerShell script generates the optimization that I want as an
end-user on the 'freedos/files/repositories/latest' tree. Only 7Z.exe
is used because 7-Zip 19.00 and AdvZip 2.1 generate outputs that are
nearly identical for a full repack.
# FDREPACK.PS1: FreeDOS repository repacking script.
$ZIPS = Get-ChildItem -Recurse -File -Filter *.ZIP
ForEach ($ZIP in $ZIPS) {
$TMPDIR = Join-Path $ENV:TMP -ChildPath $('FDREPACK_' +
$ZIP.BaseName + '_' + $(New-GUID))
$TMPDIR = New-Item -ItemType Directory -Path $TMPDIR
Push-Location $TMPDIR
7Z x $ZIP.FullName
If (Test-Path -Path 'SOURCE' -PathType Container) {
Push-Location 'SOURCE'
ForEach ($II in @(Get-ChildItem -File -Filter '*.7Z';
Get-ChildItem -File -Filter '*.ZIP')) {
NewItem -ItemType Directory -Path $II.BaseName
Push-Location -Path $II.BaseName
7Z x $II.FullName
Pop-Location
Remove-Item $II.FullName
}
ForEach ($JJ in Get-ChildItem -Directory) {
Push-Location $JJ
If (Test-Path -Path 'SOURCES.7Z' -PathType Leaf) {
7z x 'SOURCES.7Z'
Remove-Item 'SOURCES.7Z'
}
If (Test-Path -Path 'SOURCES.ZIP' -PathType Leaf) {
7z x 'SOURCES.ZIP'
Remove-Item 'SOURCES.ZIP'
}
ATTRIB -A /S
# This makes upstream sources solid in the distro package.
7Z a -tzip -mm=copy -mtc=off -stl $('..\' + $JJ + '.ZIP')
Pop-Location
Remove-Item -Recurse $JJ
}
Pop-Location # SOURCE
}
ATTRIB -A /S
# These are the best zip format 2.0 parameters at the pkzip-2.0 level.
7Z a -tzip -mm=deflate -mfb=258 -mpass=15 -mmt=on -mtc=off -stl
$($ZIP.FullName + '.repack')
Move-Item -Force $($ZIP.FullName + '.repack') ($ZIP.FullName)
Pop-Location # $TMPDIR
Remove-Item -Force -Recurse $TMPDIR
}
___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user
