Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik, 

One more thing…

The “workflow” for packages…

I download, restructure, compile, etc. the software.

It then gets staged in completely uncompressed package format as a project at 
https://gitlab.com/FDOS 
(The  sub-groups & projects there will probable be moving to 
https://gitlab.com/FreeDOS/  in the near future)

Once that is done, I zip it up using a older script I wrote a while back. 
(eventually, I’ll add support to do it to the fdvcs.sh script)

Then upload that to my personal repository.

The repo management utilities run on it hourly and will tweak a couple meta 
data fields and place in it’s repo at
https://fd.lod.bz/repos/current/pkg-html/ 


Once it appears in that repo. I download the package from my server and upload 
it to ibiblio.

The repo management utilities run on daily. It will see the metadata fields in 
the package are already modified and 
just put it in the official repository at 
https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/repositories/latest/pkg-html/
 


(This ensures that for the packages in both locations (more on my personal 
server) are identical and hash the same)

Then we come to making a release.

The RBE downloads the latest CDROM.ISO from ibiblio (created by the repo 
management utility) and does a bunch of 
stuff to the packages. Including integrating some NLS stuff from the NLS 
project at https://github.com/shidel/fd-nls 

Finally, the RBE zips those up and produces the release media.

Although with the recent addition of package staging at GitLab and several 
other things, I think a lot of the processing
done on packages in the RBE is going to go away and move further up the chain 
to the package prep stage.

:-)

Jerome





___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

Also...

> On Nov 16, 2021, at 1:00 PM, Darik Horn  wrote:
> [..]
> This bash script does the same thing.  The file is also in the shared
> folder in case the listserv mangles it here.  I will submit PRs for
> additional work now that I know about the Gitlab account.

FYI, I still need to make some updates to the RBE to account for some changes 
coming in RC5. 

And it could use improvement in several areas and tons of optimization. But, it 
works and I’ve only got
so much time to spend on it, the installers, required tools, etc.

> #!/bin/bash
> # fdrepack.sh: FreeDOS repository repacking script.
> 
> fdrepack ()
> {
>  TMPDIR=$(mktemp -d)
>  pushd "$TMPDIR" >/dev/null
>  unzip -q "$1"
> 
>  if [[ -d 'SOURCE' ]]
>  then
>pushd 'SOURCE' >/dev/null
>for ii in *.7[Zz]
>do
>  if [[ -r "$ii" ]]
>  then
># Force the source package name to uppercase.
>mkdir $(basename "${ii@U}" '.7Z')
>pushd $(basename "${ii@U}" '.7Z') >/dev/null
>7z x "../$ii"
>popd >/dev/null
>rm "$ii"
>  fi
>done
>for ii in *.[Zz][Ii][Pp]
>do
>  if [[ -r "$ii" ]]
>  then
># Force the source package name to uppercase.
>mkdir $(basename "${ii@U}" '.ZIP')
>pushd $(basename "${ii@U}" '.ZIP') >/dev/null
>unzip -q "../$ii"
>popd >/dev/null
>rm "$ii"
>  fi
>done
> 
>for ii in *
>do
>  if pushd "$ii" >/dev/null
>  then
># Unpack and delete old LFN source archives.
>find -maxdepth 1 -type f -iname sources.7z -exec 7z x {} \;
> -exec rm {} \;
>find -maxdepth 1 -type f -iname sources.zip -exec unzip -q {}
> \; -exec rm {} \;
># Using the store method here makes upstream sources solid in
> the package zip.
>zip -0Xoqr "../${ii@U}.ZIP" .
>popd >/dev/null
>rm -rf "$ii"
>  fi
>done
>popd >/dev/null
>  fi
> 
>  # Use InfoZIP here for the -k and -o switches.
>  zip -0Xkoqr "${1}.repack"
>  advzip -k -p -z -3 -i 15 "${1}.repack"
>  mv "${1}.repack" "${1}"
>  popd >/dev/null
>  rm -rf "$TMPDIR"
> }
> 
> export -f fdrepack
> find ~+ -type f -iname \*.zip -exec bash -c 'fdrepack "{}"' \;
> 

Just some notes on the script. Overall, it looks fine. I did not go through 
each switch. But, I assume they are correct.

ibiblio is the official repo, OS download site and software mirror server. It 
is graciously hosted by the University of
North Carolina. Neither 7zip or advzip are installed on that server. Installing 
linux packages is out. However, we can 
and do run some “mission critical” binaries. So, depending on dependencies we 
could run them. 

${var@U} is not supported by bash on ibiblio, nor the version on my Mac (where 
I run a lot of this stuff for prep, testing, etc).
ibiblio does support ${var^^}, however the Mac doesn’t. So, basically for 
compatibility I end up using some functions that
work on both. (yes, I know they don’t support international characters.)

[[ "$(uname)" == "Darwin" ]] && MACOS=true || unset MACOS
UPPER_CHARS='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
LOWER_CHARS='abcdefghijklmnopqrstuvwxyz'

function lowerCase () {
if [[ ${MACOS} ]] ; then
# Slower method not using ${variable,,}
local i c o
for ((i=0;i<${#1};i++)) ; do
c="${1:${i}:1}"
if [[ "${c//[${UPPER_CHARS}]}" != "${c}" ]] ; then
c="${UPPER_CHARS%%${c}*}"
c="${LOWER_CHARS:${#c}:1}"
fi
o="${o}${c}"
done
echo "${o}"
else
echo "${1,,}"
fi
}

function upperCase () {
if [[ ${MACOS} ]] ; then
# Slower method not using ${variable^^}
local i c o
for ((i=0;i<${#1};i++)) ; do
c="${1:${i}:1}"
if [[ "${c//[${LOWER_CHARS}]}" != "${c}" ]] ; then
c="${LOWER_CHARS%%${c}*}"
c="${UPPER_CHARS:${#c}:1}"
fi
o="${o}${c}"
done
echo "${o}"
else
echo "${1^^}"
fi
}

Obviously, they are no where near as fast as the built-in shell versions. 
But generally, are much faster than using tr “[:lower:]” “[:upper:]” and the 
like.
Technically, since using them through a sub-shell like fname=$(upperCase 
“${oname}”) 
it doesn’t really need to declare the vars as local.

Archive timestamps are lost. While technically a new zip file, it may be more 
useful to apply
the old archives timestamp to the new one. It is not difficult. I do something 
like that in the 
fdvcs.sh script (part of the PkgDevKit, another work in progress). It is just 
something that
we will need to consider.

While it would be good to reduce the sizes of the zip archives, there is 
currently plenty of 
room on the different release media. At present, I have several dozen critical 
things that
need done for RC5. After RC5, I will look into it more.

One area that could use volunteers now is package updates. There are roughly as 
dozen
BASE packages 

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

> On Nov 16, 2021, at 1:01 PM, Darik Horn  wrote:
> 
> Jerome,
> 
>> First, the new versions would need to be checked to ensure compatibility. 
>> They would probably be fine. But, testing would still need done. I honestly 
>> just done have the time for that at present.
> 
> This work is, in part, the result of doing coverage testing on the 
> distribution.
> 
> 
>> But, I think it is to late to consider such a large unplanned change this 
>> close to the release 1.3-RC5.
> 
> Note that this fixes the fails-to-install-all-packages-with-source bug
> in the latest release candidate.

No. 

The package for HTMLHELP used a source directory as HELP which conflicted with 
another package. I’ve corrected the package on the repository. So, there is no 
longer a conflict.

>> I appreciate the effort you put into creating a PowerShell script to repack 
>> the repository. However, I do not know if we are going to do that.
> 
> This bash script does the same thing.  The file is also in the shared
> folder in case the listserv mangles it here.  I will submit PRs for
> additional work now that I know about the Gitlab account.
> 
> 
> #!/bin/bash
> # fdrepack.sh: FreeDOS repository repacking script.
> 
> fdrepack ()
> {
>  TMPDIR=$(mktemp -d)
>  pushd "$TMPDIR" >/dev/null
>  unzip -q "$1"
> 
>  if [[ -d 'SOURCE' ]]
>  then
>pushd 'SOURCE' >/dev/null
>for ii in *.7[Zz]
>do
>  if [[ -r "$ii" ]]
>  then
># Force the source package name to uppercase.
>mkdir $(basename "${ii@U}" '.7Z')
>pushd $(basename "${ii@U}" '.7Z') >/dev/null
>7z x "../$ii"
>popd >/dev/null
>rm "$ii"
>  fi
>done
>for ii in *.[Zz][Ii][Pp]
>do
>  if [[ -r "$ii" ]]
>  then
># Force the source package name to uppercase.
>mkdir $(basename "${ii@U}" '.ZIP')
>pushd $(basename "${ii@U}" '.ZIP') >/dev/null
>unzip -q "../$ii"
>popd >/dev/null
>rm "$ii"
>  fi
>done
> 
>for ii in *
>do
>  if pushd "$ii" >/dev/null
>  then
># Unpack and delete old LFN source archives.
>find -maxdepth 1 -type f -iname sources.7z -exec 7z x {} \;
> -exec rm {} \;
>find -maxdepth 1 -type f -iname sources.zip -exec unzip -q {}
> \; -exec rm {} \;
># Using the store method here makes upstream sources solid in
> the package zip.
>zip -0Xoqr "../${ii@U}.ZIP" .
>popd >/dev/null
>rm -rf "$ii"
>  fi
>done
>popd >/dev/null
>  fi
> 
>  # Use InfoZIP here for the -k and -o switches.
>  zip -0Xkoqr "${1}.repack"
>  advzip -k -p -z -3 -i 15 "${1}.repack"
>  mv "${1}.repack" "${1}"
>  popd >/dev/null
>  rm -rf "$TMPDIR"
> }
> 
> export -f fdrepack
> find ~+ -type f -iname \*.zip -exec bash -c 'fdrepack "{}"' \;
> 
> 
> ___
> Freedos-user mailing list
> Freedos-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/freedos-user



___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Darik Horn]

Jerome,

> First, the new versions would need to be checked to ensure compatibility. 
> They would probably be fine. But, testing would still need done. I honestly 
> just done have the time for that at present.

This work is, in part, the result of doing coverage testing on the distribution.


> But, I think it is to late to consider such a large unplanned change this 
> close to the release 1.3-RC5.

Note that this fixes the fails-to-install-all-packages-with-source bug
in the latest release candidate.


> I appreciate the effort you put into creating a PowerShell script to repack 
> the repository. However, I do not know if we are going to do that.

This bash script does the same thing.  The file is also in the shared
folder in case the listserv mangles it here.  I will submit PRs for
additional work now that I know about the Gitlab account.


#!/bin/bash
# fdrepack.sh: FreeDOS repository repacking script.

fdrepack ()
{
  TMPDIR=$(mktemp -d)
  pushd "$TMPDIR" >/dev/null
  unzip -q "$1"

  if [[ -d 'SOURCE' ]]
  then
pushd 'SOURCE' >/dev/null
for ii in *.7[Zz]
do
  if [[ -r "$ii" ]]
  then
# Force the source package name to uppercase.
mkdir $(basename "${ii@U}" '.7Z')
pushd $(basename "${ii@U}" '.7Z') >/dev/null
7z x "../$ii"
popd >/dev/null
rm "$ii"
  fi
done
for ii in *.[Zz][Ii][Pp]
do
  if [[ -r "$ii" ]]
  then
# Force the source package name to uppercase.
mkdir $(basename "${ii@U}" '.ZIP')
pushd $(basename "${ii@U}" '.ZIP') >/dev/null
unzip -q "../$ii"
popd >/dev/null
rm "$ii"
  fi
done

for ii in *
do
  if pushd "$ii" >/dev/null
  then
# Unpack and delete old LFN source archives.
find -maxdepth 1 -type f -iname sources.7z -exec 7z x {} \;
-exec rm {} \;
find -maxdepth 1 -type f -iname sources.zip -exec unzip -q {}
\; -exec rm {} \;
# Using the store method here makes upstream sources solid in
the package zip.
zip -0Xoqr "../${ii@U}.ZIP" .
popd >/dev/null
rm -rf "$ii"
  fi
done
popd >/dev/null
  fi

  # Use InfoZIP here for the -k and -o switches.
  zip -0Xkoqr "${1}.repack"
  advzip -k -p -z -3 -i 15 "${1}.repack"
  mv "${1}.repack" "${1}"
  popd >/dev/null
  rm -rf "$TMPDIR"
}

export -f fdrepack
find ~+ -type f -iname \*.zip -exec bash -c 'fdrepack "{}"' \;


___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user