Re: [pacman-dev] [PATCH v3 4/8] bacman: rewrite usage page and version information

[Allan McRae]

On 01/09/16 13:59, Allan McRae wrote:
> On 01/09/16 07:05, Gordian Edenhofer wrote:
>> Signed-off-by: Gordian Edenhofer 
>> ---
>>  contrib/bacman.sh.in | 33 +
>>  1 file changed, 21 insertions(+), 12 deletions(-)
>>
> 
> The format for the header is the same for all scripts in the pacman
> codebase.  Do not change the format. You may add examples onto the end.
> 
> 
>> diff --git a/contrib/bacman.sh.in b/contrib/bacman.sh.in
>> index a0bfcf1..ca2945f 100644
>> --- a/contrib/bacman.sh.in
>> +++ b/contrib/bacman.sh.in
>> @@ -41,25 +41,33 @@ clean_up() {
>>  echo
>>  exit
>>  }
>> -
>>  # Trap termination signals
>>  trap clean_up SIGHUP SIGINT SIGTERM
>>  
>>  # Print usage information
>>  usage() {
>> -echo "${myname} (pacman) v${myver}"
>> -echo
>> -echo "Recreate a package using pacman's database and system files"
>> -echo
>> -echo "Usage: ${myname} [--nocolor] [--pacnew] "
>> -echo
>> -echo "Example: ${myname} linux-headers"
>> +cat <<-EOF
>> +$myname (pacman) v$myver
>> +This script was design to reassemble installed packages from its 
>> deliverd files.
> 
> Typo
> 
>> +It comes in handy if you have no access to an up-to-date package cache.
>> +
>> +Usage: $myname [options] 
>> +-h, --help  display this help message and exit
>> +--nocolor   turn off color in output
>> +--pacnewpackage .pacnew files
>> +
>> +Examples:   # $myname linux-headers
>> +# $myname gzip munge binutils -o ~/Downloads

Also, the -o option does not exist at this point.

Re: [pacman-dev] [PATCH v3 5/8] bacman: add option to print fewer status updates

[Allan McRae]

On 01/09/16 07:05, Gordian Edenhofer wrote:
> Signed-off-by: Gordian Edenhofer 
> ---
>  contrib/bacman.sh.in | 21 -
>  1 file changed, 16 insertions(+), 5 deletions(-)
> 

Look at the definition of msg2() etc:

scripts/library/output_format.sh

Re: [pacman-dev] [PATCH v3 3/8] bacman: proper option handling

[Allan McRae]

On 01/09/16 07:05, Gordian Edenhofer wrote:
> Switch to parseopts instead of merely checking the first argument.
> 
> Signed-off-by: Gordian Edenhofer 
> ---
>  contrib/bacman.sh.in | 63 
> ++--
>  1 file changed, 41 insertions(+), 22 deletions(-)
> 



> +# Parse arguments
> +OPT_SHORT=':v'

Why the colon?

[pacman-dev] [PATCH] Fix typo in pacman-db-upgrade usage message

[Allan McRae]

Signed-off-by: Allan McRae 
---
 scripts/pacman-db-upgrade.sh.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/pacman-db-upgrade.sh.in b/scripts/pacman-db-upgrade.sh.in
index 94085ba..79e1c59 100644
--- a/scripts/pacman-db-upgrade.sh.in
+++ b/scripts/pacman-db-upgrade.sh.in
@@ -40,7 +40,7 @@ usage() {
printf -- "$(gettext "Usage: %s [options]")\n" "$0"
echo
printf -- "$(gettext "options:")\n"
-   printf -- "$(gettext "  -b, --dbpath   set an alternate database 
location")\n"
+   printf -- "$(gettext "  -d, --dbpath   set an alternate database 
location")\n"
printf -- "$(gettext "  -h, --help   show this help message and 
exit")\n"
printf -- "$(gettext "  -r, --root set an alternate 
installation root")\n"
printf -- "$(gettext "  -V, --versionshow version information 
and exit")\n"
-- 
2.9.3

Re: [pacman-dev] [PATCH v2 1/3] bacman: allow for parallel packaging

[Allan McRae]

On 01/09/16 09:44, Gordian Edenhofer wrote:
> On Thu, 2016-09-01 at 08:28 +1000, Allan McRae wrote:
>> On 01/09/16 08:08, Dave Reisner wrote:
>>>
>>> On Wed, Aug 31, 2016 at 11:18:32PM +0200, Gordian Edenhofer wrote:

>
>>
>>>
>>> The second probably would not be accepted...
>>
>> I urge you to reconsider. Parallelization increases the speed
>> of
>> this
>
> I don't think anyone is suggesting that packaging multiple
> things in
> parallel isn't useful. I already suggested that nothing needs
> to be
> implemented in bacman proper in order for you to parallelize
> the
> work.
> You can write your own "pbacman" as simply as:
>
>   for arg; do bacman "$arg" & done; wait

 There is a huge difference between flooding your system with
 ~1000 jobs
 and tightly controlling the maximum number. Adjusting the precise
 number of jobs enables you to organize your resources which
 itself is
 desirable.
>>>
>>> Then use a program like 'parallel' which has this sort of knob. I
>>> really
>>> wonder what it is you're doing that requires running bacman with a
>>> large
>>> number of packages with any regularity.
>>>
>>
>> Gathering the files etc takes no time.  It is really the compression
>> that is being made parallel.  If only there was a way to set
>> compression
>> to use mutlithreading...
> 
> The actual compression using xz (default) is not necessary the most
> time intensive part. The linux-headers package for example is
> compressed within a few seconds but the whole process before xz is run
> takes way longer. This can be seen with top as an illustration or
> simply by running bacman one time without compression and the other
> with.
> Moreover using bacman to parallelize makes it completely independent
> from the archive format used and still brings gains when recreating
> multiple packages. At the very least it would fill the gap in between
> the compression of multiple packages. Therefore it would be beneficial
> even if compression would take the longest which is doesn't always do.
> 

So read speed is the slow part?   And trying to read more files at the
same time helps?

Allan

Re: [pacman-dev] [PATCH v2 1/3] bacman: allow for parallel packaging

[Gordian Edenhofer]

On Thu, 2016-09-01 at 08:28 +1000, Allan McRae wrote:
> On 01/09/16 08:08, Dave Reisner wrote:
> > 
> > On Wed, Aug 31, 2016 at 11:18:32PM +0200, Gordian Edenhofer wrote:
> > > 
> > > > 
> > > > > 
> > > > > > 
> > > > > > The second probably would not be accepted...
> > > > > 
> > > > > I urge you to reconsider. Parallelization increases the speed
> > > > > of
> > > > > this
> > > > 
> > > > I don't think anyone is suggesting that packaging multiple
> > > > things in
> > > > parallel isn't useful. I already suggested that nothing needs
> > > > to be
> > > > implemented in bacman proper in order for you to parallelize
> > > > the
> > > > work.
> > > > You can write your own "pbacman" as simply as:
> > > > 
> > > >   for arg; do bacman "$arg" & done; wait
> > > 
> > > There is a huge difference between flooding your system with
> > > ~1000 jobs
> > > and tightly controlling the maximum number. Adjusting the precise
> > > number of jobs enables you to organize your resources which
> > > itself is
> > > desirable.
> > 
> > Then use a program like 'parallel' which has this sort of knob. I
> > really
> > wonder what it is you're doing that requires running bacman with a
> > large
> > number of packages with any regularity.
> > 
> 
> Gathering the files etc takes no time.  It is really the compression
> that is being made parallel.  If only there was a way to set
> compression
> to use mutlithreading...

The actual compression using xz (default) is not necessary the most
time intensive part. The linux-headers package for example is
compressed within a few seconds but the whole process before xz is run
takes way longer. This can be seen with top as an illustration or
simply by running bacman one time without compression and the other
with.
Moreover using bacman to parallelize makes it completely independent
from the archive format used and still brings gains when recreating
multiple packages. At the very least it would fill the gap in between
the compression of multiple packages. Therefore it would be beneficial
even if compression would take the longest which is doesn't always do.

Best Regards,
Gordian Edenhofer

signature.asc
Description: This is a digitally signed message part

Re: [pacman-dev] [PATCH v2 1/3] bacman: allow for parallel packaging

[Allan McRae]

On 01/09/16 08:08, Dave Reisner wrote:
> On Wed, Aug 31, 2016 at 11:18:32PM +0200, Gordian Edenhofer wrote:
> The second probably would not be accepted...

 I urge you to reconsider. Parallelization increases the speed of
 this
>>>
>>> I don't think anyone is suggesting that packaging multiple things in
>>> parallel isn't useful. I already suggested that nothing needs to be
>>> implemented in bacman proper in order for you to parallelize the
>>> work.
>>> You can write your own "pbacman" as simply as:
>>>
>>>   for arg; do bacman "$arg" & done; wait
>>
>> There is a huge difference between flooding your system with ~1000 jobs
>> and tightly controlling the maximum number. Adjusting the precise
>> number of jobs enables you to organize your resources which itself is
>> desirable.
> 
> Then use a program like 'parallel' which has this sort of knob. I really
> wonder what it is you're doing that requires running bacman with a large
> number of packages with any regularity.
> 

Gathering the files etc takes no time.  It is really the compression
that is being made parallel.  If only there was a way to set compression
to use mutlithreading...

As I already stated, I'm not looking to accept the parallel patch.

A

Re: [pacman-dev] [PATCH v2 1/3] bacman: allow for parallel packaging

[Dave Reisner]

On Wed, Aug 31, 2016 at 11:18:32PM +0200, Gordian Edenhofer wrote:
> > > > The second probably would not be accepted...
> > > 
> > > I urge you to reconsider. Parallelization increases the speed of
> > > this
> > 
> > I don't think anyone is suggesting that packaging multiple things in
> > parallel isn't useful. I already suggested that nothing needs to be
> > implemented in bacman proper in order for you to parallelize the
> > work.
> > You can write your own "pbacman" as simply as:
> > 
> >   for arg; do bacman "$arg" & done; wait
> 
> There is a huge difference between flooding your system with ~1000 jobs
> and tightly controlling the maximum number. Adjusting the precise
> number of jobs enables you to organize your resources which itself is
> desirable.

Then use a program like 'parallel' which has this sort of knob. I really
wonder what it is you're doing that requires running bacman with a large
number of packages with any regularity.

> Even if there would be a perfect wrapper for bacman - which there is
> none - it would still make sense to implement easily understandable
> options into bacman for everyone to use simply to spare others the time
> of coming up with one.
> 
> Best Regards,
> Gordian Edenhofer

Re: [pacman-dev] [PATCH v2 1/3] bacman: allow for parallel packaging

[Gordian Edenhofer]

> > > The second probably would not be accepted...
> > 
> > I urge you to reconsider. Parallelization increases the speed of
> > this
> 
> I don't think anyone is suggesting that packaging multiple things in
> parallel isn't useful. I already suggested that nothing needs to be
> implemented in bacman proper in order for you to parallelize the
> work.
> You can write your own "pbacman" as simply as:
> 
>   for arg; do bacman "$arg" & done; wait

There is a huge difference between flooding your system with ~1000 jobs
and tightly controlling the maximum number. Adjusting the precise
number of jobs enables you to organize your resources which itself is
desirable.
Even if there would be a perfect wrapper for bacman - which there is
none - it would still make sense to implement easily understandable
options into bacman for everyone to use simply to spare others the time
of coming up with one.

Best Regards,
Gordian Edenhofer

signature.asc
Description: This is a digitally signed message part

[pacman-dev] [PATCH v3 7/8] bacman: add option to specify the number of jobs

[Gordian Edenhofer]

Signed-off-by: Gordian Edenhofer 
---
 contrib/bacman.sh.in | 27 +--
 1 file changed, 21 insertions(+), 6 deletions(-)

diff --git a/contrib/bacman.sh.in b/contrib/bacman.sh.in
index 706c6fb..e8e53ea 100644
--- a/contrib/bacman.sh.in
+++ b/contrib/bacman.sh.in
@@ -57,11 +57,12 @@ usage() {
-q, --quiet silence most of the status reporting
--nocolor   turn off color in output
--pacnewpackage .pacnew files
+   -j, --jobsbuild in parallel - you may want to set 
XZ_OPT
-o, --out  write output to 
 
Examples:   # $myname linux-headers
# $myname gzip munge binutils -o ~/Downloads
-   # $myname --nocolor --pacnew -o /tmp gzip munge binutils
+   # $myname --nocolor --pacnew -o /tmp -j 5 gzip munge 
binutils
# $myname \$(pacman -Qsq)
EOF
 }
@@ -89,8 +90,8 @@ for option in "$@"; do
 done
 
 # Parse arguments
-OPT_SHORT=':o:qv'
-OPT_LONG=('out:' 'quiet' 'nocolor' 'pacnew' 'version')
+OPT_SHORT=':o:j:qv'
+OPT_LONG=('out:' 'jobs:' 'quiet' 'nocolor' 'pacnew' 'version')
 if ! parseopts "$OPT_SHORT" "${OPT_LONG[@]}" -- "$@"; then
usage
exit 1
@@ -104,6 +105,14 @@ while :; do
pkg_dest="$(readlink -e $2)"
[[ $? -ne 0 ]] && echo -e "The directory 
\e[39;1m$2\e[0m does not exist!" && exit 3
shift ;;
+   -j|--jobs)
+   if [[ $2 =~ ^-?[0-9]+$ ]]; then
+   MAX_JOBS=$2
+   else
+   echo -e "\e[39;1m$2\e[0m is not a valid 
integer!"
+   exit -1
+   fi
+   shift ;;
-q|--quiet)
QUIET='y' ;;
--nocolor)
@@ -190,7 +199,9 @@ fakebuild() {
cd "$work_dir" || exit 1
 
# Assemble list of files which belong to the package and tar them
-   if [[ "$QUIET" != "y" ]]; then
+   if [[ $MAX_JOBS -gt 1 && "$QUIET" != "y" ]]; then
+   msg2 "${pkg_name}: Copying package files..."
+   elif [[ "$QUIET" != "y" ]]; then
msg2 "Copying package files..."
fi
 
@@ -267,7 +278,9 @@ fakebuild() {
 
# Reconstruct .PKGINFO from database
# TODO adopt makepkg's write_pkginfo() into this or scripts/library
-   if [[ "$QUIET" != "y" ]]; then
+   if [[ $MAX_JOBS -gt 1 && "$QUIET" != "y" ]]; then
+   msg2 "${pkg_name}: Generating .PKGINFO metadata..."
+   elif [[ "$QUIET" != "y" ]]; then
msg2 "Generating .PKGINFO metadata..."
fi
echo "# Generated by $myname $myver"> .PKGINFO
@@ -357,7 +370,9 @@ fakebuild() {
chmod 644 "$work_dir"/{.PKGINFO,.CHANGELOG,.INSTALL} 2> /dev/null
 
# Generate the package
-   if [[ "$QUIET" != "y" ]]; then
+   if [[ $MAX_JOBS -gt 1 && "$QUIET" != "y" ]]; then
+   msg2 "${pkg_name}: Generating the package..."
+   elif [[ "$QUIET" != "y" ]]; then
msg2 "Generating the package..."
fi
 
-- 
2.9.3

[pacman-dev] [PATCH v3 8/8] bacman: add manual page

[Gordian Edenhofer]

Signed-off-by: Gordian Edenhofer 
---
 contrib/doc/.gitignore   |  1 +
 contrib/doc/Makefile.am  |  5 +++-
 contrib/doc/bacman.8.txt | 71 
 3 files changed, 76 insertions(+), 1 deletion(-)
 create mode 100644 contrib/doc/bacman.8.txt

diff --git a/contrib/doc/.gitignore b/contrib/doc/.gitignore
index c5612bc..3ab2035 100644
--- a/contrib/doc/.gitignore
+++ b/contrib/doc/.gitignore
@@ -1 +1,2 @@
 verify-pacman-repo-db.1
+bacman.8
diff --git a/contrib/doc/Makefile.am b/contrib/doc/Makefile.am
index 4c316bb..d5725b1 100644
--- a/contrib/doc/Makefile.am
+++ b/contrib/doc/Makefile.am
@@ -4,12 +4,14 @@
 # man_MANS if --enable-asciidoc and/or --enable-doxygen are used.
 
 ASCIIDOC_MANS = \
-   verify-pacman-repo-db.1
+   verify-pacman-repo-db.1 \
+   bacman.8
 
 EXTRA_DIST = \
asciidoc.conf \
footer.txt \
verify-pacman-repo-db.1.txt \
+   bacman.8.txt \
$(ASCIIDOC_MANS)
 
 # Files that should be removed, but which Automake does not know.
@@ -53,5 +55,6 @@ $(ASCIIDOC_MANS): asciidoc.conf footer.txt Makefile.am
 
 # Dependency rules
 verify-pacman-repo-db.1: verify-pacman-repo-db.1.txt
+bacman.8: bacman.8.txt
 
 # vim:set noet:
diff --git a/contrib/doc/bacman.8.txt b/contrib/doc/bacman.8.txt
new file mode 100644
index 000..2c3112f
--- /dev/null
+++ b/contrib/doc/bacman.8.txt
@@ -0,0 +1,71 @@
+/
+vim:set ts=4 sw=4 syntax=asciidoc noet spell spelllang=en_us:
+/
+bacman(8)
+==
+
+Name
+
+bacman - reassemble installed packages
+
+
+Synopsis
+
+'bacman' [options] 
+
+
+Description
+---
+'bacman' was designed to reassemble installed packages from its deliverd files.
+It comes in handy if there is no internet connection available and you have no
+access to a up-to-date package cache. Otherwise packages from the Arch Linux
+Archive should always be preferred.
+
+'bacman' fully honors linkman:makepkg.conf[8] and all compression environment
+variables, notably ``XZ_OPT''.
+
+
+Options
+---
+*-h, \--help*::
+   Display usage information.
+
+*-q, \--quiet*::
+   Silence most of the status reporting.
+
+*\--nocolor*::
+   Disable colored output.
+
+*\--pacnew*::
+   Package .pacnew files instead of the concerning files currently in 
place.
+
+*-j, \--jobs *::
+   Assemble multiple packages in parallel.
+
+*-o, \--out *::
+   Write the assembled package(s) to the specified directory.
+
+
+Examples
+
+'bacman' linux-headers
+
+Recreate the package ``linux-headers''.
+
+'bacman' gzip munge binutils -o ~/Downloads
+
+Assemble the packages gzip munge binutils and place the packages at
+``~/Downloads''.
+
+'bacman' --nocolor --pacnew -o ~/backup -j 5 $(pacman -Qsq)
+
+Assemble all currently installed packages using 5 concurrent jobs, use
+``.pacnew'' whenever available, suppress colored output and place the desired
+packages in ``~/backup''.
+
+
+See Also
+
+linkman:makepkg[8], linkman:pacman[8]
+
+include::footer.txt[]
-- 
2.9.3

[pacman-dev] [PATCH v3 4/8] bacman: rewrite usage page and version information

[Gordian Edenhofer]

Signed-off-by: Gordian Edenhofer 
---
 contrib/bacman.sh.in | 33 +
 1 file changed, 21 insertions(+), 12 deletions(-)

diff --git a/contrib/bacman.sh.in b/contrib/bacman.sh.in
index a0bfcf1..ca2945f 100644
--- a/contrib/bacman.sh.in
+++ b/contrib/bacman.sh.in
@@ -41,25 +41,33 @@ clean_up() {
echo
exit
 }
-
 # Trap termination signals
 trap clean_up SIGHUP SIGINT SIGTERM
 
 # Print usage information
 usage() {
-   echo "${myname} (pacman) v${myver}"
-   echo
-   echo "Recreate a package using pacman's database and system files"
-   echo
-   echo "Usage: ${myname} [--nocolor] [--pacnew] "
-   echo
-   echo "Example: ${myname} linux-headers"
+   cat <<-EOF
+   $myname (pacman) v$myver
+   This script was design to reassemble installed packages from its 
deliverd files.
+   It comes in handy if you have no access to an up-to-date package cache.
+
+   Usage: $myname [options] 
+   -h, --help  display this help message and exit
+   --nocolor   turn off color in output
+   --pacnewpackage .pacnew files
+
+   Examples:   # $myname linux-headers
+   # $myname gzip munge binutils -o ~/Downloads
+   # $myname --nocolor --pacnew gzip munge binutils
+   # $myname \$(pacman -Qsq)
+   EOF
 }
 
 # Print version information
 version() {
printf "%s %s\n" "$myname" "$myver"
echo 'Copyright (C) 2008 locci '
+   echo 'Copyright (C) 2016 Gordian Edenhofer 
'
echo 'Copyright (C) 2008-2016 Pacman Development Team 
'
 }
 
@@ -123,7 +131,7 @@ if (( EUID )); then
fi
 fi
 
-# Source environmental variables and specify fallbacks
+# Read in environmental variables
 if [[ ! -r @sysconfdir@/pacman.conf ]]; then
error "unable to read @sysconfdir@/pacman.conf"
exit 1
@@ -139,7 +147,7 @@ if [[ -r ~/.makepkg.conf ]]; then
source ~/.makepkg.conf
 fi
 pkg_dest="${PKGDEST:-$PWD}"
-pkg_pkger=${PACKAGER:-'Unknown Packager'}
+pkg_pkger="${PACKAGER:-'Unknown Packager'}"
 
 # Check for an existing database
 if [[ ! -d $pac_db ]]; then
@@ -339,6 +347,7 @@ fakebuild() {
pkg_file="$pkg_dest/$pkg_namver-$pkg_arch${PKGEXT}"
ret=0
 
+   # Move compressed package to destination
# TODO: Maybe this can be set globally for robustness
shopt -s -o pipefail
bsdtar -cf - $comp_files * |
@@ -352,7 +361,7 @@ fakebuild() {
"$PKGEXT"; cat ;;
esac > "${pkg_file}"; ret=$?
 
-   # Move compressed package to destination
+   # Evaluate return code
if (( ret )); then
error "Unable to write package to $pkg_dest"
plain "   Maybe the disk is full or you do not have write 
access"
@@ -362,7 +371,6 @@ fakebuild() {
 
# Clean up working directory
rm -rf "$work_dir"
-   msg "Done."
 }
 
 # Run fakebuild in parralel with at maximum $MAX_JOBS jobs
@@ -391,6 +399,7 @@ else
done
 fi
 
+msg "Done."
 exit 0
 
 # vim: set noet:
-- 
2.9.3

[pacman-dev] [PATCH v3 3/8] bacman: proper option handling

[Gordian Edenhofer]

Switch to parseopts instead of merely checking the first argument.

Signed-off-by: Gordian Edenhofer 
---
 contrib/bacman.sh.in | 63 ++--
 1 file changed, 41 insertions(+), 22 deletions(-)

diff --git a/contrib/bacman.sh.in b/contrib/bacman.sh.in
index 1241718..a0bfcf1 100644
--- a/contrib/bacman.sh.in
+++ b/contrib/bacman.sh.in
@@ -32,6 +32,7 @@ INCLUDE_PACNEW='n'
 ARGS=("$@")
 
 m4_include(../scripts/library/output_format.sh)
+m4_include(../scripts/library/parseopts.sh)
 
 # Lazy recursive clean up of temporary dirs
 work_dir_root="${TMPDIR:-/tmp}/bacman"
@@ -62,35 +63,51 @@ version() {
echo 'Copyright (C) 2008-2016 Pacman Development Team 
'
 }
 
-# Check for specified arguments
-while [[ ! -z $1 ]]; do
-   if [[ $1 == "--nocolor" ]]; then
-   USE_COLOR='n'
-   shift
-   elif [[ $1 == "--pacnew" ]]; then
-   INCLUDE_PACNEW='y'
-   shift
-   else
-   break
-   fi
-done
-
 # Configure colored output
 m4_include(../scripts/library/term_colors.sh)
 
 # Break if no argument was given
-if (( $# < 1 )); then
+if (( $# == 0 )); then
+   usage
+   exit 1
+fi
+
+# Printing the usage information takes precedence over every other parameter
+for option in "$@"; do
+   [[ $option == "-h" || $option == "--help" ]] && usage && exit 0
+done
+
+# Parse arguments
+OPT_SHORT=':v'
+OPT_LONG=('nocolor' 'pacnew' 'version')
+if ! parseopts "$OPT_SHORT" "${OPT_LONG[@]}" -- "$@"; then
usage
exit 1
 fi
+set -- "${OPTRET[@]}"
+unset OPT_SHORT OPT_LONG OPTRET
+
+while :; do
+   case "$1" in
+   --nocolor)
+   USE_COLOR='n' ;;
+   --pacnew)
+   INCLUDE_PACNEW='y' ;;
+   -v|--version)
+   version
+   exit 0 ;;
+   --)
+   shift
+   break 2 ;;
+   esac
+   shift
+done
 
-# Print usage or version if requested
-if [[ $1 = -@(h|-help) ]]; then
+# Retrieve the list of packages to be assembled and break if none was specified
+pkg_list=($*)
+if [[ ${#pkg_list[@]} == 0 ]]; then
usage
-   exit 0
-elif [[ $1 = -@(V|-version) ]]; then
-   version
-   exit 0
+   exit 1
 fi
 
 # Run with fake root privileges if EUID is not root
@@ -367,9 +384,11 @@ parallelize() {
 
 # Initiate assembly function
 if [[ $MAX_JOBS -gt "1" ]]; then
-   parallelize "$@"
+   parallelize "${pkg_list[@]}"
 else
-   for PKG in $@; do fakebuild $PKG; done
+   for PKG in ${pkg_list[@]}; do
+   fakebuild $PKG
+   done
 fi
 
 exit 0
-- 
2.9.3

[pacman-dev] [PATCH v3 2/8] bacman: parallel packaging

[Gordian Edenhofer]

Signed-off-by: Gordian Edenhofer 
---
 contrib/bacman.sh.in | 23 ++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/contrib/bacman.sh.in b/contrib/bacman.sh.in
index a13f074..1241718 100644
--- a/contrib/bacman.sh.in
+++ b/contrib/bacman.sh.in
@@ -348,8 +348,29 @@ fakebuild() {
msg "Done."
 }
 
+# Run fakebuild in parralel with at maximum $MAX_JOBS jobs
+# By default only run one job
+MAX_JOBS=${MAX_JOBS:-1}
+# TODO: Use wait -n instead of this hack as soon as it is ok
+# to bump the version of the bash dependency
+parallelize() {
+   while [[ $# -gt 0 ]]; do
+   job_count=($(jobs -p))
+   if [[ ${#job_count[@]} -lt $MAX_JOBS ]] ; then
+   fakebuild "$1" &
+   shift
+   fi
+   sleep 0.1
+   done
+   wait
+}
 
-for PKG in $@; do fakebuild $PKG; done
+# Initiate assembly function
+if [[ $MAX_JOBS -gt "1" ]]; then
+   parallelize "$@"
+else
+   for PKG in $@; do fakebuild $PKG; done
+fi
 
 exit 0
 
-- 
2.9.3

[pacman-dev] [PATCH v3 1/8] bacman: allow for multiple packages as arguments

[Gordian Edenhofer]

To enable the creation of multiple packages with one command move the
assembly process into its own function.
Handle SIGHUP, SIGINT, SIGTERM and remove working directories
accordingly.
Add some comments.

Signed-off-by: Gordian Edenhofer 
---
 contrib/bacman.sh.in | 442 +--
 1 file changed, 220 insertions(+), 222 deletions(-)

diff --git a/contrib/bacman.sh.in b/contrib/bacman.sh.in
index a611c1a..a13f074 100644
--- a/contrib/bacman.sh.in
+++ b/contrib/bacman.sh.in
@@ -33,9 +33,18 @@ ARGS=("$@")
 
 m4_include(../scripts/library/output_format.sh)
 
-#
-# User Friendliness
-#
+# Lazy recursive clean up of temporary dirs
+work_dir_root="${TMPDIR:-/tmp}/bacman"
+clean_up() {
+   rm -rf "$work_dir_root".*
+   echo
+   exit
+}
+
+# Trap termination signals
+trap clean_up SIGHUP SIGINT SIGTERM
+
+# Print usage information
 usage() {
echo "${myname} (pacman) v${myver}"
echo
@@ -46,12 +55,14 @@ usage() {
echo "Example: ${myname} linux-headers"
 }
 
+# Print version information
 version() {
printf "%s %s\n" "$myname" "$myver"
echo 'Copyright (C) 2008 locci '
echo 'Copyright (C) 2008-2016 Pacman Development Team 
'
 }
 
+# Check for specified arguments
 while [[ ! -z $1 ]]; do
if [[ $1 == "--nocolor" ]]; then
USE_COLOR='n'
@@ -64,13 +75,16 @@ while [[ ! -z $1 ]]; do
fi
 done
 
+# Configure colored output
 m4_include(../scripts/library/term_colors.sh)
 
-if (( $# != 1 )); then
+# Break if no argument was given
+if (( $# < 1 )); then
usage
exit 1
 fi
 
+# Print usage or version if requested
 if [[ $1 = -@(h|-help) ]]; then
usage
exit 0
@@ -79,9 +93,7 @@ elif [[ $1 = -@(V|-version) ]]; then
exit 0
 fi
 
-#
-# Fakeroot support
-#
+# Run with fake root privileges if EUID is not root
 if (( EUID )); then
if [[ -f /usr/bin/fakeroot ]]; then
msg "Entering fakeroot environment"
@@ -94,264 +106,250 @@ if (( EUID )); then
fi
 fi
 
-#
-# Setting environmental variables
-#
+# Source environmental variables and specify fallbacks
 if [[ ! -r @sysconfdir@/pacman.conf ]]; then
error "unable to read @sysconfdir@/pacman.conf"
exit 1
 fi
-
 eval $(awk '/DBPath/ {print $1$2$3}' @sysconfdir@/pacman.conf)
 pac_db="${DBPath:-@localstatedir@/lib/pacman/}/local"
-
 if [[ ! -r @sysconfdir@/makepkg.conf ]]; then
error "unable to read @sysconfdir@/makepkg.conf"
exit 1
 fi
-
 source "@sysconfdir@/makepkg.conf"
 if [[ -r ~/.makepkg.conf ]]; then
source ~/.makepkg.conf
 fi
-
 pkg_dest="${PKGDEST:-$PWD}"
 pkg_pkger=${PACKAGER:-'Unknown Packager'}
 
-pkg_name="$1"
-pkg_dir=("$pac_db/$pkg_name"-+([^-])-+([^-]))
-pkg_namver=("${pkg_dir[@]##*/}")
-
-#
-# Checks everything is in place
-#
+# Check for an existing database
 if [[ ! -d $pac_db ]]; then
error "pacman database directory ${pac_db} not found"
exit 1
 fi
 
-if (( ${#pkg_dir[@]} != 1 )); then
-   error "%d entries for package %s found in pacman database" \
-   ${#pkg_dir[@]} "${pkg_name}"
-   msg2 "%s" "${pkg_dir[@]}"
-   exit 1
-fi
-
-if [[ ! -d $pkg_dir ]]; then
-   error "package %s is found in pacman database," "${pkg_name}"
-   plain "   but '%s' is not a directory" "${pkg_dir}"
-   exit 1
-fi
-
-#
-# Begin
-#
-msg "Package: ${pkg_namver}"
-work_dir=$(mktemp -d "${TMPDIR:-/tmp}/bacman.XX")
-cd "$work_dir" || exit 1
-
-#
-# File copying
-#
-msg2 "Copying package files..."
-
-while read i; do
-   if [[ -z $i ]]; then
-   continue
+# Assemble a single package: $1 = pkgname
+fakebuild() {
+   pkg_name="$1"
+   pkg_dir=("$pac_db/$pkg_name"-+([^-])-+([^-]))
+   pkg_namver=("${pkg_dir[@]##*/}")
+
+   # Checks database for specified package
+   if (( ${#pkg_dir[@]} != 1 )); then
+   error "%d entries for package %s found in pacman database" \
+   ${#pkg_dir[@]} "${pkg_name}"
+   msg2 "%s" "${pkg_dir[@]}"
+   exit 1
fi
-
-   if [[ $i == %+([A-Z])% ]]; then
-   current=$i
-   continue
+   if [[ ! -d $pkg_dir ]]; then
+   error "package %s is found in pacman database," "${pkg_name}"
+   plain "   but '%s' is not a directory" "${pkg_dir}"
+   exit 1
fi
 
-   case "$current" in
-   %FILES%)
-   local_file="/$i"
-   package_file="$work_dir/$i"
+   # Create working directory
+   msg "Package: ${pkg_namver}"
+   work_dir=$(mktemp -d "${work_dir_root}.XX")
+   cd "$work_dir" || exit 1
 
-   if [[ ! -e $local_file ]]; then
-   warning "package file $local_file is missing"
-   continue
-   fi
-   

Re: [pacman-dev] [PATCH v2 1/3] bacman: allow for parallel packaging

[Gordian Edenhofer]

On Tue, 2016-08-30 at 20:30 +1000, Allan McRae wrote:
> On 15/08/16 06:39, Gordian Edenhofer wrote:
> > 
> > * move the actual assembly process into its own function
> > * allow for packaging multiple packages with one command
> > * handle SIGHUP SIGINT SIGTERM and remove working dirs accordingly
> > * add some comments
> > 
> > Signed-off-by: Gordian Edenhofer 
> > ---
> > * Clean up working directories with force on abort
> > * Sleep for 100ms in between parallelization work
> 
> Why sleep?

Sleep is invoked to reduce the CPU usage caused by busy-waiting.
The approach for parallelization is straight forward and simple. It
checks every n seconds whether enough jobs are running and in case not
it starts a new one. Since this is not an elegant way to do it, I added
a note which incorporates Dave Reisner hint to use wait -n at some
point in the future when bash 4.3 is available at every system.

> Two patches are needed:
> 
> 1) allow bacman to take multiple package arguments
> 2) parallelize...

I will submit a second patch very soon which will address your
complaints concerning my commit division.

> The second probably would not be accepted...

I urge you to reconsider. Parallelization increases the speed of this
tool immensely. To proof my point I made some test on an Intel NUC with
an i5-4250U dual-core CPU. The results [1] clearly show that when using
all available four threads the time needed to assemble various chunks
of packages is sliced in half.
The graphs show the time required for a group of packages to be
recreated with bacman. To show the potential of parallelization I
chose: base + base-devel, base + base-devel + gnome and finally I
assembled all packages installed on the system. Packages contained in
multiple groups were only assembled once. All chosen package groups
were recreated using 1-6 jobs.
The respective measured values and code can be seen here [2].

Best Regards,
Gordian Edenhofer

[1] http://edh.ddns.net/pacman_ml_bacman_benchmarks/bacman:%20simple%20
benchmark.svg
[2] http://edh.ddns.net/pacman_ml_bacman_benchmarks/bacman:%20simple%20
benchmark.R.txt

signature.asc
Description: This is a digitally signed message part

Re: [pacman-dev] [PATCH v2 1/3] bacman: allow for parallel packaging

[Dave Reisner]

On Wed, Aug 31, 2016 at 10:35:37PM +0200, Gordian Edenhofer wrote:
> On Tue, 2016-08-30 at 20:30 +1000, Allan McRae wrote:
> > On 15/08/16 06:39, Gordian Edenhofer wrote:
> > > 
> > > * move the actual assembly process into its own function
> > > * allow for packaging multiple packages with one command
> > > * handle SIGHUP SIGINT SIGTERM and remove working dirs accordingly
> > > * add some comments
> > > 
> > > Signed-off-by: Gordian Edenhofer 
> > > ---
> > > * Clean up working directories with force on abort
> > > * Sleep for 100ms in between parallelization work
> > 
> > Why sleep?
> 
> Sleep is invoked to reduce the CPU usage caused by busy-waiting.
> The approach for parallelization is straight forward and simple. It
> checks every n seconds whether enough jobs are running and in case not
> it starts a new one. Since this is not an elegant way to do it, I added
> a note which incorporates Dave Reisner hint to use wait -n at some
> point in the future when bash 4.3 is available at every system.
> 
> > Two patches are needed:
> > 
> > 1) allow bacman to take multiple package arguments
> > 2) parallelize...
> 
> I will submit a second patch very soon which will address your
> complaints concerning my commit division.
> 
> > The second probably would not be accepted...
> 
> I urge you to reconsider. Parallelization increases the speed of this

I don't think anyone is suggesting that packaging multiple things in
parallel isn't useful. I already suggested that nothing needs to be
implemented in bacman proper in order for you to parallelize the work.
You can write your own "pbacman" as simply as:

  for arg; do bacman "$arg" & done; wait

> tool immensely. To proof my point I made some test on an Intel NUC with
> an i5-4250U dual-core CPU. The results [1] clearly show that when using
> all available four threads the time needed to assemble various chunks
> of packages is sliced in half.
> The graphs show the time required for a group of packages to be
> recreated with bacman. To show the potential of parallelization I
> chose: base + base-devel, base + base-devel + gnome and finally I
> assembled all packages installed on the system. Packages contained in
> multiple groups were only assembled once. All chosen package groups
> were recreated using 1-6 jobs.
> The respective measured values and code can be seen here [2].
> 
> Best Regards,
> Gordian Edenhofer
> 
> [1] http://edh.ddns.net/pacman_ml_bacman_benchmarks/bacman:%20simple%20
> benchmark.svg
> [2] http://edh.ddns.net/pacman_ml_bacman_benchmarks/bacman:%20simple%20
> benchmark.R.txt

Re: [pacman-dev] [PATCH] Replace CURLOPT_PROGRESSFUNCTION with CURLOPT_XFERINFOFUNCTION

[Will Miles]

On 2016-08-30 11:01 PM, Allan McRae wrote:


OK.   I don't think pacman is used anywhere where curl older than August
2013 is an issue...

Allan

I guess I'm the exception that proves the rule -- I'm using pacman on 
QNX 6.6, which ships with libcurl 7.24.0 from Jan 2012.Alas, RTOS 
folks are notoriously conservative thanks to long certification processes.


That said, I would still endorse this patch; it's no big deal for me to 
arrange to build an updated curl for our platform, now that I have 
pacman to manage its deployment. :)


-Will