Hello community, here is the log from the commit of package gnu_parallel for openSUSE:Factory checked in at 2020-08-03 14:16:47 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/gnu_parallel (Old) and /work/SRC/openSUSE:Factory/.gnu_parallel.new.3592 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "gnu_parallel" Mon Aug 3 14:16:47 2020 rev:57 rq:823963 version:20200722 Changes: -------- --- /work/SRC/openSUSE:Factory/gnu_parallel/gnu_parallel.changes 2020-06-05 20:29:13.069349481 +0200 +++ /work/SRC/openSUSE:Factory/.gnu_parallel.new.3592/gnu_parallel.changes 2020-08-03 14:17:30.052626334 +0200 @@ -1,0 +2,6 @@ +Thu Jul 30 13:01:42 UTC 2020 - Jan Engelhardt <jeng...@inai.de> + +- Update to release 20200722 + * Bug fixes and man page updates. + +------------------------------------------------------------------- Old: ---- parallel-20200522.tar.bz2 parallel-20200522.tar.bz2.sig New: ---- parallel-20200722.tar.bz2 parallel-20200722.tar.bz2.sig ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ gnu_parallel.spec ++++++ --- /var/tmp/diff_new_pack.HjqkaV/_old 2020-08-03 14:17:34.684630980 +0200 +++ /var/tmp/diff_new_pack.HjqkaV/_new 2020-08-03 14:17:34.688630984 +0200 @@ -17,7 +17,7 @@ Name: gnu_parallel -Version: 20200522 +Version: 20200722 Release: 0 Summary: Shell tool for executing jobs in parallel License: GPL-3.0-or-later ++++++ parallel-20200522.tar.bz2 -> parallel-20200722.tar.bz2 ++++++ ++++ 1605 lines of diff (skipped) ++++ retrying with extended exclude list diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/Makefile.am new/parallel-20200722/Makefile.am --- old/parallel-20200522/Makefile.am 2019-11-23 17:22:59.000000000 +0100 +++ new/parallel-20200722/Makefile.am 2020-05-24 14:06:04.000000000 +0200 @@ -123,4 +123,25 @@ mkdir -p urls cd urls && grep -v '(dead)' ../src/* | grep -h -Po 'https?://[^ $$<>")}]+' | perl -pe 's/(>|\{).*//;s/\\-/-/g;s/\\n//g;s/&/&/g;s/&#.*//;'"s/'.*//" | grep -Ev 'parallel-(20)?$$|coolwebsite.biz' | sort -u | egrep -v 'example.com|##|\*\(' | parallel -j0 --timeout 33 --bar --tag --joblog joblog --retries 3 neno wget -m -l1 -Q1 '{=$$_=Q($$_)=}' +reconf: + rm -fr autom4te.cache aclocal.m4 config.h config.h.in config.log Makefile.in missing install-sh + rm -rf src/Makefile.in + autoreconf --install -W gnu + ./configure + make -j + sudo make install + +pack_unpack_and_test_build: + echo '### Building tar.bz2' + ./configure + make dist + make dist-bzip2 + echo "### Unpack parallel-$(YYYYMMDD).tar.bz2" + cp parallel-$(YYYYMMDD).tar.bz2 /tmp + cd /tmp && \ + tar xjf parallel-$(YYYYMMDD).tar.bz2 && \ + cd parallel-$(YYYYMMDD) && \ + ./configure && make -j && sudo make -j install + + EXTRA_DIST = CITATION CREDITS cc-by-sa.txt fdl.txt diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/NEWS new/parallel-20200722/NEWS --- old/parallel-20200522/NEWS 2020-05-23 19:19:47.000000000 +0200 +++ new/parallel-20200722/NEWS 2020-07-22 22:50:20.000000000 +0200 @@ -1,3 +1,64 @@ +20200722 + +New in this release: + +* No new functionality + +* Bug fixes and man page updates. + +News about GNU Parallel: + +* Tips for Running Jobs on your VM + https://support.ehelp.edu.au/support/solutions/articles/6000089713-tips-for-running-jobs-on-your-vm#parallel + +* Introduction to GNU Parallel: Parallelizing Massive Individual Tasks + https://www.youtube.com/watch?v=QSFLTEyCKvo + +* Slides for same + http://www.hpc.lsu.edu/training/weekly-materials/2020-Spring/GNUParallel_Spring_2020.pdf + +* Linux-Fu: Parallel Universe + https://hackaday.com/2020/06/29/linux-fu-parallel-universe/ + +* Resumable batch jobs with gnu parallel https://igor.io/parallel/ + +* Introducing Parallel into Shell + https://www.petelawson.com/post/parallel-in-shell/ + +* AV1 encoding parallelization with GNU Parallel + https://www.reddit.com/r/AV1/comments/fjvqp9/av1_encoding_parallelization_with_gnu_parallel/ + +* Beschleunigte Befehlsbearbeitung mit GNU Parallel + https://www.linux-community.de/ausgaben/linuxuser/2013/06/beschleunigte-befehlsbearbeitung-mit-gnu-parallel/ + + +20200622 + +New in this release: + +* No new functionality + +* Bug fixes and man page updates. + +News about GNU Parallel: + +* GNU Parallel is used at NERSC: https://www.nersc.gov/assets/Uploads/08-Workflows-20200616.pdf + +* Resumable batch jobs with GNU Parallel https://igor.io/parallel/ + +* Lab Meeting technical talk https://www.slideshare.net/hoffmanlab/gnu-parallel-194030490 + +* Open Sky Software Website Credits https://www.openskysoftware.com/site-credits.htm + +* Speed up your Python Scripts using Ubuntu and GNU Parallel https://ecce.esri.ca/wpecce/2019/12/24/speed-up-your-python-scripts-using-ubuntu-and-gnu-parallel/ + +* Indispensable command-line tools https://negfeedback.blogspot.com/2020/05/indispensable-command-line-tools.html + +* GNU Parallel for simulations http://pdebuyl.be/blog/2020/gnu-parallel-for-simulations.html + +* Introduction to GNU parallel https://bioinformaticsworkbook.org/Appendix/GNUparallel/GNU_parallel_examples.html + + 20200522 New in this release: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/README new/parallel-20200722/README --- old/parallel-20200522/README 2020-05-23 20:32:43.000000000 +0200 +++ new/parallel-20200722/README 2020-07-22 22:55:10.000000000 +0200 @@ -57,11 +57,11 @@ Full installation of GNU Parallel is as simple as: - wget https://ftpmirror.gnu.org/parallel/parallel-20200522.tar.bz2 - wget https://ftpmirror.gnu.org/parallel/parallel-20200522.tar.bz2.sig - gpg parallel-20200522.tar.bz2.sig - bzip2 -dc parallel-20200522.tar.bz2 | tar xvf - - cd parallel-20200522 + wget https://ftpmirror.gnu.org/parallel/parallel-20200722.tar.bz2 + wget https://ftpmirror.gnu.org/parallel/parallel-20200722.tar.bz2.sig + gpg parallel-20200722.tar.bz2.sig + bzip2 -dc parallel-20200722.tar.bz2 | tar xvf - + cd parallel-20200722 ./configure && make && sudo make install @@ -70,11 +70,11 @@ If you are not root you can add ~/bin to your path and install in ~/bin and ~/share: - wget https://ftpmirror.gnu.org/parallel/parallel-20200522.tar.bz2 - wget https://ftpmirror.gnu.org/parallel/parallel-20200522.tar.bz2.sig - gpg parallel-20200522.tar.bz2.sig - bzip2 -dc parallel-20200522.tar.bz2 | tar xvf - - cd parallel-20200522 + wget https://ftpmirror.gnu.org/parallel/parallel-20200722.tar.bz2 + wget https://ftpmirror.gnu.org/parallel/parallel-20200722.tar.bz2.sig + gpg parallel-20200722.tar.bz2.sig + bzip2 -dc parallel-20200722.tar.bz2 | tar xvf - + cd parallel-20200722 ./configure --prefix=$HOME && make && make install Or if your system lacks 'make' you can simply copy src/parallel @@ -122,8 +122,8 @@ When using programs that use GNU Parallel to process data for publication please cite: - Tange, O. (2020, May 22). GNU Parallel 20200522 ('Kraftwerk'). - Zenodo. https://doi.org/10.5281/zenodo.3841377 + Tange, O. (2020, July 22). GNU Parallel 20200722 ('Privacy Shield'). + Zenodo. https://doi.org/10.5281/zenodo.3956817 = New versions = diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/configure.ac new/parallel-20200722/configure.ac --- old/parallel-20200522/configure.ac 2020-05-23 20:32:43.000000000 +0200 +++ new/parallel-20200722/configure.ac 2020-07-22 22:55:10.000000000 +0200 @@ -1,4 +1,4 @@ -AC_INIT([parallel], [20200522], [bug-paral...@gnu.org]) +AC_INIT([parallel], [20200722], [bug-paral...@gnu.org]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/missing new/parallel-20200722/missing --- old/parallel-20200522/missing 2020-05-23 20:33:16.000000000 +0200 +++ new/parallel-20200722/missing 2020-07-22 22:55:39.000000000 +0200 @@ -1,9 +1,9 @@ #! /bin/sh # Common wrapper for a few potentially missing GNU programs. -scriptversion=2013-10-28.13; # UTC +scriptversion=2018-03-07.03; # UTC -# Copyright (C) 1996-2014 Free Software Foundation, Inc. +# Copyright (C) 1996-2018 Free Software Foundation, Inc. # Originally written by Fran,cois Pinard <pin...@iro.umontreal.ca>, 1996. # This program is free software; you can redistribute it and/or modify @@ -17,7 +17,7 @@ # GNU General Public License for more details. # You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. +# along with this program. If not, see <https://www.gnu.org/licenses/>. # As a special exception to the GNU General Public License, if you # distribute this file as part of a program that contains a @@ -101,9 +101,9 @@ exit $st fi -perl_URL=http://www.perl.org/ -flex_URL=http://flex.sourceforge.net/ -gnu_software_URL=http://www.gnu.org/software +perl_URL=https://www.perl.org/ +flex_URL=https://github.com/westes/flex +gnu_software_URL=https://www.gnu.org/software program_details () { @@ -207,9 +207,9 @@ exit $st # Local variables: -# eval: (add-hook 'write-file-hooks 'time-stamp) +# eval: (add-hook 'before-save-hook 'time-stamp) # time-stamp-start: "scriptversion=" # time-stamp-format: "%:y-%02m-%02d.%02H" -# time-stamp-time-zone: "UTC" +# time-stamp-time-zone: "UTC0" # time-stamp-end: "; # UTC" # End: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/env_parallel.1 new/parallel-20200722/src/env_parallel.1 --- old/parallel-20200522/src/env_parallel.1 2020-02-22 20:04:55.000000000 +0100 +++ new/parallel-20200722/src/env_parallel.1 2020-07-04 11:32:19.000000000 +0200 @@ -1,4 +1,4 @@ -.\" Automatically generated by Pod::Man 4.09 (Pod::Simple 3.35) +.\" Automatically generated by Pod::Man 4.11 (Pod::Simple 3.40) .\" .\" Standard preamble: .\" ======================================================================== @@ -54,16 +54,20 @@ .\" Avoid warning from groff about undefined register 'F'. .de IX .. -.if !\nF .nr F 0 -.if \nF>0 \{\ -. de IX -. tm Index:\\$1\t\\n%\t"\\$2" +.nr rF 0 +.if \n(.g .if rF .nr rF 1 +.if (\n(rF:(\n(.g==0)) \{\ +. if \nF \{\ +. de IX +. tm Index:\\$1\t\\n%\t"\\$2" .. -. if !\nF==2 \{\ -. nr % 0 -. nr F 2 +. if !\nF==2 \{\ +. nr % 0 +. nr F 2 +. \} . \} .\} +.rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. @@ -129,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "ENV_PARALLEL 1" -.TH ENV_PARALLEL 1 "2020-01-27" "20200122" "parallel" +.TH ENV_PARALLEL 1 "2020-07-04" "20200622" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l @@ -209,7 +213,7 @@ than the maximal length of a command and smaller than half of the max if running remotely. E.g. The max size of Bash's command is 128 \s-1KB,\s0 so \&\fBenv_parallel\fR will fail if '\fBset | wc \-c\fR' is bigger than 128 -\&\s-1KB.\s0 Technically the limit is in \fIexecve\fR\|(1) which IPC::open3 uses. +\&\s-1KB.\s0 Technically the limit is in \fBexecve\fR\|(1) which IPC::open3 uses. .PP Bash completion functions are well-known for taking up well over 128 \&\s-1KB\s0 of environment space and the primary reason for causing @@ -390,7 +394,8 @@ .IX Subsection "BUGS" .PP Due to a bug in Bash, aliases containing newlines must be followed by -a newline in the command. +a newline in the command. Some systems are not affected by this bug, +but will print a warning anyway. .SS "csh" .IX Subsection "csh" \&\fBenv_parallel\fR for \fBcsh\fR breaks \fB\f(CB$PARALLEL\fB\fR, so do not use diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/env_parallel.bash new/parallel-20200722/src/env_parallel.bash --- old/parallel-20200522/src/env_parallel.bash 2020-01-27 18:17:31.000000000 +0100 +++ new/parallel-20200722/src/env_parallel.bash 2020-07-04 10:45:58.000000000 +0200 @@ -58,7 +58,7 @@ } _ignore_HARDCODED() { # These names cannot be detected - echo '(_|TIMEOUT|GROUPS|FUNCNAME|DIRSTACK|PIPESTATUS|USERNAME|BASH_[A-Z_]+)' + echo '(_|TIMEOUT|GROUPS|FUNCNAME|DIRSTACK|PIPESTATUS|USERNAME|BASHPID|BASH_[A-Z_]+)' } _ignore_READONLY() { readonly | perl -e '@r = map { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/env_parallel.fish new/parallel-20200722/src/env_parallel.fish --- old/parallel-20200522/src/env_parallel.fish 2020-01-27 18:17:31.000000000 +0100 +++ new/parallel-20200722/src/env_parallel.fish 2020-06-21 16:02:55.000000000 +0200 @@ -157,7 +157,7 @@ ($name,$val)=split(/ /,$_,2); # Ignore read-only vars $name=~/^(HOME|USER|COLUMNS|FISH_VERSION|LINES|PWD|SHLVL|_| - history|status|version)$/x and next; + fish_pid|history|hostname|status|version)$/x and next; # Single quote $val if($val =~ /[^-_.+a-z0-9\/]/i) { $val =~ s/\047/\047"\047"\047/g; # "-quote single quotes diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/env_parallel.html new/parallel-20200722/src/env_parallel.html --- old/parallel-20200522/src/env_parallel.html 2020-05-23 14:40:35.000000000 +0200 +++ new/parallel-20200722/src/env_parallel.html 2020-07-04 11:32:20.000000000 +0200 @@ -353,7 +353,7 @@ <h3 id="BUGS">BUGS</h3> -<p>Due to a bug in Bash, aliases containing newlines must be followed by a newline in the command.</p> +<p>Due to a bug in Bash, aliases containing newlines must be followed by a newline in the command. Some systems are not affected by this bug, but will print a warning anyway.</p> <h2 id="csh">csh</h2> Binary files old/parallel-20200522/src/env_parallel.pdf and new/parallel-20200722/src/env_parallel.pdf differ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/env_parallel.pod new/parallel-20200722/src/env_parallel.pod --- old/parallel-20200522/src/env_parallel.pod 2020-01-27 18:17:31.000000000 +0100 +++ new/parallel-20200722/src/env_parallel.pod 2020-07-04 11:21:35.000000000 +0200 @@ -261,7 +261,8 @@ =head3 BUGS Due to a bug in Bash, aliases containing newlines must be followed by -a newline in the command. +a newline in the command. Some systems are not affected by this bug, +but will print a warning anyway. =head2 csh diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/env_parallel.texi new/parallel-20200722/src/env_parallel.texi --- old/parallel-20200522/src/env_parallel.texi 2020-04-21 19:43:26.000000000 +0200 +++ new/parallel-20200722/src/env_parallel.texi 2020-07-04 11:21:47.000000000 +0200 @@ -352,7 +352,8 @@ @subsection BUGS Due to a bug in Bash, aliases containing newlines must be followed by -a newline in the command. +a newline in the command. Some systems are not affected by this bug, +but will print a warning anyway. @node csh @section csh diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/niceload new/parallel-20200722/src/niceload --- old/parallel-20200522/src/niceload 2020-05-23 20:32:43.000000000 +0200 +++ new/parallel-20200722/src/niceload 2020-07-22 22:55:10.000000000 +0200 @@ -23,7 +23,7 @@ use strict; use Getopt::Long; $Global::progname="niceload"; -$Global::version = 20200522; +$Global::version = 20200722; Getopt::Long::Configure("bundling","require_order"); get_options_from_array(\@ARGV) || die_usage(); if($opt::version) { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/parallel new/parallel-20200722/src/parallel --- old/parallel-20200522/src/parallel 2020-05-23 20:32:43.000000000 +0200 +++ new/parallel-20200722/src/parallel 2020-07-22 22:55:10.000000000 +0200 @@ -948,7 +948,7 @@ my $two_gb = 2**31-1; my $blocksize = $Global::blocksize; my $in = *STDIN; - my $timeout = ::multiply_time_units($opt::blocktimeout); + my $timeout = $Global::blocktimeout; my $header = find_header(\$buf,$in); my $anything_written; @@ -1762,6 +1762,13 @@ if(defined $opt::max_args) { $Global::max_number_of_args = $opt::max_args; } + if(defined $opt::blocktimeout) { + $Global::blocktimeout = int(multiply_time_units($opt::blocktimeout)); + if($Global::blocktimeout < 1) { + ::error("--block-timeout must be at least 1"); + wait_and_exit(255); + } + } if(defined $opt::timeout) { $Global::timeoutq = TimeoutQueue->new($opt::timeout); } @@ -2150,7 +2157,7 @@ sub init_globals() { # Defaults: - $Global::version = 20200522; + $Global::version = 20200722; $Global::progname = 'parallel'; $::name = "GNU Parallel"; $Global::infinity = 2**31; @@ -4535,6 +4542,7 @@ ((defined $opt::linebuffer) ? "--linebuffer" : ""), ((defined $opt::max_chars) ? "--max-chars ".$opt::max_chars : ""), ((defined $opt::plain) ? "--plain" : ""), + ((defined $opt::plus) ? "--plus" : ""), ((defined $opt::retries) ? "--retries ".$opt::retries : ""), ((defined $opt::timeout) ? "--timeout ".$opt::timeout : ""), ((defined $opt::ungroup) ? "-u" : ""), @@ -4852,6 +4860,9 @@ if($Global::unkilled_sqlworker) { waitpid($Global::unkilled_sqlworker,0); } + # Avoid: Warning: unable to close filehandle properly: No space + # left on device during global destruction. + $SIG{__WARN__} = sub {}; exit($error); } @@ -4896,8 +4907,8 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - " Tange, O. (2020, May 22). GNU Parallel 20200522 ('Kraftwerk').", - " Zenodo. https://doi.org/10.5281/zenodo.3841377", + " Tange, O. (2020, July 22). GNU Parallel 20200722 ('Privacy Shield').", + " Zenodo. https://doi.org/10.5281/zenodo.3956817", "", # Before changing this line, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice @@ -4927,8 +4938,8 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - " Tange, O. (2020, May 22). GNU Parallel 20200522 ('Kraftwerk').", - " Zenodo. https://doi.org/10.5281/zenodo.3841377", + " Tange, O. (2020, July 22). GNU Parallel 20200722 ('Privacy Shield').", + " Zenodo. https://doi.org/10.5281/zenodo.3956817", "", # Before changing this line, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and @@ -5015,11 +5026,11 @@ "GNU $Global::progname $Global::version", "Copyright (C) 2007-2020 Ole Tange, http://ole.tange.dk and Free Software", "Foundation, Inc.", - "License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>", + "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>", "This is free software: you are free to change and redistribute it.", "GNU $Global::progname comes with no warranty.", "", - "Web site: http://www.gnu.org/software/${Global::progname}\n", + "Web site: https://www.gnu.org/software/${Global::progname}\n", "When using programs that use GNU Parallel to process data for publication", "please cite as described in 'parallel --citation'.\n", ); @@ -5041,20 +5052,20 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - "\@software{tange_2020_3841377,", + "\@software{tange_2020_3956817,", " author = {Tange, Ole},", - " title = {GNU Parallel 20200522 ('Kraftwerk')},", - " month = May,", + " title = {GNU Parallel 20200722 ('Privacy Shield')},", + " month = Jul,", " year = 2020,", " note = {{GNU Parallel is a general parallelizer to run", " multiple serial command line programs in parallel", " without changing them.}},", " publisher = {Zenodo},", - " doi = {10.5281/zenodo.3841377},", - " url = {https://doi.org/10.5281/zenodo.3841377}", + " doi = {10.5281/zenodo.3956817},", + " url = {https://doi.org/10.5281/zenodo.3956817}", "}", "", - "(Feel free to use \\nocite{tange_2020_3841377})", + "(Feel free to use \\nocite{tange_2020_3956817})", "", # Before changing this line, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and @@ -5160,7 +5171,7 @@ # General Public License for more details. # # You should have received a copy of the GNU General Public License -# along with this program; if not, see <http://www.gnu.org/licenses/> +# along with this program; if not, see <https://www.gnu.org/licenses/> # or write to the Free Software Foundation, Inc., 51 Franklin St, # Fifth Floor, Boston, MA 02110-1301 USA "; @@ -8582,12 +8593,13 @@ unlink; rmdir; } - if($bash=~s/h//) { + if($bash=~s/(\d+)h/$1/) { exit $bash; } exit $csh; }). - "' ".'"$?h" "$status" '; + # `echo \$?h` is needed to make fish not complain + "' ".'"`echo \\\\\\\\\$?h`" "$status" '; } return $script } @@ -12291,8 +12303,13 @@ my $dbh; if($driver eq "CSV") { # CSV does not use normal dsn - $dbh = DBI->connect("dbi:CSV:", "", "", { f_dir => "$database", }) - or die $DBI::errstr; + if(-d $database) { + $dbh = DBI->connect("dbi:CSV:", "", "", { f_dir => "$database", }) + or die $DBI::errstr; + } else { + ::error("$database is not a directory."); + ::wait_and_exit(255); + } } else { $dbh = DBI->connect($dsn, $userid, $password, { RaiseError => 1, AutoInactiveDestroy => 1 }) @@ -12303,7 +12320,6 @@ $dbh->{'RaiseError'} = 1; $dbh->{'ShowErrorStatement'} = 1; $dbh->{'HandleError'} = sub {}; - if(not defined $options{'table'}) { ::error("The DBURL ($dburl) must contain a table."); ::wait_and_exit(255); @@ -12603,7 +12619,7 @@ my %vartype = ( "Oracle" => { "BIGINT" => "NUMBER(19,0)", "TEXT" => "CLOB", }, - "mysql" => { "TEXT" => "LONGTEXT", }, + "mysql" => { "TEXT" => "BLOB", }, "CSV" => { "BIGINT" => "INT", "FLOAT" => "REAL", }, ); @@ -12642,26 +12658,58 @@ 0, @$record_ref[1..$#$record_ref]); } + sub get_record($) { my $self = shift; my @retval; my $table = $self->table(); my @v_cols = map { ", V$_" } (1..$self->max_number_of_args()); - my $v = $self->get("SELECT Seq, Command @v_cols FROM $table ". - "WHERE Exitval = -1000 ORDER BY Seq LIMIT 1;"); - if($v->[0]) { - my $val_ref = $v->[0]; - # Mark record as taken - my $seq = shift @$val_ref; - # Save the sequence number to use when running the job - $SQL::next_seq = $seq; - $self->update("SET Exitval = ? WHERE Seq = ".$seq, -1220); - my @command = split /\257 /, shift @$val_ref; - $SQL::command_ref = \@command; - for (@$val_ref) { - push @retval, Arg->new($_); + my $rand = "Reserved-".$$.rand(); + my $v; + my $more_pending; + + do { + if($self->{'driver'} eq "CSV") { + # Sub SELECT is not supported in CSV + # So to minimize the race condition below select a job at random + my $r = $self->get("SELECT Seq, Command @v_cols FROM $table ". + "WHERE Exitval = -1000 LIMIT 100;"); + $v = [ sort { rand() > 0.5 } @$r ]; + } else { + # Avoid race condition where multiple workers get the same job + # by setting Stdout to a unique string + # (SELECT * FROM (...) AS dummy) is needed due to sillyness in MySQL + $self->update("SET Stdout = ?,Exitval = ? ". + "WHERE Seq = (". + " SELECT * FROM (". + " SELECT min(Seq) FROM $table WHERE Exitval = -1000". + " ) AS dummy". + ") AND Exitval = -1000;", $rand, -1210); + # If a parallel worker overwrote the unique string this will get nothing + $v = $self->get("SELECT Seq, Command @v_cols FROM $table ". + "WHERE Stdout = ?;", $rand); + } + if($v->[0]) { + my $val_ref = $v->[0]; + # Mark record as taken + my $seq = shift @$val_ref; + # Save the sequence number to use when running the job + $SQL::next_seq = $seq; + $self->update("SET Exitval = ? WHERE Seq = ".$seq, -1220); + # Command is encoded with '\257 space' as splitting char + my @command = split /\257 /, shift @$val_ref; + $SQL::command_ref = \@command; + for (@$val_ref) { + push @retval, Arg->new($_); + } + } else { + # If the record was updated by another job in parallel, + # then we may not be done, so see if there are more jobs pending + $more_pending = + $self->get("SELECT Seq FROM $table WHERE Exitval = ?;", -1210); } - } + } while (not $v->[0] and $more_pending->[0]); + if(@retval) { return \@retval; } else { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/parallel.1 new/parallel-20200722/src/parallel.1 --- old/parallel-20200522/src/parallel.1 2020-05-23 14:40:34.000000000 +0200 +++ new/parallel-20200722/src/parallel.1 2020-06-07 21:58:05.000000000 +0200 @@ -129,7 +129,7 @@ .\" ======================================================================== .\" .IX Title "PARALLEL 1" -.TH PARALLEL 1 "2020-05-23" "20200422" "parallel" +.TH PARALLEL 1 "2020-06-06" "20200522" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l @@ -2409,10 +2409,13 @@ \& sql:oracle://scott:ti...@ora.example.com/xe/parjob \& postgresql://scott:ti...@pg.example.com/pgdb/parjob \& pg:///parjob -\& sqlite3:///pardb/parjob.sqlite -\& csv:///%2Ftmp%2Fmydir/jobtable +\& sqlite3:///%2Ftmp%2Fpardb.sqlite/parjob +\& csv:///%2Ftmp%2Fpardb/parjob .Ve .Sp +Notice how / in the path of sqlite and \s-1CVS\s0 must be encoded as +\&\f(CW%2F\fR. Except the last / in \s-1CSV\s0 which must be a /. +.Sp It can also be an alias from ~/.sql/aliases: .Sp .Vb 1 @@ -3182,6 +3185,17 @@ .PP If \fBmy_program\fR fails a red \s-1FAIL\s0 will be printed followed by the failing command; otherwise a green \s-1OK\s0 will be printed followed by the command. +.SH "EXAMPLE: Continously show the latest line of output" +.IX Header "EXAMPLE: Continously show the latest line of output" +It can be useful to monitor the output of running jobs. +.PP +This shows the most recent output line until a job finishes. After +which the output of the job is printed in full: +.PP +.Vb 2 +\& parallel \*(Aq{} | tee >(cat >&3)\*(Aq ::: \*(Aqcommand 1\*(Aq \*(Aqcommand 2\*(Aq \e +\& 3> >(perl \-ne \*(Aq$|=1;chomp;printf"%.\*(Aq$COLUMNS\*(Aqs\er",$_." "x100\*(Aq) +.Ve .SH "EXAMPLE: Log rotate" .IX Header "EXAMPLE: Log rotate" Log rotation renames a logfile to an extension with a higher number: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/parallel.html new/parallel-20200722/src/parallel.html --- old/parallel-20200522/src/parallel.html 2020-05-23 14:40:34.000000000 +0200 +++ new/parallel-20200722/src/parallel.html 2020-06-07 21:58:05.000000000 +0200 @@ -39,6 +39,7 @@ <li><a href="#EXAMPLE:-Composed-command-with-multiple-input-sources">EXAMPLE: Composed command with multiple input sources</a></li> <li><a href="#EXAMPLE:-Calling-Bash-functions">EXAMPLE: Calling Bash functions</a></li> <li><a href="#EXAMPLE:-Function-tester">EXAMPLE: Function tester</a></li> + <li><a href="#EXAMPLE:-Continously-show-the-latest-line-of-output">EXAMPLE: Continously show the latest line of output</a></li> <li><a href="#EXAMPLE:-Log-rotate">EXAMPLE: Log rotate</a></li> <li><a href="#EXAMPLE:-Removing-file-extension-when-processing-files">EXAMPLE: Removing file extension when processing files</a></li> <li><a href="#EXAMPLE:-Removing-strings-from-the-argument">EXAMPLE: Removing strings from the argument</a></li> @@ -2281,8 +2282,10 @@ sql:oracle://scott:ti...@ora.example.com/xe/parjob postgresql://scott:ti...@pg.example.com/pgdb/parjob pg:///parjob - sqlite3:///pardb/parjob.sqlite - csv:///%2Ftmp%2Fmydir/jobtable</code></pre> + sqlite3:///%2Ftmp%2Fpardb.sqlite/parjob + csv:///%2Ftmp%2Fpardb/parjob</code></pre> + +<p>Notice how / in the path of sqlite and CVS must be encoded as %2F. Except the last / in CSV which must be a /.</p> <p>It can also be an alias from ~/.sql/aliases:</p> @@ -2969,6 +2972,15 @@ <p>If <b>my_program</b> fails a red FAIL will be printed followed by the failing command; otherwise a green OK will be printed followed by the command.</p> +<h1 id="EXAMPLE:-Continously-show-the-latest-line-of-output">EXAMPLE: Continously show the latest line of output</h1> + +<p>It can be useful to monitor the output of running jobs.</p> + +<p>This shows the most recent output line until a job finishes. After which the output of the job is printed in full:</p> + +<pre><code> parallel '{} | tee >(cat >&3)' ::: 'command 1' 'command 2' \ + 3> >(perl -ne '$|=1;chomp;printf"%.'$COLUMNS's\r",$_." "x100')</code></pre> + <h1 id="EXAMPLE:-Log-rotate">EXAMPLE: Log rotate</h1> <p>Log rotation renames a logfile to an extension with a higher number: log.1 becomes log.2, log.2 becomes log.3, and so on. The oldest log is removed. To avoid overwriting files the process starts backwards from the high number to the low number. This will keep 10 old versions of the log:</p> Binary files old/parallel-20200522/src/parallel.pdf and new/parallel-20200722/src/parallel.pdf differ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/parallel.pod new/parallel-20200722/src/parallel.pod --- old/parallel-20200522/src/parallel.pod 2020-05-23 14:40:16.000000000 +0200 +++ new/parallel-20200722/src/parallel.pod 2020-06-06 22:28:29.000000000 +0200 @@ -2380,8 +2380,11 @@ sql:oracle://scott:ti...@ora.example.com/xe/parjob postgresql://scott:ti...@pg.example.com/pgdb/parjob pg:///parjob - sqlite3:///pardb/parjob.sqlite - csv:///%2Ftmp%2Fmydir/jobtable + sqlite3:///%2Ftmp%2Fpardb.sqlite/parjob + csv:///%2Ftmp%2Fpardb/parjob + +Notice how / in the path of sqlite and CVS must be encoded as +%2F. Except the last / in CSV which must be a /. It can also be an alias from ~/.sql/aliases: @@ -2910,7 +2913,6 @@ =back - =head1 EXAMPLE: Working as xargs -n1. Argument appending GNU B<parallel> can work similar to B<xargs -n1>. @@ -3144,6 +3146,17 @@ command; otherwise a green OK will be printed followed by the command. +=head1 EXAMPLE: Continously show the latest line of output + +It can be useful to monitor the output of running jobs. + +This shows the most recent output line until a job finishes. After +which the output of the job is printed in full: + + parallel '{} | tee >(cat >&3)' ::: 'command 1' 'command 2' \ + 3> >(perl -ne '$|=1;chomp;printf"%.'$COLUMNS's\r",$_." "x100') + + =head1 EXAMPLE: Log rotate Log rotation renames a logfile to an extension with a higher number: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/parallel.texi new/parallel-20200722/src/parallel.texi --- old/parallel-20200522/src/parallel.texi 2020-05-23 14:40:39.000000000 +0200 +++ new/parallel-20200722/src/parallel.texi 2020-06-07 21:58:07.000000000 +0200 @@ -25,6 +25,7 @@ * EXAMPLE@asis{:} Composed command with multiple input sources:: * EXAMPLE@asis{:} Calling Bash functions:: * EXAMPLE@asis{:} Function tester:: +* EXAMPLE@asis{:} Continously show the latest line of output:: * EXAMPLE@asis{:} Log rotate:: * EXAMPLE@asis{:} Removing file extension when processing files:: * EXAMPLE@asis{:} Removing strings from the argument:: @@ -2674,10 +2675,13 @@ sql:oracle://scott:ti...@ora.example.com/xe/parjob postgresql://scott:ti...@pg.example.com/pgdb/parjob pg:///parjob - sqlite3:///pardb/parjob.sqlite - csv:///%2Ftmp%2Fmydir/jobtable + sqlite3:///%2Ftmp%2Fpardb.sqlite/parjob + csv:///%2Ftmp%2Fpardb/parjob @end verbatim +Notice how / in the path of sqlite and CVS must be encoded as +%2F. Except the last / in CSV which must be a /. + It can also be an alias from ~/.sql/aliases: @verbatim @@ -3542,6 +3546,19 @@ If @strong{my_program} fails a red FAIL will be printed followed by the failing command; otherwise a green OK will be printed followed by the command. +@node EXAMPLE: Continously show the latest line of output +@chapter EXAMPLE: Continously show the latest line of output + +It can be useful to monitor the output of running jobs. + +This shows the most recent output line until a job finishes. After +which the output of the job is printed in full: + +@verbatim + parallel '{} | tee >(cat >&3)' ::: 'command 1' 'command 2' \ + 3> >(perl -ne '$|=1;chomp;printf"%.'$COLUMNS's\r",$_." "x100') +@end verbatim + @node EXAMPLE: Log rotate @chapter EXAMPLE: Log rotate diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/parallel_alternatives.7 new/parallel-20200722/src/parallel_alternatives.7 --- old/parallel-20200522/src/parallel_alternatives.7 2020-05-23 18:03:47.000000000 +0200 +++ new/parallel-20200722/src/parallel_alternatives.7 2020-05-24 13:39:34.000000000 +0200 @@ -2935,6 +2935,8 @@ https://pypi.org/project/papply/ (Last checked: 2020\-04) .SS "Todo" .IX Subsection "Todo" +https://gitlab.com/netikras/bthread +.PP https://github.com/JeiKeiLim/simple_distribute_job .PP https://github.com/reggi/pkgrun diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/parallel_alternatives.html new/parallel-20200722/src/parallel_alternatives.html --- old/parallel-20200522/src/parallel_alternatives.html 2020-05-23 18:03:47.000000000 +0200 +++ new/parallel-20200722/src/parallel_alternatives.html 2020-07-04 11:32:24.000000000 +0200 @@ -2432,6 +2432,8 @@ <h2 id="Todo">Todo</h2> +<p>https://gitlab.com/netikras/bthread</p> + <p>https://github.com/JeiKeiLim/simple_distribute_job</p> <p>https://github.com/reggi/pkgrun</p> Binary files old/parallel-20200522/src/parallel_alternatives.pdf and new/parallel-20200722/src/parallel_alternatives.pdf differ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/parallel_alternatives.pod new/parallel-20200722/src/parallel_alternatives.pod --- old/parallel-20200522/src/parallel_alternatives.pod 2020-05-23 18:02:54.000000000 +0200 +++ new/parallel-20200722/src/parallel_alternatives.pod 2020-05-24 01:22:51.000000000 +0200 @@ -2616,6 +2616,8 @@ =head2 Todo +https://gitlab.com/netikras/bthread + https://github.com/JeiKeiLim/simple_distribute_job https://github.com/reggi/pkgrun diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/parallel_alternatives.texi new/parallel-20200722/src/parallel_alternatives.texi --- old/parallel-20200522/src/parallel_alternatives.texi 2020-05-23 18:03:52.000000000 +0200 +++ new/parallel-20200722/src/parallel_alternatives.texi 2020-05-24 13:39:35.000000000 +0200 @@ -3106,6 +3106,8 @@ @node Todo @section Todo +https://gitlab.com/netikras/bthread + https://github.com/JeiKeiLim/simple_distribute_job https://github.com/reggi/pkgrun diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/sem new/parallel-20200722/src/sem --- old/parallel-20200522/src/sem 2020-05-23 20:32:43.000000000 +0200 +++ new/parallel-20200722/src/sem 2020-07-22 22:55:10.000000000 +0200 @@ -948,7 +948,7 @@ my $two_gb = 2**31-1; my $blocksize = $Global::blocksize; my $in = *STDIN; - my $timeout = ::multiply_time_units($opt::blocktimeout); + my $timeout = $Global::blocktimeout; my $header = find_header(\$buf,$in); my $anything_written; @@ -1762,6 +1762,13 @@ if(defined $opt::max_args) { $Global::max_number_of_args = $opt::max_args; } + if(defined $opt::blocktimeout) { + $Global::blocktimeout = int(multiply_time_units($opt::blocktimeout)); + if($Global::blocktimeout < 1) { + ::error("--block-timeout must be at least 1"); + wait_and_exit(255); + } + } if(defined $opt::timeout) { $Global::timeoutq = TimeoutQueue->new($opt::timeout); } @@ -2150,7 +2157,7 @@ sub init_globals() { # Defaults: - $Global::version = 20200522; + $Global::version = 20200722; $Global::progname = 'parallel'; $::name = "GNU Parallel"; $Global::infinity = 2**31; @@ -4535,6 +4542,7 @@ ((defined $opt::linebuffer) ? "--linebuffer" : ""), ((defined $opt::max_chars) ? "--max-chars ".$opt::max_chars : ""), ((defined $opt::plain) ? "--plain" : ""), + ((defined $opt::plus) ? "--plus" : ""), ((defined $opt::retries) ? "--retries ".$opt::retries : ""), ((defined $opt::timeout) ? "--timeout ".$opt::timeout : ""), ((defined $opt::ungroup) ? "-u" : ""), @@ -4852,6 +4860,9 @@ if($Global::unkilled_sqlworker) { waitpid($Global::unkilled_sqlworker,0); } + # Avoid: Warning: unable to close filehandle properly: No space + # left on device during global destruction. + $SIG{__WARN__} = sub {}; exit($error); } @@ -4896,8 +4907,8 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - " Tange, O. (2020, May 22). GNU Parallel 20200522 ('Kraftwerk').", - " Zenodo. https://doi.org/10.5281/zenodo.3841377", + " Tange, O. (2020, July 22). GNU Parallel 20200722 ('Privacy Shield').", + " Zenodo. https://doi.org/10.5281/zenodo.3956817", "", # Before changing this line, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice @@ -4927,8 +4938,8 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - " Tange, O. (2020, May 22). GNU Parallel 20200522 ('Kraftwerk').", - " Zenodo. https://doi.org/10.5281/zenodo.3841377", + " Tange, O. (2020, July 22). GNU Parallel 20200722 ('Privacy Shield').", + " Zenodo. https://doi.org/10.5281/zenodo.3956817", "", # Before changing this line, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and @@ -5015,11 +5026,11 @@ "GNU $Global::progname $Global::version", "Copyright (C) 2007-2020 Ole Tange, http://ole.tange.dk and Free Software", "Foundation, Inc.", - "License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>", + "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>", "This is free software: you are free to change and redistribute it.", "GNU $Global::progname comes with no warranty.", "", - "Web site: http://www.gnu.org/software/${Global::progname}\n", + "Web site: https://www.gnu.org/software/${Global::progname}\n", "When using programs that use GNU Parallel to process data for publication", "please cite as described in 'parallel --citation'.\n", ); @@ -5041,20 +5052,20 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - "\@software{tange_2020_3841377,", + "\@software{tange_2020_3956817,", " author = {Tange, Ole},", - " title = {GNU Parallel 20200522 ('Kraftwerk')},", - " month = May,", + " title = {GNU Parallel 20200722 ('Privacy Shield')},", + " month = Jul,", " year = 2020,", " note = {{GNU Parallel is a general parallelizer to run", " multiple serial command line programs in parallel", " without changing them.}},", " publisher = {Zenodo},", - " doi = {10.5281/zenodo.3841377},", - " url = {https://doi.org/10.5281/zenodo.3841377}", + " doi = {10.5281/zenodo.3956817},", + " url = {https://doi.org/10.5281/zenodo.3956817}", "}", "", - "(Feel free to use \\nocite{tange_2020_3841377})", + "(Feel free to use \\nocite{tange_2020_3956817})", "", # Before changing this line, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and @@ -5160,7 +5171,7 @@ # General Public License for more details. # # You should have received a copy of the GNU General Public License -# along with this program; if not, see <http://www.gnu.org/licenses/> +# along with this program; if not, see <https://www.gnu.org/licenses/> # or write to the Free Software Foundation, Inc., 51 Franklin St, # Fifth Floor, Boston, MA 02110-1301 USA "; @@ -8582,12 +8593,13 @@ unlink; rmdir; } - if($bash=~s/h//) { + if($bash=~s/(\d+)h/$1/) { exit $bash; } exit $csh; }). - "' ".'"$?h" "$status" '; + # `echo \$?h` is needed to make fish not complain + "' ".'"`echo \\\\\\\\\$?h`" "$status" '; } return $script } @@ -12291,8 +12303,13 @@ my $dbh; if($driver eq "CSV") { # CSV does not use normal dsn - $dbh = DBI->connect("dbi:CSV:", "", "", { f_dir => "$database", }) - or die $DBI::errstr; + if(-d $database) { + $dbh = DBI->connect("dbi:CSV:", "", "", { f_dir => "$database", }) + or die $DBI::errstr; + } else { + ::error("$database is not a directory."); + ::wait_and_exit(255); + } } else { $dbh = DBI->connect($dsn, $userid, $password, { RaiseError => 1, AutoInactiveDestroy => 1 }) @@ -12303,7 +12320,6 @@ $dbh->{'RaiseError'} = 1; $dbh->{'ShowErrorStatement'} = 1; $dbh->{'HandleError'} = sub {}; - if(not defined $options{'table'}) { ::error("The DBURL ($dburl) must contain a table."); ::wait_and_exit(255); @@ -12603,7 +12619,7 @@ my %vartype = ( "Oracle" => { "BIGINT" => "NUMBER(19,0)", "TEXT" => "CLOB", }, - "mysql" => { "TEXT" => "LONGTEXT", }, + "mysql" => { "TEXT" => "BLOB", }, "CSV" => { "BIGINT" => "INT", "FLOAT" => "REAL", }, ); @@ -12642,26 +12658,58 @@ 0, @$record_ref[1..$#$record_ref]); } + sub get_record($) { my $self = shift; my @retval; my $table = $self->table(); my @v_cols = map { ", V$_" } (1..$self->max_number_of_args()); - my $v = $self->get("SELECT Seq, Command @v_cols FROM $table ". - "WHERE Exitval = -1000 ORDER BY Seq LIMIT 1;"); - if($v->[0]) { - my $val_ref = $v->[0]; - # Mark record as taken - my $seq = shift @$val_ref; - # Save the sequence number to use when running the job - $SQL::next_seq = $seq; - $self->update("SET Exitval = ? WHERE Seq = ".$seq, -1220); - my @command = split /\257 /, shift @$val_ref; - $SQL::command_ref = \@command; - for (@$val_ref) { - push @retval, Arg->new($_); + my $rand = "Reserved-".$$.rand(); + my $v; + my $more_pending; + + do { + if($self->{'driver'} eq "CSV") { + # Sub SELECT is not supported in CSV + # So to minimize the race condition below select a job at random + my $r = $self->get("SELECT Seq, Command @v_cols FROM $table ". + "WHERE Exitval = -1000 LIMIT 100;"); + $v = [ sort { rand() > 0.5 } @$r ]; + } else { + # Avoid race condition where multiple workers get the same job + # by setting Stdout to a unique string + # (SELECT * FROM (...) AS dummy) is needed due to sillyness in MySQL + $self->update("SET Stdout = ?,Exitval = ? ". + "WHERE Seq = (". + " SELECT * FROM (". + " SELECT min(Seq) FROM $table WHERE Exitval = -1000". + " ) AS dummy". + ") AND Exitval = -1000;", $rand, -1210); + # If a parallel worker overwrote the unique string this will get nothing + $v = $self->get("SELECT Seq, Command @v_cols FROM $table ". + "WHERE Stdout = ?;", $rand); + } + if($v->[0]) { + my $val_ref = $v->[0]; + # Mark record as taken + my $seq = shift @$val_ref; + # Save the sequence number to use when running the job + $SQL::next_seq = $seq; + $self->update("SET Exitval = ? WHERE Seq = ".$seq, -1220); + # Command is encoded with '\257 space' as splitting char + my @command = split /\257 /, shift @$val_ref; + $SQL::command_ref = \@command; + for (@$val_ref) { + push @retval, Arg->new($_); + } + } else { + # If the record was updated by another job in parallel, + # then we may not be done, so see if there are more jobs pending + $more_pending = + $self->get("SELECT Seq FROM $table WHERE Exitval = ?;", -1210); } - } + } while (not $v->[0] and $more_pending->[0]); + if(@retval) { return \@retval; } else { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/sql new/parallel-20200722/src/sql --- old/parallel-20200522/src/sql 2020-05-23 20:32:43.000000000 +0200 +++ new/parallel-20200722/src/sql 2020-07-22 22:55:10.000000000 +0200 @@ -574,7 +574,7 @@ exit ($err); sub parse_options { - $Global::version = 20200522; + $Global::version = 20200722; $Global::progname = 'sql'; # This must be done first as this may exec myself diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/parallel-20200522/src/sql.1 new/parallel-20200722/src/sql.1 --- old/parallel-20200522/src/sql.1 2020-05-23 20:33:18.000000000 +0200 +++ new/parallel-20200722/src/sql.1 2020-07-22 22:55:40.000000000 +0200 @@ -1,4 +1,4 @@ -.\" Automatically generated by Pod::Man 4.09 (Pod::Simple 3.35) +.\" Automatically generated by Pod::Man 4.11 (Pod::Simple 3.40) .\" .\" Standard preamble: .\" ======================================================================== @@ -54,16 +54,20 @@ .\" Avoid warning from groff about undefined register 'F'. .de IX .. -.if !\nF .nr F 0 -.if \nF>0 \{\ -. de IX -. tm Index:\\$1\t\\n%\t"\\$2" +.nr rF 0 +.if \n(.g .if rF .nr rF 1 +.if (\n(rF:(\n(.g==0)) \{\ +. if \nF \{\ +. de IX +. tm Index:\\$1\t\\n%\t"\\$2" .. -. if !\nF==2 \{\ -. nr % 0 -. nr F 2 +. if !\nF==2 \{\ +. nr % 0 +. nr F 2 +. \} . \} .\} +.rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. @@ -129,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "SQL 1" -.TH SQL 1 "2020-05-23" "20200522" "parallel" +.TH SQL 1 "2020-07-22" "20200722" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l Binary files old/parallel-20200522/src/sql.pdf and new/parallel-20200722/src/sql.pdf differ ++++++ parallel-20200522.tar.bz2.sig -> parallel-20200722.tar.bz2.sig ++++++ --- /work/SRC/openSUSE:Factory/gnu_parallel/parallel-20200522.tar.bz2.sig 2020-06-05 20:29:13.237349977 +0200 +++ /work/SRC/openSUSE:Factory/.gnu_parallel.new.3592/parallel-20200722.tar.bz2.sig 2020-08-03 14:17:34.144630438 +0200 @@ -2,7 +2,7 @@ # To check the signature run: # echo | gpg -# gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve parallel-20200522.tar.bz2.sig +# gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve parallel-20200722.tar.bz2.sig echo | gpg 2>/dev/null gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve $0 @@ -10,32 +10,32 @@ -----BEGIN PGP SIGNATURE----- -iQUHBAABCgAdFiEEzaAaQgjE90UGEH570atFFoiIiIgFAl7JbI0ACgkQ0atFFoiI -iIgx/yagx60zqVyc6TKNteaZ+vU3aIL5yB6kpxwzxy0QrA2RBV9/v2f1fjroorda -w33MIFEXkqKhpVM07jTlsZy05ZK/llI1lUKmyh1jB9qVncZNl5vwZuemZfDigscO -xFwc+5NdI1MQpE64xYoc0g1eFH0ZJzZ3GYjLTRJ1Sn6L0WNUUXXMl+qoy67MRFAS -NFPZ1uCEhdW3T6d5bJdb5c5tK+u6yf0OFKTfAwcFf8BdTBfC2lIx69pWPsedv+XS -naojvnqC3PkTn3Gt/O3/eL36i6yBd+AytVpJBxFDdBWa1t9SzcrokfXvJFj1YcLu -o+U271in5wCX4jblEpkVRRLzVs97UfIMM2QYVpQljTC0iXyDxgb/6eqo9IDP6NRp -AIGStCyy26Ttps73PbtrtKBeSZww1HbrAfZxn/o6zzuRLNwHWApC40JUrYxvhBv1 -JNLa2kSvcLeQ5Yey4RO4/qdeV/bhiLOJZB0/0gVb6Q7vjW7PzYKeLEVwfzJ4kljY -jG8kTSHeKXfCq63ouuZEKE5oeE3GDg/pmq0G32PUNIHezcPF3gk0A5lyy6UOHHRV -V5Y+bFir0BCC+BFHobFaIfSIj6Ov8YK5fsC/C4YsZsv6MKzsPhOKNyTWlT6z/CQz -l8MgHFAFB9f1b5qWeD5e+c/xWXsyKSYorRswY42kFx3cYf0vwKX1t4aPFXEueKVs -rqkKVxX/IbDMYN89BT7ONYpglPLKqWr136JeHiTE1Zk+uCv9car4HjDdgT+VZ0Aa -bMtHOcY1n4Q8JBVzq31PZ2nRWHZXlSJOfzGLvGtfweueg6TZbXPaPhirwtpgE0Eo -zKWFP4JYbYPLVRZVfWntFU5O+mvb683rABs1k6ZGrKkyiNkOmjRd3KJk2Gl9xvgp -YekWEqh8RlORSMn6M9QyfZGLGuL/ND7zp256qWSXVOObjQuh3rG4f3yYYlaKgQbA -INaQ7Cj8ahZoCDWfR+7r5ZmubCz0udZA8jFKO1Mbik8YL3id0ItPvavOn7o9002K -axKtb7arY1P++rEElQ10j5ODwVnhPGx7G952bUzdusf9c0p5evzcnJaEAQNsnvFL -ZwKwg26LEwjw0YOJ57ZXzT1kUrdPQNC/Gj9XsuSOEc97GBl2d6SNQ145mtYjVaho -80BU2mzYmPPi1xZaZB5JNpOkRhu5PoZRl0Sxpx+8Bmu/aaINspR5rhdxDXoi+H0O -92DllIWmLofqGkg9RK9/8h+WEDZoyr6Y4xXeilm8Xa27Y1mwqwuDnyvrf2D2I0PY -Qo/umAjNjYRdiJgRxfE+Fcuo6ubpaqwEF/YZj7pSEurjXPsUVQGN+TC1XEgq/fih -EFkSCk0+v6W1NBOZioGPMet0jfeuvYcBYawq3H5SZ+2OPzAbSS3Ve1ixqDB7Pzpo -N87sV8JPJyoWpvT68gGTqoef0TVk/Q1gHDTjLSbkC3Ra5XyuG53zS8dQZZUEPLkW -41VXO1Pr+1KUgMNjmLhVNhCUKD8rHuQSomeaKXFAYW2oi1krR6U0W+V7sI02XUwK -RmODYG2K1nmkMUNEkmUCZhXjRJRWBZyPI1Vcv8JDK/xvXMvrTvMSWpe+HFDbc0ES -tfVlnrrzY0pb0PRP/1mn4YluvBnTFJuTgzFNknl1GdMq6TFR+tyuypPh -=qcHS +iQUHBAABCgAdFiEEzaAaQgjE90UGEH570atFFoiIiIgFAl8YqBIACgkQ0atFFoiI +iIgb7yagtq4p8frx0/OinsuCZrGkkARAhRPdSVU9KndJ0nHqmHcuy87fMtRAgoA4 ++RVIUlYu/ctbn96PmfOWky9AHO0G+9qwI0KEapP8cWJloRH+Nsnr+njLulb3GAHw +EukalhBlRiQ0dUolbHBRQR1RU75gnagIBePsPanc2hKAD2Z/QFbx80i6m8KfGoZF +IFNhsXaysM2mVt4G9bzY8U1CFrjH2v80u9voYMbtU7K6Kpdaxc0uYNI1yHJvXgX4 +EXyJeGXL05ZliPFezi3OoVdk2WBQh529LIXof3I5btHtRrlr4ueFbIx/wk/LI8ND +pd8BPghVGKuTx/Q7v93JWLEc4MfEJGseD4XSq4WI34Dl3aPux8ueMUDru6hYagl+ +BxiWVh2xHkaTpuBothuqNC+C92ZQB4WLd+EFJde2OPkw2v7ME/A9hnE3qxoMRd7S +U955RtfIeL93hCGvVfAuNv1V6olQIwMhG8zL8KUVtoNRjW6QC3qX+PrgFVpKT80V +w3i1uFmYOZ4MElqR5VYfeWOjBlHeOuBRLD6Jh1Z8NDwr/G6dP6zHvA2e5+OwJBU/ +WvxzH9TJ+7sj95S9LCUenAW3xf7t/VJhutA9PA/uyqCXglQ3x7zKamndA8pXNp9L +oujxNBya5034KaVdN/ah4v2Q85xaM9aE4g1b/HZeImYPWuzqrjcNsvqDLeG4YEJ4 +i+1JzMagqoxOVxr+YxpMIoY47+MsspscnG7p5i+Pm0nT8jtVli070nCnarZH4lqQ +GqOG1NFh9SzRGo86vRrKR/tcE3cKPqYUgRDa4uj2GXBhVgRyMsNrVgfgc3ItUtqP +6SjRTGhHz9Xz45e4VI4dpKf2rXw2DS7y1XFrhpbUN28I/KOFs/6fWxL4MdimnnEY +1YICdJ9FPTgIZC2FB1n5TfiqF5ESKKjDqIAUtKNvKOSn6mWL4JEVwrVYcO6eexIg +tsbq18vq0Ntp6kH/OXR3a0M7unFIBmF9pqMQQ+BgeirmbRdXRjKbvJNm9x3SIpZz +c2GGY9d74Pqn889avc49PkeCOOFH7QF6dUPaXXvE3Phpz9bI/Fwgmn3BUUFE5pGr +xx2OavPP5bwUo6cznQO3oF2YIdpDeh2XufFQ+xArPEWP2PgqXAWiDXoon7kAwovO +TvzNQjPJZvvGBMz95mcsH0GCwQWyjH5aCuh+mKD0uV1GsWRBHTw4c4tF/QJFzlxE +/b7aSfhj/EgY3WQ53rMUUhgMX/fQUrWX/BORAAfZgsF0oMOsplz5sVk51oK07bL9 +IgTBGsY3cOw+VsudLPqhFKaczIHRCjnVh6OZQnslHRxGRXGL309kDLW1atBsLl4I +0arIu391JVpA/Oa/aQi6Zc+jp656gzEzh9NhFBcF5kDitnh7BZjvzD+OYNfBbPcc +ED1Z4gDsl8D6/I0wCgCt6v0PxAJU3neVjMgXaMx2pbjHq81/KwiizL5La4MXeoeX +eK1fdUihfZeWsL11qpTy+7F4nOKfII5u2kB9NwuQx9rNns4AaHYMfDD1eB3pSYUW +Ekm81fHgFWu0VFvBdRPf0dqMEHr+BnJi2FZ2ErD/RxmRhJySFUjTtX+fE3uvSQG7 +EgyiSjPNwBpJlsFFuwcup8bNfd7bGBNEpVnI4MgYdTcBQXFIZJ1EG9ez +=pW61 -----END PGP SIGNATURE-----