Re: portability of xargs

[Dan Kegel]

Oh, absolutely, python is a nonstarter for an autotools replacement.

Autotools is/was great because it worked everywhere, even during early
bootstrap where all you have is shell, basic shell utilities, and a C
compiler.
Python doesn't, and never will.

The C ports of meson and ninja could potentially be part of a future,
modernized, bootstrap-the-world effort.
Switching build systems is crazy hard, so you want to take advantage
of what people are already moving to.
My impression is that meson is gaining momentum.

In 2020, I spent some time porting zlib-ng's build system to meson.
That was fun.
But rather than submit that port, I spent my time setting up a script to
run at CI time that verified their Make and CMake builds produce
bit-for-bit identical outputs.
That's still in use, and makes it a *lot* easier for them to support
both build systems.
I figured having a verifier like that would make it easier to add
Meson support later, should someone want to do it.
And I recommend doing that when adding a new build system to any core tool.
It's not common practice, but it's not that hard, especially if you
have an example to work from.
- Dan


On Tue, Feb 15, 2022 at 7:52 PM Mike Frysinger  wrote:
>
> On 15 Feb 2022 20:25, Jacob Bachmeyer wrote:
> > Dan Kegel wrote:
> > > Meson is a candidate for such a next-gen config system.  It is in python,
> > > which does not quite qualify as usable during early uplift/bootstrap, but
> > > there are C ports in progress, see e.g. https://sr.ht/~lattis/muon/
> >
> > *Please* do not introduce a dependency on Python; they do not worry much
> > about backwards compatibility.  If there is ever a Python 4 with a 3->4
> > transition anything like the 2->3 transition, you could end up with
> > every past release relying on current Python becoming unbuildable.
>
> Python 3.0 isn't even compatible with Python 3.10 in some ways.  it's a
> sliding window of time/releases, not a major version skew.
>
> > Having complex dependencies for creating the build scripts is one thing,
> > but needing major packages (like Python) to *use* the build scripts is a
> > serious problem for anything below the "user application" tier,
> > especially the "base system" tier.
>
> yeah, i see no path forward for requiring Python in the generated configure
> or Makefile.in files.  i wouldn't feel as bad about replacing the current
> perl code with python to run `automake`, but i don't think the current set
> of Automake maintainers would agree :).
> -mike

Re: type errors, command length limits, and Awk (was: portability of xargs)

[Dan Kegel]

FWIW, commandline length limits are a real thing, I've run into them
with Make, CMake, and Meson.
I did some work to help address them in Meson, see e.g.
https://github.com/mesonbuild/meson/issues/7212

And just for fun, here's a vaguely related changelog entry from long
ago, back when things were much worse:

Tue Jun  8 15:24:14 1993  Paul Eggert  (egg...@twinsun.com)
* inp.c (plan_a): Check that RCS and working files are not the
same.  This check is needed on hosts that do not report file
name length limits and have short limits.

:-)

Re: type errors, command length limits, and Awk (was: portability of xargs)

[Mike Frysinger]

On 15 Feb 2022 21:17, Jacob Bachmeyer wrote:
> Mike Frysinger wrote:
> > context: https://bugs.gnu.org/53340
> >   
> Looking at the highlighted line in the context:

thanks for getting into the weeds with me

> > >   echo "$$py_files" | $(am__pep3147_tweak) | $(am__base_list) | \
> It seems that the problem is that am__base_list expects ListOf/File (and 
> produces ChunkedListOf/File) but am__pep3147_tweak emits ListOf/Glob.  
> This works in the usual case because the shell implicitly converts Glob 
> -> ListOf/File and implicitly flattens argument lists, but results in 
> the overall command line being longer than expected if the globs expand 
> to more filenames than expected, as described there.
> 
> It seems that the proper solution to the problem at hand is to have 
> am__pep3147_tweak expand globs itself somehow and thus provide 
> ListOf/File as am__base_list expects.
> 
> Do I misunderstand?  Is there some other use for xargs?

if i did not care about double expansion, this might work.  the pipeline
quoted here handles the arguments correctly (other than whitespace splitting
on the initial input, but that's a much bigger task) before passing them to
the rest of the pipeline.  so the full context:

  echo "$$py_files" | $(am__pep3147_tweak) | $(am__base_list) | \
  while read files; do \
$(am__uninstall_files_from_dir) || st=$$?; \
  done || exit $$?; \
...
am__uninstall_files_from_dir = { \
  test -z "$$files" \
|| { test ! -d "$$dir" && test ! -f "$$dir" && test ! -r "$$dir"; } \
|| { echo " ( cd '$$dir' && rm -f" $$files ")"; \
 $(am__cd) "$$dir" && rm -f $$files; }; \
  }

leveraging xargs would allow me to maintain a single shell expansion.
the pathological situation being:
  bar.py
  __pycache__/
bar.pyc
bar*.pyc
bar**.pyc

py_files="bar.py" which turns into "__pycache__/bar*.pyc" by the pipeline,
and then am__uninstall_files_from_dir will expand it when calling `rm -f`.

if the pipeline expanded the glob, it would be:
  __pycache__/bar.pyc __pycache__/bar*.pyc __pycache__/bar**.pyc
and then when calling rm, those would expand a 2nd time.

i would have to change how the pipeline outputs the list of files such that
the final subshell could safely consume & expand.  since this is portable
shell, i don't have access to arrays & fancy things like readarray.  if the
pipeline switched to newline delimiting, and i dropped $(am__base_list), i
could use positionals to construct an array and safely expand that.  but i
strongly suspect that it's not going to be as performant, and i might as
well just run `rm` once per file :x.

  echo "$$py_files" | $(am__pep3147_tweak) | \
  ( set --
while read file; do
  set -- "$@" "$file"
  if test $# -ge 40; then
rm -f "$@"
set --
  fi
done
if test $# -gt 0; then
  rm -f "$@"
fi
  )

which at this point i've written `xargs -n40`, but not as fast :p.

> > automake jumps through some hoops to try and limit the length of generated
> > command lines, like deleting output objects in a non-recursive build.  it's
> > not perfect -- it breaks arguments up into 40 at a time (akin to xargs -n40)
> > and assumes that it won't have 40 paths with long enough names to exceed the
> > command line length.  it also has some logic where it's deleting paths by
> > globs, but the process to partition the file list into groups of 40 happens
> > before the glob is expanded, so there are cases where it's 40 globs that can
> > expand into many many more files and then exceed the command line length.
> 
> First, I thought that GNU-ish systems were not supposed to have such 
> arbitrary limits,

one person's "arbitrary limits" is another person's "too small limit" :).
i'm most familiar with Linux, so i'll focus on that.

xargs --show-limits on my Linux-5.15 system says:
Your environment variables take up 5934 bytes
POSIX upper limit on argument length (this system): 2089170
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2083236

2MB ain't too shabby.  but if we consult execve(2), it has more details:
https://man7.org/linux/man-pages/man2/execve.2.html
   On Linux prior to kernel 2.6.23, the memory used to store the
   environment and argument strings was limited to 32 pages (defined
   by the kernel constant MAX_ARG_PAGES).  On architectures with a
   4-kB page size, this yields a maximum size of 128 kB.

i've def seen "Argument list too long" errors in Gentoo from a variety of
packages due to this 128 kB limit (which includes the environ strings).
users are very hostile to packages :p.

   On kernel 2.6.23 and later, most architectures support a size
   limit derived from the soft RLIMIT_STACK resource limit (see
   getrlimit(2)) that is in force at the time of the execve() call.
   (Architectures with no memory management unit are excepted: they
   maintain the limit that was in effect before kern

Re: portability of xargs

[Mike Frysinger]

On 15 Feb 2022 20:25, Jacob Bachmeyer wrote:
> Dan Kegel wrote:
> > Meson is a candidate for such a next-gen config system.  It is in python,
> > which does not quite qualify as usable during early uplift/bootstrap, but
> > there are C ports in progress, see e.g. https://sr.ht/~lattis/muon/
> 
> *Please* do not introduce a dependency on Python; they do not worry much 
> about backwards compatibility.  If there is ever a Python 4 with a 3->4 
> transition anything like the 2->3 transition, you could end up with 
> every past release relying on current Python becoming unbuildable.

Python 3.0 isn't even compatible with Python 3.10 in some ways.  it's a
sliding window of time/releases, not a major version skew.

> Having complex dependencies for creating the build scripts is one thing, 
> but needing major packages (like Python) to *use* the build scripts is a 
> serious problem for anything below the "user application" tier, 
> especially the "base system" tier.

yeah, i see no path forward for requiring Python in the generated configure
or Makefile.in files.  i wouldn't feel as bad about replacing the current
perl code with python to run `automake`, but i don't think the current set
of Automake maintainers would agree :).
-mike


signature.asc
Description: PGP signature

Re: portability of xargs

[Mike Frysinger]

On 15 Feb 2022 08:29, Bob Friesenhahn wrote:
> A problem with xargs is that without using the GNU -O or --null 
> argument and null-terminated arguments, file names containing spaces 
> won't be handled properly.  File names containing spaces is an issue 
> for Autotools in general.  This is again an issue under Microsoft 
> Windows where users typically are provided with directory paths which 
> contain a space and they need to take additional administrative 
> measures in order to provide directory paths which work with 
> Autotools.

there's a few related issues here.  i'm not trying to fix everything, just
not actively make the current situation worse.

Automake fundamentally uses whitespace delimited variables, so there's no
support for installing or building a file that has spaces in its name.

building in a directory that has spaces is a bit more tractable.

installing/uninstalling to a directory that has spaces is also a bit more
tractable.  and i think we do an OK job here (by quoting $DESTDIR and
related path expansions).

so wrt xargs usage in uninstall, since we have no way of supporting files
themselves with spaces, it's not an issue.  the code will cd to the base
path first, and then delete things from there (including letting the
shell do globs expansion before calling the underlying tool).
-mike


signature.asc
Description: PGP signature

type errors, command length limits, and Awk (was: portability of xargs)

[Jacob Bachmeyer]

Mike Frysinger wrote:

context: https://bugs.gnu.org/53340
  

Looking at the highlighted line in the context:

>   echo "$$py_files" | $(am__pep3147_tweak) | $(am__base_list) | \
It seems that the problem is that am__base_list expects ListOf/File (and 
produces ChunkedListOf/File) but am__pep3147_tweak emits ListOf/Glob.  
This works in the usual case because the shell implicitly converts Glob 
-> ListOf/File and implicitly flattens argument lists, but results in 
the overall command line being longer than expected if the globs expand 
to more filenames than expected, as described there.


It seems that the proper solution to the problem at hand is to have 
am__pep3147_tweak expand globs itself somehow and thus provide 
ListOf/File as am__base_list expects.


Do I misunderstand?  Is there some other use for xargs?

I note that the current version of standards.texi also allows configure 
and make rules to use awk(1); could that be useful here instead? (see below)



[...]

automake jumps through some hoops to try and limit the length of generated
command lines, like deleting output objects in a non-recursive build.  it's
not perfect -- it breaks arguments up into 40 at a time (akin to xargs -n40)
and assumes that it won't have 40 paths with long enough names to exceed the
command line length.  it also has some logic where it's deleting paths by
globs, but the process to partition the file list into groups of 40 happens
before the glob is expanded, so there are cases where it's 40 globs that can
expand into many many more files and then exceed the command line length.
  


First, I thought that GNU-ish systems were not supposed to have such 
arbitrary limits, and this issue (the context) originated from Gentoo 
GNU/Linux.  Is this a more fundamental bug in Gentoo or still an issue 
because Automake build scripts are supposed to be portable to foreign 
system that do have those limits?


Second, counting files in the list, as you note, does not necessarily 
actually conform to the system limits, while Awk can track both number 
of elements in the list and the length of the list as a string, allowing 
to break the list to meet both command tail length limits (on Windows or 
total size of block to transfer with execve on POSIX) and argument count 
limits (length of argv acceptable to execve on POSIX).


POSIX Awk should be fairly widely available, although at least Solaris 
10 has a non-POSIX awk in /usr/bin and a POSIX awk in /usr/xpg4/bin; I 
found this while working on DejaGnu.  I ended up using this test to 
ensure that "awk" is suitable:


8<--
# The non-POSIX awk in /usr/bin on Solaris 10 fails this test
if echo | "$awkbin" '1 && 1 {exit 0}' > /dev/null 2>&1 ; then
   have_awk=true
else
   have_awk=false
fi
8<--


Another "gotcha" with Solaris 10 /usr/bin/awk is that it will accept 
"--version" as a valid Awk program, so if you use that to test whether 
"awk" is GNU Awk, you must redirect input from /dev/null or it will hang.


Automake may want to do more extensive testing to find a suitable Awk; 
the above went into a script that remains generic when installed and so 
must run its tests every time the user invokes it, so "quick" was a high 
priority.



-- Jacob

Re: portability of xargs

[NightStrike]

On Tue, Feb 15, 2022 at 6:01 PM Bob Friesenhahn
 wrote:
> The people who tell me it is more portable are very interested in
> targeting Microsoft Windows.

Introduce them to mingw-w64.sf.net + msys2 :)  I heard good things
about those :P

> The "Makefiles" that Cmake generates are self-referential in that
> almost all rules invoke Cmake to do the work.

> I find that the function of Autoconf is quite useful.  When compared
> with Cmake, it is much more user-friendly since the configure script
> responds to --help and the help output is very helpful.  The configure
> script nicely tells me what it is doing and what it found.  Autotools
> is very structured and projects work in a common way whereas I find
> that Cmake projects are heavily customized with options added by the
> project developer.  Just doing a build entirely outside of the source
> tree is extremely clumsy with Cmake.
>
> Regardless, I don't think that Autotools developers should waste time
> on assuring compatability with systems which are no longer in active
> use.  It is much better so spend time improving areas where Autotools
> is weak.

"Portability" is impossible to quantify.  It expands to far more than
support of ancient, esoteric tools.

I would argue that Cmake is less portable, since it requires that
Cmake be available for a user to build the software.  That can be
onerous, especially if the user doesn't have permission to install
Cmake.  Autoconf requires 'sh', and automake requires 'make'.

I would further argue that Cmake is less portable because it's less
user friendly.  The poor man's distinction is that autotools makes
life for the project owner hard so that the life of the user is easy.
Cmake does the exact opposite, and makes the user's life a difficult
experience so that the developer's life is simpler.  I would
extrapolate that to the realm of "less portable", since it requires
that I have more system requirements in my own brain (skills,
knowledge, experience).  Whereas I can handle any autotools package
trivially with "./configure CFLAGS=-myflag CXX=myC++Compiler
--enable-option --with-package=/custom/path", and know very little
about the upstream project, I can't do so with a Cmake based build
system.

For me, this isn't about 32-year old artifacts.  This is about the
infinitely diverse nature of any modern, still maintained system.  In
a world of corporate policies, shifting environments, environment
modules, embedded devices, cyber security, firewalls, user
permissions, and all sorts of other mitigating factors, all of which I
personally must deal with when trying to deploy any given package to
an environment, I will pick autotools every single time.  The workload
required for me to deploy an autotools package is exponentially
smaller than that required of a Cmake package.

Re: portability of xargs

[Jacob Bachmeyer]

Dan Kegel wrote:

Meson is a candidate for such a next-gen config system.  It is in python,
which does not quite qualify as usable during early uplift/bootstrap, but
there are C ports in progress, see e.g. https://sr.ht/~lattis/muon/
  


*Please* do not introduce a dependency on Python; they do not worry much 
about backwards compatibility.  If there is ever a Python 4 with a 3->4 
transition anything like the 2->3 transition, you could end up with 
every past release relying on current Python becoming unbuildable.


Having complex dependencies for creating the build scripts is one thing, 
but needing major packages (like Python) to *use* the build scripts is a 
serious problem for anything below the "user application" tier, 
especially the "base system" tier.



-- Jacob

Re: portability of xargs

[Paul Smith]

On Tue, 2022-02-15 at 16:59 -0600, Bob Friesenhahn wrote:
> The people who tell me it is more portable are very interested in 
> targeting Microsoft Windows.

Unsurprising :).

Just to be clear, cmake can't/won't help you write programs that are
portable to Windows.  If you are targeting the W32 system API, you may
have to modify a lot of code to make it work and cmake will be of zero
assistance to creating this code "portably".  And if you are going to
use a POSIX layer on top of Windows (for example, cygwin) then you have
a POSIX environment already and can just run configure anyway.

If your project is basically pretty portable (doesn't use many system
APIs outside of C and C++ standard runtimes), so that the main problem
is the configure and build systems, then cmake can be a big benefit.

For example, you see a lot of cmake usage in library projects or simple
programs on Git forges, where the code is pretty stand-alone and people
want to make it easy to build on different systems.

> 
> The "Makefiles" that Cmake generates are self-referential in that 
> almost all rules invoke Cmake to do the work.

True; CMake includes a kind of "busybox" set of operations, to increase
portability (so it doesn't have to rely on "echo", "cp", etc. in its
rules, that wouldn't work everywhere).  But, the types of operations it
supports are limited.  It can echo, copy files/directories, make
directories, remove files/directories, rename things, touch files,
create tar files, and generate checksums with builtin operations (and a
few other things).

If you have cmake installed you can run "cmake -E --help" to see a
complete list.

Re: portability of xargs

[Bob Friesenhahn]

On Tue, 15 Feb 2022, Paul Smith wrote:


On Tue, 2022-02-15 at 15:37 -0600, Bob Friesenhahn wrote:

I have been told by several people (who have much more self-esteem
than me) that a build tool called 'cmake' is far more portable than
Autotools so maybe we should make support for 32 year old systems
cmake's responsibility?


That is not accurate.  Or at least, cmake uses a much different
definition of "portable" than autoconf / automake.  Certainly cmake
cannot possibly support 32-year old systems (if that's needed).


Thanks for the nice response.  I did not for a minute think that 
what I was told was accurate.


The people who tell me it is more portable are very interested in 
targeting Microsoft Windows.



cmake is a tool that, given an input definition of build dependencies,
can generate build control files for multiple different types of build
tools.  It can generate makefiles, Ninja files, Xcode project files,
and Visual Studio project files.  Maybe others, I'm not sure.


The "Makefiles" that Cmake generates are self-referential in that 
almost all rules invoke Cmake to do the work.



The main idea of autoconf is that it allows configuring the package for
systems that the author never even heard of, much less has access to
(as long as it has a POSIX interface).  cmake only tries to generate
output files for systems it has been developed for (for example, each
new version of Visual Studio often requires a new release of cmake to
support it).


The above is a perfect description of the situation.


Of course maybe autoconf is not necessary anymore: I don't know about
that.  I do know that even though GNU make itself is relatively simple,


I find that the function of Autoconf is quite useful.  When compared 
with Cmake, it is much more user-friendly since the configure script 
responds to --help and the help output is very helpful.  The configure 
script nicely tells me what it is doing and what it found.  Autotools 
is very structured and projects work in a common way whereas I find 
that Cmake projects are heavily customized with options added by the 
project developer.  Just doing a build entirely outside of the source 
tree is extremely clumsy with Cmake.


Regardless, I don't think that Autotools developers should waste time 
on assuring compatability with systems which are no longer in active 
use.  It is much better so spend time improving areas where Autotools 
is weak.


If 'xargs' has worked consistently for 20 years, it should be ok to 
use.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: portability of xargs

[Paul Smith]

On Tue, 2022-02-15 at 15:37 -0600, Bob Friesenhahn wrote:
> I have been told by several people (who have much more self-esteem 
> than me) that a build tool called 'cmake' is far more portable than 
> Autotools so maybe we should make support for 32 year old systems 
> cmake's responsibility?

That is not accurate.  Or at least, cmake uses a much different
definition of "portable" than autoconf / automake.  Certainly cmake
cannot possibly support 32-year old systems (if that's needed).

cmake is a tool that, given an input definition of build dependencies,
can generate build control files for multiple different types of build
tools.  It can generate makefiles, Ninja files, Xcode project files,
and Visual Studio project files.  Maybe others, I'm not sure.

So in that sense it's "more portable" because given a single input
definition you can build the project with both make-based systems and
Visual Studio-based systems (etc.) without extra maintenance work.

However, of course because the definition files must be able to
generate all these different outputs they are necessarily "least common
denominator" definitions; you can't just define a new make rule to do
something obscure like you can with automake. Cmake works great for
simple projects but you will absolutely struggle to encode more complex
build requirements.

And, cmake has really not much comparable to autoconf.  It does provide
some very minimal support for choosing some compiler options and some
facilities sort of like pkg-config but nothing even approaching the
kinds of checks and configuration that autoconf supports.

The main idea of autoconf is that it allows configuring the package for
systems that the author never even heard of, much less has access to
(as long as it has a POSIX interface).  cmake only tries to generate
output files for systems it has been developed for (for example, each
new version of Visual Studio often requires a new release of cmake to
support it).

Of course maybe autoconf is not necessary anymore: I don't know about
that.  I do know that even though GNU make itself is relatively simple,
people still build and use it on older, and non-GNU/Linux, systems, and
that they want to use cross-compilers (cmake can do this, kind of, but
it's much more difficult than autotools support) and other more
advanced operations.

Re: portability of xargs

[Bob Friesenhahn]

It it really expected that Autotools should support 32 year old 
systems?


This feels counter-productive to me.

I have been told by several people (who have much more self-esteem 
than me) that a build tool called 'cmake' is far more portable than 
Autotools so maybe we should make support for 32 year old systems 
cmake's responsibility?


I am fond of systems from the early/mid '90s but it seems better to 
support POSIX compliant systems with newer software.


People who have somehow figured out how to keep old hardware running 
without disk drives, electrolytic capacitors, or fans, can install 
older GNU software first in order to bootstrap sufficiently to a 
sufficient level of POSIX compliance.


The well-built systems I bought prior to 2007 are already dead or 
difficult to repair.


I do not see any reason to spend any time at all supporting an OS 
older than 2008.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: portability of xargs

[Nick Bowler]

On 2022-02-14, Mike Frysinger  wrote:
> context: https://bugs.gnu.org/53340
>
> how portable is xargs ?  like, beyond POSIX, as autoconf & automake both
> support non-POSIX compliant systems.  i want to use it in its simplest
> form: `echo $var | xargs rm -f`.

As far as I can tell xargs was introduced in the original System V UNIX
(ca. 1983).  This utility subsequently made its way back into V10 UNIX
(ca. 1989) and subsequently 4.3BSD-Reno (ca. 1990) and from there to
basically everywhere.  The original implementation from System V
supports the "-x", "-l", "-i", "-t", "-e", "-s", "-n" and "-p" options.
Of these, POSIX only chose to standardize "-x", "-t", "-s", "-n" and
"-p" suggesting possible incompatibilities with other options.

HP-UX 11 xargs expects the last filename to be followed by a white-space
character, or it will be ignored:

  gnu% printf 'no blank at the end' | xargs printf '[%s]'; echo
  [no][blank][at][the][end]

  hpux11% printf 'no blank at the end' | xargs printf '[%s]'; echo
  [no][blank][at][the]

The HP-UX 11 behaviour is also observed on Ultrix 4.5, but not on
4.3BSD-Reno.  Since xargs input typically ends with a newline, this is
not a serious practical problem.

Cheers,
 Nick

Re: portability of xargs

[Roman Neuhauser]

# egg...@cs.ucla.edu / 2022-02-14 19:53:17 -0800:
> On 2/14/22 19:45, Mike Frysinger wrote:
> > how portable is xargs ?
> 
> It can be a porting problem, unfortunately. There are several corner 
> cases that various implementations don't get right. I expect this is why 
> the GNU Coding Standards exclude xargs from the list of programs that 
> 'configure' and Makefile rules can use.

It seems that current circumstances notwithstanding,
this project has no way to recognize when a workaround
is no longer necessary.  In other words, it may well be
past many such transitions already.  I'm reminded of
the herd of monkeys beating up any one of them who reached
for the banana even though none of them knew what was wrong
with it.

Back when I was involved with FreeBSD ports (~ 4.7 - 6.2),
the extreme[*] complexity of autotools was the greater
hindrance to porting -- actual incompatibilities of the
underlying environment were a distant second (IMO - I speak
for now one else).

[*] It was disproportional to the achievements - certainly,
given the complexity present in the tools and leaking into
their interfaces, building and installing autotools-using
code shouldn't have required so much constant effort.


I didn't mean to whine here, the two paragraps above are
meant as an anecdote in support of another take on the
situation.

I've only seen the last twenty years, and am an external
observer to anything FSF or GNU, but it seems to me that:

* the OS ecosystem has fewer species than it had when
  autotools were conceived
* it would be perfectly fine for autotools maintainers
  to say "if you insist on running SunOS 2, HPUX 10 or
  similar, feel free to put in the work or the funds"
* same for minority/fringe FOSS environments: the
  message could and IMO should be "you can deviate if
  you take on the cost"

These things came across my mind when I read the thread,
I'm probably missing or a vital angle or two, or maybe
just valuing them differently.

Thanks for reading and back to lurking,

-- 
roman

Re: portability of xargs

[Alex Ameen]

If the goal is to limit command line length while maintaining or increasing
the number of files passed to various compilation tools, it may be worth
writing object lists to files to be passed as command scripts - in cases
where the tool supports it. Obviously whether the majority of tools support
command files is going to be the deciding factor.

Libtool is pretty equipped to deal with this category of tool portability;
but allowing `automake' to make this assumption is something to approach
more cautiously.

On Mon, Feb 14, 2022, 11:29 PM Paul Eggert  wrote:

> On 2/14/22 20:03, Mike Frysinger wrote:
> > are the corner cases known ?
>
> I don't know of a catalog, no.
>
> These days you might have better luck with "find ... -exec ... {} +".
> Although standardized more recently than xargs, my vague impression is
> that there's less variation among implementations. It works only on file
> names, though.
>
>

Re: portability of xargs

[Bob Friesenhahn]

On Mon, 14 Feb 2022, Mike Frysinger wrote:


On 14 Feb 2022 19:53, Paul Eggert wrote:

On 2/14/22 19:45, Mike Frysinger wrote:

how portable is xargs ?


It can be a porting problem, unfortunately. There are several corner
cases that various implementations don't get right. I expect this is why
the GNU Coding Standards exclude xargs from the list of programs that
'configure' and Makefile rules can use.


are the corner cases known ?  if it's such that xargs doesn't always correctly
limit itself to the system limit based on other factors, i can live with that
assuming that the -n option is reliable.


This morning I read this discussion thread and did not see any actual 
problems with current (on systems that people are still using) xargs 
portability mentioned.


Microsoft Windows (e.g. POSIX environments which run under Windows 
such as Cygwin, MSYS, etc.) is surely an existing corner case which 
demands more attention than archaic systems.  It seems that the 
command line length when using Microsoft Windows may depend on the 
number of bytes in the arguments (not the number of arguments) as well 
as the number of bytes in the current environment variable data.


A problem with xargs is that without using the GNU -O or --null 
argument and null-terminated arguments, file names containing spaces 
won't be handled properly.  File names containing spaces is an issue 
for Autotools in general.  This is again an issue under Microsoft 
Windows where users typically are provided with directory paths which 
contain a space and they need to take additional administrative 
measures in order to provide directory paths which work with 
Autotools.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: portability of xargs

[Daniel Herring]

On Tue, 15 Feb 2022, Jan Engelhardt wrote:



On Tuesday 2022-02-15 07:16, Daniel Herring wrote:


Maybe a next-generation configuration tool should start by defining interfaces
for user interactions and build tools.  This would allow CLI and easy GUI and
IDE users, integration with multiple build systems, static and dynamic probing,
...


Obligatory https://xkcd.com/927/ response.



:)  Not necessarily define entirely NEW interfaces, just recognize where 
newer tech like JSON (?) could be applied to improve the functionality of 
older tech like Autotools.  Something like the Language Server Protocol, 
but for build configuration.  In fact, this has the potential to greatly 
reduce the dependencies for a cross-compile target.  You could treat it as 
a very dumb terminal and push all the logic to the host a la Expect / 
Ansible.  Don't even need a POSIX shell...


For all its stupid connectors and cables, I much prefer USB to what it 
displaced many years ago.  Similar story with Git.  Running Systemd but 
don't fully trust it yet.


-- Daniel

Re: portability of xargs

[Jan Engelhardt]

On Tuesday 2022-02-15 07:16, Daniel Herring wrote:
>
> Maybe a next-generation configuration tool should start by defining interfaces
> for user interactions and build tools.  This would allow CLI and easy GUI and
> IDE users, integration with multiple build systems, static and dynamic 
> probing,
> ...

Obligatory https://xkcd.com/927/ response.

Re: portability of xargs

[Dan Kegel]

Meson is a candidate for such a next-gen config system.  It is in python,
which does not quite qualify as usable during early uplift/bootstrap, but
there are C ports in progress, see e.g. https://sr.ht/~lattis/muon/

In the meantime, I agree that find is more portable than cards ;-)

- Dan

Daniel Herring  schrieb am Mo., 14. Feb. 2022, 22:17:

> Hi Mike,
>
> It often makes sense to change a project that uses Autotools rather than
> modifying the Autotools.  Can this argument-splitting behavior be
> documented as expected?  Are no workarounds available?
>
> Replacing "overly fancy but proven shell script" with "dependency on a new
> tool" may replace an obscure problem for one project with an intractable
> problem that breaks portability (to obscure platforms) for all projects.
> I have distant memories of working through similar problems by tracing
> configure scripts and makefiles, redirecting the problematic command lines
> to files, and monkey-patching them before execution.  Tedious but doable
> due to the simplicity of the tooling, even in a cross compile.  Getting a
> "functional" xargs would have been MUCH harder in that environment.
>
> IMO, you are touching on the fundamental issue of Autotools.  Portable
> shell scripting is one of the worst programming environments available
> today, and it hinders progress and adoption, but it is the mostly widely
> available bootstrapping environment available.  Autotools embodies decades
> of hard-won experience, and people rightly want to use it on new projects,
> but they are weighed down by all the baggage of history, even when it is
> not even remotely relevant.
>
> As the years go by, this tradeoff is getting less justifiable.  I hope a
> better bootstrapping environment will come along.  Something that ports
> the spirit of Autotools to a much better language.  Maybe a subset of
> Python that can bootstrap from shell?  Many would argue for cmake or one
> of the newer build systems, but they all seem to miss one or more
> fundamentals of the Autotools.  At a minimum, they should address the
> concerns about imake in the Autoconf FAQ.  ;)
>
>
> https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.71/html_node/Why-Not-Imake.html
>
> Maybe a next-generation configuration tool should start by defining
> interfaces for user interactions and build tools.  This would allow CLI
> and easy GUI and IDE users, integration with multiple build systems,
> static and dynamic probing, ...
>
> -- Daniel
>
>
> On Mon, 14 Feb 2022, Mike Frysinger wrote:
>
> > context: https://bugs.gnu.org/53340
> >
> > how portable is xargs ?  like, beyond POSIX, as autoconf & automake both
> > support non-POSIX compliant systems.  i want to use it in its simplest
> > form: `echo $var | xargs rm -f`.
> >
> > automake jumps through some hoops to try and limit the length of
> generated
> > command lines, like deleting output objects in a non-recursive build.
> it's
> > not perfect -- it breaks arguments up into 40 at a time (akin to xargs
> -n40)
> > and assumes that it won't have 40 paths with long enough names to exceed
> the
> > command line length.  it also has some logic where it's deleting paths by
> > globs, but the process to partition the file list into groups of 40
> happens
> > before the glob is expanded, so there are cases where it's 40 globs that
> can
> > expand into many many more files and then exceed the command line length.
> >
> > if i can assume `xargs` is available, then i think i can replace all of
> this
> > custom logic with a simple call to that.
> > -mike
> >
>
>

Re: portability of xargs

[Daniel Herring]

Hi Mike,

It often makes sense to change a project that uses Autotools rather than 
modifying the Autotools.  Can this argument-splitting behavior be 
documented as expected?  Are no workarounds available?


Replacing "overly fancy but proven shell script" with "dependency on a new 
tool" may replace an obscure problem for one project with an intractable 
problem that breaks portability (to obscure platforms) for all projects. 
I have distant memories of working through similar problems by tracing 
configure scripts and makefiles, redirecting the problematic command lines 
to files, and monkey-patching them before execution.  Tedious but doable 
due to the simplicity of the tooling, even in a cross compile.  Getting a 
"functional" xargs would have been MUCH harder in that environment.


IMO, you are touching on the fundamental issue of Autotools.  Portable 
shell scripting is one of the worst programming environments available 
today, and it hinders progress and adoption, but it is the mostly widely 
available bootstrapping environment available.  Autotools embodies decades 
of hard-won experience, and people rightly want to use it on new projects, 
but they are weighed down by all the baggage of history, even when it is 
not even remotely relevant.


As the years go by, this tradeoff is getting less justifiable.  I hope a 
better bootstrapping environment will come along.  Something that ports 
the spirit of Autotools to a much better language.  Maybe a subset of 
Python that can bootstrap from shell?  Many would argue for cmake or one 
of the newer build systems, but they all seem to miss one or more 
fundamentals of the Autotools.  At a minimum, they should address the 
concerns about imake in the Autoconf FAQ.  ;)


https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.71/html_node/Why-Not-Imake.html

Maybe a next-generation configuration tool should start by defining 
interfaces for user interactions and build tools.  This would allow CLI 
and easy GUI and IDE users, integration with multiple build systems, 
static and dynamic probing, ...


-- Daniel


On Mon, 14 Feb 2022, Mike Frysinger wrote:


context: https://bugs.gnu.org/53340

how portable is xargs ?  like, beyond POSIX, as autoconf & automake both
support non-POSIX compliant systems.  i want to use it in its simplest
form: `echo $var | xargs rm -f`.

automake jumps through some hoops to try and limit the length of generated
command lines, like deleting output objects in a non-recursive build.  it's
not perfect -- it breaks arguments up into 40 at a time (akin to xargs -n40)
and assumes that it won't have 40 paths with long enough names to exceed the
command line length.  it also has some logic where it's deleting paths by
globs, but the process to partition the file list into groups of 40 happens
before the glob is expanded, so there are cases where it's 40 globs that can
expand into many many more files and then exceed the command line length.

if i can assume `xargs` is available, then i think i can replace all of this
custom logic with a simple call to that.
-mike

Re: portability of xargs

[Paul Eggert]

On 2/14/22 20:03, Mike Frysinger wrote:

are the corner cases known ?


I don't know of a catalog, no.

These days you might have better luck with "find ... -exec ... {} +". 
Although standardized more recently than xargs, my vague impression is 
that there's less variation among implementations. It works only on file 
names, though.

Re: portability of xargs

[Mike Frysinger]

On 14 Feb 2022 19:53, Paul Eggert wrote:
> On 2/14/22 19:45, Mike Frysinger wrote:
> > how portable is xargs ?
> 
> It can be a porting problem, unfortunately. There are several corner 
> cases that various implementations don't get right. I expect this is why 
> the GNU Coding Standards exclude xargs from the list of programs that 
> 'configure' and Makefile rules can use.

are the corner cases known ?  if it's such that xargs doesn't always correctly
limit itself to the system limit based on other factors, i can live with that
assuming that the -n option is reliable.

the trouble here is that in order to implement its own xargs logic, Automake
has to expand things exactly once.  atm, it happens when invoking rm, but if
we want to partition things up, they have to be expanded first once, and then
passed to the command directly.  without shell arrays or other fancy tools,
this is difficult to pull off :(.
-mike


signature.asc
Description: PGP signature

Re: portability of xargs

[Paul Eggert]

On 2/14/22 19:45, Mike Frysinger wrote:

how portable is xargs ?


It can be a porting problem, unfortunately. There are several corner 
cases that various implementations don't get right. I expect this is why 
the GNU Coding Standards exclude xargs from the list of programs that 
'configure' and Makefile rules can use.

portability of xargs

[Mike Frysinger]

context: https://bugs.gnu.org/53340

how portable is xargs ?  like, beyond POSIX, as autoconf & automake both
support non-POSIX compliant systems.  i want to use it in its simplest
form: `echo $var | xargs rm -f`.

automake jumps through some hoops to try and limit the length of generated
command lines, like deleting output objects in a non-recursive build.  it's
not perfect -- it breaks arguments up into 40 at a time (akin to xargs -n40)
and assumes that it won't have 40 paths with long enough names to exceed the
command line length.  it also has some logic where it's deleting paths by
globs, but the process to partition the file list into groups of 40 happens
before the glob is expanded, so there are cases where it's 40 globs that can
expand into many many more files and then exceed the command line length.

if i can assume `xargs` is available, then i think i can replace all of this
custom logic with a simple call to that.
-mike


signature.asc
Description: PGP signature