Re: Parallelization of shell scripts for 'configure' etc.

[Alex Ameen]

I've been telling folks about the config site file every time this thread
comes up. Good on you for actually trying it haha.

It can make a huge difference. You can short circuit a lot of checks this
way.

Now, the disclaimer: you still shouldn't share a cache file between
projects, and if you use a `config.site` don't stash internal values. Be
sure you keep an eye on your `config.site` values as your system is updated
over time, and if you use containers or build in chroots keep in mind how
that can effect the validity of your cache and `config.site` settings.



On Fri, Jul 8, 2022, 3:05 PM Simon Josefsson via Discussion list for the
autoconf build system  wrote:

> Tim Rühsen  writes:
>
> > a) The maintainer/contributor/hacker setup
> > This is when you re-run configure relatively often for the same
> project(s).
> > I do this normally and and came up with
> >
> https://gitlab.com/gnuwget/wget2/-/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain.
>
> > It may be a bit outdated, but may help one or the other here.
> > Btw, I am down to 2.5s for a ./configure run from 25s originally.
>
> Wow, I think more developers should known about your final suggestion:
>
>
> https://gitlab.com/gnuwget/wget2/-/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain#cccflags-dependent-usage-of-configure-caching
>
> That is, put this in ~/.bash_aliases:
>
> export CONFIG_SITE=~/src/config.site
>
> and this in ~/src/config.site:
>
> if test "$cache_file" = /dev/null; then
>   hash=`echo $CFLAGS $LDFLAGS $host_alias $build_alias|md5sum|cut -d' '
> -f1`
>   cache_file=~/src/config.cache.$CC.$hash
> fi
>
> The top of config.log says which cache file was used, so you can remove
> it when you hack on autoconf/M4 macros.
>
> This appears to save me tons of build time, and I'll run with this now
> since it is non-obtrusive and doesn't require changes in each project...
> maybe the CWD should be put into the cache_file string to avoid cache
> poisining between projects, but that is minor.
>
> /Simon
>

Re: Parallelization of shell scripts for 'configure' etc.

[Simon Josefsson via Gnulib discussion list]

Tim Rühsen  writes:

> a) The maintainer/contributor/hacker setup
> This is when you re-run configure relatively often for the same project(s).
> I do this normally and and came up with
> https://gitlab.com/gnuwget/wget2/-/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain.
>  
> It may be a bit outdated, but may help one or the other here.
> Btw, I am down to 2.5s for a ./configure run from 25s originally.

Wow, I think more developers should known about your final suggestion:

https://gitlab.com/gnuwget/wget2/-/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain#cccflags-dependent-usage-of-configure-caching

That is, put this in ~/.bash_aliases:

export CONFIG_SITE=~/src/config.site

and this in ~/src/config.site:

if test "$cache_file" = /dev/null; then
  hash=`echo $CFLAGS $LDFLAGS $host_alias $build_alias|md5sum|cut -d' ' -f1`
  cache_file=~/src/config.cache.$CC.$hash
fi

The top of config.log says which cache file was used, so you can remove
it when you hack on autoconf/M4 macros.

This appears to save me tons of build time, and I'll run with this now
since it is non-obtrusive and doesn't require changes in each project...
maybe the CWD should be put into the cache_file string to avoid cache
poisining between projects, but that is minor.

/Simon


signature.asc
Description: PGP signature

Re: Parallelization of shell scripts for 'configure' etc.

[Alexandre Oliva]

On Jun 13, 2022, Paul Eggert  wrote:

> In many Gnu projects, the 'configure' script is the biggest barrier to
> building because it takes s long to run. Is there some way that we
> could improve its performance without completely reengineering it, by
> improving Bash so that it can parallelize 'configure' scripts?

I don't know what counts as "completely reengineering it", but I gather
we could get quite a long way by introducing "parallelizable regions",
with explicitly named shell variables and other properties (say config.h
matter) as inputs and outputs (plus enabling macros to add to inputs and
outputs), and having autoconf generate a configure script, script
fragments for the regions, and a Makefile with the dependencies inferred
from explicit and implicit inputs and outputs.  The configure script
would then attempt to run make, or run the fragments serially.

-- 
Alexandre Oliva, happy hackerhttps://FSFLA.org/blogs/lxo/
   Free Software Activist   GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about 

Re: Parallelization of shell scripts for 'configure' etc.

[Chet Ramey]

On 6/18/22 3:05 PM, Tim Rühsen wrote:

[Configure for wget2 runs 'rm' and 'cat' each roughly 2000x - so I came up 
with enabling plugins for those two commands (had to write a plugin for 
'rm', not sure if it never has been accepted by bash upstream).]


The loadable `rm' has been in bash since bash-5.0. There was, let's say,
pushback about it, mostly because it was inefficient in pathological cases
(I'm sure you remember the criticism), but it works fine for configure and
most other cases.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Parallelization of shell scripts for 'configure' etc.

[Tim Rühsen]

Hi all,

On 14.06.22 00:39, Paul Eggert wrote:
In many Gnu projects, the 'configure' script is the biggest barrier to 
building because it takes s long to run. Is there some way that we 
could improve its performance without completely reengineering it, by 
improving Bash so that it can parallelize 'configure' scripts?


A faster configure script execution indeed is something I'd love to see.
The title of this thread infers that we *only* want to discuss 
parallelization - maybe we can generalize this to "Making configure 
scripts run faster" ?


[A little side-note: the invocation of gnulib-tool is *far* slower than 
the running the configure scripts, for the projects that I work on.

But surely this is a problem on it own.]

I see two main setups when running configure scripts. How to speeding up 
the execution has several possible solutions for each of the setups (but 
with overlaps of course).


a) The maintainer/contributor/hacker setup
This is when you re-run configure relatively often for the same project(s).
I do this normally and and came up with 
https://gitlab.com/gnuwget/wget2/-/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain. 
It may be a bit outdated, but may help one or the other here.

Btw, I am down to 2.5s for a ./configure run from 25s originally.

b) The one-time build setup
This is people building + installing from tarball and automated build 
systems (e.g. CI) with regular OS updates. I also think of systems like 
Gentoo where you build everything from source.
As Alex Ameen pointed out, using a global configure cache across 
different projects may be insecure.

Also people often want to use optimization in this case.
Installing ccache is also not likely when people just want to 
build+install a single project.


I personally see a) as solved, at least for me.

b) is a problem because
1. People start to complain about the slow GNU build system (autotools), 
which drives new projects away from using autotools and possible it 
drives people away from GNU in general. Or in other words: let's not eat 
up people's precious time unnecessarily when building our software.


2. Building software in large scale eats tons of energy. If we could 
reduce the energy consumption, it gives us at least a better feeling.



What can we do to solve b)
I guess we first need to analyze/profile the configure execution.
For this I wrote a little tool some years ago: 
https://gitlab.com/rockdaboot/librusage.
It's simple to build and use and gives some number of which (external) 
commands are executed - fork+exec are pretty heavy.
[Configure for wget2 runs 'rm' and 'cat' each roughly 2000x - so I came 
up with enabling plugins for those two commands (had to write a plugin 
for 'rm', not sure if it never has been accepted by bash upstream).]
Maybe be we can create plugins for other highly used commands as well 
e.g. for sed !?


The output of the tool also roughly allows to see where the time goes - 
it's beyond my spare time to go deeper into this right now.

Please test yourself and share some numbers.

Another option is to group tests, e.g. if test 1 is X, we also know the 
results for tests 2,3,4,... Or we group several tests into a single C 
file, if possible. Just an idea (sounds a bit tedious, though).


Parallelism... can't we do that with &, at least for well-known / 
often-used tests ?


Family calls...

Regards, Tim


OpenPGP_signature
Description: OpenPGP digital signature

Re: Parallelization of shell scripts for 'configure' etc.

[L A Walsh]

On 2022/06/13 15:39, Paul Eggert wrote:
In many Gnu projects, the 'configure' script is the biggest barrier to 
building because it takes s long to run. Is there some way that we 
could improve its performance without completely reengineering it, by 
improving Bash so that it can parallelize 'configure' scripts?
  


I don't know what type of instrumentations you've done over configure,
but before investing a much time in optimization, it might be interesting
to know where most of the time is being spent.

I.e cpu, I/O -- what types of I/O -- actual test-I/O or executable load 
time.


The reason I say that is that having run configure for the same projects
on linux and on cygwin -- I note that the cygwin version is MUCH slower
doing the same work on the same machine. 


A big slowdown in cygwin is loading & starting a new executable/binary.
I.e loading 100 programs 10x each will take a disproportionately higher time
on cygwin due to its exec-load penalty (Since windows has no fork, all 
of the

memory space duplication (and later copies on write) has to be done manually
in cygwin -- very painful.  But noting that one of the big boosts in
shell scripts can come from using a parallel option of a util vs. feeding in
file/pathnames one at a time like using 'find -exec "rm {}" \; or similar.

Similarly a big speed up in configure might be to use the bundled version of
coreutils (all binaries in 1 image invoked via different command names), and
put that in the same binary as bash, perhaps via a loadable command, with
any following core-util calls being routed "in-binary" to the already loaded
version.  Of course it would likely not be trivial assuring all the
commands can be re-invoked to assure they had their necessary 
initializations

redone on each "in-image" launch, but keeping all the coreutil binaries
"in-memory", I think would be a big win even if it wasn't multi-threaded. 

Of course it might be of benefit if the various utils were all thread 
safe, so

a more powerful dispatcher could use multi-threading w/o worries about
thread safety, but just eliminating most of the redundant util-loads 
might be
a huge win by itself.  That's sorta why I was wondering how much 
perf-profiling

had been done on config(.sh)...

Anyway -- just some random thoughts...


For ideas about this, please see PaSh-JIT:

Kallas K, Mustafa T, Bielak J, Karnikis D, Dang THY, Greenberg M, 
Vasilakis N. Practically correct, just-in-time shell script 
parallelization. Proc OSDI 22. July 2022. 
https://nikos.vasilak.is/p/pash:osdi:2022.pdf


I've wanted something like this for *years* (I assigned a simpler 
version to my undergraduates but of course it was too much to expect 
them to implement it) and I hope some sort of parallelization like this 
can get into production with Bash at some point (or some other shell if 
Bash can't use this idea).


  

Re: Parallelization of shell scripts for 'configure' etc.

[Bruno Haible]

Paul Eggert wrote:
> It appears that PaSh itself isn't suitable for prime-time use.

That's my current conclusion too: I tried running PaSh on
  - the configure script of a coreutils release,
  - gnulib-tool (a large shell script),
and got a parse error in both cases [1].

Bruno

[1] https://github.com/binpash/pash/issues/573

Re: Parallelization of shell scripts for 'configure' etc.

[Jeffrey Walton]

On Tue, Jun 14, 2022 at 11:15 PM Ángel  wrote:
>
> On 2022-06-13 at 18:32 -0700, Paul Eggert wrote:
> > Yes, all that could be done in theory, but it'd take a lot of
> > hacking and it's been decades and it hasn't happened.
> >
> > I'd rather have shell scripts "just work" in parallel with a minimum
> > of fuss.
>
> If this hasn't happened, that might be because nobody found it
> important enough to dive into such task, relatively to the effort
> required (or estimated).
>
> Everyone wishes their configure scripts to finish quickly but, first
> and foremost, the result must be right. Something which is easier with
> serial script than with a parallel implementation.
>
> And in the case of configure, they are a mix of boilerplate code, m4
> and shell scripting, so not many people would be, in addition to
> motivated by that task, proficient enough in all of those. Furthermore,
> configure typically target really classic dialects, which makes even
> more difficult to use modern features which could speed up the script.
>
> Optimizing that at bash would likely be even more complex, since it
> could only automatically optimize the cases that would never have side
> effects.
>
> I think that most gainings should probably come from autoconf blocks
> with optimizations. Still, it might be possible through some loadable
> builtin to improve the configure times.
>
> Do you have any handy example of configure that takes too long to run?

Try Emacs.

Jeff

Re: Parallelization of shell scripts for 'configure' etc.

[Paul Eggert]

On 6/14/22 18:55, Ángel wrote:

Do you have any handy example of configure that takes too long to run?


Sure. Coreutils, emacs. Pretty much any nontrivial configure script 
takes longer than it should.


I understand that parallelization of shell scripts is nontrivial.

Re: Parallelization of shell scripts for 'configure' etc.

[Ángel]

On 2022-06-13 at 18:32 -0700, Paul Eggert wrote:
> Yes, all that could be done in theory, but it'd take a lot of
> hacking and it's been decades and it hasn't happened.
> 
> I'd rather have shell scripts "just work" in parallel with a minimum
> of fuss.

If this hasn't happened, that might be because nobody found it
important enough to dive into such task, relatively to the effort
required (or estimated).

Everyone wishes their configure scripts to finish quickly but, first
and foremost, the result must be right. Something which is easier with
serial script than with a parallel implementation.

And in the case of configure, they are a mix of boilerplate code, m4
and shell scripting, so not many people would be, in addition to
motivated by that task, proficient enough in all of those. Furthermore,
configure typically target really classic dialects, which makes even
more difficult to use modern features which could speed up the script.

Optimizing that at bash would likely be even more complex, since it
could only automatically optimize the cases that would never have side
effects.

I think that most gainings should probably come from autoconf blocks
with optimizations. Still, it might be possible through some loadable
builtin to improve the configure times.

Do you have any handy example of configure that takes too long to run?

Re: Parallelization of shell scripts for 'configure' etc.

[Paul Eggert]

On 6/14/22 10:11, Nick Bowler wrote:


The resulting config.h is correct but pa.sh took almost 1 minute to run
the configure script, about ten times longer than dash takes to run the
same script.  More than half of that time appears to be spent just
loading the program into pa.sh, before a single shell command is
actually executed.


Ouch. Thanks for looking into this, and sorry it took so much of your 
time. It appears that PaSh itself isn't suitable for prime-time use. 
This isn't too surprising, though, as it's a research project and I 
wouldn't expect it to compete with production shells. The main question 
here is whether the ideas of PaSh are good ones for improving Bash.

Re: Parallelization of shell scripts for 'configure' etc.

[Chet Ramey]

On 6/13/22 6:39 PM, Paul Eggert wrote:
> In many Gnu projects, the 'configure' script is the biggest barrier to
> building because it takes s long to run. Is there some way that we
> could improve its performance without completely reengineering it, by
> improving Bash so that it can parallelize 'configure' scripts?

Previous iterations of this research rewrote existing shell scripts based
on a database of annotations about individual commands, without any changes
to the shell itself. Unless that's changed, you can experiment with Pash
and Pash-JIT immediately.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Parallelization of shell scripts for 'configure' etc.

[Steffen Nurpmeso]

Chet Ramey wrote in
 <211b74c0-caed-227f-ffae-b85868ef7...@case.edu>:
 |On 6/13/22 6:39 PM, Paul Eggert wrote:
 |> In many Gnu projects, the 'configure' script is the biggest barrier to
 |> building because it takes s long to run. Is there some way that we
 |> could improve its performance without completely reengineering it, by
 |> improving Bash so that it can parallelize 'configure' scripts?
 |
 |Previous iterations of this research rewrote existing shell scripts based
 |on a database of annotations about individual commands, without any changes
 |to the shell itself. Unless that's changed, you can experiment with Pash
 |and Pash-JIT immediately.

I wondered how they handle (file-descriptor) resource exhaustion.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Parallelization of shell scripts for 'configure' etc.

[Alex Ameen]

You can try to use the `requires` toposort routine to identify "Strongly
Connected Sub-Components", which is where I imagine you'll get the
best results. What you'll need to watch out for is undeclared ordering
requirements that parallelism would break.

The `m4sh` and `m4sugar` source code is documented in a lot of detail. The
manuals exclude that type of documentation because it's internal; but you
could keep yourself occupied for at least a month or two before you ran out
of topics to explore.

On Mon, Jun 13, 2022, 8:45 PM Dale R. Worley  wrote:

> Paul Eggert  writes:
> > In many Gnu projects, the 'configure' script is the biggest barrier to
> > building because it takes s long to run. Is there some way that we
> > could improve its performance without completely reengineering it, by
> > improving Bash so that it can parallelize 'configure' scripts?
>
> It seems to me that bash provides the needed tools -- "( ... ) &" lets
> you run things in parallel.  Similarly, if you've got a lot of small
> tasks with a complex web of dependencies, you can encode that in a
> "makefile".
>
> It seems to me that the heavy work is rebuilding how "configure" scripts
> are constructed based on which items can be run in parallel.  I've never
> seen any "metadocumentation" that laid out how all that worked.
>
> Dale
>
>

Re: Parallelization of shell scripts for 'configure' etc.

[Alex Ameen]

Yeah honestly splitting most of the `configure` checks into multiple
threads is definitely possible.

Caching between projects is even a straightforward extension with systems
like `Nix`.

The "gotcha" here in both cases is that existing scripts that are living in
source tarballs are not feasible to "regenerate" in the general case. You
could have this ship out with future projects though if project authors
updated to new versions of Autoconf.


If you have a particularly slow package, you can optimize it in a few
hours. Largely this means "identify which tests 100% match the standard
implementation of a check" in which case you can fill in a cached value.
But what I think y'all are asking about is "can I safely use a cache from
one project in another project?" and the answer there is "no not really -
and please don't because it will be a nightmare to debug".

The nasty part about trying to naively share caches is that it will
probably work fine ~90% of the time. The problem is that the 10% that
misbehave are high risk for undefined behavior. My concern is the 0.5% that
appear to work fine, but "whoops we didn't know project X extended a macro
without changing the name - and now an ABI conflict in `gpgp` appears on
the third Sunday of every October causing it skip encryption silently" or
some absurd edge case.


I think optimizating "freshly generated" scripts is totally doable though.

On Mon, Jun 13, 2022, 5:40 PM Paul Eggert  wrote:

> In many Gnu projects, the 'configure' script is the biggest barrier to
> building because it takes s long to run. Is there some way that we
> could improve its performance without completely reengineering it, by
> improving Bash so that it can parallelize 'configure' scripts?
>
> For ideas about this, please see PaSh-JIT:
>
> Kallas K, Mustafa T, Bielak J, Karnikis D, Dang THY, Greenberg M,
> Vasilakis N. Practically correct, just-in-time shell script
> parallelization. Proc OSDI 22. July 2022.
> https://nikos.vasilak.is/p/pash:osdi:2022.pdf
>
> I've wanted something like this for *years* (I assigned a simpler
> version to my undergraduates but of course it was too much to expect
> them to implement it) and I hope some sort of parallelization like this
> can get into production with Bash at some point (or some other shell if
> Bash can't use this idea).
>
>

Re: Parallelization of shell scripts for 'configure' etc.

[Dale R. Worley]

Paul Eggert  writes:
> In many Gnu projects, the 'configure' script is the biggest barrier to 
> building because it takes s long to run. Is there some way that we 
> could improve its performance without completely reengineering it, by 
> improving Bash so that it can parallelize 'configure' scripts?

It seems to me that bash provides the needed tools -- "( ... ) &" lets
you run things in parallel.  Similarly, if you've got a lot of small
tasks with a complex web of dependencies, you can encode that in a
"makefile".

It seems to me that the heavy work is rebuilding how "configure" scripts
are constructed based on which items can be run in parallel.  I've never
seen any "metadocumentation" that laid out how all that worked.

Dale

Re: Parallelization of shell scripts for 'configure' etc.

[Paul Eggert]

On 6/13/22 18:25, Dale R. Worley wrote:

It seems to me that bash provides the needed tools -- "( ... ) &" lets
you run things in parallel.  Similarly, if you've got a lot of small
tasks with a complex web of dependencies, you can encode that in a
"makefile".

It seems to me that the heavy work is rebuilding how "configure" scripts
are constructed based on which items can be run in parallel.


Yes, all that could be done in theory, but it'd take a lot of hacking 
and it's been decades and it hasn't happened.


I'd rather have shell scripts "just work" in parallel with a minimum of 
fuss.

Parallelization of shell scripts for 'configure' etc.

[Paul Eggert]

In many Gnu projects, the 'configure' script is the biggest barrier to 
building because it takes s long to run. Is there some way that we 
could improve its performance without completely reengineering it, by 
improving Bash so that it can parallelize 'configure' scripts?


For ideas about this, please see PaSh-JIT:

Kallas K, Mustafa T, Bielak J, Karnikis D, Dang THY, Greenberg M, 
Vasilakis N. Practically correct, just-in-time shell script 
parallelization. Proc OSDI 22. July 2022. 
https://nikos.vasilak.is/p/pash:osdi:2022.pdf


I've wanted something like this for *years* (I assigned a simpler 
version to my undergraduates but of course it was too much to expect 
them to implement it) and I hope some sort of parallelization like this 
can get into production with Bash at some point (or some other shell if 
Bash can't use this idea).