Re: gcc parallel make check
On Wed, 3 Sep 2014, VandeVondele Joost wrote: I've noticed that make -j -k check-fortran results in a serialized checking, while make -j32 -k check-fortran goes parallel. Somehow the explicit 'N' in -jN seems to be needed for the check target, while the other targets seem to do just fine. Is that a feature, or should I file a PR for that... ? https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53155 -- Marc Glisse
Re: gcc parallel make check
On Wed, Sep 03, 2014 at 09:15:51AM +, VandeVondele Joost wrote: > I've noticed that > > make -j -k check-fortran > > results in a serialized checking, while > > make -j32 -k check-fortran > > goes parallel. Somehow the explicit 'N' in -jN seems to be needed for the > check target, while the other targets seem to do just fine. Is that a > feature, or should I file a PR for that... ? It is intentional. With -j it is essentially a fork bomb, just don't use it. > Somewhat related is there a rule of thumb on how is the granularity of > parallel check decided ? E.g. check-fortran seems to be limited to about > ~5 parallel targets, which is few for a typical server (but of course a > welcome speedup already). The splitting has some cost (e.g. lots of various checks are cached, with split jobs they need to be done in each separate goal), and the goal of the split is toplevel make check parallelization, not individual directory or language testing. For the latter perhaps more fine grained split could be useful, but how would one find out if it is a toplevel make check, or say make -C gcc check where you test many languages, or check-gfortran? Jakub
RE: gcc parallel make check
> It is intentional. With -j it is essentially a fork bomb, just don't use it. well, silently ignoring it for just this target did cost me a lot of time, while an eventual fork bomb would have been dealt with much more quickly. >> Somewhat related is there a rule of thumb on how is the granularity of >> parallel check decided ? E.g. check-fortran seems to be limited to about >> ~5 parallel targets, which is few for a typical server (but of course a >> welcome speedup already). > >The splitting has some cost (e.g. lots of various checks are cached, with >split jobs they need to be done in each separate goal), and the goal of the >split is toplevel make check parallelization, not individual directory or >language testing. For the latter perhaps more fine grained split could be >useful, but how would one find out if it is a toplevel make check, or say >make -C gcc check where you test many languages, or check-gfortran? the cost must be small compared to the possible gain... on a 32 core server, testing of fortran FE changes would be 4x larger. I notice that even on a full check, the Fortran tests are still running when the number of processes is already way below 32. However, the longest running (by a few minutes) are those: expect -- /usr/share/dejagnu/runtest.exp --tool gcc lto.exp weak.exp tls.exp ipa.exp tree-ssa.exp debug.exp dwarf2.exp fixed-point.exp vxworks.exp cilk-plus.exp vmx.exp pch.exp simulate-thread.exp x86_64-costmodel-vect.exp i386-costmodel-vect.exp spu-costmodel-vect.exp ppc-costmodel-vect.exp charset.exp noncompile.exp tsan.exp graphite.exp compat.exp expect -- /usr/share/dejagnu/runtest.exp --tool g++ lto.exp tls.exp gcov.exp debug.exp dwarf2.exp cilk-plus.exp pch.exp bprob.exp simulate-thread.exp vect.exp charset.exp tsan.exp graphite.exp compat.exp struct-layout-1.exp ubsan.exp tm.exp gomp.exp dfp.exp tree-prof.exp stackalign.exp plugin.exp guality.exp asan.exp ecos.exp so can those be run more independently ?
RE: gcc parallel make check
> What did you expect for -j alone? an error? No, as is standard in gnu make, a new process for any target that can be processed (i.e. unlimited). >> ... check-fortran seems to be limited to about ~5 parallel targets ... > >Running the make with -j8 gives 7 directories gfortran[1-6]? in gcc/testsuite/. >Note that the load balancing could be improved: few minutes with a single >thread >over ~20 minutes. I'd like to have roughly 32 directories (or as many of the -jN allows for).
Re: gcc parallel make check
On Wed, Sep 03, 2014 at 09:37:19AM +, VandeVondele Joost wrote: > > It is intentional. With -j it is essentially a fork bomb, just don't use > > it. > > well, silently ignoring it for just this target did cost me a lot of time, > while an eventual fork bomb would have been dealt with much more quickly. > > >> Somewhat related is there a rule of thumb on how is the granularity of > >> parallel check decided ? E.g. check-fortran seems to be limited to about > >> ~5 parallel targets, which is few for a typical server (but of course a > >> welcome speedup already). > > > >The splitting has some cost (e.g. lots of various checks are cached, with > >split jobs they need to be done in each separate goal), and the goal of the > >split is toplevel make check parallelization, not individual directory or > >language testing. For the latter perhaps more fine grained split could be > >useful, but how would one find out if it is a toplevel make check, or say > >make -C gcc check where you test many languages, or check-gfortran? > > the cost must be small compared to the possible gain... on a 32 core > server, testing of fortran FE changes would be 4x larger. I notice that It depends. For make -j2 if you split check-gfortran alone into 32 pieces, check-gcc into 32 pieces, check-g++ into 32 pieces, libstdc++ into 32 pieces etc., it might be too much. The problem with too fine-grained split beyond some cost to start the testing in the goal, and running various cached tests, is also that once you want to split a single *.exp job into smaller parts, you need to use wildcards, but then it is a maintainance problem, you don't want to test anything more than once, or not at all, even if new tests with weirdo names are added later. > even on a full check, the Fortran tests are still running when the number > of processes is already way below 32. However, the longest running (by a > few minutes) are those: > > expect -- /usr/share/dejagnu/runtest.exp --tool gcc lto.exp weak.exp tls.exp > ipa.exp tree-ssa.exp debug.exp dwarf2.exp fixed-point.exp vxworks.exp > cilk-plus.exp vmx.exp pch.exp simulate-thread.exp x86_64-costmodel-vect.exp > i386-costmodel-vect.exp spu-costmodel-vect.exp ppc-costmodel-vect.exp > charset.exp noncompile.exp tsan.exp graphite.exp compat.exp > expect -- /usr/share/dejagnu/runtest.exp --tool g++ lto.exp tls.exp gcov.exp > debug.exp dwarf2.exp cilk-plus.exp pch.exp bprob.exp simulate-thread.exp > vect.exp charset.exp tsan.exp graphite.exp compat.exp struct-layout-1.exp > ubsan.exp tm.exp gomp.exp dfp.exp tree-prof.exp stackalign.exp plugin.exp > guality.exp asan.exp ecos.exp > > so can those be run more independently ? It is a moving target, new tests are added every day. I'm trying to adjust it during stage3/stage4 occassionally, but it also very much depends on which target it is (e.g. i?86/x86_64 has many more tests in i386.exp then other targets in their gcc.target), how fast the compiler is on the target (e.g. on some targets -g is much slower than on others, etc.). Jakub
Re: gcc parallel make check
Hi Joost, VandeVondele Joost wrote: > I've noticed that > make -j -k check-fortran > results in a serialized checking, while > make -j32 -k check-fortran > goes parallel. I have to admit that I don't know why that's the case. However, I can answer the next question, is presumably related to this one: > Somewhat related is there a rule of thumb on how is the granularitys > of parallel check decided ? DejaGNU is not able to run checks in parallel - thus, we have only makefile parallelization (check-gcc, check-gfortran). As that wasn't suifficient, Jakub (?) split the single tests into multiple ones, trying to do ensure that those subtargets all take about the same time. See: gcc/fortran/Make-lang.in, which has: # For description see comment above check_gcc_parallelize in gcc/Makefile.in. check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \ dg.exp=gfortran.dg/\[bcBC\]* \ dg.exp=gfortran.dg/\[nopNOP\]* \ dg.exp=gfortran.dg/\[isuvISUV\]* \ dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \ dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]* Thus, you currently get 6 parallel check-gfortran checks - followed by one which tries to combine the results. I think Diego has some means to run GCC's in a vastly parallel way, which break due to a test-framework issue / gfortran.dg-dependency issues. See PR56408. Thus, you could asks him how he does it. Additionally, I wouldn't mind if some lispy person could look at the PR - my attempts failed, but, admittedly, I didn't spend much time on it. Tobias PS: There was/is the reoccuring thought of replacing DejaGNU by a different framework or to enhance it, but not much substantial work has happened, despite some occasional effort. At least DejaGNU is now back under maintaince, cf. http://git.savannah.gnu.org/gitweb/?p=dejagnu.git;a=summary
RE: gcc parallel make check
> I have to admit that I don't know why that's the case. Actually Marc answered that one (I had the wrong mail address for gcc@ so repeat here): https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53155 > See: gcc/fortran/Make-lang.in, which has: I'll have a look and do some testing what the gains/costs of a further split are. Joost
RE: gcc parallel make check
>> expect -- /usr/share/dejagnu/runtest.exp --tool gcc lto.exp weak.exp tls.exp >> ipa.exp tree-ssa.exp debug.exp >dwarf2.exp fixed-point.exp vxworks.exp >> cilk-plus.exp vmx.exp pch.exp simulate-thread.exp x86_64-costmodel-vect.exp >> i386-costmodel-vect.exp spu-costmodel-vect.exp ppc-costmodel-vect.exp >> charset.exp noncompile.exp tsan.exp graphite.exp compat.exp >> expect -- /usr/share/dejagnu/runtest.exp --tool g++ lto.exp tls.exp gcov.exp >> debug.exp dwarf2.exp cilk-plus.exp pch.exp bprob.exp simulate-thread.exp >> vect.exp charset.exp tsan.exp graphite.exp compat.exp struct-layout-1.exp >> ubsan.exp tm.exp gomp.exp dfp.exp tree-prof.exp stackalign.exp plugin.exp >> guality.exp asan.exp ecos.exp >> >> so can those be run more independently ? >It is a moving target, new tests are added every day. I'm trying to adjust >it during stage3/stage4 occassionally, but it also very much depends on >which target it is (e.g. i?86/x86_64 has many more tests in i386.exp then >other targets in their gcc.target), how fast the compiler is on the target >(e.g. on some targets -g is much slower than on others, etc.). could you point me to the right file (or example commit) for trying to adjust this ? I can try to do some testing and come back with some numbers.
Re: gcc parallel make check
On Wed, Sep 03, 2014 at 10:35:41AM +, VandeVondele Joost wrote: > >> expect -- /usr/share/dejagnu/runtest.exp --tool gcc lto.exp weak.exp > >> tls.exp ipa.exp tree-ssa.exp debug.exp >dwarf2.exp fixed-point.exp > >> vxworks.exp cilk-plus.exp vmx.exp pch.exp simulate-thread.exp > >> x86_64-costmodel-vect.exp i386-costmodel-vect.exp spu-costmodel-vect.exp > >> ppc-costmodel-vect.exp charset.exp noncompile.exp tsan.exp graphite.exp > >> compat.exp > >> expect -- /usr/share/dejagnu/runtest.exp --tool g++ lto.exp tls.exp > >> gcov.exp debug.exp dwarf2.exp cilk-plus.exp pch.exp bprob.exp > >> simulate-thread.exp vect.exp charset.exp tsan.exp graphite.exp compat.exp > >> struct-layout-1.exp ubsan.exp tm.exp gomp.exp dfp.exp tree-prof.exp > >> stackalign.exp plugin.exp guality.exp asan.exp ecos.exp > >> > >> so can those be run more independently ? > > >It is a moving target, new tests are added every day. I'm trying to adjust > >it during stage3/stage4 occassionally, but it also very much depends on > >which target it is (e.g. i?86/x86_64 has many more tests in i386.exp then > >other targets in their gcc.target), how fast the compiler is on the target > >(e.g. on some targets -g is much slower than on others, etc.). > > could you point me to the right file (or example commit) for trying to adjust > this ? I can try to do some testing and come back with some numbers. The splits are in the Makefiles, see check_gcc_parallelize var in gcc/Makefile.in (where there is a big comment with documentation), check_g++_parallelize var in gcc/cp/Make-lang.in, check_gfortran_parallelize var in gcc/fortran/Make-lang.in, or check-DEJAGNU goal in libstdc++-v3/testsuite/Makefile.am. Jakub
[PATCH] RE: gcc parallel make check
> The splits are in the Makefiles, see check_gcc_parallelize
attached is a patch to improve the parallel performance of 'make -jXX -k
check-fortran'. For XX=16, this yields ~50% speedup, and even with XX=4 we
still have 15%, the measured slowdown at XX=1 (<2%) is in the noise of testing.
The patch is a simple update of the 'check_gfortran_parallelize' variable,
updating it from its 2008 values to a set that I found +- optimal based on
several tests. Detailed timings are :
# timings/trunk-check-fortran
#cores averagestd. dev. #tests
1 2955.3275.06 3
2 1735.30 122.26 3
4 929.5154.19 3
8 470.29 7.85 3
16 468.09 4.29 3
32 466.06 1.24 3
# timings/patched-check-fortran
#cores averagestd. dev. #tests
1 3008.8916.38 3
2 1534.17 118.33 3
4 800.1831.71 3
8 418.71 0.20 2
16 298.29 5.86 3
32 299.84 1.34 3
There is no effect on a full 'make -j32 -k check' as other goals run for much
longer (to be looked at in a followup).
A second part of the patch is a new file 'contrib/generate_tcl_patterns.sh'
which generates the needed regexp to do the split based on an input of the
files in the target directory. It basically groups the initial characters such
that each regexp tries not to exceed a maximum number of files. So, the number
of files is used as a proxy for the runtime. While I don't feel to strong about
adding this (shell/gawk) script, it certainly is convenient, and makes sure
that no characters are missing from the regexp. The maximum number of files per
regexp is an input, testing (-j16) with 200, 300, 400 I found that 300 was
optimal for testsuite/gfortran.dg, but this will depend on many things.
A sample run would look like
gcc/gcc/testsuite/gfortran.dg> ls -1 |
../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
Adding label: p matching files:499
Adding label: c matching files:497
Adding label: a matching files:448
Adding label: i matching files:350
Adding label: d matching files:245
Adding label: s matching files:211
Adding label: b matching files:206
Adding label: t matching files:180
Adding label: f matching files:173
Adding label: e matching files:166
Adding label: r matching files:165
Adding label: n matching files:162
Adding label: mu matching files:278
Adding label: wlgo matching files:284
Adding label: vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj matching files:94
patterns:
dg.exp=gfortran.dg/p* \
dg.exp=gfortran.dg/c* \
dg.exp=gfortran.dg/a* \
dg.exp=gfortran.dg/i* \
dg.exp=gfortran.dg/\[wlgo\]* \
dg.exp=gfortran.dg/\[mu\]* \
dg.exp=gfortran.dg/d* \
dg.exp=gfortran.dg/s* \
dg.exp=gfortran.dg/b* \
dg.exp=gfortran.dg/t* \
dg.exp=gfortran.dg/f* \
dg.exp=gfortran.dg/e* \
dg.exp=gfortran.dg/r* \
dg.exp=gfortran.dg/n* \
dg.exp=gfortran.dg/\[vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj\]* \
Is the current attached patch OK for trunk ?
contrib/ChangeLog
2014-09-05 Joost VandeVondele
* generate_tcl_patterns.sh: New file.
gcc/fortran/ChangeLog
2014-09-05 Joost VandeVondele
* Make-lang.in (check_gfortran_parallelize): improved parallelism.
Index: contrib/generate_tcl_patterns.sh
===
--- contrib/generate_tcl_patterns.sh (revision 0)
+++ contrib/generate_tcl_patterns.sh (revision 0)
@@ -0,0 +1,86 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input,
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+# cd gcc/gcc/testsuite/gfortran.dg
+# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING. If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+
Re: [PATCH] RE: gcc parallel make check
On Fri, Sep 05, 2014 at 02:26:45PM +, VandeVondele Joost wrote: > Adding label: mu matching files:278 > Adding label: wlgo matching files:284 > Adding label: vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj matching > files:94 > patterns: > dg.exp=gfortran.dg/p* \ > dg.exp=gfortran.dg/c* \ > dg.exp=gfortran.dg/a* \ > dg.exp=gfortran.dg/i* \ > dg.exp=gfortran.dg/\[wlgo\]* \ > dg.exp=gfortran.dg/\[mu\]* \ > dg.exp=gfortran.dg/d* \ > dg.exp=gfortran.dg/s* \ > dg.exp=gfortran.dg/b* \ > dg.exp=gfortran.dg/t* \ > dg.exp=gfortran.dg/f* \ > dg.exp=gfortran.dg/e* \ > dg.exp=gfortran.dg/r* \ > dg.exp=gfortran.dg/n* \ > dg.exp=gfortran.dg/\[vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj\]* \ > > Is the current attached patch OK for trunk ? > gcc/fortran/ChangeLog > > 2014-09-05 Joost VandeVondele > >* Make-lang.in (check_gfortran_parallelize): improved parallelism. Capital I. > --- gcc/fortran/Make-lang.in (revision 214949) > +++ gcc/fortran/Make-lang.in (working copy) > @@ -168,12 +168,22 @@ check-fortran-subtargets : check-gfortra > lang_checks += check-gfortran > lang_checks_parallelized += check-gfortran > # For description see comment above check_gcc_parallelize in gcc/Makefile.in. > -check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \ > - dg.exp=gfortran.dg/\[bcBC\]* \ > - dg.exp=gfortran.dg/\[nopNOP\]* \ > - dg.exp=gfortran.dg/\[isuvISUV\]* \ > - dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \ > - dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]* > +check_gfortran_parallelize = execute.exp \ > + dg.exp=gfortran.dg/p* \ > + dg.exp=gfortran.dg/c* \ > + dg.exp=gfortran.dg/a* \ > + dg.exp=gfortran.dg/i* \ > + dg.exp=gfortran.dg/\[wlgo\]* \ > + dg.exp=gfortran.dg/\[mu\]* \ > + dg.exp=gfortran.dg/d* \ > + dg.exp=gfortran.dg/s* \ > + dg.exp=gfortran.dg/b* \ > + dg.exp=gfortran.dg/t* \ > + dg.exp=gfortran.dg/f* \ > + dg.exp=gfortran.dg/e* \ > + dg.exp=gfortran.dg/r* \ > + dg.exp=gfortran.dg/n* \ > + > dg.exp=gfortran.dg/\[vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj\]* Please sort the letters (LC_ALL=C sort) and where consecutive, use ranges. Thus \[0-9A-Zhjqvx-z\]* or so. What is - doing in your list? _-9 is quite unexpected range. Also, please tweak the script so that it will do the sorting and ranges handling for you. Jakub
Re: [PATCH] RE: gcc parallel make check
On Fri, Sep 05, 2014 at 02:48:01PM +, VandeVondele Joost wrote: > > > Please sort the letters (LC_ALL=C sort) and where consecutive, use ranges. > > Thus \[0-9A-Zhjqvx-z\]* or so. What is - doing in your list? _-9 is quite > > unexpected range. > > the '-' is a bug indeed. I added this to the list of allowed starting > characters, but this clearly this is not a good idea. Thanks, removed. > > > Also, please tweak the script so that it will do the > > sorting and ranges handling for you. > > The current order is the number of files that start with this letter, which > is somewhat informative, but let me see if I can have ranges easily (in > awk...). BTW, number of tests might help as initial guess on how to split things, but might be severely off. What I'd recommend after you split with your script that you perform a make -j1 check and diff the Test run by and runtest completed times in each of the */*.log.sep files and see what time each job takes, then perhaps reshuffle stuff a little bit until it is roughly even. Some letters could start more dg-do run tests while others more dg-do compile once, some tests take a short time to compile, some very long, etc. Jakub
RE: [PATCH] RE: gcc parallel make check
>> > Please sort the letters (LC_ALL=C sort) and where consecutive, use ranges.
>> > Thus \[0-9A-Zhjqvx-z\]*
OK, works fine with the attached patch, and looks cleaner in Make-lang.in.
Now, with the proper email address for gcc-patches... I wonder how many time
I'll be punished for typos.
unmodified CL.
Joost
Index: contrib/generate_tcl_patterns.sh
===
--- contrib/generate_tcl_patterns.sh (revision 0)
+++ contrib/generate_tcl_patterns.sh (revision 0)
@@ -0,0 +1,108 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input,
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+# cd gcc/gcc/testsuite/gfortran.dg
+# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING. If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+ # list of allowed starting chars for a file name in a dir to split
+ achars="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+ ranget="112233"
+}
+{
+ nfiles++ ; files[nfiles]=$1
+}
+END{
+ for(i=1; i<=length(achars); i++) count[substr(achars,i,1)]=0
+ for(i=1; i<=nfiles; i++) {
+ if (length(files[i]>0)) { count[substr(files[i],1,1)]++ }
+ };
+ asort(count,ordered)
+ countsingle=0
+ groups=0
+ label=""
+ for(i=length(achars);i>=1;i--) {
+countsingle=countsingle+ordered[i]
+for(j=1;j<=length(achars);j++) {
+ if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+}
+count[found]=-1
+label=label found
+if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+if(countsingle+val>maxcount) {
+ subset[label]=countsingle
+ print "Adding label: ", label, "matching files:" countsingle
+ groups++
+ countsingle=0
+ label=""
+}
+ }
+ print "patterns:"
+ asort(subset,ordered)
+ for(i=groups;i>=1;i--) {
+for(j in subset){
+ if(subset[j]==ordered[i]) found=j
+}
+subset[found]=-1
+if (length(found)==1) {
+ printf("%s%s* \\\n",prefix,found)
+} else {
+ sortandcompress()
+ printf("%s\\[%s\\]* \\\n",prefix,found)
+}
+ }
+}
+function sortandcompress(i,n,tmp,bestj)
+{
+ n=length(found)
+ for(i=1; i<=n; i++) tmp[i]=substr(found,i,1)
+ asort(tmp)
+ for(i=1;i<=n;i++){
+ipos=index(achars,tmp[i])
+for(j=i;j<=n;j++){
+ jpos=index(achars,tmp[j])
+ if (jpos-ipos==j-i && substr(ranget,ipos,1)==substr(ranget,jpos,1)) bestj=j
+}
+if (bestj-i>3) {
+ tmp[i+1]="-"
+ for(j=i+2;j
RE: [PATCH] RE: gcc parallel make check
Attached is an extended version of the patch, it brings a 100% improvement in
make -j32 -k check-gcc (down from 20min to <10min) by modification of
check_gcc_parallelize.
It includes one non-trivial part, namely a split of the target exps. They are
now all split using a common choice (based on i386), which I believe is
reasonable as it is the target with most tests, and the patterns will be
somewhat similar for other targets (e.g. split of p(rxxx)). The implementation
of this in the makefile uses an odd looking technique to substitute spaces with
commas in a variable, if this can be done more elegantly, I'm happy to make the
change.
Bootstrap and testing revealed one issue, i386.exp hard-codes a loop for the
testcase 'vect-args.c' in order to test 10 different combinations of options.
With the current split (i.e. target x4) this test will thus be executed 4
times. There are two easy options
1) keep the current setup, overhead is small
2) keep the .exp file simple and just replicate this test 10x
I've selected 1), but I can update a patch with 2). Ideally dg-options in the
testcase file itself could be repeated, but I haven't found an example of this.
The script now includes sorting and compression of the ranges, and an
additional sanity check on the input, i.e. that file names start with
[0-9A-Za-z]. Some (few) files seem to start with _ or # (in ./gcc.dg/cpp/).
I'll follow up with a separate patch to improve check_g++_parallelize.
Full 'make -j k32 check' is now dominated by libstdc++ testing, which contains
single goals that run ~1100s (e.g. regex related tests). These uses a slightly
different syntax (see gcc/libstdc++-v3/testsuite/Makefile.am) and I'm not yet
sure how to deal with the .am files.
current patch OK for trunk ?
Joost
patch-speedup-checkfortran-v05.CL
Description: patch-speedup-checkfortran-v05.CL
Index: contrib/generate_tcl_patterns.sh
===
--- contrib/generate_tcl_patterns.sh (revision 0)
+++ contrib/generate_tcl_patterns.sh (revision 0)
@@ -0,0 +1,114 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input, starting with [0-9A-Za-z],
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+# cd gcc/gcc/testsuite/gfortran.dg
+# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING. If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+ # list of allowed starting chars for a file name in a dir to split
+ achars="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+ ranget="112233"
+}
+{
+ if (index(achars,substr($1,1,1))==0){
+ print "file : " $1 " does not start with an allowed character."
+ _assert_exit = 1
+ exit 1
+ }
+ nfiles++ ; files[nfiles]=$1
+}
+END{
+ if (_assert_exit) exit 1
+ for(i=1; i<=length(achars); i++) count[substr(achars,i,1)]=0
+ for(i=1; i<=nfiles; i++) {
+ if (length(files[i]>0)) { count[substr(files[i],1,1)]++ }
+ };
+ asort(count,ordered)
+ countsingle=0
+ groups=0
+ label=""
+ for(i=length(achars);i>=1;i--) {
+countsingle=countsingle+ordered[i]
+for(j=1;j<=length(achars);j++) {
+ if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+}
+count[found]=-1
+label=label found
+if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+if(countsingle+val>maxcount) {
+ subset[label]=countsingle
+ print "Adding label: ", label, "matching files:" countsingle
+ groups++
+ countsingle=0
+ label=""
+}
+ }
+ print "patterns:"
+ asort(subset,ordered)
+ for(i=groups;i>=1;i--) {
+for(j in subset){
+ if(subset[j]==ordered[i]) found=j
+}
+subset[found]=-1
+if (length(found)==1) {
+ printf("%s%s* \\\n",prefix,found)
+} else {
+ sortandcompress()
+ pri
Re: [PATCH] RE: gcc parallel make check
On 09/09/2014 10:51 AM, VandeVondele Joost wrote: > Attached is an extended version of the patch, > it brings a 100% improvement in make -j32 -k check-gcc First of all, many thanks for working on this. +# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/" How does this work with subdirectories? Can we replace ls with find? -check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \ + 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 $(shell seq 1 40) ? + if (_assert_exit) exit 1 Haven't you already exited above? > A second part of the patch is a new file 'contrib/generate_tcl_patterns.sh' > which generates the needed regexp Can we provide a Makefile target to automatically update Makefile.in? -Y
Re: [PATCH] RE: gcc parallel make check
On Tue, Sep 09, 2014 at 02:02:18PM +0400, Yury Gribov wrote: > On 09/09/2014 10:51 AM, VandeVondele Joost wrote: > > Attached is an extended version of the patch, > > it brings a 100% improvement in make -j32 -k check-gcc > > First of all, many thanks for working on this. > > +# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 > "dg.exp=gfortran.dg/" > > How does this work with subdirectories? Can we replace ls with find? Generally, if the argument to *.exp doesn't contain a particular subdirectory, then the wildcard is taken against basenames of the tests. > -check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 > +check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \ > + 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 > > $(shell seq 1 40) ? Would that be sufficiently portable to weirdo hosts (M$Win, Darwin, ...)? We require GNU make, so if it can be written in GNU make text functions, fine, otherwise it is better to keep as is. > > + if (_assert_exit) exit 1 > > Haven't you already exited above? > > > A second part of the patch is a new file > 'contrib/generate_tcl_patterns.sh' > > which generates the needed regexp > > Can we provide a Makefile target to automatically update Makefile.in? No. As I wrote earlier, splitting on filenames and test counts only is only very rough split, all the splits really need to be backed out by real timing data from popular targets. Also, I'm afraid of some tests being left out unintentionally (e.g. the wildcards created at some point, then a new test is added with a weird starting character that hasn't been used before and suddenly it will not be tested with make -j?). Jakub
RE: [PATCH] RE: gcc parallel make check
> +# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300
> "dg.exp=gfortran.dg/"
>
> How does this work with subdirectories? Can we replace ls with find?
The input to the script is general, you can use this to your advantage. For
example, I've been using:
ls -1 g++.*/* | cut -c5- | ../../../contrib/generate_tcl_patterns.sh 700
old-deja.exp=g++.old-deja/g++.
to split at a deeper level or
find . -name "[0-9A-Za-z]*" -type f -printf "%f\n" |
../../../../contrib/generate_tcl_patterns.sh 300 dg-torture.exp=torture/
to collect statistics also from subdirs.
> + if (_assert_exit) exit 1
>
> Haven't you already exited above?
yes, but the END{} block in awk is nevertheless executed, unless protected as
above.
RE: [PATCH] RE: gcc parallel make check
> No. As I wrote earlier, splitting on filenames and test counts only is only > very rough split, all the splits really need to be backed out by real timing > data from popular targets. I'm actually doing quite some testing trying to get a reasonable balance, checking 'completed in' in all *.log.sep files. However, it is important that the procedure is semi-automatic, otherwise few people will be interested in doing so. Furthermore, for parallel performance, it is not so important that times are distributed evenly (it is anyway unlikely the number of goals is exactly divided by N of -jN), but rather that the goals are ordered (executed) from slow to fast (similar to omp schedule guided). Most of the real bottlenecks are single letter patterns (e.g. p* since pr is such a common filename), and this is ultimately limiting. In the project (CP2K) I'm working on, we also parallelize testing over directories, but we keep a list of approximate runtimes per directory, and keep that (global) list sorted. Testing follows that list. As a result, we have near perfect parallel speedup, despite (or because) timings per directory ranging from a few 100s to 1s. > Also, I'm afraid of some tests being left out > unintentionally (e.g. the wildcards created at some point, then a new test > is added with a weird starting character that hasn't been used before and > suddenly it will not be tested with make -j?). I agree this is an issue, partially addressed by not having to write patterns by hand anymore (i.e. a script does this), and by having the script check its input. There are something like 10 testnames that do not fall in [0-9A-Za-z], as mentioned in a previous email.
Re: [PATCH] RE: gcc parallel make check
On Tue, Sep 09, 2014 at 10:57:09AM +, VandeVondele Joost wrote: > > No. As I wrote earlier, splitting on filenames and test counts only is only > > very rough split, all the splits really need to be backed out by real timing > > data from popular targets. > > Furthermore, for parallel performance, it is not > so important that times are distributed evenly (it is anyway unlikely the > number of goals is exactly divided by N of -jN), but rather that the goals > are ordered (executed) from slow to fast (similar to omp schedule guided). > Most of the real bottlenecks are single letter patterns (e.g. p* since > pr is such a common filename), and this is ultimately limiting. I disagree. If e.g. in gcc.dg/ more than a third of testcases are pr*.c, then running dg.exp=p* in one job and dg.exp=a* in another one etc. is simply a bad idea, the pr*.c should be split more and some other letters just be done together. Even that can be done semi-automatically. If you get whitespace right, one can provide multiple different wildcards to a single *.exp file, e.g. make check-gcc RUNTESTFLAGS="dg.exp='p[0-9A-Za-qs-z]* pr[9A-Za-z]*'" should cover all tests starting with p other than pr[0-8]*.c (where you could split say pr[0-2]* into another job, pr[3-5]* into another and pr[6-8]* into another. The fact that some check-gcc or check-gfortran test job is early in the list doesn't mean it will be started early, you need to consider also all other potentially long jobs like check-g++, check-target-libgomp, check-target-libstdc++-v3 etc. Jakub
RE: [PATCH] RE: gcc parallel make check
> If you get whitespace right, one can provide multiple different wildcards to > a single *.exp file, e.g. > make check-gcc RUNTESTFLAGS="dg.exp='p[0-9A-Za-qs-z]* pr[9A-Za-z]*'" should > cover all tests starting with p other than pr[0-8]*.c (where you could split > say pr[0-2]* into another job, pr[3-5]* into another and pr[6-8]* into > another. I think this confirms that it becomes very delicate to try and write these more complex patterns. The above would miss p_test.c, p-1.c, etc ? For other classes of files the difference is even further down the filename (e.g. using dates as in 20020508-3.c going from 2000 to 2014, or avx*), making the automatic generation of the patterns more complicated. I certainly don't want to claim that the patch I have now is perfect, it is rather an incremental improvement on the current setup.
Re: [PATCH] RE: gcc parallel make check
On 09/09/2014 06:14 PM, VandeVondele Joost wrote: I certainly don't want to claim that the patch I have now is perfect, it is rather an incremental improvement on the current setup. I'd second this. Writing patterns manually seems rather inefficient and error-prone (not undoable of course but unnecessarily complicated). And with current (crippled) version Joost already got 100% test time improvement. -Y
Re: [PATCH] RE: gcc parallel make check
On Tue, Sep 09, 2014 at 06:27:10PM +0400, Yury Gribov wrote: > On 09/09/2014 06:14 PM, VandeVondele Joost wrote: > >I certainly don't want to claim that the patch I have now is perfect, > >it is rather an incremental improvement on the current setup. > > I'd second this. Writing patterns manually seems rather inefficient and > error-prone > (not undoable of course but unnecessarily complicated). And with current > (crippled) > version Joost already got 100% test time improvement. But if there are jobs that just take 1s to complete, then clearly it doesn't make sense to split them off as separate job. I think we don't need 100% even split, but at least roughly is highly desirable. Jakub
Re: [PATCH] RE: gcc parallel make check
On 09/09/2014 06:33 PM, Jakub Jelinek wrote: On Tue, Sep 09, 2014 at 06:27:10PM +0400, Yury Gribov wrote: On 09/09/2014 06:14 PM, VandeVondele Joost wrote: I certainly don't want to claim that the patch I have now is perfect, it is rather an incremental improvement on the current setup. I'd second this. Writing patterns manually seems rather inefficient and error-prone (not undoable of course but unnecessarily complicated). And with current (crippled) version Joost already got 100% test time improvement. But if there are jobs that just take 1s to complete, then clearly it doesn't make sense to split them off as separate job. I think we don't need 100% even split, but at least roughly is highly desirable. You mean enhancing the script to split across arbitrarily long prefixes? That would be great. -Y
RE: [PATCH] RE: gcc parallel make check
Now with gzipped figure.. why do these bounce ? > But if there are jobs that just take 1s to complete, then clearly it doesn't > make sense to split them off as separate job. I think we don't need 100% > even split, but at least roughly is highly desirable. Let me add some data, attached is a graph (logscale y) showing the runtime of tests before and after my changes (including a new patch for c++). There is virtually no change for tests running shorter than 50s, only slowly running tests have been split. Now, there are only very few slow tests remaining: gcc_trunk/obj.new> find . -name "*.log" | xargs grep " completed in " | sort -n -k 5 | tail -n 10 ./gcc/testsuite/gcc/gcc.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gcc.dg/torture/dg-torture.exp completed in 521 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 530 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 553 seconds ./x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libgomp/testsuite/libgomp.fortran/fortran.exp completed in 561 seconds ./gcc/testsuite/gcc/gcc.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gcc.c-torture/compile/compile.exp completed in 625 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 683 seconds ./gcc/testsuite/g++/g++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/g++.dg/dg.exp completed in 702 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 726 seconds ./gcc/testsuite/gcc/gcc.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gcc.c-torture/execute/execute.exp completed in 752 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 904 seconds They, of course, limit the ultimate speedup. timings.png.gz Description: timings.png.gz
RE: [PATCH] RE: gcc parallel make check
Attached is a further revision of the patch, now dealing with check-c++. Roughly 50% speedup here at '-j32' (18m vs 12m). For my setup (--enable-languages=c,c++,fortran) I have now improved all targets called in 'make -j32 -k check'. The latter is now 30% faster (15m vs 20m). Note that there are +- 1m fluctuations in these numbers, easily. I currently have no plans to work on other check targets before this patch is committed. OK for trunk ? Joost contrib/ChangeLog 2014-09-09 Joost VandeVondele * generate_tcl_patterns.sh: New file. gcc/fortran/ChangeLog 2014-09-09 Joost VandeVondele * Make-lang.in (check_gfortran_parallelize): Improved parallelism. gcc/Changelog 2014-09-09 Joost VandeVondele * Makefile.in (check_gcc_parallelize): Improved parallelism. (check_p_numbers): Increase maximum value. (dg_target_exps): Mention targets as separate words only. (null,space,comma,dg_target_exps_p1,dg_target_exps_p2, dg_target_exps_p3,dg_target_exps_p4): New variables. gcc/cp/ChangeLog 2014-09-09 Joost VandeVondele * Make-lang.in (check_g++_parallelize): Improved parallelism. libstdc++-v3/ChangeLog 2014-09-09 Joost VandeVondele * testsuite/Makefile.am (check_DEJAGNU_normal_targets): Add check-DEJAGNUnormal[11-15]. (check-DEJAGNU): Split into 15 jobs for parallel testing. * testsuite/Makefile.in: Regenerated. Index: libstdc++-v3/testsuite/Makefile.am === --- libstdc++-v3/testsuite/Makefile.am (revision 215017) +++ libstdc++-v3/testsuite/Makefile.am (working copy) @@ -101,7 +101,7 @@ new-abi-baseline: @test ! -f $*/site.exp || mv $*/site.exp $*/site.bak @mv $*/site.exp.tmp $*/site.exp -check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10) +check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15) $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp # Run the testsuite in normal mode. @@ -111,7 +111,7 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \ && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \ $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \ - for idx in 0 1 2 3 4 5 6 7 8 9 10; do \ + for idx in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do \ mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \ mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \ done; \ @@ -138,25 +138,35 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar fi; \ dirs="`cd $$srcdir; echo [013-9][0-9]_*/*`";; \ normal1) \ - dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \ + dirs="`cd $$srcdir; echo e*/*`";; \ normal2) \ - dirs="`cd $$srcdir; echo 2[01]_*/*`";; \ + dirs="`cd $$srcdir; echo 28_*/a*`";; \ normal3) \ - dirs="`cd $$srcdir; echo 22_*/*`";; \ + dirs="`cd $$srcdir; echo 23_*/[lu]*`";; \ normal4) \ - dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \ + dirs="`cd $$srcdir; echo 2[459]_*/*`";; \ normal5) \ - dirs="`cd $$srcdir; echo 23_*/[luv]*`";; \ + dirs="`cd $$srcdir; echo 2[01]_*/*`";; \ normal6) \ - dirs="`cd $$srcdir; echo 2[459]_*/*`";; \ + dirs="`cd $$srcdir; echo 23_*/[m-tw-z]*`";; \ normal7) \ - dirs="`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`";; \ + dirs="`cd $$srcdir; echo 26_*/*`";; \ normal8) \ dirs="`cd $$srcdir; echo 27_*/*`";; \ normal9) \ - dirs="`cd $$srcdir; echo 28_*/[ab]*`";; \ + dirs="`cd $$srcdir; echo 22_*/*`";; \ normal10) \ dirs="`cd $$srcdir; echo t*/*`";; \ + normal11) \ + dirs="`cd $$srcdir; echo 28_*/b*`";; \ + normal12) \ + dirs="`cd $$srcdir; echo 28_*/[c-z]*`";; \ + normal13) \ + dirs="`cd $$srcdir; echo de* p*/*`";; \ + normal14) \ + dirs="`cd $$srcdir; echo [ab]* 23_*/v*`";; \ + normal15) \ + dirs="`cd $$srcdir; echo 23_*/[a-k]*`";; \ esac; \ if [ -n "$*" ]; then cd "$*"; fi; \ if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \ Index: libstdc++-v3/testsuite/Makefile.in === --- libstdc++-v3/testsuite/Makefile.in (revision 215017) +++ libstdc++-v3/testsuite/Makefile.in (working copy) @@ -301,7 +301,7 @@ lists_of_files = \ extract_symvers = $(glibcxx_builddir)/scripts/extract_symvers baseline_subdir := $(shell $(CXX) $(baseline_subdir_switch)) -check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10) +check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15) # Runs the testsuite, but in compile only mode. # Can be used to test sources with non-GNU FE's at various warning @@ -562,7 +562,7 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar if [ -z "$*$(filter-out --target_board=%, $(RUNTES
Re: [PATCH] RE: gcc parallel make check
On Tue, Sep 09, 2014 at 03:14:08PM +, VandeVondele Joost wrote: > Attached is a further revision of the patch, now dealing with check-c++. > Roughly 50% speedup here at '-j32' (18m vs 12m). For my setup > (--enable-languages=c,c++,fortran) I have now improved all targets called in > 'make -j32 -k check'. The latter is now 30% faster (15m vs 20m). Note that > there are +- 1m fluctuations in these numbers, easily. > > I currently have no plans to work on other check targets before this patch is > committed. > > OK for trunk ? So, I've tested your patch on a few years old 16way AMD, --enable-checking=yes and with make check -j1 and make check -j2 (the former doesn't parallelize, the latter does). First of all, the -j2 testing shows more tests tested in gcc and libstdc++: === gcc Summary === -# of expected passes 108100 +# of expected passes 108130 # of unexpected failures 175 # of unexpected successes 33 # of expected failures 262 ... === libstdc++ Summary === -# of expected passes 10133 +# of expected passes 10152 In gcc testsuite the difference seems to be (according to sorted PASS lines from *.sum files): +PASS: gcc.target/i386/vect-args.c -m3dnow -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse (test for excess errors) +PASS: gcc.target/i386/vect-args.c -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c (test for excess errors) +PASS: gcc.target/i386/vect-args.c (test for excess errors) +PASS: gcc.target/i386/vect-args.c (test for excess errors) and in libstdc++: +PASS: 23_containers/map/modifiers/erase/abi_tag.cc scan-assembler _ZNSt3mapIiiSt4lessIiESaISt4pairIKiiEEE5eraseB5cxx11ESt17_Rb_tree_iteratorIS4_E +PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors) +PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc scan-assembler _ZNSt8multimapIiiSt4lessIiESaISt4pairIKiiEEE5eraseB5cxx11ESt17_Rb_tree_iteratorIS4_E +PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors) +PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc scan-assembler _ZNSt8multisetIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiE +PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc scan-assembler _ZNSt8multisetIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiES5_ +PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors) +PASS: 23_containers/set/modifiers/erase/abi_tag.cc scan-assembler _ZNSt3setIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiE +PASS: 23_containers/set/modifiers/erase/abi_tag.cc scan-assembler _ZNSt3setIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiES5_ +PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIdE4imagB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIdE4realB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexI[eg]E4imagB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexI[eg]E4realB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIfE4imagB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complex
RE: [PATCH] RE: gcc parallel make check
Thanks for testing. The vect-args.c I explained earlier, and is indeed due to i386.exp hardcoding those. The libstdc++ double counts didn't appear in my testing, but I'll have a look. Note that these patterns are handwritten, so error prone. The long tests in libstdc++ come from (in timing order, from my machine): normal1) \ dirs="`cd $$srcdir; echo e*/*`";; \ normal2) \ dirs="`cd $$srcdir; echo 28_*/a*`";; \ normal3) \ dirs="`cd $$srcdir; echo 23_*/[lu]*`";; \ normal4) \ dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
RE: [PATCH] RE: gcc parallel make check
> You mean enhancing the script to split across arbitrarily long prefixes? > That would be great. I've now a script that does something like that: ~/test$ find /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gfortran.dg/ -maxdepth 1 -type f -printf "%f\n" | ./generate_patterns.py 500 foo All 3947 files matched the pattern ^[0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+ without exception Final 12 patterns and match count: (^[j-z_#+-][p-z_#+-][0-9A-Za-i][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-][0-9A-Za-o][0-9A-Za-m]([.][0-9A-Za-z_#+-]+)+) matching 469 files (^[0-9A-Za-i][0-9A-Za-n][0-9A-Za-n][0-9A-Za-o][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^([.][0-9A-Za-z_#+-]+)+) matching 433 files (^[j-z_#+-][0-9A-Za-o][n-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][0-9A-Za-n][o-z_#+-]([.][0-9A-Za-z_#+-]+)+) matching 400 files (^[j-z_#+-][p-z_#+-][j-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i]([.][0-9A-Za-z_#+-]+)+) matching 371 files (^[0-9A-Za-i][o-z_#+-][s-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][0-9A-Za-n][0-9A-Za-n]([.][0-9A-Za-z_#+-]+)+) matching 323 files (^[0-9A-Za-i][o-z_#+-][0-9A-Za-r][o-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-][p-z_#+-]([.][0-9A-Za-z_#+-]+)+) matching 314 files (^[0-9A-Za-i][o-z_#+-][0-9A-Za-r][0-9A-Za-n][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-][0-9A-Za-o]([.][0-9A-Za-z_#+-]+)+) matching 314 files (^[j-z_#+-][0-9A-Za-o][0-9A-Za-m][0-9A-Za-i][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-]([.][0-9A-Za-z_#+-]+)+) matching 272 files (^[0-9A-Za-i][0-9A-Za-n][0-9A-Za-n][p-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][o-z_#+-]([.][0-9A-Za-z_#+-]+)+) matching 270 files (^[0-9A-Za-i][0-9A-Za-n][o-z_#+-][0-9A-Za-l][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][0-9A-Za-n]([.][0-9A-Za-z_#+-]+)+) matching 265 files (^[0-9A-Za-i][0-9A-Za-n][o-z_#+-][m-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][o-z_#+-][0-9A-Za-r]([.][0-9A-Za-z_#+-]+)+) matching 260 files ^[j-z_#+-][0-9A-Za-o][0-9A-Za-m][j-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+ matching 256 files It is a set of patterns that will match any file of the form '^[0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+', but such that it splits a list of input files roughly in equal chunks (e.g. between 500 and 500/2 in this example), even if files have long overlapping prefixes. However, I'm unsure if/how this can be integrated, i.e. what precisely is allowed for testsuite filenames, and if this regexp format can be employed in gcc makefiles / tcl / expect harness, suggestions/help appreciated.
RE: [PATCH] RE: gcc parallel make check
Jakub, > First of all, the -j2 testing shows more tests tested in gcc and libstdc++: > >-# of expected passes 10133 >+# of expected passes 10152 > >+PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) >[...] > >Not sure where the bug is, could be e.g. in i386.exp for gcc, but for >libstdc++ less likely to be there rather than in the split. I looked into this, and believe this problem is already in current trunk, and not due to my patch. I.e. unmodified trunk also has these tests executed several times: libstdc++-v3/testsuite/normal4/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc libstdc++-v3/testsuite/normal1/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc I believe the current trunk pattern could indeed match those twice (Makefile.in in trunk): normal1) \ dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \ normal4) \ dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \ could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ? Joost
RE: [PATCH] RE: gcc parallel make check
> could it be that the pattern in normal1 should have been '[ab]*/ de*/ > [ep]*/*' ? I've checked that this fixes the bug in the current trunk split. I.e. files are stil tested, but now only once. Consider this change added to the previously submitted patch.
Re: [PATCH] RE: gcc parallel make check
On 11 September 2014 07:22, VandeVondele Joost wrote: > Jakub, > >> First of all, the -j2 testing shows more tests tested in gcc and libstdc++: >> >>-# of expected passes 10133 >>+# of expected passes 10152 >> >>+PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) >>[...] >> >>Not sure where the bug is, could be e.g. in i386.exp for gcc, but for >>libstdc++ less likely to be there rather than in the split. > > I looked into this, and believe this problem is already in current trunk, and > not due to my patch. I.e. unmodified trunk also has these tests executed > several times: > > libstdc++-v3/testsuite/normal4/libstdc++.log.sep:PASS: > 23_containers/map/modifiers/erase/abi_tag.cc > libstdc++-v3/testsuite/normal1/libstdc++.log.sep:PASS: > 23_containers/map/modifiers/erase/abi_tag.cc > > I believe the current trunk pattern could indeed match those twice > (Makefile.in in trunk): > normal1) \ > dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \ > normal4) \ > dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \ > > could it be that the pattern in normal1 should have been '[ab]*/ de*/ > [ep]*/*' ? Yes, we are running these tests multiple times: PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 26_numerics/complex/abi_tag.cc (test for excess errors) I'll fix that.
RE: [PATCH] RE: gcc parallel make check
>> could it be that the pattern in normal1 should have been '[ab]*/ de*/ >> [ep]*/*' ? > >Yes, we are running these tests multiple times: > >PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors) >PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess >errors) >PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess >errors) >PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) >PASS: 26_numerics/complex/abi_tag.cc (test for excess errors) > >I'll fix that. Actually, the proper pattern should presumably be '[ab]*/* de*/* [ep]*/*' even though it seems to make no difference in testing. I'll have this included in yet another version of the parallel make check patch (plus some further reschuffling as requested by Jakub), so I think there is no need for you to fix this now.
Re: [PATCH] RE: gcc parallel make check
On 11 September 2014 15:45, VandeVondele Joost wrote: > >>> could it be that the pattern in normal1 should have been '[ab]*/ de*/ >>> [ep]*/*' ? >> >>Yes, we are running these tests multiple times: >> >>PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors) >>PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess >>errors) >>PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess >>errors) >>PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) >>PASS: 26_numerics/complex/abi_tag.cc (test for excess errors) >> >>I'll fix that. > > Actually, the proper pattern should presumably be '[ab]*/* de*/* [ep]*/*' > even though it seems to make no difference in testing. Yes, that's what I'm testing. > I'll have this included in yet another version of the parallel make check > patch (plus some further reschuffling as requested by Jakub), so I think > there is no need for you to fix this now. This can (and should) be fixed now, without waiting for some other change.
