dejagnu testsuite bug?
I was looking through my 'make check' output (from a MIPS cross compiler)
and saw this error. Has anyone else run into something like this? I am
not entirely sure where to start looking for this problem and I am also
not sure if this is a new problem or not. Normally I just grep for FAIL
and don't examine the testing output that closely. I see the 'usual' C
and C++ faliiures after this error and the rest of the testsuite seems to
run fine.
Steve Ellcey
sell...@mips.com
Test Run By sellcey on Fri Sep 5 03:08:58 2014
Native configuration is x86_64-unknown-linux-gnu
=== tests ===
Schedule of variations:
multi-sim
Running target multi-sim
Using
/scratch/sellcey/nightly/src/gcc/dejagnu/testsuite/../config/base-config.exp as
tool-and-target-specific interface file.
Using /scratch/sellcey/nightly/src/gcc/./dejagnu/baseboards/multi-sim.exp as
board description file for target.
Using /scratch/sellcey/nightly/src/gcc/./dejagnu/config/sim.exp as generic
interface file for target.
Using /scratch/sellcey/nightly/src/gcc/./dejagnu/baseboards/basic-sim.exp as
board description file for target.
Using /scratch/sellcey/nightly/src/gcc/gcc/testsuite/config/default.exp as
tool-and-target-specific interface file.
Using /scratch/sellcey/nightly/src/gcc/dejagnu/testsuite/config/default.exp as
tool-and-target-specific interface file.
Running /scratch/sellcey/nightly/src/gcc/dejagnu/testsuite/libdejagnu/tunit.exp
...
send: spawn id exp0 not open
while executing
"send_user -- "$message\n""
("default" arm line 2)
invoked from within
"switch -glob "$firstword" {
"PASS:" -
"XFAIL:" -
"KFAIL:" -
"UNRESOLVED:" -
"UNSUPPORTED:" -
"UNTESTED:" {
if {$all_flag} {
send_user -- ..."
(procedure "clone_output" line 10)
invoked from within
"clone_output "Running $test_file_name ...""
(procedure "runtest" line 7)
invoked from within
"runtest $test_name"
("foreach" body line 42)
invoked from within
"foreach test_name [lsort [find ${dir} *.exp]] {
if { ${test_name} == "" } {
continue
}
# Ignore this one if asked to.
if { ${ignore..."
("foreach" body line 54)
invoked from within
"foreach dir "${test_top_dirs}" {
if { ${dir} != ${srcdir} } {
# Ignore this directory if is a directory to be
# ignored.
if {[info..."
("foreach" body line 121)
invoked from within
"foreach pass $multipass {
# multipass_name is set for `record_test' to use (see framework.exp).
if { [lindex $pass 0] != "" } {
set multipass_..."
("foreach" body line 51)
invoked from within
"foreach current_target $target_list {
verbose "target is $current_target"
set current_target_name $current_target
set tlist [split $curren..."
(file "/scratch/sellcey/nightly/src/gcc/./dejagnu/runtest.exp" line 1627)^M
make[3]: *** [check-DEJAGNU] Error 1
make[3]: Leaving directory
`/scratch/sellcey/nightly/obj-mips-mti-linux-gnu/gcc/dejagnu'
make[2]: *** [check-am] Error 2
make[2]: Target `check' not remade because of errors.
make[2]: Leaving directory
`/scratch/sellcey/nightly/obj-mips-mti-linux-gnu/gcc/dejagnu'
make[1]: *** [check-dejagnu] Error 2
Re: libstdc++ testsuite "misbehaving"
On Sep 5, 2014, at 4:05 AM, Thomas Schwinge wrote:
> Updating my GCC trunk working tree from r212389 (2014-07-09) to r214918
> (2014-09-04), I notice that (only) in libstdc++ testing, and only for the
> second multilib of »RUNTESTFLAGS='--target_board=unix\{,-m32\}'« (so, the
> 32-bit x86 one), a lot of PASSes "disappear" (compile tests only?). Has
> anyone seen this before, and/or an explanation for this?
So, the tcl, dejagnu versions might play some role. Also, check to see if:
2014-08-22 Max Ostapenko
* lib/gcc-dg.exp: Change pattern.
is in your tree, and if it is, try without it. If that fixes it, someone will
have to puzzle out why that patch is wrong, or we just revert it.
RE: [PATCH] RE: gcc parallel make check
>> > Please sort the letters (LC_ALL=C sort) and where consecutive, use ranges.
>> > Thus \[0-9A-Zhjqvx-z\]*
OK, works fine with the attached patch, and looks cleaner in Make-lang.in.
Now, with the proper email address for gcc-patches... I wonder how many time
I'll be punished for typos.
unmodified CL.
Joost
Index: contrib/generate_tcl_patterns.sh
===
--- contrib/generate_tcl_patterns.sh (revision 0)
+++ contrib/generate_tcl_patterns.sh (revision 0)
@@ -0,0 +1,108 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input,
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+# cd gcc/gcc/testsuite/gfortran.dg
+# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING. If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+ # list of allowed starting chars for a file name in a dir to split
+ achars="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+ ranget="112233"
+}
+{
+ nfiles++ ; files[nfiles]=$1
+}
+END{
+ for(i=1; i<=length(achars); i++) count[substr(achars,i,1)]=0
+ for(i=1; i<=nfiles; i++) {
+ if (length(files[i]>0)) { count[substr(files[i],1,1)]++ }
+ };
+ asort(count,ordered)
+ countsingle=0
+ groups=0
+ label=""
+ for(i=length(achars);i>=1;i--) {
+countsingle=countsingle+ordered[i]
+for(j=1;j<=length(achars);j++) {
+ if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+}
+count[found]=-1
+label=label found
+if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+if(countsingle+val>maxcount) {
+ subset[label]=countsingle
+ print "Adding label: ", label, "matching files:" countsingle
+ groups++
+ countsingle=0
+ label=""
+}
+ }
+ print "patterns:"
+ asort(subset,ordered)
+ for(i=groups;i>=1;i--) {
+for(j in subset){
+ if(subset[j]==ordered[i]) found=j
+}
+subset[found]=-1
+if (length(found)==1) {
+ printf("%s%s* \\\n",prefix,found)
+} else {
+ sortandcompress()
+ printf("%s\\[%s\\]* \\\n",prefix,found)
+}
+ }
+}
+function sortandcompress(i,n,tmp,bestj)
+{
+ n=length(found)
+ for(i=1; i<=n; i++) tmp[i]=substr(found,i,1)
+ asort(tmp)
+ for(i=1;i<=n;i++){
+ipos=index(achars,tmp[i])
+for(j=i;j<=n;j++){
+ jpos=index(achars,tmp[j])
+ if (jpos-ipos==j-i && substr(ranget,ipos,1)==substr(ranget,jpos,1)) bestj=j
+}
+if (bestj-i>3) {
+ tmp[i+1]="-"
+ for(j=i+2;j
Re: [PATCH] RE: gcc parallel make check
On Fri, Sep 05, 2014 at 02:48:01PM +, VandeVondele Joost wrote: > > > Please sort the letters (LC_ALL=C sort) and where consecutive, use ranges. > > Thus \[0-9A-Zhjqvx-z\]* or so. What is - doing in your list? _-9 is quite > > unexpected range. > > the '-' is a bug indeed. I added this to the list of allowed starting > characters, but this clearly this is not a good idea. Thanks, removed. > > > Also, please tweak the script so that it will do the > > sorting and ranges handling for you. > > The current order is the number of files that start with this letter, which > is somewhat informative, but let me see if I can have ranges easily (in > awk...). BTW, number of tests might help as initial guess on how to split things, but might be severely off. What I'd recommend after you split with your script that you perform a make -j1 check and diff the Test run by and runtest completed times in each of the */*.log.sep files and see what time each job takes, then perhaps reshuffle stuff a little bit until it is roughly even. Some letters could start more dg-do run tests while others more dg-do compile once, some tests take a short time to compile, some very long, etc. Jakub
Re: [PATCH] RE: gcc parallel make check
On Fri, Sep 05, 2014 at 02:26:45PM +, VandeVondele Joost wrote: > Adding label: mu matching files:278 > Adding label: wlgo matching files:284 > Adding label: vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj matching > files:94 > patterns: > dg.exp=gfortran.dg/p* \ > dg.exp=gfortran.dg/c* \ > dg.exp=gfortran.dg/a* \ > dg.exp=gfortran.dg/i* \ > dg.exp=gfortran.dg/\[wlgo\]* \ > dg.exp=gfortran.dg/\[mu\]* \ > dg.exp=gfortran.dg/d* \ > dg.exp=gfortran.dg/s* \ > dg.exp=gfortran.dg/b* \ > dg.exp=gfortran.dg/t* \ > dg.exp=gfortran.dg/f* \ > dg.exp=gfortran.dg/e* \ > dg.exp=gfortran.dg/r* \ > dg.exp=gfortran.dg/n* \ > dg.exp=gfortran.dg/\[vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj\]* \ > > Is the current attached patch OK for trunk ? > gcc/fortran/ChangeLog > > 2014-09-05 Joost VandeVondele > >* Make-lang.in (check_gfortran_parallelize): improved parallelism. Capital I. > --- gcc/fortran/Make-lang.in (revision 214949) > +++ gcc/fortran/Make-lang.in (working copy) > @@ -168,12 +168,22 @@ check-fortran-subtargets : check-gfortra > lang_checks += check-gfortran > lang_checks_parallelized += check-gfortran > # For description see comment above check_gcc_parallelize in gcc/Makefile.in. > -check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \ > - dg.exp=gfortran.dg/\[bcBC\]* \ > - dg.exp=gfortran.dg/\[nopNOP\]* \ > - dg.exp=gfortran.dg/\[isuvISUV\]* \ > - dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \ > - dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]* > +check_gfortran_parallelize = execute.exp \ > + dg.exp=gfortran.dg/p* \ > + dg.exp=gfortran.dg/c* \ > + dg.exp=gfortran.dg/a* \ > + dg.exp=gfortran.dg/i* \ > + dg.exp=gfortran.dg/\[wlgo\]* \ > + dg.exp=gfortran.dg/\[mu\]* \ > + dg.exp=gfortran.dg/d* \ > + dg.exp=gfortran.dg/s* \ > + dg.exp=gfortran.dg/b* \ > + dg.exp=gfortran.dg/t* \ > + dg.exp=gfortran.dg/f* \ > + dg.exp=gfortran.dg/e* \ > + dg.exp=gfortran.dg/r* \ > + dg.exp=gfortran.dg/n* \ > + > dg.exp=gfortran.dg/\[vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj\]* Please sort the letters (LC_ALL=C sort) and where consecutive, use ranges. Thus \[0-9A-Zhjqvx-z\]* or so. What is - doing in your list? _-9 is quite unexpected range. Also, please tweak the script so that it will do the sorting and ranges handling for you. Jakub
[PATCH] RE: gcc parallel make check
> The splits are in the Makefiles, see check_gcc_parallelize
attached is a patch to improve the parallel performance of 'make -jXX -k
check-fortran'. For XX=16, this yields ~50% speedup, and even with XX=4 we
still have 15%, the measured slowdown at XX=1 (<2%) is in the noise of testing.
The patch is a simple update of the 'check_gfortran_parallelize' variable,
updating it from its 2008 values to a set that I found +- optimal based on
several tests. Detailed timings are :
# timings/trunk-check-fortran
#cores averagestd. dev. #tests
1 2955.3275.06 3
2 1735.30 122.26 3
4 929.5154.19 3
8 470.29 7.85 3
16 468.09 4.29 3
32 466.06 1.24 3
# timings/patched-check-fortran
#cores averagestd. dev. #tests
1 3008.8916.38 3
2 1534.17 118.33 3
4 800.1831.71 3
8 418.71 0.20 2
16 298.29 5.86 3
32 299.84 1.34 3
There is no effect on a full 'make -j32 -k check' as other goals run for much
longer (to be looked at in a followup).
A second part of the patch is a new file 'contrib/generate_tcl_patterns.sh'
which generates the needed regexp to do the split based on an input of the
files in the target directory. It basically groups the initial characters such
that each regexp tries not to exceed a maximum number of files. So, the number
of files is used as a proxy for the runtime. While I don't feel to strong about
adding this (shell/gawk) script, it certainly is convenient, and makes sure
that no characters are missing from the regexp. The maximum number of files per
regexp is an input, testing (-j16) with 200, 300, 400 I found that 300 was
optimal for testsuite/gfortran.dg, but this will depend on many things.
A sample run would look like
gcc/gcc/testsuite/gfortran.dg> ls -1 |
../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
Adding label: p matching files:499
Adding label: c matching files:497
Adding label: a matching files:448
Adding label: i matching files:350
Adding label: d matching files:245
Adding label: s matching files:211
Adding label: b matching files:206
Adding label: t matching files:180
Adding label: f matching files:173
Adding label: e matching files:166
Adding label: r matching files:165
Adding label: n matching files:162
Adding label: mu matching files:278
Adding label: wlgo matching files:284
Adding label: vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj matching files:94
patterns:
dg.exp=gfortran.dg/p* \
dg.exp=gfortran.dg/c* \
dg.exp=gfortran.dg/a* \
dg.exp=gfortran.dg/i* \
dg.exp=gfortran.dg/\[wlgo\]* \
dg.exp=gfortran.dg/\[mu\]* \
dg.exp=gfortran.dg/d* \
dg.exp=gfortran.dg/s* \
dg.exp=gfortran.dg/b* \
dg.exp=gfortran.dg/t* \
dg.exp=gfortran.dg/f* \
dg.exp=gfortran.dg/e* \
dg.exp=gfortran.dg/r* \
dg.exp=gfortran.dg/n* \
dg.exp=gfortran.dg/\[vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj\]* \
Is the current attached patch OK for trunk ?
contrib/ChangeLog
2014-09-05 Joost VandeVondele
* generate_tcl_patterns.sh: New file.
gcc/fortran/ChangeLog
2014-09-05 Joost VandeVondele
* Make-lang.in (check_gfortran_parallelize): improved parallelism.
Index: contrib/generate_tcl_patterns.sh
===
--- contrib/generate_tcl_patterns.sh (revision 0)
+++ contrib/generate_tcl_patterns.sh (revision 0)
@@ -0,0 +1,86 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input,
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+# cd gcc/gcc/testsuite/gfortran.dg
+# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING. If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+
Re: libstdc++ testsuite "misbehaving"
On 05/09/14 13:05 +0200, Thomas Schwinge wrote:
Updating my GCC trunk working tree from r212389 (2014-07-09) to r214918
(2014-09-04), I notice that (only) in libstdc++ testing, and only for the
second multilib of »RUNTESTFLAGS='--target_board=unix\{,-m32\}'« (so, the
32-bit x86 one), a lot of PASSes "disappear" (compile tests only?). Has
anyone seen this before, and/or an explanation for this?
I don't know what could cause it and I don't see it:
=== libstdc++ Summary for unix ===
# of expected passes9434
# of unexpected failures8
# of expected failures 69
# of unsupported tests 537
=== libstdc++ Summary for unix/-m32 ===
# of expected passes9442
# of unexpected failures8
# of expected failures 69
# of unsupported tests 533
=== libstdc++ Summary ===
# of expected passes18876
# of unexpected failures16
# of expected failures 138
# of unsupported tests 1070
GNU Tools Cauldron 2014 - Videos for presentations
All the videos for Cauldron 2014 are now available at the conference page: https://gcc.gnu.org/wiki/cauldron2014 Thanks to Simon Cook for processing the videos and setting them up! If you notice anything missing or mis-labeled, please let me know. Thanks. Diego.
libstdc++ testsuite "misbehaving"
Hi!
Updating my GCC trunk working tree from r212389 (2014-07-09) to r214918
(2014-09-04), I notice that (only) in libstdc++ testing, and only for the
second multilib of »RUNTESTFLAGS='--target_board=unix\{,-m32\}'« (so, the
32-bit x86 one), a lot of PASSes "disappear" (compile tests only?). Has
anyone seen this before, and/or an explanation for this?
Here's a diff of the sum file that shows how the »unix« variant behaves
as expected, but for the »unix/-m32« one a lot of PASSes disappear, and
in the latter's summary the »# of expected passes« is basically cut into
half:
[...]
Native configuration is x86_64-unknown-linux-gnu
=== libstdc++ tests ===
Schedule of variations:
unix
unix/-m32
Running target unix
[...]
=== libstdc++ Summary for unix ===
-# of expected passes 9368
-# of unexpected failures 1
+# of expected passes 9552
# of expected failures 69
-# of unsupported tests 536
+# of unsupported tests 535
Running target unix/-m32
Running
/home/thomas/tmp/source/gcc/trunk/source/libstdc++-v3/testsuite/libstdc++-abi/abi.exp
...
-FAIL: libstdc++-abi/abi_check
+PASS: libstdc++-abi/abi_check
Running
/home/thomas/tmp/source/gcc/trunk/source/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp
...
-PASS: 17_intro/freestanding.cc (test for excess errors)
PASS: 17_intro/freestanding.cc execution test
-PASS: 17_intro/headers/c++1998/49745.cc (test for excess errors)
-PASS: 17_intro/headers/c++1998/all.cc (test for excess errors)
-PASS: 17_intro/headers/c++1998/all_c++200x_compatibility.cc (test for
excess errors)
-PASS: 17_intro/headers/c++1998/all_no_exceptions.cc (test for excess
errors)
-PASS: 17_intro/headers/c++1998/all_no_rtti.cc (test for excess errors)
-PASS: 17_intro/headers/c++1998/all_pedantic_errors.cc (test for excess
errors)
-PASS: 17_intro/headers/c++1998/complex.cc (test for excess errors)
-PASS: 17_intro/headers/c++1998/operator_names.cc (test for excess errors)
-PASS: 17_intro/headers/c++1998/stdc++.cc (test for excess errors)
PASS: 17_intro/headers/c++1998/stdc++_assert_neg.cc (test for errors,
line 34)
-PASS: 17_intro/headers/c++1998/stdc++_assert_neg.cc (test for excess
errors)
-PASS: 17_intro/headers/c++1998/stdc++_multiple_inclusion.cc (test for
excess errors)
-PASS: 17_intro/headers/c++200x/42319.cc (test for excess errors)
-PASS: 17_intro/headers/c++200x/all_no_exceptions.cc (test for excess
errors)
-PASS: 17_intro/headers/c++200x/all_no_rtti.cc (test for excess errors)
-PASS: 17_intro/headers/c++200x/all_pedantic_errors.cc (test for excess
errors)
-PASS: 17_intro/headers/c++200x/operator_names.cc (test for excess errors)
-PASS: 17_intro/headers/c++200x/stdc++.cc (test for excess errors)
-PASS: 17_intro/headers/c++200x/stdc++_multiple_inclusion.cc (test for
excess errors)
-PASS: 17_intro/shared_with_static_deps.cc (test for excess errors)
-PASS: 17_intro/static.cc (test for excess errors)
-PASS: 17_intro/using_namespace_std.cc (test for excess errors)
PASS: 17_intro/using_namespace_std_tr1_neg.cc (test for errors, line 66)
PASS: 17_intro/using_namespace_std_tr1_neg.cc (test for errors, line 66)
-PASS: 17_intro/using_namespace_std_tr1_neg.cc (test for excess errors)
-PASS: 18_support/50594.cc (test for excess errors)
PASS: 18_support/50594.cc execution test
-PASS: 18_support/bad_alloc/cons_virtual_derivation.cc (test for excess
errors)
PASS: 18_support/bad_alloc/cons_virtual_derivation.cc execution test
[...]
=== libstdc++ Summary for unix/-m32 ===
-# of expected passes 9376
-# of unexpected failures 1
-# of expected failures 69
-# of unsupported tests 532
+# of expected passes 4146
+# of unexpected successes 1
+# of expected failures 61
+# of unsupported tests 531
=== libstdc++ Summary ===
-# of expected passes 18744
-# of unexpected failures 2
-# of expected failures 138
-# of unsupported tests 1068
+# of expected passes 13698
+# of unexpected successes 1
+# of expected failures 130
+# of unsupported tests 1066
Grüße,
Thomas
pgpFlN7g1rsbG.pgp
Description: PGP signature
Re: non-reproducible g++.dg/ubsan/align-2.C -Os execution failure
On 09/04/2014 11:12 AM, Tom de Vries wrote: > I ran into this non-reproducible failure while testing a non-bootstrap > build on x86_64: > ... > PASS: g++.dg/ubsan/align-2.C -Os (test for excess errors) Added UBSan folks. Can this be related to http://llvm.org/bugs/show_bug.cgi?id=20721 ? It has been causing sporadic align-4 errors. -Y
