Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing
Patch proposed at https://gcc.gnu.org/pipermail/gcc-patches/2023-November/635000.html -- Maxim Kuvyrkov https://www.linaro.org > On Sep 27, 2023, at 18:47, Maxim Kuvyrkov wrote: > > Hi Bernhard, > > Thanks, I meant to fix this, but forgot. > > The underlying problem here is that we want to detect which sub-testsuites > had failures. Current regex doesn't match go's case because there is no > "..." at the end: "Running foo" vs "Running foo ..." . > > My preferred way of fixing this is to make go's testsuite print out "..." . > We have a similar patch for glibc [1]. > > [1] https://sourceware.org/pipermail/libc-alpha/2023-June/148702.html > > -- > Maxim Kuvyrkov > https://www.linaro.org > >> On Sep 26, 2023, at 19:46, Bernhard Reutner-Fischer >> wrote: >> >> Hi Maxim! >> >> On Mon, 5 Jun 2023 18:06:25 +0400 >> Maxim Kuvyrkov via Gcc-patches wrote: >> On Jun 3, 2023, at 19:17, Jeff Law wrote: On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote: > This patch adds tracking of current testsuite "tool" and "exp" > to the processing of .sum files. This avoids aliasing between > tests from different testsuites with same name+description. > E.g., this is necessary for testsuite/c-c++-common, which is ran > for both gcc and g++ "tools". > This patch changes manifest format from ... > > FAIL: gcc_test > FAIL: g++_test > > ... to ... > > === gcc tests === > Running gcc/foo.exp ... > FAIL: gcc_test > === gcc Summary == > === g++ tests === > Running g++/bar.exp ... > FAIL: g++_test > === g++ Summary == > . > The new format uses same formatting as DejaGnu's .sum files > to specify which "tool" and "exp" the test belongs to. I think the series is fine. You're not likely to hear from Diego or Doug I suspect, I don't think either are involved in GNU stuff anymore. >>> >>> Thanks, Jeff. I'll wait for a couple of days and will merge if there are >>> no new comments. >> >> Maxim, may i ask you to have a look at the following problem, please? >> >> ISTM that your exp code does not work as expected for go, maybe you >> forgot to test the changes with go enabled? >> >> Ever since your changes in summer i see the following: >> >> gcc-14.mine$ >> /scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py >> --clean_build ../gcc-14.orig/ >> Getting actual results from build directory . >> ./gcc/testsuite/go/go.sum >> ./gcc/testsuite/gcc/gcc.sum >> ./gcc/testsuite/objc/objc.sum >> ./gcc/testsuite/jit/jit.sum >> ./gcc/testsuite/gdc/gdc.sum >> ./gcc/testsuite/gnat/gnat.sum >> ./gcc/testsuite/ada/acats/acats.sum >> ./gcc/testsuite/g++/g++.sum >> ./gcc/testsuite/obj-c++/obj-c++.sum >> ./gcc/testsuite/rust/rust.sum >> ./gcc/testsuite/gfortran/gfortran.sum >> ./x86_64-pc-linux-gnu/libgomp/testsuite/libgomp.sum >> ./x86_64-pc-linux-gnu/libphobos/testsuite/libphobos.sum >> ./x86_64-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum >> ./x86_64-pc-linux-gnu/libffi/testsuite/libffi.sum >> ./x86_64-pc-linux-gnu/libitm/testsuite/libitm.sum >> ./x86_64-pc-linux-gnu/libgo/libgo.sum >> ./x86_64-pc-linux-gnu/libatomic/testsuite/libatomic.sum >> ./gotools/gotools.sum >> .sum file seems to be broken: tool="gotools", exp="None", >> summary_line="FAIL: TestScript" >> Traceback (most recent call last): >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 732, in >> retval = Main(sys.argv) >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 721, in Main >> retval = CompareBuilds() >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 622, in CompareBuilds >> actual = GetResults(sum_files) >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 466, in GetResults >> build_results.update(ParseSummary(sum_fname)) >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 405, in ParseSummary >> result = result_set.MakeTestResult(line, ordinal) >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 239, in MakeTestResult >> return TestResult(summary_line, ordinal, >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 151, in __init__ >> raise >> RuntimeError: No active exception to reraise >> >> >> The problem seems to be that gotools.sum does not mention any ".exp" >> files. >> >> $ grep "Running " gotools/gotools.sum >> Running cmd/go >> Running runtime >> Running cgo >> Running carchive >> Running cmd/vet >> Running embed >> $ grep -c "\.exp" gotools/gotools.sum >> 0 >> >> The .sum files looks like this: >> ---8<--- >> Test Run By foo on Tue Sep 26 14:46:48 CEST 2023 >> Native configuration is x86_64-foo-linux-gnu >> >> === gotools tests
Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing
On 27 September 2023 16:47:27 CEST, Maxim Kuvyrkov wrote: >Hi Bernhard, > >Thanks, I meant to fix this, but forgot. np. >The underlying problem here is that we want to detect which sub-testsuites had >failures. Current regex doesn't match go's case because there is no "..." at >the end: "Running foo" vs "Running foo ..." . > >My preferred way of fixing this is to make go's testsuite print out "..." . >We have a similar patch for glibc [1]. > >[1] https://sourceware.org/pipermail/libc-alpha/2023-June/148702.html Which asks: ---8<--- >> WDYT? > > I looked at the gcc-testresults mailing list, and there appear no > === … failures === lines at all? What was the motivation for adding it > in the first place? The only motivation is that it looks like a nice header for the following FAILs. What's your preference for the line -- drop it entirely or print out: === glibc failures === no unexpected failures ? ---8<--- I'd drop the above entirely if there are no failures, it's pretty superfluous, isn't it. And concerning gotools and the missing trailing ".exp ...", I guess it's fine to add that to streamline the gotools output to all the other existing sum output. TIA, > >-- >Maxim Kuvyrkov >https://www.linaro.org > >> On Sep 26, 2023, at 19:46, Bernhard Reutner-Fischer >> wrote: >> >> Hi Maxim! >> >> On Mon, 5 Jun 2023 18:06:25 +0400 >> Maxim Kuvyrkov via Gcc-patches wrote: >> On Jun 3, 2023, at 19:17, Jeff Law wrote: On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote: > This patch adds tracking of current testsuite "tool" and "exp" > to the processing of .sum files. This avoids aliasing between > tests from different testsuites with same name+description. > E.g., this is necessary for testsuite/c-c++-common, which is ran > for both gcc and g++ "tools". > This patch changes manifest format from ... > > FAIL: gcc_test > FAIL: g++_test > > ... to ... > > === gcc tests === > Running gcc/foo.exp ... > FAIL: gcc_test > === gcc Summary == > === g++ tests === > Running g++/bar.exp ... > FAIL: g++_test > === g++ Summary == > . > The new format uses same formatting as DejaGnu's .sum files > to specify which "tool" and "exp" the test belongs to. I think the series is fine. You're not likely to hear from Diego or Doug I suspect, I don't think either are involved in GNU stuff anymore. >>> >>> Thanks, Jeff. I'll wait for a couple of days and will merge if there are >>> no new comments. >> >> Maxim, may i ask you to have a look at the following problem, please? >> >> ISTM that your exp code does not work as expected for go, maybe you >> forgot to test the changes with go enabled? >> >> Ever since your changes in summer i see the following: >> >> gcc-14.mine$ >> /scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py >> --clean_build ../gcc-14.orig/ >> Getting actual results from build directory . >> ./gcc/testsuite/go/go.sum >> ./gcc/testsuite/gcc/gcc.sum >> ./gcc/testsuite/objc/objc.sum >> ./gcc/testsuite/jit/jit.sum >> ./gcc/testsuite/gdc/gdc.sum >> ./gcc/testsuite/gnat/gnat.sum >> ./gcc/testsuite/ada/acats/acats.sum >> ./gcc/testsuite/g++/g++.sum >> ./gcc/testsuite/obj-c++/obj-c++.sum >> ./gcc/testsuite/rust/rust.sum >> ./gcc/testsuite/gfortran/gfortran.sum >> ./x86_64-pc-linux-gnu/libgomp/testsuite/libgomp.sum >> ./x86_64-pc-linux-gnu/libphobos/testsuite/libphobos.sum >> ./x86_64-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum >> ./x86_64-pc-linux-gnu/libffi/testsuite/libffi.sum >> ./x86_64-pc-linux-gnu/libitm/testsuite/libitm.sum >> ./x86_64-pc-linux-gnu/libgo/libgo.sum >> ./x86_64-pc-linux-gnu/libatomic/testsuite/libatomic.sum >> ./gotools/gotools.sum >> .sum file seems to be broken: tool="gotools", exp="None", >> summary_line="FAIL: TestScript" >> Traceback (most recent call last): >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 732, in >>retval = Main(sys.argv) >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 721, in Main >>retval = CompareBuilds() >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 622, in CompareBuilds >>actual = GetResults(sum_files) >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 466, in GetResults >>build_results.update(ParseSummary(sum_fname)) >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 405, in ParseSummary >>result = result_set.MakeTestResult(line, ordinal) >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 239, in MakeTestResult >>return TestResult(summary_line, ordinal, >> File >> "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", >> line 151,
Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing
Hi Bernhard, Thanks, I meant to fix this, but forgot. The underlying problem here is that we want to detect which sub-testsuites had failures. Current regex doesn't match go's case because there is no "..." at the end: "Running foo" vs "Running foo ..." . My preferred way of fixing this is to make go's testsuite print out "..." . We have a similar patch for glibc [1]. [1] https://sourceware.org/pipermail/libc-alpha/2023-June/148702.html -- Maxim Kuvyrkov https://www.linaro.org > On Sep 26, 2023, at 19:46, Bernhard Reutner-Fischer > wrote: > > Hi Maxim! > > On Mon, 5 Jun 2023 18:06:25 +0400 > Maxim Kuvyrkov via Gcc-patches wrote: > >>> On Jun 3, 2023, at 19:17, Jeff Law wrote: >>> >>> On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote: This patch adds tracking of current testsuite "tool" and "exp" to the processing of .sum files. This avoids aliasing between tests from different testsuites with same name+description. E.g., this is necessary for testsuite/c-c++-common, which is ran for both gcc and g++ "tools". This patch changes manifest format from ... FAIL: gcc_test FAIL: g++_test ... to ... === gcc tests === Running gcc/foo.exp ... FAIL: gcc_test === gcc Summary == === g++ tests === Running g++/bar.exp ... FAIL: g++_test === g++ Summary == . The new format uses same formatting as DejaGnu's .sum files to specify which "tool" and "exp" the test belongs to. >>> I think the series is fine. You're not likely to hear from Diego or Doug I >>> suspect, I don't think either are involved in GNU stuff anymore. >>> >> >> Thanks, Jeff. I'll wait for a couple of days and will merge if there are no >> new comments. > > Maxim, may i ask you to have a look at the following problem, please? > > ISTM that your exp code does not work as expected for go, maybe you > forgot to test the changes with go enabled? > > Ever since your changes in summer i see the following: > > gcc-14.mine$ > /scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py > --clean_build ../gcc-14.orig/ > Getting actual results from build directory . > ./gcc/testsuite/go/go.sum > ./gcc/testsuite/gcc/gcc.sum > ./gcc/testsuite/objc/objc.sum > ./gcc/testsuite/jit/jit.sum > ./gcc/testsuite/gdc/gdc.sum > ./gcc/testsuite/gnat/gnat.sum > ./gcc/testsuite/ada/acats/acats.sum > ./gcc/testsuite/g++/g++.sum > ./gcc/testsuite/obj-c++/obj-c++.sum > ./gcc/testsuite/rust/rust.sum > ./gcc/testsuite/gfortran/gfortran.sum > ./x86_64-pc-linux-gnu/libgomp/testsuite/libgomp.sum > ./x86_64-pc-linux-gnu/libphobos/testsuite/libphobos.sum > ./x86_64-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum > ./x86_64-pc-linux-gnu/libffi/testsuite/libffi.sum > ./x86_64-pc-linux-gnu/libitm/testsuite/libitm.sum > ./x86_64-pc-linux-gnu/libgo/libgo.sum > ./x86_64-pc-linux-gnu/libatomic/testsuite/libatomic.sum > ./gotools/gotools.sum > .sum file seems to be broken: tool="gotools", exp="None", summary_line="FAIL: > TestScript" > Traceback (most recent call last): > File > "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", > line 732, in >retval = Main(sys.argv) > File > "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", > line 721, in Main >retval = CompareBuilds() > File > "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", > line 622, in CompareBuilds >actual = GetResults(sum_files) > File > "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", > line 466, in GetResults >build_results.update(ParseSummary(sum_fname)) > File > "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", > line 405, in ParseSummary >result = result_set.MakeTestResult(line, ordinal) > File > "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", > line 239, in MakeTestResult >return TestResult(summary_line, ordinal, > File > "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", > line 151, in __init__ >raise > RuntimeError: No active exception to reraise > > > The problem seems to be that gotools.sum does not mention any ".exp" > files. > > $ grep "Running " gotools/gotools.sum > Running cmd/go > Running runtime > Running cgo > Running carchive > Running cmd/vet > Running embed > $ grep -c "\.exp" gotools/gotools.sum > 0 > > The .sum files looks like this: > ---8<--- > Test Run By foo on Tue Sep 26 14:46:48 CEST 2023 > Native configuration is x86_64-foo-linux-gnu > >=== gotools tests === > > Running cmd/go > UNTESTED: TestAccidentalGitCheckout > PASS: TestAlwaysLinkSysoFiles > ... > UNTESTED: TestParallelTest > FAIL: TestScript > ... > ---8<--- > > May i ask you to have a look, please? > > TIA,
Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing
Hi Maxim! On Mon, 5 Jun 2023 18:06:25 +0400 Maxim Kuvyrkov via Gcc-patches wrote: > > On Jun 3, 2023, at 19:17, Jeff Law wrote: > > > > On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote: > >> This patch adds tracking of current testsuite "tool" and "exp" > >> to the processing of .sum files. This avoids aliasing between > >> tests from different testsuites with same name+description. > >> E.g., this is necessary for testsuite/c-c++-common, which is ran > >> for both gcc and g++ "tools". > >> This patch changes manifest format from ... > >> > >> FAIL: gcc_test > >> FAIL: g++_test > >> > >> ... to ... > >> > >> === gcc tests === > >> Running gcc/foo.exp ... > >> FAIL: gcc_test > >> === gcc Summary == > >> === g++ tests === > >> Running g++/bar.exp ... > >> FAIL: g++_test > >> === g++ Summary == > >> . > >> The new format uses same formatting as DejaGnu's .sum files > >> to specify which "tool" and "exp" the test belongs to. > > I think the series is fine. You're not likely to hear from Diego or Doug I > > suspect, I don't think either are involved in GNU stuff anymore. > > > > Thanks, Jeff. I'll wait for a couple of days and will merge if there are no > new comments. Maxim, may i ask you to have a look at the following problem, please? ISTM that your exp code does not work as expected for go, maybe you forgot to test the changes with go enabled? Ever since your changes in summer i see the following: gcc-14.mine$ /scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py --clean_build ../gcc-14.orig/ Getting actual results from build directory . ./gcc/testsuite/go/go.sum ./gcc/testsuite/gcc/gcc.sum ./gcc/testsuite/objc/objc.sum ./gcc/testsuite/jit/jit.sum ./gcc/testsuite/gdc/gdc.sum ./gcc/testsuite/gnat/gnat.sum ./gcc/testsuite/ada/acats/acats.sum ./gcc/testsuite/g++/g++.sum ./gcc/testsuite/obj-c++/obj-c++.sum ./gcc/testsuite/rust/rust.sum ./gcc/testsuite/gfortran/gfortran.sum ./x86_64-pc-linux-gnu/libgomp/testsuite/libgomp.sum ./x86_64-pc-linux-gnu/libphobos/testsuite/libphobos.sum ./x86_64-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum ./x86_64-pc-linux-gnu/libffi/testsuite/libffi.sum ./x86_64-pc-linux-gnu/libitm/testsuite/libitm.sum ./x86_64-pc-linux-gnu/libgo/libgo.sum ./x86_64-pc-linux-gnu/libatomic/testsuite/libatomic.sum ./gotools/gotools.sum .sum file seems to be broken: tool="gotools", exp="None", summary_line="FAIL: TestScript" Traceback (most recent call last): File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 732, in retval = Main(sys.argv) File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 721, in Main retval = CompareBuilds() File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 622, in CompareBuilds actual = GetResults(sum_files) File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 466, in GetResults build_results.update(ParseSummary(sum_fname)) File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 405, in ParseSummary result = result_set.MakeTestResult(line, ordinal) File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 239, in MakeTestResult return TestResult(summary_line, ordinal, File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 151, in __init__ raise RuntimeError: No active exception to reraise The problem seems to be that gotools.sum does not mention any ".exp" files. $ grep "Running " gotools/gotools.sum Running cmd/go Running runtime Running cgo Running carchive Running cmd/vet Running embed $ grep -c "\.exp" gotools/gotools.sum 0 The .sum files looks like this: ---8<--- Test Run By foo on Tue Sep 26 14:46:48 CEST 2023 Native configuration is x86_64-foo-linux-gnu === gotools tests === Running cmd/go UNTESTED: TestAccidentalGitCheckout PASS: TestAlwaysLinkSysoFiles ... UNTESTED: TestParallelTest FAIL: TestScript ... ---8<--- May i ask you to have a look, please? TIA,
Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing
> On Jun 3, 2023, at 19:17, Jeff Law wrote: > > On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote: >> This patch adds tracking of current testsuite "tool" and "exp" >> to the processing of .sum files. This avoids aliasing between >> tests from different testsuites with same name+description. >> E.g., this is necessary for testsuite/c-c++-common, which is ran >> for both gcc and g++ "tools". >> This patch changes manifest format from ... >> >> FAIL: gcc_test >> FAIL: g++_test >> >> ... to ... >> >> === gcc tests === >> Running gcc/foo.exp ... >> FAIL: gcc_test >> === gcc Summary == >> === g++ tests === >> Running g++/bar.exp ... >> FAIL: g++_test >> === g++ Summary == >> . >> The new format uses same formatting as DejaGnu's .sum files >> to specify which "tool" and "exp" the test belongs to. > I think the series is fine. You're not likely to hear from Diego or Doug I > suspect, I don't think either are involved in GNU stuff anymore. > Thanks, Jeff. I'll wait for a couple of days and will merge if there are no new comments. Kind regards, -- Maxim Kuvyrkov https://www.linaro.org
Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing
On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote: This patch adds tracking of current testsuite "tool" and "exp" to the processing of .sum files. This avoids aliasing between tests from different testsuites with same name+description. E.g., this is necessary for testsuite/c-c++-common, which is ran for both gcc and g++ "tools". This patch changes manifest format from ... FAIL: gcc_test FAIL: g++_test ... to ... === gcc tests === Running gcc/foo.exp ... FAIL: gcc_test === gcc Summary == === g++ tests === Running g++/bar.exp ... FAIL: g++_test === g++ Summary == . The new format uses same formatting as DejaGnu's .sum files to specify which "tool" and "exp" the test belongs to. I think the series is fine. You're not likely to hear from Diego or Doug I suspect, I don't think either are involved in GNU stuff anymore. jeff
[PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing
This patch adds tracking of current testsuite "tool" and "exp"
to the processing of .sum files. This avoids aliasing between
tests from different testsuites with same name+description.
E.g., this is necessary for testsuite/c-c++-common, which is ran
for both gcc and g++ "tools".
This patch changes manifest format from ...
FAIL: gcc_test
FAIL: g++_test
... to ...
=== gcc tests ===
Running gcc/foo.exp ...
FAIL: gcc_test
=== gcc Summary ==
=== g++ tests ===
Running g++/bar.exp ...
FAIL: g++_test
=== g++ Summary ==
.
The new format uses same formatting as DejaGnu's .sum files
to specify which "tool" and "exp" the test belongs to.
---
.../testsuite-management/validate_failures.py | 137 +++---
1 file changed, 115 insertions(+), 22 deletions(-)
diff --git a/contrib/testsuite-management/validate_failures.py
b/contrib/testsuite-management/validate_failures.py
index 43d9d50af8d..94ba2e58b51 100755
--- a/contrib/testsuite-management/validate_failures.py
+++ b/contrib/testsuite-management/validate_failures.py
@@ -64,6 +64,16 @@ import sys
_VALID_TEST_RESULTS = [ 'FAIL', 'UNRESOLVED', 'XPASS', 'ERROR' ]
_VALID_TEST_RESULTS_REX = re.compile("%s" % "|".join(_VALID_TEST_RESULTS))
+# Formats of .sum file sections
+_TOOL_LINE_FORMAT = '\t\t=== %s tests ===\n'
+_EXP_LINE_FORMAT = '\nRunning %s ...\n'
+_SUMMARY_LINE_FORMAT = '\n\t\t=== %s Summary ===\n'
+
+# ... and their compiled regexs.
+_TOOL_LINE_REX = re.compile('^\t\t=== (.*) tests ===\n')
+_EXP_LINE_REX = re.compile('^Running (.*\.exp) \.\.\.\n')
+_SUMMARY_LINE_REX = re.compile('^\t\t=== (.*) Summary ===\n')
+
# Subdirectory of srcdir in which to find the manifest file.
_MANIFEST_SUBDIR = 'contrib/testsuite-management'
@@ -111,9 +121,11 @@ class TestResult(object):
ordinal: Monotonically increasing integer.
It is used to keep results for one .exp file sorted
by the order the tests were run.
+tool: Top-level testsuite name (aka "tool" in DejaGnu parlance) of the
test.
+exp: Name of .exp testsuite file.
"""
- def __init__(self, summary_line, ordinal=-1):
+ def __init__(self, summary_line, ordinal, tool, exp):
try:
(self.attrs, summary_line) = SplitAttributesFromSummaryLine(summary_line)
try:
@@ -125,6 +137,12 @@ class TestResult(object):
print('Failed to parse summary line: "%s"' % summary_line)
raise
self.ordinal = ordinal
+ if tool == None or exp == None:
+# .sum file seem to be broken. There was no "tool" and/or "exp"
+# lines preceding this result.
+raise
+ self.tool = tool
+ self.exp = exp
except ValueError:
Error('Cannot parse summary line "%s"' % summary_line)
@@ -133,14 +151,27 @@ class TestResult(object):
self.state, summary_line, self))
def __lt__(self, other):
-return (self.name < other.name or
-(self.name == other.name and self.ordinal < other.ordinal))
+if (self.tool != other.tool):
+ return self.tool < other.tool
+if (self.exp != other.exp):
+ return self.exp < other.exp
+if (self.name != other.name):
+ return self.name < other.name
+return self.ordinal < other.ordinal
def __hash__(self):
-return hash(self.state) ^ hash(self.name) ^ hash(self.description)
-
+return (hash(self.state) ^ hash(self.tool) ^ hash(self.exp)
+^ hash(self.name) ^ hash(self.description))
+
+ # Note that we don't include "attrs" in this comparison. This means that
+ # result entries "FAIL: test" and "flaky | FAIL: test" are considered
+ # the same. Therefore the ResultSet will preserve only the first occurence.
+ # In practice this means that flaky entries should preceed expected fails
+ # entries.
def __eq__(self, other):
return (self.state == other.state and
+self.tool == other.tool and
+self.exp == other.exp and
self.name == other.name and
self.description == other.description)
@@ -174,6 +205,43 @@ class TestResult(object):
return now > expiration_date
+class ResultSet(set):
+ """Describes a set of DejaGNU test results.
+ This set can be read in from .sum files or emitted as a manifest.
+
+ Attributes:
+current_tool: Name of the current top-level DejaGnu testsuite.
+current_exp: Name of the current .exp testsuite file.
+ """
+
+ def __init__(self):
+super().__init__()
+self.ResetToolExp()
+
+ def ResetToolExp(self):
+self.current_tool = None
+self.current_exp = None
+
+ def MakeTestResult(self, summary_line, ordinal=-1):
+return TestResult(summary_line, ordinal,
+ self.current_tool, self.current_exp)
+
+ def Print(self, outfile=sys.stdout):
+current_tool = None
+current_exp = None
+
+for result in sorted(self):
+ if current_tool != result.tool:
+current_tool = result.tool
+
