On 1/14/21 2:22 PM, Rainer Orth wrote:
Hi Martin,
* Besides, the test outcomes are not generic message facilities but are
supposed to follow a common format:
<result>: <testname> [<subtest>]
with <testname> the pathname to the test relative to (in this case)
gcc/testsuite. In this case, this might be something like
UNSUPPORTED: g++.dg/gcov/pr98273.C run-gcov-pytest
Currently, you don't have the pathname in run-gcov-pytest, though.
All right, now one will see:
UNSUPPORTED: g++.dg/gcov/pr98273.C run-gcov-pytest could not find Python
interpreter and (or) pytest module
Hello.
please shorten this quite a bit: maybe
... run-gcov-pytest python3 pytest missing
Sure, done.
* If we now have an (even optional) dependency on python/pytest, this
(with the exact versions and use) needs to be documented in
install.texi.
Done that.
+be installed. Some optional tests also require Python3 and pytest module.
It would be better to be more specific here. Or would Python 3.0 and
pytest 2.0.0 do ;-)
I would leave it as it is. Python3 is a well established term. About pytest:
I don't know how to investigate a minimal version right now.
* On to the implementation: your test for the presence of pytest is
wrong:
set result [remote_exec host "pytest -m pytest --version"]
has nothing to do with what you actually use later: on all of Fedora
29, Ubuntu 20.04, and Solaris 11.4 (with a caveat) pytest is Python
2.7 based, but you don't check that. It is well possible that pytest
for 2.7 is installed, but pytest for Python 3.x isn't.
Besides, while Solaris 11.4 does bundle pytest, they don't deliver
pytest, but only py.test due to a conflict with a different pytest from
logilab-common, cf. https://github.com/pytest-dev/pytest/issues/1833.
This is immaterial, however, since what you actually run is
spawn -noecho python3 -m pytest --color=no -rA -s --tb=no
$srcdir/$subdir/$pytest_script
So you should just run python3 -m pytest --version instead to check
for the presence of the version you're going to use.
Btw., there's a mess with pytest on Fedora 29: running the above gives
I must confirm this is mess. I definitely don't want to support Python2 and
I think
the best way would be to use 'env python3', hope it's portable enough.
@David: What do you think?
As I mentioned, it's not: Solaris 11.3 has no python3, only (for the 3.x
series) python3.4.
However, I don't understand what you expect to gain from running
$ env python3
rather than just
$ python3
(or a suitable Python 3.x version by any name)?
All right, let's replace it just with 'python3'.
I just had a quick look and the autoconf-archive has AX_PYTHON which
claims to do that:
https://www.gnu.org/software/autoconf-archive/ax_python.html
Unfortunately, it doesn't know about Python 3.8+ yet.
When running the test on Solaris 11.4 (with the bundled pytest 4.4.0),
I get
============================= test session starts
==============================
platform sunos5 -- Python 3.7.9, pytest-4.4.0, py-1.8.0, pluggy-0.9.0
rootdir: /vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov
collected 2 items
../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py
..
=========================== 2 passed in 0.04 seconds
===========================
while 4.6.9 on Linux gives
============================= test session starts
==============================
platform linux -- Python 3.8.2, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
rootdir: /vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov
collected 2 items
../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py
..
=========================== short test summary info
============================
PASSED
../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_basics
PASSED
../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_lines
=========================== 2 passed in 0.17 seconds
===========================
Obviously pytest -rA was introduced only after 4.4.0 and the 'A' is
silently ignored. Fortunately, I can just use -rap instead which
works with both versions.
This will be fixed by this:
env python3 -m pytest --color=no -rA -s --tb=no --version
No, as I already wrote: pytest 4.4.0 silently ignores -rA and doesn't
print the PASSED (or FAILED) lines. With both versions, pytest -rap
worked for me instead.
Ah, all right, I fixed that.
After this has been processed by gcov.exp, g++.sum contains
PASS:
../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_basic
PASS:
../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_line
which is again completely wrong in light of what I wrote above on the
format of test names: it lacks the testname part completely and
contains absolute pathnames which makes it impossible to compare
testresults from different systems. Instead, there should be some
sort of tag, perhaps patterned after what the various scan-* functions
do.
Fixed that too.
Thanks; I'll give the patch a fresh whirl tonight.
I'll appreciate your help. Let me know how it goes with the attached patch.
Thanks,
Martin
Rainer
>From 22862617f972109a46b944f90d9ff5e4979960a3 Mon Sep 17 00:00:00 2001
From: Martin Liska <mli...@suse.cz>
Date: Thu, 14 Jan 2021 09:09:32 +0100
Subject: [PATCH] Pytest in tests: improve
gcc/ChangeLog:
* doc/install.texi: Document that some tests need pytest module.
* doc/sourcebuild.texi: Likewise.
gcc/testsuite/ChangeLog:
* lib/gcov.exp: Use 'env python3' for execution of pytests.
Check that pytest accepts all needed options first.
Improve formatting of PASS/FAIL lines.
---
gcc/doc/install.texi | 2 +-
gcc/doc/sourcebuild.texi | 4 ++++
gcc/testsuite/lib/gcov.exp | 31 ++++++++++++++++++++++++-------
3 files changed, 29 insertions(+), 8 deletions(-)
diff --git a/gcc/doc/install.texi b/gcc/doc/install.texi
index 17b5382010e..4c38244ae58 100644
--- a/gcc/doc/install.texi
+++ b/gcc/doc/install.texi
@@ -2990,7 +2990,7 @@ Second, you must have the testing tools installed. This includes
the DejaGnu site has links to these. For running the BRIG frontend
tests, a tool to assemble the binary BRIGs from HSAIL text,
@uref{https://github.com/HSAFoundation/HSAIL-Tools/,,HSAILasm} must
-be installed.
+be installed. Some optional tests also require Python3 and pytest module.
If the directories where @command{runtest} and @command{expect} were
installed are not in the @env{PATH}, you may need to set the following
diff --git a/gcc/doc/sourcebuild.texi b/gcc/doc/sourcebuild.texi
index 3d0873dd074..b9cbe21a4bb 100644
--- a/gcc/doc/sourcebuild.texi
+++ b/gcc/doc/sourcebuild.texi
@@ -3092,6 +3092,10 @@ Check line counts in @command{gcov} tests.
@item run-gcov [branches] [calls] @{ @var{opts} @var{sourcefile} @}
Check branch and/or call counts, in addition to line counts, in
@command{gcov} tests.
+
+@item run-gcov-pytest @{ @var{sourcefile} @var{pytest_file} @}
+Check output of @command{gcov} intermediate format with a pytest
+script.
@end table
@subsubsection Clean up generated test files
diff --git a/gcc/testsuite/lib/gcov.exp b/gcc/testsuite/lib/gcov.exp
index 4bcab1d3f1d..f3309653b77 100644
--- a/gcc/testsuite/lib/gcov.exp
+++ b/gcc/testsuite/lib/gcov.exp
@@ -247,6 +247,19 @@ proc verify-calls { testname testcase file } {
return $failed
}
+proc gcov-pytest-format-line { args } {
+ global subdir
+
+ set testcase [lindex $args 0]
+ set pytest_script [lindex $args 1]
+ set output_line [lindex $args 2]
+
+ set index [string first "::" $output_line]
+ set test_output [string range $output_line [expr $index + 2] [string length $output_line]]
+
+ return "$subdir/$testcase ${pytest_script}::${test_output}"
+}
+
# Call by dg-final to run gcov --json-format which produces a JSON file
# that is later analysed by a pytest Python script.
# We pass filename of a test via GCOV_PATH environment variable.
@@ -261,30 +274,34 @@ proc run-gcov-pytest { args } {
set testcase [remote_download host $testcase]
set result [remote_exec host $GCOV "$testcase -i"]
- set result [remote_exec host "pytest -m pytest --version"]
+ set pytest_cmd "python3 -m pytest --color=no -rap -s --tb=no"
+ set result [remote_exec host "$pytest_cmd --version"]
set status [lindex $result 0]
if { $status != 0 } then {
- unresolved "could not find Python interpreter and (or) pytest module for $testcase"
+ unsupported "$subdir/$testcase run-gcov-pytest python3 pytest missing"
return
}
set pytest_script [lindex $args 1]
setenv GCOV_PATH $testcase
- verbose "pytest_script: $pytest_script" 2
- spawn -noecho python3 -m pytest --color=no -rA -s --tb=no $srcdir/$subdir/$pytest_script
+ verbose "pytest_script: $srcdir $subdir $pytest_script" 2
+ spawn -noecho python3 -m pytest --color=no -rap -s --tb=no $srcdir/$subdir/$pytest_script
set prefix "\[^\r\n\]*"
expect {
-re "FAILED($prefix)\[^\r\n\]+\r\n" {
- fail "$expect_out(1,string)"
+ set output [gcov-pytest-format-line $testcase $pytest_script $expect_out(1,string)]
+ fail $output
exp_continue
}
-re "ERROR($prefix)\[^\r\n\]+\r\n" {
- fail "$expect_out(1,string)"
+ set output [gcov-pytest-format-line $testcase $pytest_script $expect_out(1,string)]
+ fail $output
exp_continue
}
-re "PASSED($prefix)\[^\r\n\]+\r\n" {
- pass "$expect_out(1,string)"
+ set output [gcov-pytest-format-line $testcase $pytest_script $expect_out(1,string)]
+ pass $output
exp_continue
}
}
--
2.29.2
