On 10/24/2023 1:56 PM, Jacob Hoffman-Andrews via curl-library wrote:
I often have trouble efficiently debugging curl tests, and I wanted
to ask if I'm doing it wrong, and/or if there are improvements possible.
My workflow is like this: If I notice a set of tests are failing, I
will usually run, e.g.:
make; cd tests; ./runtests.pl <http://runtests.pl/> 719
One issue I run into is that `make` must be run in the curl repository
root, while `runtests.pl <http://runtests.pl/>` must be run in the
`tests` directory. Is it possible to make `runtests.pl
<http://runtests.pl/>` agnostic as to the current directory from which
it is run? This would save some overhead changing directory back and
forth.
Another issue is that if tests are running long (for instance, if I
ran `make test` and then changed my mind), hitting Ctrl-C takes a very
long time to shut down the test runner. At least 10 seconds, and often
much more. Sometimes the Ctrl-C doesn't register with the test runner
at all.
Lastly, when tests fail in CI, it can be hard to find the cause. For
instance, in this test run
(https://github.com/curl/curl/actions/runs/6628686909/job/18006346659),
I saw that test 719 failed, so I did Ctrl-F 719 to find it, with no
results. Later I realized that GitHub cuts off super long log output,
so my browser did not have a copy of the lines related to test case
719. Using the "Search logs" text box in GitHub's log viewer did turn
up the relevant log lines. A couple possible fixes here: When tests
fail we could emit a final log line that says to use the search box
rather than the browser's built-in search. Also, one of the major
culprits was the (failing) test1474, which emits ~6000 lines of mostly
"xxxxxx". Perhaps we could suppress or summarize output for certain
tests that have super long output?
You can run individual tests from the top level directory by passing the
arguments to runtests via TFLAGS.
make test TFLAGS=719
make test-full TFLAGS=719
By default 'test' configuration in the top level Makefile [1] will
invoke the tests/Makefile [2] with configuration 'quiet-test'
(./runtests.pl -a -s [tflags]). 'test-full' is invoked as 'full-test'
configuration (./runtests.pl -a -p -r [tflags]).
It is only necessary to run make again from the top level directory if
you have modified curl because a test failed. In that case curl needs to
be rebuilt and the lib/unit tests may need to be rebuilt as well. If
curl does not need to be rebuilt and only tests need to be rebuilt (eg
you modified a libtest but not curl) then you can run make from the
tests directory. If only the XML of a test (ie tests/data dir files) was
changed then, in most cases, you do not need to run make at all to
rebuild anything because the XML is not used by the make system. After
that you run runtests.pl.
Regarding the GitHub Actions CI output, sometimes I can easily find what
I'm looking for and sometimes I can't. I'm not sure if it's my browser
or GitHub. Sometimes I will go to the failed section (like in the URL
you gave) and the expanded output of the failed section (eg run tests)
will be blank, so I have to reload the page. Then, even if I get the
output and I search for the failed test in Chrome (CTRL+F) it often
can't find the test. For example your failed CI job output says:
TESTFAIL: These test cases failed: 371 719 728 1148 1268 1278 1471 1472
2053
So I search for "test 0371" and nothing is found by Chrome. Instead I
have to scroll up until I see test 0371. The text is there but there's
some issue with the asynchronous loading that the text does not always
parse or show properly, or is delayed so it can't be found. A workaround
for that is to view the raw log which will download the whole thing at
once. Click the sprocket icon in that panel choose "View raw logs" then
search for "test 0371" and Chrome finds it instantly.
Regarding the long output, we set it that way for the CI purposely so
that there would be a postmortem of every failed test. Refer to
configuration test-ci/ci-test in the makefiles to see the exact options
used.
[1]: https://github.com/curl/curl/blob/curl-8_4_0/Makefile.am#L197-L216
[2]: https://github.com/curl/curl/blob/curl-8_4_0/tests/Makefile.am#L67-L108
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
