Patrick
This is very good feed back and understanding. Since you are discovering
all this partially documented material you can add the work flows as discussed
below to the developers guide (and put your name on the guide) so the future
developers don't need to guess and extrapolate as you have. Better to have a
little too much in the developers guide than not enough.
Barry
> On Jan 25, 2018, at 5:35 AM, Patrick Sanan <patrick.sa...@gmail.com> wrote:
>
> Awesome!
>
> As a contributor, what I am I supposed to do when a test fails due to some
> error of mine?
> What's the procedure to be able to run a test "under my own control" (be able
> to see what's actually being executed, and what's failing in the diff) so
> that I can fix my error?
> That is, I want to be able to "back out of the test harness" to be able to
> debug a failure myself.
>
> This following is what I would try, given my limited knowledge of how things
> work and after a quick scan of the docs. Is this as intended or is there an
> easier way?
>
> --------------------------------------------------
>
> I make some seemingly-innocuous changes to the code. (In this case I
> literally change nothing in master.)
>
> I want to run all the tests to make sure I didn't break something that I
> don't understand, so I look at the User Manual and copy and run this command:
>
> make -f gmakefile.test test
>
> I go away for a while and come back and see that some (3) tests have failed,
> and that the harness gives me instructions on how to re-run them.
>
> # -------------
> # Summary
> # -------------
> # FAILED ts_tutorials-ex11_adv_2d_quad_0
> diff-sys_classes_viewer_tests-ex4_4 ts_tutorials-ex11_adv_2d_quad_1
> # success 3051/3915 tests (77.9%)
> # failed 3/3915 tests (0.1%)
> # todo 91/3915 tests (2.3%)
> # skip 770/3915 tests (19.7%)
> #
> # To rerun failed tests:
> # /opt/local/bin/gmake -f gmakefile test
> search='ts_tutorials-runex11_adv_2d_quad_0 sys_classes_viewer_tests-runex4_4
> ts_tutorials-runex11_adv_2d_quad'
>
> I want to take a look at the output so I can see what's happening - I see
> that I'm given instructions on how to get more verbose output, so I run one
> of the tests again with V=1 :
>
> /opt/local/bin/gmake V=1 -f gmakefile test
> search='ts_tutorials-runex11_adv_2d_quad_0'
>
> This tells me that this is failing because of a timeout
>
> # Exceeded timeout limit of 60 s
>
> I poke around in the dev manual (note that "make -f gmakefile help" and "make
> -f gmakefile.test help" say nothing about TIMEOUT as promised)
> and eventually run
>
> /opt/local/bin/gmake V=1 TIMEOUT=1000 -f gmakefile test
> search='ts_tutorials-runex11_adv_2d_quad_0'
>
> and get success. Sweet - my assumption that this failed because I'm running a
> debug build on a slow laptop is confirmed.
>
> # -------------
> # Summary
> # -------------
> # success 2/2 tests (100.0%)
> # failed 0/2 tests (0.0%)
> # todo 0/2 tests (0.0%)
> # skip 0/2 tests (0.0%)
>
> I then look at the other failed test.
>
> /opt/local/bin/gmake V=1 -f gmakefile test
> search='sys_classes_viewer_tests-runex4_4'
>
> This one seems to be failing because of the diff:
>
>
> arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests/runex4_4.sh
> -v
> ok sys_classes_viewer_tests-ex4_4 cat ex4_4.tmp | > /dev/null; cat
> ex4a1.tmp > ex4_4.tmp.tmp 2>> runex4_4.err && mv ex4_4.tmp.tmp ex4_4.tmp
> not ok diff-sys_classes_viewer_tests-ex4_4
> /Users/patrick/petsc-master/bin/petscdiff
> /Users/patrick/petsc-master/src/sys/classes/viewer/examples/tests/output/ex4a.out
> ex4_4.tmp > diff-runex4_4.out 2> diff-runex4_4.out
> # 2a3,14
> # > Testing PetscViewerASCIIPrintf 0
> # > Testing PetscViewerASCIIPrintf 1
> # > Testing PetscViewerASCIIPrintf 0
> # > Testing PetscViewerASCIIPrintf 1
> # > Testing PetscViewerASCIIPrintf 0
> # > Testing PetscViewerASCIIPrintf 1
> # > Testing PetscViewerASCIIPrintf 0
> # > Testing PetscViewerASCIIPrintf 1
> # > Testing PetscViewerASCIIPrintf 0
> # > Testing PetscViewerASCIIPrintf 1
> # > Testing PetscViewerASCIIPrintf 0
> # > Testing PetscViewerASCIIPrintf 1
>
> I go to take a look at the output of the test to see if I can run the actual
> executable myself.
>
> cd
> arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests
>
> runex4_4.sh in that directory isn't so informative as far as the command I
> should run. It does tell me where I might look for reference output, though
>
>
> /Users/patrick/petsc-master/src/sys/classes/viewer/examples/tests/output/ex4a.out
>
> If I look in the runex4_4 directory, I find
> sys_classes_viewer_tests-ex4_4.sh which gives me something to copy and paste
> and use my actual knowledge/reasoning to fix:
>
> /Users/patrick/petsc-master/arch-darwin-master-double-debug/bin/mpiexec
> -n 1 ../ex4 -myviewer ascii:ex4a1.tmp::append
>
> (I'm not going to try to debug this failure further, but I'm confident I'd be
> able to given the information I have now)
>
> This is pretty close to what I want, which is a way to "back out of the test
> harness" to be able to debug the failed test myself. I was hoping that
> running this from the PETSc root directory would more directly tell me that:
>
>
> arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests/runex4_4.sh
> -v
>
>
>
>
>
> 2018-01-25 5:47 GMT+01:00 Smith, Barry F. <bsm...@mcs.anl.gov>:
>
> PETSc developers,
>
> We have completed moving all PETSc examples over from the old test
> system (where tests were written in the makefile) to a new system, provided
> by Scott Kruger, where the test rules are written in bottom of the source
> file of the example. Directions for usage and adding new tests can be found
> in the PETSc developers manual
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/developers.pdf chapter 7.
>
> Barry
>
>
>
