You should look at one of the many testing packages that are out there for further inspiration:
https://groups.google.com/forum/#!searchin/julia-users/testing$20packages/julia-users/fSLuCXpk_1M/wkDLA-ZfOQYJ On Saturday, April 26, 2014 9:49:06 AM UTC-4, Tomas Lycken wrote: > > In many situations my test suites are such that if more than one test > fails, they typically provide more information together than just their > individual results ("the total is greater than the sum of the parts", if > you will), so it takes less time for me to track down the actual problem if > I see all tests that fail, rather than just the first one. This is not > currently the default behavior of the macros in Base.Test: > > using Base.Test > > function throws_error() > error("oops") > end > > @test 2 == 3 > @test 2 == 2 > @test 4 < 3 > @test throws_error() > > Output: > > ERROR: test failed: 2 == 3 > in error at error.jl:21 > in default_handler at test.jl:19 > in do_test at test.jl:39 > in include at boot.jl:244 > while loading [...]/example.jl, in expression starting on line 7 > > Note that no information is given about the third test. > > I suppose this could be remedies using a custom handler, similar to this > example in the > docs<http://docs.julialang.org/en/latest/stdlib/test/#handlers>, > so I started building one. Running tests with this handler is much more > informative, in some aspects, but much less in others: > > using Base.Test > > nonfatal_handler(r::Test.Success) = Test.default_handler(r) > function nonfatal_handler(r::Test.Failure) > print_with_color(:red, "Test FAILED: $(r.expr)\n") > end > function nonfatal_handler(r::Test.Error) > print_with_color(:red, "ERROR during test: $(r.expr)\n") > showerror(STDOUT, r.err, r.backtrace) > end > > function throws_error() > error("oops") > end > > Test.with_handler(nonfatal_handler) do > @test 2 == 3 > @test 2 == 2 > @test 4 < 3 > @test throws_error() > end > > Output: > > Test FAILED: 2 == 3 > Test FAILED: 4 < 3 > ERROR during test: throws_error() > oops > in throws_error at [...]/example.jl:13 > > Now I see all failures, and not just the first one - but I get much less > information about them. In particular, I don't even get a line number in > the test file for test cases that fail but don't throw errors. I'm also not > entirely very about hard-coding STDOUT in the error handler, but I couldn't > figure out a way around it without interrupting control flow. > > This isn't perfect, though - if we re-order the tests, so that the > throwing case is in the middle, control flow is interrupted anyway, and the > tests after don't run. Can this even be fixed without altering the @test > macro(s)? > > Any suggestions on how to improve this? And if this is improved enough to > give substantial information about each case, without interrupting flow, > would it be an interesting addition to the library? > > Best regards > > // Tomas >
