----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviewboard.asterisk.org/r/3489/#review11858 -----------------------------------------------------------
/asterisk/trunk/runtests.py <https://reviewboard.asterisk.org/r/3489/#comment21738> Assert that run_num > 0 - Mark Michelson On April 28, 2014, 8:18 p.m., Matt Jordan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviewboard.asterisk.org/r/3489/ > ----------------------------------------------------------- > > (Updated April 28, 2014, 8:18 p.m.) > > > Review request for Asterisk Developers. > > > Repository: testsuite > > > Description > ------- > > This patch changes up the logging in the Asterisk Test Suite. It dramatically > reduces the amount of logging done in the logs/ directory by default > (although this can always be bumped up), and it redirects the vast majority > of the test suite logs to the actual sandboxed test run directories > themselves. > > When we first added Python logging (it didn't always used to be there!) there > were far fewer tests. Having all tests dump into the same log files actually > felt like a benefit, since during test development (or when developing many > tests) it made it easier to debug test failures. Over time, however, this > approach has run into a number of problems: > (1) Many of the tests are much 'chattier' than they used to be. Having > twisted, starpy, websockets, requests, and other libraries dump information > has greatly increased the number of log messages. Not to mention logging out > every SIPp screen received... > (2) There are a lot more tests than there were when this was added. > (3) Tailing a log file is not always the best way to debug. Sometimes you > have to search through an entire log file for one run. Finding the error > becomes problematic when your editor of choice chokes on the size of the file. > > Hence, this patch. > > Logging now works in the following way: > (1) The test_case.TestCase class now always sets up the logging. The logging > set up done by test_runner was removed (as it only logged out a few messages > before an instance of TestCase was created anyway). > (2) A global logger can still be configured in logger.conf. It now only sets > up a logger of INFO messages and higher. This allows a test executor to watch > which tests are being run, without getting spammed. During test development, > the log message level can be increased to DEBUG. > (3) TestCase now places the Asterisk directories created by the test > execution in a further subfolder, run_N (where N increases with each > execution of that test). Where before you might have: > > tests/my_test/ast1 > tests/my_test/ast2 > tests/my_test/ast3 > tests/my_test/ast4 > > You will now have: > > tests/my_test/run_1/ast1 > tests/my_test/run_1/ast2 > tests/my_test/run_2/ast1 > tests/my_test/run_2/ast2 > > This lets you determine which Asterisk instances belong together, and also > makes it possible for only the erroring test run to be archived when a test > fails (as opposed to every Asterisk directory) > (4) TestCase now creates a DEBUG and INFO log file(s) in the run directory. > These contain the full Asterisk logs for the test run. > > > Diffs > ----- > > /asterisk/trunk/runtests.py 5002 > /asterisk/trunk/logger.conf 5002 > /asterisk/trunk/lib/python/asterisk/test_runner.py 5002 > /asterisk/trunk/lib/python/asterisk/test_case.py 5002 > /asterisk/trunk/lib/python/asterisk/asterisk.py 5002 > > Diff: https://reviewboard.asterisk.org/r/3489/diff/ > > > Testing > ------- > > I've actually been running an instance of the test suite with this set up for > several months, using it for development of several tests and generally > tweaking it. I finally felt like it was "good enough". > > Tested: > * Failing tests archive appropriately > * Crashing Asterisk gets archived > * Global log file is still created with expected message levels > * Test specific log files are created appropriately with expected message > levels > * Tested a pluggable module based test (pbx/dialplan) and a 'traditional' > Python test (channels/SIP/sip_hold) > > > Thanks, > > Matt Jordan > >
-- _____________________________________________________________________ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
