Hey all, A few folks had an IRC discussion about improving the trial & hypothesis experience. I took some notes and thought I'd share them here in case anyone else is interested.
We're planning to pick up the discussion next Wednesday. Feel free to join in #twisted-dev on Freenode (we might set up a Google Hangout, details will be on IRC if so). Jean-Paul
* Goals ** Make Twisted / Hypothesis as convenient as Anything Else / Hypothesis. *** trial doesn't discover "runTest" so Hypothesis stateful tests aren't discovery **** Make trial discover "runTest" ***** This might break things ***** Also TestLoader.getTestCaseNames is incompatible with test methods without a test_ prefix (so we need a new interface) **** Make Hypothesis name its test "test_something" ***** Name could be an argument to a stateful TestCase factory *** trial TestCases have extra helper methods not availabe to Hypothesis stateful tests **** Make base TestCase class an argument to a stateful TestCase factory *** setUp / tearDown are called just once even though test body might be called many times; and *** tests that return Deferred are simply incompatible with Hypothesis **** Hypothesis has an "executor" hook (eg new_style_executor) that might be a good place to hang reactor knowledge **** this could also be a place to hang better setUp / tearDown handling ***** this would be incompatible with current operation which has separate setup_example/teardown_example hooks ***** and there's still only a single TestCase instance having methods called on it repeatedly which could be surprising ****** `trial -u` does this already though, so maybe less so for trial users ***** but it might be better, anyway *** Hypothesis does a lot of work within a single test method. **** A consequence is a ton of log data mixed together from runs with different values. ***** It would be nice to be able to see minimal logging from just the most shrunk iteration. **** Maybe there could be a custom HypothesisSuite that knows how repeatedly instantiate a TestCase and run it more normally ***** This replaces the loop inside a test_ method with a loop around something more like TestCase().run() ****** might solve many problems at once ****** confuses things like `trial -j` ****** causes each test run to have a different number of tests, which is weird
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
