This may not be exactly what you want, but you could generate the spec files and then shell out to rspec to run it. You'd probably then tell rspec to use a custom formatter and have it fail on first failure. The exit code would tell you if it passed and the output to stdout could be captured to read in any metadata necessary to decide what to do next. Allen Madsen http://www.allenmadsen.com
On Mon, Aug 31, 2015 at 11:00 PM, Rob Howard <[email protected]> wrote: > Hello, > > Thanks for the reply. :-) > > Specifically regarding: >> However, I think you’ll have better success if you model each property as >> a >> separate RSpec example, and then have that RSpec example delegate to a bit >> of >> code you’ve written that dynamically generates and runs as many assertions >> as >> necessary against as many input examples as necessary — and then reports >> the >> results back to RSpec in the form of no exception (if the property passed >> all >> checks) or an exception with a good, detailed message if the property >> failed a >> check. > > Understood. That's pretty much how you'd use, say, Rantly. For example: > > specify "+ and - reverse each other" do > property_of { > integer > }.check { |n| > expect(n + 1 - 1).to eq n > } > end > > That's fine and all (neatened up so you have a shortcut with, say, property > "+ and - ..." or something), but the thing I was really hoping to keep was > the running of the before and after hooks after each property test. > > At the moment, as soon as you try something with ActiveRecord in a way that > hits the database, you then only get the benefit of your existing "RSpec > Example lifecycle" before and after the entire run of properties. Same with > any setup for Rack testing, for example. > > (I *could* come up with my own mini-DSL that'd let you specify before/after > property runs, with the existing before/after(:each) acting effectively as > before(:all) for the properties, but I was very much hoping to avoid > something like that.) > > Thanks again, > Rob > > > > > On Tue, Sep 1, 2015 at 4:56 AM, Myron Marston <[email protected]> > wrote: >> >> Hey Rob, >> >> Sounds like an interesting, useful project! >> >> Dynamically generating examples works fine as long as they are all >> generated during the “load spec files” phase, before the first spec runs. >> Defining additional examples while the examples run isn’t going to work very >> well. A lot of RSpec’s features (e.g. randomization and metadata filtering) >> need to be able to get the complete list of examples before the first one >> runs so that these features can do their thing. In addition, other features >> in 3.3 (bisect and —only-failures) rely the example IDs being stable — that >> is, spec/foo_spec.rb[1:10] needs to consistently reference the same example >> (in this case, the 10th example in the 1st example group in >> spec/foo_spec.rb) on each run of the suite to work properly. If you are >> generating additional examples as the specs run the ids may not be >> consistent and you’re going to break some of RSpec’s features. >> >> My suggestion is to change how you model the problem. From what you >> described, you are modeling each assertion as a separate RSpec example. This >> is natural, given that each assertion represents an example of the property >> being verified and RSpec has something called an example as well. However, I >> think you’ll have better success if you model each property as a separate >> RSpec example, and then have that RSpec example delegate to a bit of code >> you’ve written that dynamically generates and runs as many assertions as >> necessary against as many input examples as necessary — and then reports the >> results back to RSpec in the form of no exception (if the property passed >> all checks) or an exception with a good, detailed message if the property >> failed a check. >> >> Does that make sense? >> >> BTW, I’m leaving for a long vacation tonight and will have sporadic access >> to email so I may not see or respond to replies for quite some time. I’m >> sure others can help out, though :). >> >> HTH, >> Myron >> >> >> On Sun, Aug 30, 2015 at 4:05 PM, Rob Howard <[email protected]> wrote: >>> >>> Hello, >>> >>> I'm looking to write a library that adds property-based testing to RSpec, >>> in the style of QuickCheck[1]. >>> >>> The particular difficulties of this integration is that there there are a >>> number of repeated tests, these tests being generated or cut short on the >>> fly, as opposed to being all being generated in advance. >>> >>> For example, when testing that a sorting function works correctly (eg. >>> [3,1,2].my_sort == [1,2,3]), I'd generate 100 tests with different arrays of >>> items in ever-increasing sizes. If one of the tests were to fail, I'd want >>> to stop running further tests, and then swap over to a separate "shrinking" >>> phase where I generate further tests with the failing output until I can get >>> the smallest input I can generate to fail the example. >>> >>> The test runs could look like (each line being an input, and an output of >>> a test run with its result): >>> >>> Initial run (up to 100 attempts): >>> [] == [], Passed >>> [1] == [1], Passed >>> [2,1] == [1,2], Passed >>> [2,1,9,4,5,5,6,7] == [1,2,4,5,6,7,9], Failed! Stopping here; no further >>> data generation to be done. >>> >>> Shrinking failed value... >>> [2,1,9,4,5,5,6] == [1,2,4,5,6,9], Failed >>> [2,1,9,4,5,5] == [1,2,4,5,9], Failed >>> [2,1,9,4,5] == [1,2,4,5,9], Passed >>> >>> Failed, with smallest known failing value: [2,1,9,4,5,5] >>> >>> To do the above with RSpec means being able to, in either an around() >>> block or in an example itself, add further examples to an example group to >>> run, on the fly. This doesn't appear to be possible. Even directly monkeying >>> with an example.example_group.examples list is already too late; nothing >>> picks up the change. >>> >>> I've had the suggestion of having the ExampleGroup examples be an >>> iterator instead of a plain list. Would I be on the right track there, or >>> does anyone have any other suggestions? >>> >>> (I'm particularly interested in an RSpec integration so as to take >>> advantage of the existing test integrations people have, eg. database resets >>> for ActiveRecord models, Capybara Rack-based tests, etc. There are other >>> Quickcheck ports/adaptations (eg. Rantly[2], Rubycheck[3], Queencheck[4]), >>> yet none of them have this RSpec lifecycle integration; the only thing that >>> *does* is Generative[5], which just replicates single examples a certain >>> number of times, with none of the early-stop or shrinking smarts that I'm >>> trying to implement.) >>> >>> Thanks, >>> Rob >>> >>> 1. >>> https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing >>> 2. https://github.com/hayeah/rantly >>> 3. https://github.com/mcandre/rubycheck >>> 4. https://github.com/rosylilly/QueenCheck >>> 5. https://github.com/justincampbell/generative >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "rspec" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/rspec/CALx9tg3ixVUQOrnu6yCdpr1TSWQhNRW_ohOJn8p0X0LCPNCE0w%40mail.gmail.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "rspec" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/rspec/CADUxQmuQK9riLFoZ50pEJpERaY97FU7Yuza9mkmzbNw%2BR9uMSg%40mail.gmail.com. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "rspec" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/rspec/CALx9tg3DaWNt%2BsG6-5uxtQ_cVxna6DB4uxCORRPhqT8gF4E5ew%40mail.gmail.com. > > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "rspec" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/CAK-y3Cthyi-CYQsvL3zk8_ZaxuA%3DcUS7%3DBoMdb50Y5KELrDGfA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
