Hello,
Thanks for the reply. :-)
Specifically regarding:
> However, I think you’ll have better success if you model each property as
a
> separate RSpec example, and then have that RSpec example delegate to a
bit of
> code you’ve written that dynamically generates and runs as many
assertions as
> necessary against as many input examples as necessary — and then reports
the
> results back to RSpec in the form of no exception (if the property passed
all
> checks) or an exception with a good, detailed message if the property
failed a
> check.
Understood. That's pretty much how you'd use, say, Rantly. For example:
specify "+ and - reverse each other" do
property_of {
integer
}.check { |n|
expect(n + 1 - 1).to eq n
}
end
That's fine and all (neatened up so you have a shortcut with, say, property
"+ and - ..." or something), but the thing I was really hoping to keep was
the running of the before and after hooks after each property test.
At the moment, as soon as you try something with ActiveRecord in a way that
hits the database, you then only get the benefit of your existing "RSpec
Example lifecycle" before and after the entire run of properties. Same with
any setup for Rack testing, for example.
(I *could* come up with my own mini-DSL that'd let you specify before/after
property runs, with the existing before/after(:each) acting effectively as
before(:all) for the properties, but I was very much hoping to avoid
something like that.)
Thanks again,
Rob
On Tue, Sep 1, 2015 at 4:56 AM, Myron Marston <[email protected]>
wrote:
> Hey Rob,
>
> Sounds like an interesting, useful project!
>
> Dynamically generating examples works fine *as long as they are all
> generated during the “load spec files” phase, before the first spec runs*.
> Defining additional examples while the examples run isn’t going to work
> very well. A lot of RSpec’s features (e.g. randomization and metadata
> filtering) need to be able to get the complete list of examples *before*
> the first one runs so that these features can do their thing. In addition,
> other features in 3.3 (bisect and —only-failures) rely the example IDs
> being stable — that is, spec/foo_spec.rb[1:10] needs to consistently
> reference the same example (in this case, the 10th example in the 1st
> example group in spec/foo_spec.rb) on each run of the suite to work
> properly. If you are generating additional examples as the specs run the
> ids may not be consistent and you’re going to break some of RSpec’s
> features.
>
> My suggestion is to change how you model the problem. From what you
> described, you are modeling each assertion as a separate RSpec example.
> This is natural, given that each assertion represents an example of the
> property being verified and RSpec has something called an example as well.
> However, I think you’ll have better success if you model each property as a
> separate RSpec example, and then have that RSpec example delegate to a bit
> of code you’ve written that dynamically generates and runs as many
> assertions as necessary against as many input examples as necessary — and
> then reports the results back to RSpec in the form of no exception (if the
> property passed all checks) or an exception with a good, detailed message
> if the property failed a check.
>
> Does that make sense?
>
> BTW, I’m leaving for a long vacation tonight and will have sporadic access
> to email so I may not see or respond to replies for quite some time. I’m
> sure others can help out, though :).
>
> HTH,
> Myron
>
>
> On Sun, Aug 30, 2015 at 4:05 PM, Rob Howard <[email protected]> wrote:
>
>> Hello,
>>
>> I'm looking to write a library that adds property-based testing to RSpec,
>> in the style of QuickCheck[1].
>>
>> The particular difficulties of this integration is that there there are a
>> number of repeated tests, these tests being generated or cut short on the
>> fly, as opposed to being all being generated in advance.
>>
>> For example, when testing that a sorting function works correctly (eg.
>> [3,1,2].my_sort == [1,2,3]), I'd generate 100 tests with different arrays
>> of items in ever-increasing sizes. If one of the tests were to fail, I'd
>> want to stop running further tests, and then swap over to a separate
>> "shrinking" phase where I generate further tests with the failing output
>> until I can get the smallest input I can generate to fail the example.
>>
>> The test runs could look like (each line being an input, and an output of
>> a test run with its result):
>>
>> Initial run (up to 100 attempts):
>> [] == [], Passed
>> [1] == [1], Passed
>> [2,1] == [1,2], Passed
>> [2,1,9,4,5,5,6,7] == [1,2,4,5,6,7,9], Failed! Stopping here; no further
>> data generation to be done.
>>
>> Shrinking failed value...
>> [2,1,9,4,5,5,6] == [1,2,4,5,6,9], Failed
>> [2,1,9,4,5,5] == [1,2,4,5,9], Failed
>> [2,1,9,4,5] == [1,2,4,5,9], Passed
>>
>> Failed, with smallest known failing value: [2,1,9,4,5,5]
>>
>> To do the above with RSpec means being able to, in either an around()
>> block or in an example itself, add further examples to an example group to
>> run, on the fly. This doesn't appear to be possible. Even directly
>> monkeying with an example.example_group.examples list is already too late;
>> nothing picks up the change.
>>
>> I've had the suggestion of having the ExampleGroup examples be an
>> iterator instead of a plain list. Would I be on the right track there, or
>> does anyone have any other suggestions?
>>
>> (I'm particularly interested in an RSpec integration so as to take
>> advantage of the existing test integrations people have, eg. database
>> resets for ActiveRecord models, Capybara Rack-based tests, etc. There are
>> other Quickcheck ports/adaptations (eg. Rantly[2], Rubycheck[3],
>> Queencheck[4]), yet none of them have this RSpec lifecycle integration; the
>> only thing that *does* is Generative[5], which just replicates single
>> examples a certain number of times, with none of the early-stop or
>> shrinking smarts that I'm trying to implement.)
>>
>> Thanks,
>> Rob
>>
>> 1.
>> https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing
>> 2. https://github.com/hayeah/rantly
>> 3. https://github.com/mcandre/rubycheck
>> 4. https://github.com/rosylilly/QueenCheck
>> 5. https://github.com/justincampbell/generative
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "rspec" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/rspec/CALx9tg3ixVUQOrnu6yCdpr1TSWQhNRW_ohOJn8p0X0LCPNCE0w%40mail.gmail.com
>> <https://groups.google.com/d/msgid/rspec/CALx9tg3ixVUQOrnu6yCdpr1TSWQhNRW_ohOJn8p0X0LCPNCE0w%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "rspec" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/rspec/CADUxQmuQK9riLFoZ50pEJpERaY97FU7Yuza9mkmzbNw%2BR9uMSg%40mail.gmail.com
> <https://groups.google.com/d/msgid/rspec/CADUxQmuQK9riLFoZ50pEJpERaY97FU7Yuza9mkmzbNw%2BR9uMSg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"rspec" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/rspec/CALx9tg3DaWNt%2BsG6-5uxtQ_cVxna6DB4uxCORRPhqT8gF4E5ew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
