Yes, that is a major selling point. Generative testing is really cool, and you should definitely not be disheartened - it's a tool like any other, with its strong points and weak points (like static typing, too). It's definitely not universally applicable, and even more than standard testing, it requires you to write your code keeping in mind how you will test it. It happens that in my case I believe that static typing is a better fit, but that doesn't mean that it won't work well for your use case. You should definitely be excited to try it and see - if it works well in your case, it's pretty magical.
The limitation in my case is that generative testing works best when testing pure functions. Because so much of my system involves calls out to a huge mutable blob of Java, it doesn't work well for me. i actually don't have a lot of experience with generative testing, but here are two examples from the little experience I have, one which worked well and one which didn't. The one that worked well was testing a binary object serialiser. This was in Java several years ago, so a lot of the complexity in my code was writing generators - spec (via test.check) really helps with this and it would be relatively trivial to achieve nowadays what took me a long time back then. But once I had a generator which generated random objects, I then just ran a bunch of tests which generated an object, serialised it, deserialised the resulting bytes, and checked that the deserialised object was equal to the original one. It was really useful, and caught a whole lot of edge case bugs I would never have caught otherwise. Things to note are that the serialisation and deserialisation are pure functions, and the success condition is very easy to define. One that I have not yet found a good way to test generatively is the paredit code in Cursive, which occasionally suffers from edge case bugs. I'd also like to test code refactorings. However here, the generation is much much harder (I'd have to generate valid code in some way) and then I'd have to randomly refactor it and somehow check that the refactored version is equivalent. Here both the generation and the success condition are extremely hard to define, and the risk is that in your success condition you really just end up reimplementing your original algorithm and then what you're testing is actually that those two algorithms are the same, not necessarily that they do what you expect. I was lucky enough to talk at length to Thomas Arts at Curry On last year and I asked him about this. He had done something similar for Erlang code, and had a very complicated code generator. They generated code that produced some output, then randomly refactored it, compiled both versions, ran them both and checked that the output was the same. His conclusion was that in that case, the tests weren't worth it - they took forever to run and were very brittle. On 22 October 2016 at 02:54, Daniel <doubleagen...@gmail.com> wrote: > > In this sort of situation, a static type system which provides universal > guarantees (this value can never be null) is more useful than a contract > system (no null values have been seen yet for the test inputs you've > tried). There's simply no way I can test all combinations, or reproduce all > combinations that users might have running. > > Isn't a major selling point of generative testing was that it creates > loads of unique cases you can't invent on your own? > > You don't trust it to do that? Is that from personal experience? > Genuinely curious because I am a little excited about using it in a project > at work but this is disheartening. > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
