Colin, I think the points you bring up here are very interesting, and reflect ideas that I have not yet been able to formulate.
I certainly would think there is enough "stuff" in here for a talk, and also some ideas into how to grnerative test "bread and butter" code, as can I, and if so, how, generative-test my REST-api. Can I do it in some more meaningful way than just checking for a non 5xx status code? How do I property base test my functions which take my domain "objects". Do I have to write generators for them, and am I then ending up in the same pain as I do writing mocks in Java? I know that I can compose generators, but I might end up needing users with e.g. genuine Norwegian addresses. These are my questions wrt generative testing in my world. Erik. -- i farta > Den 22. okt. 2016 kl. 23.48 skrev Colin Fleming <colin.mailingl...@gmail.com>: > > Yes, that is a major selling point. Generative testing is really cool, and > you should definitely not be disheartened - it's a tool like any other, with > its strong points and weak points (like static typing, too). It's definitely > not universally applicable, and even more than standard testing, it requires > you to write your code keeping in mind how you will test it. It happens that > in my case I believe that static typing is a better fit, but that doesn't > mean that it won't work well for your use case. You should definitely be > excited to try it and see - if it works well in your case, it's pretty > magical. > > The limitation in my case is that generative testing works best when testing > pure functions. Because so much of my system involves calls out to a huge > mutable blob of Java, it doesn't work well for me. > > i actually don't have a lot of experience with generative testing, but here > are two examples from the little experience I have, one which worked well and > one which didn't. > > The one that worked well was testing a binary object serialiser. This was in > Java several years ago, so a lot of the complexity in my code was writing > generators - spec (via test.check) really helps with this and it would be > relatively trivial to achieve nowadays what took me a long time back then. > But once I had a generator which generated random objects, I then just ran a > bunch of tests which generated an object, serialised it, deserialised the > resulting bytes, and checked that the deserialised object was equal to the > original one. It was really useful, and caught a whole lot of edge case bugs > I would never have caught otherwise. Things to note are that the > serialisation and deserialisation are pure functions, and the success > condition is very easy to define. > > One that I have not yet found a good way to test generatively is the paredit > code in Cursive, which occasionally suffers from edge case bugs. I'd also > like to test code refactorings. However here, the generation is much much > harder (I'd have to generate valid code in some way) and then I'd have to > randomly refactor it and somehow check that the refactored version is > equivalent. Here both the generation and the success condition are extremely > hard to define, and the risk is that in your success condition you really > just end up reimplementing your original algorithm and then what you're > testing is actually that those two algorithms are the same, not necessarily > that they do what you expect. I was lucky enough to talk at length to Thomas > Arts at Curry On last year and I asked him about this. He had done something > similar for Erlang code, and had a very complicated code generator. They > generated code that produced some output, then randomly refactored it, > compiled both versions, ran them both and checked that the output was the > same. His conclusion was that in that case, the tests weren't worth it - they > took forever to run and were very brittle. > >> On 22 October 2016 at 02:54, Daniel <doubleagen...@gmail.com> wrote: >> > In this sort of situation, a static type system which provides universal >> > guarantees (this value can never be null) is more useful than a contract >> > system (no null values have been seen yet for the test inputs you've >> > tried). There's simply no way I can test all combinations, or reproduce >> > all combinations that users might have running. >> >> Isn't a major selling point of generative testing was that it creates loads >> of unique cases you can't invent on your own? >> >> You don't trust it to do that? Is that from personal experience? Genuinely >> curious because I am a little excited about using it in a project at work >> but this is disheartening. >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with your >> first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
