I've played around now with implementing specs for a couple of my projects.
What I'm finding is that writing specs to check the inputs of a function is easy-ish, but building useful random generators is very hard -- in my projects, this seems too hard to do with any reasonable amount of effort. This isn't due to anything inherent in clojure.spec, it's just that for non-trivial functions, coming up with relevant random input is a very hard problem. For example, let's say I have a function that takes two integers. Odds are that not any two randomly chosen integers will work. Some combinations of integers are non-sensical and could trigger an error, other combinations may cause the function to run way too long. As a concrete example, I just tried to spec out a SAT solver (which tries to solve NP-complete problems). The input should be a vector of vectors of ints, but many combinations of inputs will just run forever. How to generate "good" SAT problems? I have no idea. So for the most part, I've ignored the generation aspect of specs because it just feels like too much work. But without the generators, clojure.spec's utility is significantly diminished. 1. No way to test function output specs. For documentation purposes, I want to have an output spec on my function. However, as far as I know, after instrumentation-triggered checking of output specs was removed a couple of alphas ago, the only way remaining to check against output specs is to use randomly generated tests. So if I can't make good generators, I have no way to confirm that my output spec works the way I think it does. My documentation could be totally out of touch with reality, and that displeases me. 2. Limited ability for testing that functions take and receive what you expect. Whereas a static type system can prove certain properties about whether functions are called with valid inputs, with spec, you only get those guarantees if you pump a function with enough valid random data to trigger the function calling all its callees with interesting combinations of data. But if I use the naive generators, the function will never even complete with most of the randomly generated input, let alone call other functions in a useful way. And in many cases I don't see how to generate something of better quality. So looking back at my experiments, my preliminary impression is that by adding specs to my public APIs, I've gained some useful documentation, and I've given users the ability to instrument functions in order to get high-quality assertion-checking of the inputs. In some cases, the error messages for bad input when instrumented are also more useful than I would have otherwise gotten, but in some cases they aren't. Overall, I've struggled to write generators, and without them, the value proposition isn't as great. One other issue I've had, unrelated to generators, is that I'm struggling to express higher-order type constraints, for example, this function takes a vector v1 of anything and a vector v2 of anything, but the type of the things in vector v1 better match the type of the things in vector v2. What are other people finding? Do you find it easy/hard to write generators? (If you think it's easy, I'd love to know your tricks). Do you find it easy/hard to read specs as a form of documentation about the contract of a function? Do you find it frustrating that there's no way to turn on instrumentation of function outputs for manual testing? Do you feel your generators are providing sufficient code coverage when exercising callee functions? -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
