I tried your idea of generating the size first, then passing it to the matrix & vector generators. This does seem to work better. The shrunk cases that return are actually worse, but so far it hasn't wedged itself, which is a great improvement. They all return within a few seconds. I don't yet understand the shrinking well enough to know why it's better.
Example shrunk state: :samples ["0" "0" "0" "0" "0" "0" "0" "0" "0"], :probes ["0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0"] (def gen-matrix-size (gen/tuple gen/s-pos-int gen/s-pos-int)) (def gen-mostly-ints (gen/frequency [[9 gen/int] [1 (gen/return Double/NaN)]])) (defn gen-matrix2 [x y] (gen/vector (gen/vector gen-mostly-ints x) y)) (defn gen-names [n] (gen/vector (gen/such-that not-empty gen/string-alpha-numeric) n)) (def gen-tsv2 (gen/bind gen-matrix-size (fn [[x y]] (gen/hash-map :probes (gen-names x) :samples (gen-names y) :matrix (gen-matrix2 x y))))) On Wednesday, November 12, 2014 12:47:37 PM UTC-8, Lucas Bradstreet wrote: > > I'm pretty sure I did encounter the performance problem you're talking > about, but I killed it and re-ran until I hit cases that shrink quickly. > I'm afraid I'm not much help with those, although I agree that the bad > shrinking is probably related to the performance issues. > > On 13 Nov 2014, at 04:22, Brian Craft <craft...@gmail.com <javascript:>> > wrote: > > Interesting that you don't see a performance problem. What version did you > try? I'm using 0.5.9. > > I just re-ran this example to make sure I wasn't imagining it. On the 11th > run, it wedged, not returning, and burning a lot of cpu, presumably trying > to shrink. > > It's a larger problem with the real test cases. One test ran for about > eight hours yesterday before I killed it. Hard to tell what it's doing, and > not sure what sort of times I should expect for shrinking larger test cases. > > > On Wednesday, November 12, 2014 9:05:40 AM UTC-8, Lucas Bradstreet wrote: >> >> I've also had some tricky shrinking type issues with recursive generators >> using bind. I had a play with your generators, using such-that to reduce >> the row/column name length and also preventing some generator shrinking by >> using no-shrink, but I didn't have much luck improving the resulting >> shrinks (though the shrinks did finish in a timely fashion). >> >> You might have more luck if you generated an m x n matrix size >> independently of the matrix itself and feed the size into the matrix >> generator and the samples and probes generators. This may allow the matrix >> size to be shrunk down more easily, while also shrinking the samples and >> probes vectors without them having to be regenerated for ever new matrix >> shrink (i.e. where the matrix size is maintained but the gen/int values >> within it are shrunk). >> >> However, this is all guess work and conjecture, as I don't understand the >> shrinking algorithm well enough, and I haven't tried my suggestion. Also, >> modifying your generators in this way may make your model less general if >> you wish to use it in other ways later. >> >> Cheers >> >> Lucas >> >> >> >> On 12 Nov 2014, at 02:43, Brian Craft <craft...@gmail.com> wrote: >> >> Using test.check, I'm finding the shrinking to be very, very slow. >> Running a hundred cases takes a few seconds, unless it hits an error, in >> which case it takes 40-60 minutes to shrink, and the shrinking is not very >> effective (resulting test case is much larger than necessary). Sometimes >> the shrinking is much faster. It behaves a bit like it's occasionally >> getting into a pathological state, or a difficult shrinking scenario. >> >> Are there any docs on generators or the shrinking algorithm that would >> help build tests that shrink more effectively? >> >> The problematic generator builds a randomly-sized matrix of integers, >> with randomly assigned names for the rows and columns. The failure case is >> when either a column or row name is repeated. I expect the slow shrinking >> has something to do with it being rare for the generator to emit the same >> name twice. >> >> ; Generator of randomly sized matrices of random numbers. >> (def gen-matrix >> (gen/bind >> gen/s-pos-int >> (fn [x] (gen/bind >> gen/s-pos-int >> (fn [y] (gen/vector (gen/vector gen/int x) y)))))) >> >> ; Generator of matrix with probe and sample ids. >> (def gen-tsv >> (gen/bind >> gen-matrix >> (fn [m] >> (gen/hash-map >> :probes (gen/vector >> (gen/such-that not-empty gen/string-alpha-numeric) >> (count m)) >> :samples (gen/vector >> (gen/such-that not-empty gen/string-alpha-numeric) >> (count (first m))) >> :matrix (gen/return m))))) >> >> Shrinking will result in a case like >> >> {:matrix [[1 4 -3] [-4 -3 -5] [-5 2 3] [4 -5 -5] [1 -2 3] [1 4 1]], >> :samples ["0" "0" "0"], :probes ["0" "0" "0" "0" "0" "0"]} >> >> where :samples ["0" "0"] :probes["0"] would do. >> >> The following test will exhibit the behavior, sometimes succeeding, >> sometimes failing quickly, sometimes shrinking for a very long time: >> >> (tc/quick-check 100 (prop/for-all [tsv gen-tsv] (= (count (set (:probes >> tsv))) (count (:probes tsv))))) >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> >> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clo...@googlegroups.com <javascript:> > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+u...@googlegroups.com <javascript:> > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+u...@googlegroups.com <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.