Heya Adam, Amazing! Thank you! You're correct that it's slower.
I'd completely forgotten about FactoryBoy! though used it a fair bit a few years ago. Interesting to note that the original ruby project renamed itself to FactoryBot <https://github.com/thoughtbot/factory_bot/blob/master/NAME.md> ("offensive or problematic" indeed). The conversation continued about Factory* libraries, and Zac agrees that these are useful, but has reservations that examples generated by them are fewer, and more limited in range, eg omitting empty string, control characters -- which is important for "real" bug finding, this is usualy bug gold. He also reckons that they have weaker support for reproducing failures. Long and the short is possibly that FactoryBoy could be improved and maybe there might be cases for Django projects to use Hypothesis too :) Whatever happens I'm going to keep plugging away at hypothesis examples, and maybe making the official docs, er, more penetrable. Personally I'm doing something nuts, for which FactoryBoy is not suitable, so this makes sense in my case. It's great to hear about the OSS-Fuzz project. I've passed the details along. It's always nice to cross-polinate and delighted Zac'll be packing for PyCon with this background under his belt. It seems likely more conversation will be happening there. Thank you so much again, --- Elena Williams Github: elena <http://github.com/elena/> On Fri, 21 Feb 2020 at 20:22, Adam Johnson <[email protected]> wrote: > Hi Elena, > > I like fuzz testing and how Hypothesis does it. I have used it on a couple > of projects and it always found bugs. In fact it was my first Pycon that I > sat with David and sprinted on adding it to a project :) > > That said I've always found it quite a leap from straightforward tests, > being slower and non-deterministic. There is a hard balance to make writing > fuzz tests versus writing individual test cases to ensure edge cases keep > being covered. And also we generally want tests to be less complex than the > code they are testing - short functions with several parameters can easily > need many lines of hypothesis code to set up accurate testing. > > Personally I tend to settle on using a little bit of fuzzing with random > data from factory boy, but adding control to the random seed used in > testing using pytest-randomly. I gave this approach a bit of a write up > when I started using it in 2014: > https://adamj.eu/tech/2014/09/03/factory-boy-fun/ . > > As for Django's test sutie: I don't believe there is any fuzz testing > there. But it is being tested separately instead with Google's OSS-Fuzz > project: https://github.com/google/oss-fuzz/tree/master/projects/django . > This was set up by Guido Vranken who announced it in this mailing list > thread: > https://groups.google.com/forum/#!topic/django-developers/-WweB07YiVQ . > Apparently it has caught a few bugs since implementation, and there's scope > to expand it quite a lot: > https://groups.google.com/d/msg/django-developers/PMtvmMlsjyw/_rOj_LzvCQAJ > . I *think* if we were to increase our fuzz testing, we'd probably want to > concentrate efforts there. Google also offer $ rewards to critical projects > like Django for reaching certain levels of coverage, which is nice of them. > > Thanks, > > Adam > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CAMyDDM2%2Bbz%2BwVgiZiVG7YijwO0zzxZcfVE%3Dpxmcper7D3iUJQg%40mail.gmail.com > <https://groups.google.com/d/msgid/django-developers/CAMyDDM2%2Bbz%2BwVgiZiVG7YijwO0zzxZcfVE%3Dpxmcper7D3iUJQg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMRRKVe8uq3nVaukgW5iN3cBVnzFuQ4k7Fq4R99-U4DfRVWAXA%40mail.gmail.com.
