Thanks for the pointer to sqlsmith. I had seen it in the past but its existence slipped from my memory. (I was also just looking for an excuse to play around with afl.) Sounds like it might be helpful to have a test harness to enable us to easily run sqlsmith (I wouldn't expect this to execute as part of the normal test suite). I like the second option you proposed for this.
The third option is closest to what I'm currently doing although the need for a bridge between Java land and afl definitely slows things down significantly which makes it much harder to realize the benefits of fuzz testing. I haven't found a great Java native library for fuzz testing that's anywhere near as mature and robust as afl. > it does find multiple bugs in RexSimplify in a minute. I assume all these bugs have been reported. Either way, it sounds like this could be a good contribution if it's so adept at finding bugs :) -- Michael Mior mm...@apache.org Le lun. 10 sept. 2018 à 08:09, Vladimir Sitnikov < sitnikov.vladi...@gmail.com> a écrit : > Michael>Tests are running quite slowly, but I left it overnight and haven't > found any crashes > > There's https://github.com/anse1/sqlsmith which is supposed to be used > against PostgreSQL. > However, we could configure Calcite to be a proxy to the PostgreSQL, so > sqlsmith would use Calcite as if it was PostgreSQL. > > Second option is to call sqlsmith to generate some queries, and then > execute it via Java-based API. > > The third option is to implement Java-based (or Kotlin-based) fuzzer. I > think it might stress more aspects. > For instance, Calcite has various quoting rules, and sqlsmith just does not > support that. > > Michael>Any suggestions on other things that might be relatively > straightforward to test? > > My RexFuzzer.java is about 200-300 lines of straight-forward Java code that > produces random RexNodes, and it does find multiple bugs in RexSimplify in > a minute. > I think we might want to have a fuzzer for RelBuilder as well. > It should be more-or-less trivial, and it might find interesting cases. > > As a bonus point, we can run the same SQL through regular PostgreSQL and > compare the outputs. > > Vladimir >
