Roman,

Probably you are right that the SQL benchmarks need to be incorporated into
the existing 'ignite-benchmarks' module. Please disregard my suggestion of
setting a dedicated repository.

-
Denis


On Sat, May 23, 2020 at 6:12 AM Roman Kondakov <kondako...@mail.ru.invalid>
wrote:

> Hi Denis,
>
> I'm not sure we need a separate repository for it. What would be the
> benefit of using a separate repo?
>
> BTW I noticed that Ignite has `ignite-benchmarks` module. It contains
> JMH/JOL benchmarks for now. We can also put the SQL benchmark to this
> module. What do you think?
>
>
> --
> Kind Regards
> Roman Kondakov
>
>
> On 22.05.2020 22:36, Denis Magda wrote:
> > Hi Roman,
> >
> > +1 for sure. On a side note, should we create a separate ASF/Git
> repository
> > for the project? Not sure we need to put the suite in the main Ignite
> repo.
> >
> > -
> > Denis
> >
> >
> > On Fri, May 22, 2020 at 8:54 AM Roman Kondakov
> <kondako...@mail.ru.invalid>
> > wrote:
> >
> >> Hi everybody!
> >>
> >> Currently Ignite doesn't have an ability to detect SQL performance
> >> regressions between different versions. We have a Yardstick benchmark
> >> module, but it has several drawbacks:
> >> - it doesn't compare different Ignite versions
> >> - it doesn't check the query result
> >> - it doesn't have an ability to execute randomized SQL queries (aka
> >> fuzzy testing)
> >>
> >> So, Yardstick is not very helpful for detecting SQL performance
> >> regressions.
> >>
> >> I think we need a brand-new framework for this task and I propose to
> >> implement it by adopting the ideas taken from the Apollo tool paper [1].
> >> The Apollo tool pipeline works like like this:
> >>
> >> 1. Apollo start two different versions of databases simultaneously.
> >> 2. Then Apollo populates them with the same dataset
> >> 3. Apollo generates random SQL queries using external library (i.e.
> >> SQLSmith [2])
> >> 4. Each query is executed in both database versions. Execution time is
> >> measured by the framework.
> >> 5. If the execution time difference for the same query exceeds some
> >> threshold (say, 2x slower), the query is logged.
> >> 6. Apollo then tries to simplify the problematic queries in order to
> >> obtain the minimal reproducer.
> >> 7. Apollo also has an ability to automatically perform git history
> >> binary search to find the bad commit
> >> 8. It also can localize a root cause of the regression by carrying out
> >> the statistical debugging.
> >>
> >> I think we don't have to implement all these Apollo steps. First 4 steps
> >> will be enough for our needs.
> >>
> >> My proposal is to create a new module called 'sql-testing'. We need a
> >> separate module because it should be suitable for both query engines:
> >> H2-based and upcoming Calcite-based. This module will contain a test
> >> suite which works in the following way:
> >> 1. It starts two Ignite clusters with different versions (current
> >> version and the previous release version).
> >> 2. Framework then runs randomly generated queries in both clusters and
> >> checks the execution time for each cluster. We need to port SQLSmith [2]
> >> library from C++ to java for this step. But initially we can start with
> >> some set of hardcoded queries and postpone the SQLSmith port. Randomized
> >> queries can be added later.
> >> 3. All problematic queries are then reported as performance issues. In
> >> this way we can manually examine the problems.
> >>
> >> This tool will bring a certain amount of robustness to our SQL layer as
> >> well as some portion of confidence in absence of SQL query regressions.
> >>
> >> What do you think?
> >>
> >>
> >> [1] http://www.vldb.org/pvldb/vol13/p57-jung.pdf
> >> [2] https://github.com/anse1/sqlsmith
> >>
> >>
> >> --
> >> Kind Regards
> >> Roman Kondakov
> >>
> >>
> >
>

Reply via email to