I'd love to see arrow-rs and arrow-datafusion also publish results to
Arrow's Conbench server.
If you point me to the existing benchmarks for each project and
instructions on how to execute them, I can let you know the easiest
integration path.
If the arrow-rs benchmarks are executable from command line and return
parsable results (like json), it should be pretty easy to publish the
results.
And if any of the existing benchmarks are written in Python (perhaps
arrow-datafusion?), then there may be a different (but also easy)
integration path.
There are a few caveats though:
- The benchmarks are running on dedicated machines at Wes' house, so
we would need his blessing (or additional dedicated machines).
- The arrow-rs and arrow-datafusion GitHub repositories must use
squash merges (or Conbench would have to be extended to understand the
2 other GitHub merge methods).
- I'm not sure what the security implications are with respect to
adding our ursabot integration and buildkite hooks to other
repositories.
In the meantime, here's a blog post that initially announced the
Arrow/Conbench integration and contains a brief functionality
overview.
https://ursalabs.org/blog/announcing-conbench/
That blog post is a bit outdated now – we've since started publishing
the Arrow Java and JavaScript benchmarks to https://conbench.ursa.dev/
as well.
And Conbench now collects and plots benchmark distribution history,
and computes z-scores (rather than just a basic percent changed).
Example distribution history (note the regression):
https://conbench.ursa.dev/compare/benchmarks/7efda5a661a348ebb66e021dc9aa7bba...6a4122865bba4b6b9045390d4d1eb7cc/
Example simple benchmark result plotting:
https://conbench.ursa.dev/batches/cbe57f13e51f46019f207853421db764/
Example run of the Arrow C++ and Java benchmarks:
https://conbench.ursa.dev/runs/82308256f17a4d059db902b07d4d4530/
Example run of the Arrow R, Python, and JavaScript benchmarks:
https://conbench.ursa.dev/runs/b56fdd1375ed4b0cbf928dd8dafe8322/
Finally, here's a link to the Conbench repository (as well as the
repositories with the Arrow benchmarks):
- Conbench:
https://github.com/ursacomputing/conbench
- Arrow Python benchmarks + external benchmark wrappers/runners:
https://github.com/ursacomputing/benchmarks
- Arrow R benchmarks:
https://github.com/ursacomputing/arrowbench
- Arrow Java/JavaScript/C++ benchmarks:
https://github.com/apache/arrow
Looking forward to collaborating on this!
--diana
On Sat, Sep 11, 2021 at 5:01 AM Andrew Lamb <al...@influxdata.com> wrote:
>
> I wasn't even aware of conbench.ursa.dev -- I agree it looks like a really
> neat tool and could be very valuable for the rust language implementation
>
> On Fri, Sep 10, 2021 at 11:16 PM QP Hou <ho...@apache.org> wrote:
>
> > Hi,
> >
> > I think conbench.ursa.dev works really well for the main arrow repo,
> > especially the ability to request on demand benchmarks during PR
> > reviews by mentioning usrabot in a comment.
> >
> > I am wondering if it is something that arrow-rs and arrow-datafusion
> > could leverage as well to help speed up PR reviews and guard against
> > performance degradation? If so, what's the process for setting it up?
> >
> > Thanks,
> > QP
> >
