On Wed, Jun 24, 2020 at 9:48 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > > In that case I would propose the following: > 1. Standardize on clang for performance generating numbers for performance > related PRs > 2. Adjust our binary artifact builds to use clang where feasible (I think > should wait until after our next release).
Note that the manylinux* standards mandate gcc so we'll be stuck on gcc for the Python wheels and perhaps some other builds. Otherwise I'd be in favor of using clang when possible. > 3. Add to the contributors guide summarizing the discussion. > > Sound reasonable? > > Thanks, > Micah > > On Tue, Jun 23, 2020 at 12:53 AM Uwe L. Korn <uw...@xhochy.com> wrote: > > > FTR: We can use the latest(!) clang for all platform for conda and wheels. > > It isn't probably even that much of a complicated setup. > > > > On Mon, Jun 22, 2020, at 5:42 PM, Francois Saint-Jacques wrote: > > > We should aim to improve the performance of the most widely used > > > *default* packages, which are python pip, python conda and R (all > > > platforms). AFAIK, both pip (manywheel) and conda use gcc on Linux by > > > default. R uses gcc on Linux and mingw (gcc) on Windows. I suppose > > > (haven't checked) that clang is used on OSX via brew. Thus, by > > > default, almost all users are going to use a gcc compiled version of > > > arrow on Linux. > > > > > > François > > > > > > On Mon, Jun 22, 2020 at 9:47 AM Wes McKinney <wesmck...@gmail.com> > > wrote: > > > > > > > > Based on some of my performance work recently, I'm growing > > > > uncomfortable with using gcc as the performance baseline since the > > > > results can be significantly different (sometimes 3-4x or more on > > > > certain fast algorithms) from clang and MSVC. The perf results on > > > > https://github.com/apache/arrow/pull/7506 were really surprising -- > > > > some benchmarks that showed 2-5x performance improvement on both clang > > > > and MSVC shows small regressions (20-30%) with gcc. > > > > > > > > I don't think we need a hard-and-fast rule about whether to accept PRs > > > > based on benchmarks but there are a few guiding criteria: > > > > > > > > * How much binary size does the new code add? I think many of us would > > > > agree that a 20% performance increase on some algorithm might not be > > > > worth adding 500KB to libarrow.so > > > > * Is the code generally faster across the major compiler targets (gcc, > > > > clang, MSVC)? > > > > > > > > I think that using clang as a baseline for informational benchmarks > > > > would be good, but ultimately we need to be systematically collecting > > > > data on all the major compiilers. Some time ago I proposed building a > > > > Continuous Benchmarking framework > > > > (https://github.com/conbench/conbench/blob/master/doc/REQUIREMENTS.md) > > > > for use with Arrow (and outside of Arrow, too) so I hope that this > > > > will be able to help. > > > > > > > > - Wes > > > > > > > > On Mon, Jun 22, 2020 at 5:12 AM Yibo Cai <yibo....@arm.com> wrote: > > > > > > > > > > On 6/22/20 5:07 PM, Antoine Pitrou wrote: > > > > > > > > > > > > Le 22/06/2020 à 06:27, Micah Kornfield a écrit : > > > > > >> There has been significant effort recently trying to optimize our > > C++ > > > > > >> code. One thing that seems to come up frequently is different > > benchmark > > > > > >> results between GCC and Clang. Even different versions of the > > same > > > > > >> compiler can yield significantly different results on the same > > code. > > > > > >> > > > > > >> I would like to propose that we choose a specific compiler and > > version on > > > > > >> Linux for evaluating performance related PRs. PRs would only be > > accepted > > > > > >> if they improve the benchmarks under the selected version. > > > > > > > > > > > > Would this be a hard rule or just a guideline? There are many > > ways in > > > > > > which benchmark numbers can be improved or deteriorated by a PR, > > and in > > > > > > some cases that doesn't matter (benchmarks are not always > > realistic, and > > > > > > they are not representative of every workload). > > > > > > > > > > > > > > > > I agree that microbenchmark is not always useful, focusing too much > > on > > > > > improving microbenchmark result gives me feeling of "overfit" (to > > some > > > > > specific microarchitecture, compiler, or use case). > > > > > > > > > > > Regards > > > > > > > > > > > > Antoine. > > > > > > > > > > >
