Re: status-benchmark.cc compilation time

2017-02-23 Thread Henry Robinson 
I think the main problem I want to avoid is paying the cost of linking,
which is expensive for Impala as it often generates multi-hundred-MB
binaries per benchmark or test.

Building the benchmarks during GVO seems the best solution to that to me.

On 23 February 2017 at 10:23, Todd Lipcon  wrote:

> One thing we've found useful in Kudu to prevent bitrot of benchmarks is to
> actually use gtest and gflags for the benchmark programs.
>
> We set some flag like --benchmark_num_rows or --benchmark_num_iterations
> with a default that's low enough to only run for a second or two, and run
> it as part of our normal test suite. Rarely catches any bugs, but serves to
> make sure that the code keeps working. Then, when a developer wants to
> actually test a change for performance, they can run it with
> --num_iterations=.
>
> Doesn't help the weird case of status-benchmark where *compiling* takes 10
> minutes... but I think the manual unrolling of 1000 status calls in there
> is probably unrealistic anyway regarding how the different options perform
> in a whole-program setting.
>
> -Todd
>
> On Thu, Feb 23, 2017 at 10:20 AM, Zachary Amsden 
> wrote:
>
> > Yes.  If you take a look at the benchmark, you'll notice the JNI call to
> > initialize the frontend doesn't even have the right signature anymore.
> > That's one easy way to bitrot while still compiling.
> >
> > Even fixing that isn't enough to get it off the ground.
> >
> >  - Zach
> >
> > On Tue, Feb 21, 2017 at 11:44 AM, Henry Robinson 
> > wrote:
> >
> > > Did you run . bin/set-classpath.sh before running expr-benchmark?
> > >
> > > On 21 February 2017 at 11:30, Zachary Amsden 
> > wrote:
> > >
> > > > Unfortunately some of the benchmarks have actually bit-rotted.  For
> > > > example, expr-benchmark compiles but immediately throws JNI
> exceptions.
> > > >
> > > > On Tue, Feb 21, 2017 at 10:55 AM, Marcel Kornacker <
> > mar...@cloudera.com>
> > > > wrote:
> > > >
> > > > > I'm also in favor of not compiling it on the standard commandline.
> > > > >
> > > > > However, I'm very much against allowing the benchmarks to bitrot.
> As
> > > > > was pointed out, those benchmarks can be valuable tools during
> > > > > development, and keeping them in working order shouldn't really
> > impact
> > > > > the development process.
> > > > >
> > > > > In other words, let's compile them as part of gvo.
> > > > >
> > > > > On Tue, Feb 21, 2017 at 10:50 AM, Alex Behm <
> alex.b...@cloudera.com>
> > > > > wrote:
> > > > > > +1 for not compiling the benchmarks in -notests
> > > > > >
> > > > > > On Mon, Feb 20, 2017 at 7:55 PM, Jim Apple  >
> > > > wrote:
> > > > > >
> > > > > >> > On which note, would anyone object if we disabled benchmark
> > > > > compilation
> > > > > >> by
> > > > > >> > default when building the BE tests? I mean separating out
> > -notests
> > > > > into
> > > > > >> > -notests and -build_benchmarks (the latter false by default).
> > > > > >>
> > > > > >> I think this is a great idea.
> > > > > >>
> > > > > >> > I don't mind if the benchmarks bitrot as a result, because we
> > > don't
> > > > > run
> > > > > >> > them regularly or pay attention to their output except when
> > > > > developing a
> > > > > >> > feature. Of course, maybe an 'exhaustive' run should build the
> > > > > benchmarks
> > > > > >> > as well just to keep us honest, but I'd be happy if 95% of
> > Jenkins
> > > > > builds
> > > > > >> > didn't bother.
> > > > > >>
> > > > > >> The pre-merge (aka GVM aka GVO) testing builds
> > > > > >> http://jenkins.impala.io:8080/job/all-build-options, which
> builds
> > > > > >> without the "-notests" flag.
> > > > > >>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Henry Robinson
> > > Software Engineer
> > > Cloudera
> > > 415-994-6679
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Re: status-benchmark.cc compilation time

2017-02-23 Thread Todd Lipcon 
One thing we've found useful in Kudu to prevent bitrot of benchmarks is to
actually use gtest and gflags for the benchmark programs.

We set some flag like --benchmark_num_rows or --benchmark_num_iterations
with a default that's low enough to only run for a second or two, and run
it as part of our normal test suite. Rarely catches any bugs, but serves to
make sure that the code keeps working. Then, when a developer wants to
actually test a change for performance, they can run it with
--num_iterations=.

Doesn't help the weird case of status-benchmark where *compiling* takes 10
minutes... but I think the manual unrolling of 1000 status calls in there
is probably unrealistic anyway regarding how the different options perform
in a whole-program setting.

-Todd

On Thu, Feb 23, 2017 at 10:20 AM, Zachary Amsden 
wrote:

> Yes.  If you take a look at the benchmark, you'll notice the JNI call to
> initialize the frontend doesn't even have the right signature anymore.
> That's one easy way to bitrot while still compiling.
>
> Even fixing that isn't enough to get it off the ground.
>
>  - Zach
>
> On Tue, Feb 21, 2017 at 11:44 AM, Henry Robinson 
> wrote:
>
> > Did you run . bin/set-classpath.sh before running expr-benchmark?
> >
> > On 21 February 2017 at 11:30, Zachary Amsden 
> wrote:
> >
> > > Unfortunately some of the benchmarks have actually bit-rotted.  For
> > > example, expr-benchmark compiles but immediately throws JNI exceptions.
> > >
> > > On Tue, Feb 21, 2017 at 10:55 AM, Marcel Kornacker <
> mar...@cloudera.com>
> > > wrote:
> > >
> > > > I'm also in favor of not compiling it on the standard commandline.
> > > >
> > > > However, I'm very much against allowing the benchmarks to bitrot. As
> > > > was pointed out, those benchmarks can be valuable tools during
> > > > development, and keeping them in working order shouldn't really
> impact
> > > > the development process.
> > > >
> > > > In other words, let's compile them as part of gvo.
> > > >
> > > > On Tue, Feb 21, 2017 at 10:50 AM, Alex Behm 
> > > > wrote:
> > > > > +1 for not compiling the benchmarks in -notests
> > > > >
> > > > > On Mon, Feb 20, 2017 at 7:55 PM, Jim Apple 
> > > wrote:
> > > > >
> > > > >> > On which note, would anyone object if we disabled benchmark
> > > > compilation
> > > > >> by
> > > > >> > default when building the BE tests? I mean separating out
> -notests
> > > > into
> > > > >> > -notests and -build_benchmarks (the latter false by default).
> > > > >>
> > > > >> I think this is a great idea.
> > > > >>
> > > > >> > I don't mind if the benchmarks bitrot as a result, because we
> > don't
> > > > run
> > > > >> > them regularly or pay attention to their output except when
> > > > developing a
> > > > >> > feature. Of course, maybe an 'exhaustive' run should build the
> > > > benchmarks
> > > > >> > as well just to keep us honest, but I'd be happy if 95% of
> Jenkins
> > > > builds
> > > > >> > didn't bother.
> > > > >>
> > > > >> The pre-merge (aka GVM aka GVO) testing builds
> > > > >> http://jenkins.impala.io:8080/job/all-build-options, which builds
> > > > >> without the "-notests" flag.
> > > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: status-benchmark.cc compilation time

2017-02-23 Thread Henry Robinson 
Fair enough, always worth checking the easy things first :)

On 23 February 2017 at 10:20, Zachary Amsden  wrote:

> Yes.  If you take a look at the benchmark, you'll notice the JNI call to
> initialize the frontend doesn't even have the right signature anymore.
> That's one easy way to bitrot while still compiling.
>
> Even fixing that isn't enough to get it off the ground.
>
>  - Zach
>
> On Tue, Feb 21, 2017 at 11:44 AM, Henry Robinson 
> wrote:
>
> > Did you run . bin/set-classpath.sh before running expr-benchmark?
> >
> > On 21 February 2017 at 11:30, Zachary Amsden 
> wrote:
> >
> > > Unfortunately some of the benchmarks have actually bit-rotted.  For
> > > example, expr-benchmark compiles but immediately throws JNI exceptions.
> > >
> > > On Tue, Feb 21, 2017 at 10:55 AM, Marcel Kornacker <
> mar...@cloudera.com>
> > > wrote:
> > >
> > > > I'm also in favor of not compiling it on the standard commandline.
> > > >
> > > > However, I'm very much against allowing the benchmarks to bitrot. As
> > > > was pointed out, those benchmarks can be valuable tools during
> > > > development, and keeping them in working order shouldn't really
> impact
> > > > the development process.
> > > >
> > > > In other words, let's compile them as part of gvo.
> > > >
> > > > On Tue, Feb 21, 2017 at 10:50 AM, Alex Behm 
> > > > wrote:
> > > > > +1 for not compiling the benchmarks in -notests
> > > > >
> > > > > On Mon, Feb 20, 2017 at 7:55 PM, Jim Apple 
> > > wrote:
> > > > >
> > > > >> > On which note, would anyone object if we disabled benchmark
> > > > compilation
> > > > >> by
> > > > >> > default when building the BE tests? I mean separating out
> -notests
> > > > into
> > > > >> > -notests and -build_benchmarks (the latter false by default).
> > > > >>
> > > > >> I think this is a great idea.
> > > > >>
> > > > >> > I don't mind if the benchmarks bitrot as a result, because we
> > don't
> > > > run
> > > > >> > them regularly or pay attention to their output except when
> > > > developing a
> > > > >> > feature. Of course, maybe an 'exhaustive' run should build the
> > > > benchmarks
> > > > >> > as well just to keep us honest, but I'd be happy if 95% of
> Jenkins
> > > > builds
> > > > >> > didn't bother.
> > > > >>
> > > > >> The pre-merge (aka GVM aka GVO) testing builds
> > > > >> http://jenkins.impala.io:8080/job/all-build-options, which builds
> > > > >> without the "-notests" flag.
> > > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679
> >
>



-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Re: status-benchmark.cc compilation time

2017-02-23 Thread Zachary Amsden 
Yes.  If you take a look at the benchmark, you'll notice the JNI call to
initialize the frontend doesn't even have the right signature anymore.
That's one easy way to bitrot while still compiling.

Even fixing that isn't enough to get it off the ground.

 - Zach

On Tue, Feb 21, 2017 at 11:44 AM, Henry Robinson  wrote:

> Did you run . bin/set-classpath.sh before running expr-benchmark?
>
> On 21 February 2017 at 11:30, Zachary Amsden  wrote:
>
> > Unfortunately some of the benchmarks have actually bit-rotted.  For
> > example, expr-benchmark compiles but immediately throws JNI exceptions.
> >
> > On Tue, Feb 21, 2017 at 10:55 AM, Marcel Kornacker 
> > wrote:
> >
> > > I'm also in favor of not compiling it on the standard commandline.
> > >
> > > However, I'm very much against allowing the benchmarks to bitrot. As
> > > was pointed out, those benchmarks can be valuable tools during
> > > development, and keeping them in working order shouldn't really impact
> > > the development process.
> > >
> > > In other words, let's compile them as part of gvo.
> > >
> > > On Tue, Feb 21, 2017 at 10:50 AM, Alex Behm 
> > > wrote:
> > > > +1 for not compiling the benchmarks in -notests
> > > >
> > > > On Mon, Feb 20, 2017 at 7:55 PM, Jim Apple 
> > wrote:
> > > >
> > > >> > On which note, would anyone object if we disabled benchmark
> > > compilation
> > > >> by
> > > >> > default when building the BE tests? I mean separating out -notests
> > > into
> > > >> > -notests and -build_benchmarks (the latter false by default).
> > > >>
> > > >> I think this is a great idea.
> > > >>
> > > >> > I don't mind if the benchmarks bitrot as a result, because we
> don't
> > > run
> > > >> > them regularly or pay attention to their output except when
> > > developing a
> > > >> > feature. Of course, maybe an 'exhaustive' run should build the
> > > benchmarks
> > > >> > as well just to keep us honest, but I'd be happy if 95% of Jenkins
> > > builds
> > > >> > didn't bother.
> > > >>
> > > >> The pre-merge (aka GVM aka GVO) testing builds
> > > >> http://jenkins.impala.io:8080/job/all-build-options, which builds
> > > >> without the "-notests" flag.
> > > >>
> > >
> >
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>

[Toolchain-CR] Update Kudu version to 2b0edbe

2017-02-23 Thread Dimitris Tsirogiannis (Code Review) 
Dimitris Tsirogiannis has posted comments on this change.

Change subject: Update Kudu version to 2b0edbe
..


Patch Set 1: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/6110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I175133b1841449e311d637d10efc9d503612a3cf
Gerrit-PatchSet: 1
Gerrit-Project: Toolchain
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-HasComments: No

[Toolchain-CR] Update Kudu version to 2b0edbe

2017-02-23 Thread Matthew Jacobs (Code Review) 
Matthew Jacobs has submitted this change and it was merged.

Change subject: Update Kudu version to 2b0edbe
..


Update Kudu version to 2b0edbe

Change-Id: I175133b1841449e311d637d10efc9d503612a3cf
---
M buildall.sh
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Matthew Jacobs: Verified
  Dimitris Tsirogiannis: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/6110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I175133b1841449e311d637d10efc9d503612a3cf
Gerrit-PatchSet: 1
Gerrit-Project: Toolchain
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Matthew Jacobs

[Toolchain-CR] Update Kudu version to 2b0edbe

2017-02-23 Thread Matthew Jacobs (Code Review) 
Matthew Jacobs has posted comments on this change.

Change subject: Update Kudu version to 2b0edbe
..


Patch Set 1: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/6110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I175133b1841449e311d637d10efc9d503612a3cf
Gerrit-PatchSet: 1
Gerrit-Project: Toolchain
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-HasComments: No

New Impala committer: Thomas Marshall

2017-02-23 Thread Matthew Jacobs 
The Podling Project Management Committee (PPMC) for Apache Impala
(incubating) has invited Thomas Marshall to become a committer and we
are pleased to announce that he has accepted.

Congratulations and welcome, Thomas!

New Impala committer: Michael Brown

2017-02-23 Thread Jim Apple 
The Podling Project Management Committee (PPMC) for Apache Impala
(incubating) has invited Michael Brown to become a committer and we
are pleased to announce that he has accepted.

Congratulations and welcome, Michael!