Re: jenkins.impala.io maintenance

[Thomas Tauber-Marshall]

The update has been applied and Jenkins should be available again.

Please let me know if you encounter any issues.

On Wed, Jan 13, 2021 at 10:47 AM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> The public Jenkins instance will be taken offline shortly for a security
> update.
>
> There are no jobs running currently, so hopefully the impact will be
> minimal.
>
> An email will be sent when the service is available again.
>
> Thank you for your patience,
> - Thomas
>

jenkins.impala.io maintenance

[Thomas Tauber-Marshall]

The public Jenkins instance will be taken offline shortly for a security
update.

There are no jobs running currently, so hopefully the impact will be
minimal.

An email will be sent when the service is available again.

Thank you for your patience,
- Thomas

Breaking change in impala-shell

[Thomas Tauber-Marshall]

Impala devs,

There is a patch that has just gone into master which switches the default
protocol for impala-shell from beeswax to hiveserver2, as part of the
ongoing process of deprecating and removing beeswax, see
https://issues.apache.org/jira/browse/IMPALA-10074 Because this is a
breaking change, it was timed to be included with Impala 4.0, per previous
discussions.

I wanted to bring attention to this patch because it has the potential to
break tests and dev workflows in any situation where impala-shell is being
used with the --impalad/-i parameter with the beeswax port specified (by
default 21000) but without the --protocol parameter. In this situation, the
expected error message is "shell_exceptions.MissingThriftMethodException:
Invalid method name: 'OpenSession'"

The fix is to either explicitly specify "--protocol=beeswax" or to switch
to using the hiveserver2 port (by default, 21050)

Re: [VOTE] 3.4.0 release candidate 2

[Thomas Tauber-Marshall]

+1 (binding)

I ran: https://jenkins.impala.io/job/release-test-ub1604/34/


On Fri, Apr 17, 2020 at 10:43 AM Joe McDonnell 
wrote:

> Correction: this is the link:
> https://dist.apache.org/repos/dist/dev/impala/3.4.0/RC2/
>
> On Fri, Apr 17, 2020 at 10:42 AM Joe McDonnell 
> wrote:
>
> > This is a vote for Impala 3.4.0.
> >
> > The artifacts for testing can be downloaded from:
> > https://dist.apache.org/repos/dist/dev/impala/3.4.0/RC2/
> > 
> > Git tag: 3.4.0-rc2
> > Tree hash: b2ac7f44172d292a5743d23ff4710e49e5de688b
> >
> > Please vote +1 or -1. -1 votes should be accompanied by an explanation of
> > the reason. Only PMC members have binding votes, but other
> > community members are encouraged to cast non-binding votes. This vote
> will
> > pass if there are 3 binding +1 votes and more binding +1 votes than -1
> > votes.
> >
> > This wiki page describes how to check the release before you vote:
> > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> >
> > I tested this by running exhaustive jobs for both USE_CDP_HIVE=false and
> > true. I ran the normal release check here:
> > https://jenkins.impala.io/job/release-test-ub1604/33/
> >
> > As compared to RC1, this added the fixes for IMPALA-9529 and IMPALA-8361
> > (needed by IMPALA-9529).
> >
> > Thanks,
> > Joe
> >
>

Re: Branching/tagging native-toolchain projects's state for the Impala 3.4.0 branch

[Thomas Tauber-Marshall]

Agreed that it's definitely a good idea to branch native-toolchain for the
release. We've done this in the past in a way that corresponded with
Cloudera releases, since native-toolchain is a Cloudera maintained project
and Cloudera has historically always had a close match between Cloudera
releases and Apache releases.

That match may not always be kept, though, and it's good from an Apache
perspective to decouple things from Cloudera's decisions anyways.

I would suggest using a branch, for the flexibility of adding commits later
if needed  - we've had issues in the past where we needed to fix
interaction with an external dependency that doesn't actually change what
the toolchain is building (eg. when github stopped allowing older tls
versions).

On Mon, Apr 6, 2020 at 2:04 PM Laszlo Gaal  wrote:

> Hi,
>
> As there is pretty tight coupling between Impala and native-toolchain,
> I think it would be useful to mark the commit on native-toolchain
> that corresponds to the current state used by the Impala 3.4.0 branch.
> Doing so would make it  easier if someone wanted to evolve Impala 3.x in a
> way
> that requires a toolchain change.
> Also, if there is agreement on such a mark, should it be a tag, or a
> branch?
>
> Thoughts?
>
> Thanks,
>
>   - LaszloG
>

Re: [VOTE] 3.3.0 release candidate 1

[Thomas Tauber-Marshall]

+1 (binding)
https://jenkins.impala.io/view/Utility/job/release-test-ub1604/27/

On Wed, Aug 21, 2019 at 8:55 PM Lars Volker  wrote:

> +1 (binding)
>
>- Checked the signature
>- Ran buildall.sh
>- Ran a handful of queries, created and dropped a table
>
>
> On Wed, Aug 21, 2019 at 1:31 PM Bharath Vissapragada <
> bhara...@cloudera.com>
> wrote:
>
> > +1 (binding)
> > https://jenkins.impala.io/view/Utility/job/release-test-ub1604/26/
> >
> > The job failed because of a flaky test for which there is a patch in
> flight
> > [1]. It shouldn't block the release IMO.
> >
> > [1] https://gerrit.cloudera.org/#/c/14117/
> >
> > On Tue, Aug 20, 2019 at 7:38 PM Jim Apple  wrote:
> >
> > > +1 (binding), based on
> > > https://jenkins.impala.io/view/Utility/job/release-test-ub1604/25/
> > >
> > > On Mon, Aug 19, 2019 at 1:01 PM Quanlong Huang <
> huangquanl...@gmail.com>
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > This is a vote for Impala 3.3.0
> > > >
> > > > The artifacts for testing can be downloaded from:
> > > > https://dist.apache.org/repos/dist/dev/impala/3.3.0/RC1/
> > > > Git tag: 3.3.0-rc1
> > > > Tree hash: 5ef67bca619d19402f8c7186b2ab6895bd0603ba
> > > >
> > > > Please vote +1 or -1. -1 votes should be accompanied by an
> explanation
> > of
> > > > the reason. Only PMC members have binding votes, but other
> > > > community members are encouraged to cast non-binding votes. This vote
> > > will
> > > > pass if there are 3 binding +1 votes and more binding +1 votes than
> -1
> > > > votes.
> > > >
> > > > This wiki page describes how to check the release before you vote:
> > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> > > >
> > > > Cheers,
> > > > Quanlong
> > > >
> > >
> >
>

Re: Query status "Session Closed"

[Thomas Tauber-Marshall]

Right, if the connection is being closed you should see that logging.

If the session is being closed due to the idle session timeout you should
see some logging like "Expiring session: " and the status of
queries that are cancelled as a result should be something like ""Session
expired due to inactivity""

You can also turn on verbose logging at the VLOG_QUERY level to see if
CloseSession() is being explicitly called over hiveserver2

On Tue, Aug 6, 2019 at 11:51 AM Antoni Ivanov  wrote:

> Thanks
> This helps
>
> Is there any way to figure out which case is which ? Either from the query
> profile or summary ?
> I guess one pointer below was to look in the logs for "Connection from
> clientclosed, closing  associated sessions "
>
> Thanks,
> Antoni
>
> -Original Message-
> From: Tim Armstrong 
> Sent: Monday, August 5, 2019 5:51 PM
> To: u...@impala.apache.org
> Cc: dev@impala 
> Subject: Re: Query status "Session Closed"
>
> You could also be hitting a timeout on an idle session - if the client
> doesn't perform any operations in a session, then you can configure Impala
> to close the session in this fashion. See
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fimpala.apache.org%2Fdocs%2Fbuild%2Fhtml%2Ftopics%2Fimpala_timeouts.html&data=02%7C01%7Caivanov%40vmware.com%7C7e725bcae8ef4b220f9d08d71a084a4a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637006495239689446&sdata=zT%2BlqH0orz4KiyknWbGD%2FvQUSnESBg7byz9cRefbNrs%3D&reserved=0
>
> On Mon, Aug 5, 2019 at 5:15 PM Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
> > Impala has two client interfaces with slightly different session
> behavior:
> >
> > Beewax (default port 21000) - sessions are created when the client
> > connects
> > Hiveserver2 (default port 21050) - sessions are created when
> > OpenSession() is called
> >
> > In both cases, sessions can be closed either if
> > - the connection ends
> > - someone presses "close" on the /sessions page of the debug webui
> >
> > For hiveserver2, sessions can also be closed if CloseSession() is
> > explicitly called by the client
> >
> > So probably the most likely cause of your issue is that the
> > connections are being dropped somehow, possibly due to the load balancer.
> >
> > If sessions are in fact getting closed due to connections being
> > dropped, you should see some lines of the form "Connection from client
> >  > hostname> closed, closing  associated sessions"
> >
> > Fwiw, this behavior was recently changed so that hiveserver2 sessions
> > are not closed immediately when the connection ends but timeout if
> > there hasn't been an associated connection for a configurable timeout:
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu
> > es.apache.org%2Fjira%2Fbrowse%2FIMPALA-1653&data=02%7C01%7Caivanov
> > %40vmware.com%7C7e725bcae8ef4b220f9d08d71a084a4a%7Cb39138ca3cee4b4aa4d
> > 6cd83d9dd62f0%7C0%7C0%7C637006495239689446&sdata=vhbsC4v26p5drWvTG
> > Ozfx44H2kx47DJPkWnv0Ib43Fs%3D&reserved=0 though we haven't done a
> > release since that work went in
> >
> > On Mon, Aug 5, 2019 at 3:40 PM Antoni Ivanov  wrote:
> >
> >> Hi,
> >>
> >> I am investigating the most common errors we see in our Impala Cluster.
> >> The most common is with query status = 'Session Closed'
> >>
> >> I can see from the code (
> >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> >> hub.com%2Fapache%2Fimpala%2Fblob%2F72c9370856d7436885adbee3e8da7e7d93
> >> 36df15%2Fbe%2Fsrc%2Fservice%2Fimpala-server.cc%23L1435&data=02%7C
> >> 01%7Caivanov%40vmware.com%7C7e725bcae8ef4b220f9d08d71a084a4a%7Cb39138
> >> ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637006495239689446&sdata=IQO
> >> RbGfQV8HeTUU%2Fgu6xULUW7mZThm%2FY38hQLLRarug%3D&reserved=0
> >> )
> >> that it is set when Session is closed and this happens when
> >> connection is closed (ConnectionEnd<
> >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> >> hub.com%2Fapache%2Fimpala%2Fblob%2F72c9370856d7436885adbee3e8da7e7d93
> >> 36df15%2Fbe%2Fsrc%2Fservice%2Fimpala-server.cc%23L2094&data=02%7C
> >> 01%7Caivanov%40vmware.com%7C7e725bcae8ef4b220f9d08d71a084a4a%7Cb39138
> >> ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637006495239689446&sdata=fbH
> >> 1W3zXymjI%2FrGwU5WEnrdwAtLT04PkXmr0zIJoFeo%3D&reserved=0
> >> >)
> >> and this is called when Th

Re: Query status "Session Closed"

[Thomas Tauber-Marshall]

Impala has two client interfaces with slightly different session behavior:

Beewax (default port 21000) - sessions are created when the client connects
Hiveserver2 (default port 21050) - sessions are created when OpenSession()
is called

In both cases, sessions can be closed either if
- the connection ends
- someone presses "close" on the /sessions page of the debug webui

For hiveserver2, sessions can also be closed if CloseSession() is
explicitly called by the client

So probably the most likely cause of your issue is that the connections are
being dropped somehow, possibly due to the load balancer.

If sessions are in fact getting closed due to connections being dropped,
you should see some lines of the form "Connection from client  closed, closing  associated sessions"

Fwiw, this behavior was recently changed so that hiveserver2 sessions are
not closed immediately when the connection ends but timeout if there hasn't
been an associated connection for a configurable timeout:
https://issues.apache.org/jira/browse/IMPALA-1653 though we haven't done a
release since that work went in

On Mon, Aug 5, 2019 at 3:40 PM Antoni Ivanov  wrote:

> Hi,
>
> I am investigating the most common errors we see in our Impala Cluster.
> The most common is with query status = 'Session Closed'
>
> I can see from the code (
> https://github.com/apache/impala/blob/72c9370856d7436885adbee3e8da7e7d9336df15/be/src/service/impala-server.cc#L1435
> )
> that it is set when Session is closed and this happens when connection is
> closed (ConnectionEnd<
> https://github.com/apache/impala/blob/72c9370856d7436885adbee3e8da7e7d9336df15/be/src/service/impala-server.cc#L2094
> >)
> and this is called when Thrift transport is closed<
> https://github.com/apache/impala/blob/82f753e3044bd2482f35d137fbb28516fc0ef86c/be/src/rpc/TAcceptQueueServer.cpp>
> (and query has not completed or failed in some way it would be marked as
> Session Closed
>
> Does this mean that the remote end has simply dropped the connection ?
> E.g there has been network interruption or someone killed (SIGKILL) the
> remote process ?
> We have (TCP) load balancer (HaProxy) and I am wondering if for example
> Load Balancer tcp timeout can cause such error. Or can client socket
> timeout cause it?
>
> I'd be grateful for any insides into the semantics of when "Session
> Closed" is set.
>
>
>
> Thanks,
> Antoni
>

New Committer - Andrew Sherman

[Thomas Tauber-Marshall]

Hi All,

The Project Management Committee (PMC) for Apache Impala
has invited Andrew Sherman to become a committer and we are pleased
to announce that he has accepted.

Welcome and congratulations, Andrew!

Re: [VOTE] 3.2.0 release candidate 1

[Thomas Tauber-Marshall]

+1 (binding)

Jenkins job: https://jenkins.impala.io/job/release-test-ub1604/23/

On Thu, Mar 21, 2019 at 9:05 AM Jim Apple  wrote:

> +1. I checked the things in
>
> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#HowtoRelease-HowtoVoteonaReleaseCandidate
> , plus a run of the usual pre-merge testing for every patch, using a
> Jenkins job: https://jenkins.impala.io/job/release-test-ub1604/22/
>
> On Wed, Mar 20, 2019 at 10:01 AM Gabor Kaszab 
> wrote:
> >
> > Thanks for the observation, Jim!
> > I've removed the prefix and commited it to svn. I think it takes some
> time
> > to sync, though.
> >
> > Gabor
> >
> > On Wed, Mar 20, 2019 at 5:50 PM Jim Apple  wrote:
> >
> > >
> > >
> https://dist.apache.org/repos/dist/dev/impala/3.2.0/RC1/apache-impala-3.2.0.tar.gz.sha512
> > > specifies the absolute directory of the tarball as being in /tmp/. Can
> you
> > > remove that part so the checks succeed on a tarball not in /tmp/?
> > >
> > > + wget -q
> > >
> https://dist.apache.org/repos/dist/dev/impala/3.2.0/RC1/apache-impala-3.2.0.tar.gz
> > > + wget -q
> > >
> https://dist.apache.org/repos/dist/dev/impala/3.2.0/RC1/apache-impala-3.2.0.tar.gz.sha512
> > > + pwd
> > > /tmp/tmp.Kxr8QPOD94
> > > + sha512sum --check apache-impala-3.2.0.tar.gz.sha512
> > > sha512sum: /tmp/apache-impala-3.2.0.tar.gz: No such file or directory
> > > /tmp/apache-impala-3.2.0.tar.gz: FAILED open or read
> > > sha512sum: WARNING: 1 listed file could not be read
> > >
> > >
> > >
> > > On Wed, Mar 20, 2019 at 7:50 AM Gabor Kaszab 
> > > wrote:
> > >
> > > > Hey,
> > > >
> > > > This is a vote for Impala 3.2.0
> > > >
> > > > The artifacts for testing can be downloaded from:
> > > > https://dist.apache.org/repos/dist/dev/impala/3.2.0/RC1/
> > > > Git tag: 3.2.0-rc1
> > > > Tree hash: c3b697dd2b943bc6c7937d8113172848e27840d3
> > > >
> > > > Please vote +1 or -1. -1 votes should be accompanied by an
> explanation of
> > > > the reason. Only PMC members have binding votes, but other
> > > > community members are encouraged to cast non-binding votes. This vote
> > > will
> > > > pass if there are 3 binding +1 votes and more binding +1 votes than
> -1
> > > > votes.
> > > >
> > > > This wiki page describes how to check the release before you vote:
> > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> > > >
> > > > Cheers,
> > > > Gabor
> > > >
> > >
>

Re: [DISCUSS] 3.2.0 release

[Thomas Tauber-Marshall]

If its not too late, it would be great to include this fix:
https://issues.apache.org/jira/browse/IMPALA-8299

On Thu, Mar 14, 2019 at 5:38 AM Gabor Kaszab 
wrote:

> (Sorry, accidentally sent out the mail too early)
> - Alex sent in a number of doc commits covering changes that are already
> in, so I'll include them as well:
> - 76286bf2c0320cce0eb1bf0c269d344255b4dd0e IMPALA-7974
> - 9cc75c59d5a09bd898bdf05cc64a98a70ffd IMPALA-8133
> - 535d286ee16721a35bda1861e4d011ea99a8c02f IMPALA-8067
> - 0d3d3258d2762887c61bf8e64e5df33dc2419817 IMPALA-8298
> - 668bb73e1153afcb50ffc8b45267232b926a2258 IMPALA-7974
> - ca98d6649d700517e90359d39c94d3356c6c092e IMPALA-7718
>
> Alex, I see 2 doc changes remaining according to your list. Do you feel
> they can get to the repo in the next 1-2 days? I can wait for them, then.
> IMPALA-8296
> IMPALA-8297
>
> If no other request comes in until then, then I'll take the above mentioned
> as the content of the 3.2.0 release and advance with the next steps of
> release creation.
>
> Cheers,
> Gabor
>
>
> On Thu, Mar 14, 2019 at 1:18 PM Gabor Kaszab 
> wrote:
>
> > Hey,
> >
> > Just for the record I gather the requests and conclusions here in one
> mail:
> > - It's fine to cut at 9686545bfd7236350d165813621bf783b42d1c18
> IMPALA-6503:
> > Support reading complex types from ORC
> > <
> https://github.com/apache/impala/commit/9686545bfd7236350d165813621bf783b42d1c18
> >
> >  (inclusive)
> > - As Csaba requested, I'll
> > include 26639963e0ea66bea716c0238cdf1ff086159c7c IMPALA-8300 to correct a
> > test issue introduced by the commit right above.
> > - Alex sent in a number of doc commits covering changes that are already
> > in, so I'll include them as well:
> >
> >
> > On Mon, Mar 11, 2019 at 6:54 PM Alexandra Rodoni 
> > wrote:
> >
> >> I have 10 doc tickets for 3.2. Out of 18, 4 in progress, and 3 still
> open.
> >> The rest is done.
> >>
> >> *In progress: *
> >> IMPALA-8133
> >> IMPALA-8296
> >> IMPALA-8297
> >> IMPALA-7974
> >>
> >> *Open:*
> >> IMPALA-8067
> >> IMPALA-7718
> >> IMPALA-8298
> >>
> >> Thanks.
> >> alex
> >>
> >> On Mon, Mar 11, 2019 at 6:33 AM Gabor Kaszab 
> >> wrote:
> >>
> >> > Thanks for the feedback!
> >> > No objections came agains the release so let me continue with the next
> >> > steps. I see that the ORC complex types just got into the repo so I
> >> propose
> >> > to cut the release there (inclusive):
> >> >
> >> > 9686545 IMPALA-6503: Support reading complex types from ORC
> >> >
> >> > In case there is anything before this commit that you are not sure and
> >> > prefer to omit from the release please let me know. Similarly, if
> there
> >> is
> >> > anything that is not ready but is right around the corner and you
> >> insist to
> >> > include it to the release, let me know.
> >> >
> >> > From doc's perspective I'm curious how far we are from having coverage
> >> for
> >> > the proposed changes.
> >> >
> >> > Having a release notes doc is a good idea. Once we agree on the
> content
> >> > I'll create one and share it with the community so that each
> contributor
> >> > can fill in their change's description.
> >> >
> >> > Cheers,
> >> > Gabor
> >> >
> >> >
> >> > On Mon, Mar 4, 2019 at 11:28 PM Tim Armstrong <
> tarmstr...@cloudera.com>
> >> > wrote:
> >> >
> >> > > +1 - what Jim said!
> >> > >
> >> > > On Mon, Mar 4, 2019 at 9:53 AM Jim Apple 
> >> wrote:
> >> > >
> >> > > > +1 to both a release and a habit of curated release notes as
> >> Quanlong
> >> > > > suggests.
> >> > > >
> >> > > > On Mon, Mar 4, 2019 at 6:54 AM Quanlong Huang <
> >> huangquanl...@gmail.com
> >> > >
> >> > > > wrote:
> >> > > > >
> >> > > > > +1. Thanks for your volunteer!
> >> > > > >
> >> > > > > If we decide to release, could we manage a doc that everyone can
> >> > write
> >> > > > down
> >> > > > > his/her notable works in this version? Got this idea from the
> Kudu
> >> > > > > community and feel it very helpful:
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/8a10a4d244369f43b97c79513f13766c393a98252d3a2dade7611cc3@%3Cdev.kudu.apache.org%3E
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Quanlong
> >> > > > >
> >> > > > > On Mon, Mar 4, 2019 at 10:18 PM Gabor Kaszab <
> >> > gaborkas...@cloudera.com
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hey,
> >> > > > > >
> >> > > > > > 3.1.0 has been released at the beginning of December and I
> feel
> >> > that
> >> > > > since
> >> > > > > > then there are a number of bug fixes and improvements checked
> >> in.
> >> > > > > > What do you think about releasing 3.2.0 soon? I volunteer as a
> >> > > release
> >> > > > > > manager in case no one is against.
> >> > > > > >
> >> > > > > > Cheers,
> >> > > > > > Gabor
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [VOTE] 3.1.0 release candidate 1

[Thomas Tauber-Marshall]

+1 (binding)

On Wed, Nov 28, 2018 at 10:24 AM Michael Ho  wrote:

> +1 (binding)
>
> Ran this job to verify the release candidate:
> https://jenkins.impala.io/job/release-test-ub1604/18
>
> On Tue, Nov 27, 2018 at 9:40 AM Zoltan Borok-Nagy  >
> wrote:
>
> > This is a vote to release Impala 3.1.0
> >
> > The artifacts for testing can be downloaded from:
> > https://dist.apache.org/repos/dist/dev/impala/3.1.0/RC1/
> > Git tag: 3.1.0-rc1
> > Tree hash: 0fb7b90d5dad7aeedd48db28939b1999a7a3
> >
> > Please vote +1 or -1. -1 votes should be accompanied by an explanation of
> > the reason. Only PMC members have binding votes, but other
> > community members are encouraged to cast non-binding votes. This vote
> will
> > pass if there are 3 binding +1 votes and more binding +1 votes than -1
> > votes.
> >
> > This wiki page describes how to check the release before you vote:
> > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> >
>
>
> --
> Thanks,
> Michael
>

Re: Improving Kudu Build Support

[Thomas Tauber-Marshall]

This has gone in now. You should be able to rebase, re-source
impala-config.sh, restart the minicluster (eg. with ./buildall.sh -noclean
-notests -start_minicluster), and continue working as normal.

If you rebase and fail to re-source impala-config.sh, you may see an error
like:
Traceback (most recent call last):
  File "/home/thomas/Impala/infra/python/bootstrap_virtualenv.py", line
383, in 
kudu_client_dir = find_kudu_client_install_dir()
  File "/home/thomas/Impala/infra/python/bootstrap_virtualenv.py", line
316, in find_kudu_client_install_dir
kudu_base_dir = os.environ["IMPALA_KUDU_HOME"]
  File "/usr/lib/python2.7/UserDict.py", line 40, in __getitem__
raise KeyError(key)
KeyError: 'IMPALA_KUDU_HOME'

Let me know if you encounter any other issues.

On Thu, Aug 30, 2018 at 1:31 PM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> The patch for this is finally out: https://gerrit.cloudera.org/#/c/11363/
>
> I put some work into trying to make sure this will be as seamless as
> possible for everyone.
> - For people on Ubuntu 16 or the other CDH supported distros, you'll
> automatically start using the CDH Kudu.
> - For people on Ubuntu 14, Impala will automatically set
> USE_CDH_KUDU=false and fall back to the toolchain Kudu. You should see no
> change.
> - For people who want to develop on other OSes, KUDU_IS_SUPPORTED will get
> set to false, we will build a stub Kudu client, and you should be able to
> build and run as normal, except that Kudu related operations will fail. Or
> you can build Kudu locally and link Impala against it using the
> KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables
>
> I also filed IMPALA-7515
> <https://issues.apache.org/jira/browse/IMPALA-7515> to track the eventual
> removal of toolchain Kudu support
>
> On Wed, Aug 22, 2018 at 11:31 AM Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
>> So the toolchain binaries are provided for: centos5,6,7, debian7,8,
>> sles11,12, ubuntu14,16
>> The new CDH component binaries will be available for: redhat6,7, debian8,
>> sles12, ubuntu16
>> so we would be dropping easy support for building on basically centos5,
>> debian7, sles11, and ubuntu14
>>
>> For people who want to develop on other oses like Ubuntu 18.04, it would
>> still be possible by compiling Kudu locally and using the
>> KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables.
>>
>> I'm also strongly leaning towards hiding this functionality behind a flag
>> for now (not many people chimed in on this thread, but I suspect there are
>> still a decent number of people developing on Ubuntu 14 who I'll get angry
>> emails from if they rebase and discover they have to upgrade their os), in
>> which case building the toolchain would remain a viable option for using
>> other oses as well, as least for now.
>>
>> On Tue, Aug 21, 2018 at 4:47 PM Tim Armstrong 
>> wrote:
>>
>>> Is there a path to building a version of Kudu locally for an arbitrary
>>> linux distro?
>>>
>>> Personally I am less concerned about 14.04 support and more concerned
>>> about
>>> what the path to upgrading to 18.04. It would also be nice for it to be
>>> at
>>> least possible to develop on RedHat-derived distros even if it requires
>>> some extra effort.
>>>
>>> On Tue, Aug 21, 2018 at 6:48 AM, Laszlo Gaal 
>>> wrote:
>>>
>>> > +1 for simplifying Kudu updates.
>>> >
>>> > I am also still on Ubuntu 14.04, but I am all for simplifying Kudu
>>> > integration:
>>> > I agree with Thomas that Kudu snapshots should be grouped with the
>>> other
>>> > CDH components.
>>> > Given that Ubuntu 14.04 will be EOL'd next spring, upgrading the
>>> > development OS
>>> > is a reasonably small price to pay -- especially that it will soon
>>> become
>>> > necessary anyway.
>>> >
>>> > Thanks for doing this Thomas!
>>> >
>>> >   - Laszlo
>>> >
>>> > On Tue, Aug 21, 2018 at 12:34 AM Lars Volker  wrote:
>>> >
>>> > > I'm in favor of not spending developer time and effort to maintain
>>> > > compatibility with 14.04. Personally I'm still developing on Ubuntu
>>> 14.04
>>> > > so I'd be happy if we can support it without much pain. On the other
>>> hand
>>> > > it EOLs in April 2019, so I might as well go to 18.04 now, should we
>>> > decide
>>> > > to drop support. Maybe not many other folks

Re: Improving Kudu Build Support

[Thomas Tauber-Marshall]

The patch for this is finally out: https://gerrit.cloudera.org/#/c/11363/

I put some work into trying to make sure this will be as seamless as
possible for everyone.
- For people on Ubuntu 16 or the other CDH supported distros, you'll
automatically start using the CDH Kudu.
- For people on Ubuntu 14, Impala will automatically set USE_CDH_KUDU=false
and fall back to the toolchain Kudu. You should see no change.
- For people who want to develop on other OSes, KUDU_IS_SUPPORTED will get
set to false, we will build a stub Kudu client, and you should be able to
build and run as normal, except that Kudu related operations will fail. Or
you can build Kudu locally and link Impala against it using the
KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables

I also filed IMPALA-7515 <https://issues.apache.org/jira/browse/IMPALA-7515>
to track the eventual removal of toolchain Kudu support

On Wed, Aug 22, 2018 at 11:31 AM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> So the toolchain binaries are provided for: centos5,6,7, debian7,8,
> sles11,12, ubuntu14,16
> The new CDH component binaries will be available for: redhat6,7, debian8,
> sles12, ubuntu16
> so we would be dropping easy support for building on basically centos5,
> debian7, sles11, and ubuntu14
>
> For people who want to develop on other oses like Ubuntu 18.04, it would
> still be possible by compiling Kudu locally and using the
> KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables.
>
> I'm also strongly leaning towards hiding this functionality behind a flag
> for now (not many people chimed in on this thread, but I suspect there are
> still a decent number of people developing on Ubuntu 14 who I'll get angry
> emails from if they rebase and discover they have to upgrade their os), in
> which case building the toolchain would remain a viable option for using
> other oses as well, as least for now.
>
> On Tue, Aug 21, 2018 at 4:47 PM Tim Armstrong 
> wrote:
>
>> Is there a path to building a version of Kudu locally for an arbitrary
>> linux distro?
>>
>> Personally I am less concerned about 14.04 support and more concerned
>> about
>> what the path to upgrading to 18.04. It would also be nice for it to be at
>> least possible to develop on RedHat-derived distros even if it requires
>> some extra effort.
>>
>> On Tue, Aug 21, 2018 at 6:48 AM, Laszlo Gaal 
>> wrote:
>>
>> > +1 for simplifying Kudu updates.
>> >
>> > I am also still on Ubuntu 14.04, but I am all for simplifying Kudu
>> > integration:
>> > I agree with Thomas that Kudu snapshots should be grouped with the other
>> > CDH components.
>> > Given that Ubuntu 14.04 will be EOL'd next spring, upgrading the
>> > development OS
>> > is a reasonably small price to pay -- especially that it will soon
>> become
>> > necessary anyway.
>> >
>> > Thanks for doing this Thomas!
>> >
>> >   - Laszlo
>> >
>> > On Tue, Aug 21, 2018 at 12:34 AM Lars Volker  wrote:
>> >
>> > > I'm in favor of not spending developer time and effort to maintain
>> > > compatibility with 14.04. Personally I'm still developing on Ubuntu
>> 14.04
>> > > so I'd be happy if we can support it without much pain. On the other
>> hand
>> > > it EOLs in April 2019, so I might as well go to 18.04 now, should we
>> > decide
>> > > to drop support. Maybe not many other folks are on 14.04 after all?
>> > >
>> > >
>> > >
>> > > On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
>> > > tmarsh...@cloudera.com> wrote:
>> > >
>> > > > Impala community,
>> > > >
>> > > > For years now, Impala has utilized tarballs built by Cloudera and
>> > > uploaded
>> > > > to S3 for running most of the Hadoop components in the testing
>> > > minicluster.
>> > > > The one exception to this is Kudu, which is instead provided by the
>> > > > toolchain.
>> > > >
>> > > > This was never ideal - native-toolchain makes more sense for
>> libraries
>> > > > where we want to build against a fairly static version, but Kudu is
>> > under
>> > > > active development and we'd like to always build against a
>> relatively
>> > > > up-to-date version. As a result, patches just bumping the version of
>> > Kudu
>> > > > make up a significant portion of the commit history of
>> > native-toolchain.
>> > > >
>> > > > Thank

Re: Improving Kudu Build Support

[Thomas Tauber-Marshall]

So the toolchain binaries are provided for: centos5,6,7, debian7,8,
sles11,12, ubuntu14,16
The new CDH component binaries will be available for: redhat6,7, debian8,
sles12, ubuntu16
so we would be dropping easy support for building on basically centos5,
debian7, sles11, and ubuntu14

For people who want to develop on other oses like Ubuntu 18.04, it would
still be possible by compiling Kudu locally and using the
KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables.

I'm also strongly leaning towards hiding this functionality behind a flag
for now (not many people chimed in on this thread, but I suspect there are
still a decent number of people developing on Ubuntu 14 who I'll get angry
emails from if they rebase and discover they have to upgrade their os), in
which case building the toolchain would remain a viable option for using
other oses as well, as least for now.

On Tue, Aug 21, 2018 at 4:47 PM Tim Armstrong 
wrote:

> Is there a path to building a version of Kudu locally for an arbitrary
> linux distro?
>
> Personally I am less concerned about 14.04 support and more concerned about
> what the path to upgrading to 18.04. It would also be nice for it to be at
> least possible to develop on RedHat-derived distros even if it requires
> some extra effort.
>
> On Tue, Aug 21, 2018 at 6:48 AM, Laszlo Gaal 
> wrote:
>
> > +1 for simplifying Kudu updates.
> >
> > I am also still on Ubuntu 14.04, but I am all for simplifying Kudu
> > integration:
> > I agree with Thomas that Kudu snapshots should be grouped with the other
> > CDH components.
> > Given that Ubuntu 14.04 will be EOL'd next spring, upgrading the
> > development OS
> > is a reasonably small price to pay -- especially that it will soon become
> > necessary anyway.
> >
> > Thanks for doing this Thomas!
> >
> >   - Laszlo
> >
> > On Tue, Aug 21, 2018 at 12:34 AM Lars Volker  wrote:
> >
> > > I'm in favor of not spending developer time and effort to maintain
> > > compatibility with 14.04. Personally I'm still developing on Ubuntu
> 14.04
> > > so I'd be happy if we can support it without much pain. On the other
> hand
> > > it EOLs in April 2019, so I might as well go to 18.04 now, should we
> > decide
> > > to drop support. Maybe not many other folks are on 14.04 after all?
> > >
> > >
> > >
> > > On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
> > > tmarsh...@cloudera.com> wrote:
> > >
> > > > Impala community,
> > > >
> > > > For years now, Impala has utilized tarballs built by Cloudera and
> > > uploaded
> > > > to S3 for running most of the Hadoop components in the testing
> > > minicluster.
> > > > The one exception to this is Kudu, which is instead provided by the
> > > > toolchain.
> > > >
> > > > This was never ideal - native-toolchain makes more sense for
> libraries
> > > > where we want to build against a fairly static version, but Kudu is
> > under
> > > > active development and we'd like to always build against a relatively
> > > > up-to-date version. As a result, patches just bumping the version of
> > Kudu
> > > > make up a significant portion of the commit history of
> > native-toolchain.
> > > >
> > > > Thanks to work I'm currently doing at Cloudera, there will soon be
> > > snapshot
> > > > tarballs of Kudu getting uploaded to S3 along with the other Hadoop
> > > > components. I would like to propose that Impala switch to using those
> > > > instead of the toolchain Kudu.
> > > >
> > > > One problem here is that the new Kudu tarballs will not be getting
> > build
> > > > for Ubuntu 14.04, only 16.04, but we still officially say we support
> > > > development on 14.04.
> > > >
> > > > One option here would be to maintain the toolchain Kudu for now and
> > hide
> > > > downloading of the new tarballs behind a flag. We could also postpone
> > > some
> > > > of this work until 14.04 is less common. Or, given that the
> > > > bootstrap_development script already only supports 16.04, we might
> want
> > > to
> > > > just drop support for building on 14.04.
> > > >
> > > > Thoughts?
> > > >
> > >
> >
>

Improving Kudu Build Support

[Thomas Tauber-Marshall]

Impala community,

For years now, Impala has utilized tarballs built by Cloudera and uploaded
to S3 for running most of the Hadoop components in the testing minicluster.
The one exception to this is Kudu, which is instead provided by the
toolchain.

This was never ideal - native-toolchain makes more sense for libraries
where we want to build against a fairly static version, but Kudu is under
active development and we'd like to always build against a relatively
up-to-date version. As a result, patches just bumping the version of Kudu
make up a significant portion of the commit history of native-toolchain.

Thanks to work I'm currently doing at Cloudera, there will soon be snapshot
tarballs of Kudu getting uploaded to S3 along with the other Hadoop
components. I would like to propose that Impala switch to using those
instead of the toolchain Kudu.

One problem here is that the new Kudu tarballs will not be getting build
for Ubuntu 14.04, only 16.04, but we still officially say we support
development on 14.04.

One option here would be to maintain the toolchain Kudu for now and hide
downloading of the new tarballs behind a flag. We could also postpone some
of this work until 14.04 is less common. Or, given that the
bootstrap_development script already only supports 16.04, we might want to
just drop support for building on 14.04.

Thoughts?

Re: Assign JIRA

[Thomas Tauber-Marshall]

On Mon, Aug 6, 2018 at 1:18 PM Jim Apple 
wrote:

> Thomas, I added you as an admin so you can fix this once you know Andrew
> Sheman's username - there are two "Andrew Sherman"s in the ASF JIRA system,
> it seems.
>
> Project Settings > Users and roles > Add users to a role. Use the
> "Contributors" role.
>

Thanks!


>
> On Mon, Aug 6, 2018 at 12:54 PM, Thomas Tauber-Marshall <
> tmarsh...@cloudera.com.invalid> wrote:
>
> > I'm trying to assign a JIRA to a new contributor (Andrew Sherman), but
> > they're not showing up as an option. I guess I need admin privileges on
> > JIRA to add them to the project? Can anyone help me out with that? Thanks
> >
>

Assign JIRA

[Thomas Tauber-Marshall]

I'm trying to assign a JIRA to a new contributor (Andrew Sherman), but
they're not showing up as an option. I guess I need admin privileges on
JIRA to add them to the project? Can anyone help me out with that? Thanks

Re: boost::scoped_ptr vs std::unique_ptr

[Thomas Tauber-Marshall]

I'm definitely in favor of using more standard c++ to reduce both confusion
and our reliance on boost, especially as I suspect a lot of people (eg. me)
don't know the subtle difference between scoped_ptr and unique_ptr off the
top of their head anyways.

Fwiw, I was under the impression from talking with people in the past that
we were already trying to make this move, and the
PartitionedAggregationNode refactor that just went in made the switch to
unique_ptr, though no one commented on it in the review.

On Thu, Jul 5, 2018 at 4:39 PM Tim Armstrong
 wrote:

> I was just talking with Michael Ho on a review about this
> https://gerrit.cloudera.org/#/c/10810/7/be/src/exec/scan-node.h@271
>
> For a while we've continued using scoped_ptr in some places because it
> supports a smaller set of operators and implies that the pointer isn't
> movable. See
>
> https://cwiki.apache.org/confluence/display/IMPALA/Resource+Management+Best+Practices+in+Impala
> .
>
> I don't think we're consistently following this pattern and it seems to
> cause some confusion about what the best practice is, particularly for
> people coming from other code bases. I personally like the distinction, but
> I don't feel that strongly about it.
>
> What do people think? Should we continue using scoped_ptr or move away from
> it. There is already a JIRA to make the change but we haven't done it
> because of the above reasons:
> https://issues.apache.org/jira/browse/IMPALA-3444
>
> - Tim
>

Re: Broken/Flaky Tests

[Thomas Tauber-Marshall]

So while its definitely better, there are still a large number of failing
builds. We've been hit by at least: IMPALA-6642
<https://issues.apache.org/jira/browse/IMPALA-6642>, IMPALA-6956
<https://issues.apache.org/jira/browse/IMPALA-6956>, IMPALA-7101
<https://issues.apache.org/jira/browse/IMPALA-7101> and IMPALA-3040
<https://issues.apache.org/jira/browse/IMPALA-3040>
all within the last day, along with some mysterious crashes that I haven't
filed anything for with Apache yet as there's very little info about what's
actually going on. There are still multiple builds that haven't been green
in over a month.
<https://issues.apache.org/jira/browse/IMPALA-6642>

Of course, if we hold commits for too long, there's a danger that when we
open things back up a bunch of changes will all land at the same time and
destabilize the builds again, putting back in the same situation. So, I
would say at a minimum that any changes that are relatively minor and low
risk can go in now.

My preference would be to hold off on major changes until we have more
stability.

On Fri, Jun 1, 2018 at 10:30 AM Lars Volker  wrote:

> Hi Thomas,
>
> Can you give an update on where we are with the builds?
>
> We currently have ~15 changes with a +2:
>
> https://gerrit.cloudera.org/#/q/status:open+project:Impala-ASF+branch:master+label:Code-Review%253D2
>
> Thanks, Lars
>
> On Fri, May 25, 2018 at 5:20 PM, Henry Robinson  wrote:
>
> > +1 - thanks for worrying about build health.
> >
> > On 25 May 2018 at 17:18, Jim Apple  wrote:
> >
> > > Sounds good to me. Thanks for taking ownership!
> > >
> > > On Fri, May 25, 2018 at 5:10 PM Thomas Tauber-Marshall <
> > > tmarsh...@cloudera.com> wrote:
> > >
> > > > Hey Impala community,
> > > >
> > > > There seems to have been an unusually large number of flaky or broken
> > > tests
> > > > <
> > > > https://issues.apache.org/jira/browse/IMPALA-7073?jql=
> > > project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%
> > > 22In%20Progress%22%2C%20Reopened)%20AND%20labels%
> > > 20in%20(flaky%2C%20broken-build)
> > > > >
> > > > cropping up in the last few weeks. I'd like to suggest that we hold
> off
> > > on
> > > > merging new changes that aren't related to fixing those testing
> issues
> > > for
> > > > at least a few days until things become more stable.
> > > >
> > > > Does anyone have any objections? If not, I'll send out another email
> > when
> > > > more of the issues have been addressed.
> > > >
> > > > Thanks,
> > > > Thomas Tauber-Marshall
> > > >
> > >
> >
>

Re: Does Impala support grouping sets or some other operation

[Thomas Tauber-Marshall]

No, Impala does not support grouping sets. There is work ongoing to allow
for multiple groupings in a single query (see
https://issues.apache.org/jira/browse/IMPALA-110), which will eventually be
used to implement grouping sets, but there is no timeline on when that work
will be done.

We of course always welcome contributions if there's any part of this work
that you'd like to take on.

On Tue, May 29, 2018 at 8:00 PM 陈 小健  wrote:

> Hi,
> Does Impala support grouping sets?
> If so,I can use sql1 instead of sql2 to avoid extra scan operation
> Or does impala support some operation like it?
>
> sql1:
> SELECT A,
>B,
>C,
>count(1)
> FROM tableName
> GROUP BY A,
>  B,
>  C
> grouping sets
> (
>
>(A,B),
>(B,C)
>
> )
>
>
> sql2:
> SELECT A,
>B,
>NULL,
>count(1)
> FROM tableName
> GROUP BY A,
>  B
> UNION ALL
> SELECT NULL,
>B,
>C,
>count(1)
> FROM tableName
> GROUP BY B,
>  C
>
>
>
>
>
> Best regards,
> Xiaojian
>
>

Broken/Flaky Tests

[Thomas Tauber-Marshall]

Hey Impala community,

There seems to have been an unusually large number of flaky or broken tests
<https://issues.apache.org/jira/browse/IMPALA-7073?jql=project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20labels%20in%20(flaky%2C%20broken-build)>
cropping up in the last few weeks. I'd like to suggest that we hold off on
merging new changes that aren't related to fixing those testing issues for
at least a few days until things become more stable.

Does anyone have any objections? If not, I'll send out another email when
more of the issues have been addressed.

Thanks,
Thomas Tauber-Marshall

Re: Statically link Kudu client

[Thomas Tauber-Marshall]

There isn't a built in way in buildall to statically link the Kudu client,
and I'm not aware of any way to do it.

We don't provide any guarantees or do any testing around compatibility of
Impala with Kudu clients of different versions other than the versions that
correspond, i.e. the value of IMPALA_KUDU_VERSION for the branch of Impala
that you want to run. If you upgraded Impala but were dynamically linking
against an older version of the Kudu client that may have been what caused
the crash.

And Todd is right as well that you'll need to use a libkudu that was built
with Impala's toolchain, either by building the native-toolchain project
yourself, or by downloading the pre-built toolchain with
bin/bootstrap-toolchain.py, which will automatically grab the right libkudu
version for the branch of Impala that you're building.

On Tue, May 22, 2018 at 10:14 PM Todd Lipcon  wrote:

> Hi Quanlong,
>
> I think the comment you found must be out of date. I think Thomas may know
> the latest state of how the Kudu client is built as part of
> "native-toolchain".
>
> One thing to note is that, since Impala builds with its own toolchain, you
> need to make sure that it dynamic-links against the libkudu_client that
> comes from the toolchain. If you end up linking against one built using
> your system toolchain there is a chance that you'll get ABI mismatches and
> strange crashes. Perhaps that's what you saw when you first tried.
>
> -Todd
>
> On Tue, May 22, 2018 at 3:21 PM, Quanlong Huang 
> wrote:
>
>> Hi all,
>>
>>
>> Recently when I tried to upgrade our Impala cluster in CDH5.7.1 to
>> Impala-2.12, impalad crashed when inserting data to kudu tables. However,
>> it works when reading from kudu tables.
>> Finally, I found out that the kudu client
>> (/usr/lib/x86_64-linux-gnu/libkudu_client.so.0) is still linked
>> dynamically. Issues are resolved after I update libkudu_client.so.0.
>>
>>
>> In kudu-table-sink.cc, the comments said the kudu-client can be linked
>> statically:
>> https://github.com/apache/impala/blob/2.12.0/be/src/exec/kudu-table-sink.cc#L151
>> However, when building with -static in ./buildall.sh, the kudu-client is
>> still linked dynamically (see `ldd be/build/latest/service/impalad`). Is
>> there a build option to link it statically?
>>
>
>>
>> Thanks,
>> Quanlong
>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: [VOTE] 3.0.0 release candidate 1

[Thomas Tauber-Marshall]

+1 binding

Ran the release test job:
https://jenkins.impala.io/job/release-test-ub1604/13/

On Wed, May 2, 2018 at 9:17 PM Jim Apple  wrote:

> +1, binding
>
> I looked at the output of the release testing job that Sailesh appears to
> have run on jenkins.impala.io.
>
> On Tue, May 1, 2018 at 4:34 PM Sailesh Mukil  wrote:
>
> > This is a vote to release Impala 3.0.0
> >
> > The artifacts for testing can be downloaded from:
> > https://dist.apache.org/repos/dist/dev/impala/3.0.0/RC1/
> > Git tag: 3.0.0-rc1
> > Tree hash: d5bf1fdeb20ac332ce48c6a2b0588bcaf7ac9cb0
> >
> > Please vote +1 or -1. -1 votes should be accompanied by an explanation of
> > the reason. Only PMC members have binding votes, but other
> > community members are encouraged to cast non-binding votes. This vote
> will
> > pass if there are 3 binding +1 votes and more binding +1 votes than -1
> > votes.
> >
> > Please also note that this release is a new major release, and does not
> > override the recently released 2.12.0 as the latest release. Rather,
> 2.12.0
> > is the latest release from the 2.x branch and 3.0.0 is the first and
> latest
> > release from the 3.x branch.
> >
> > This wiki page describes how to check the release before you vote:
> > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> >
> > The vote will be open until 04:30 PM PST on Friday, May 4
> >
>

Re: [VOTE] 2.12.0 release candidate 1

[Thomas Tauber-Marshall]

+1 (binding)

Ran the release test job:
https://jenkins.impala.io/job/release-test-ub1604/9/


On Fri, Apr 20, 2018 at 11:58 AM Lars Volker  wrote:

> +1 (binding)
>
> * Validated that the sha512 file has been fixed
> * Validated the GPG signature
> * Ran the release test job on jenkins.impala.io, result is here
> 
> * Unpacked and compiled the tarball on my development machine, made sure
> the daemon starts, and ran some queries.
>
> I noticed that the build identifies itself as "impalad version
> 2.12.0-RELEASE DEBUG (build Could not obtain git hash)".
> This is tracked in IMPALA-4744
>  and we're released
> with
> this before. Should we require a subsequent RC, we can manually change
> GIT_HASH in bin/save-version.sh.
>
> Thanks for creating the RC, Sailesh!
>
>
> On Thu, Apr 19, 2018 at 4:57 PM, Sailesh Mukil 
> wrote:
>
> > I've already updated that file to not have an absolute path. Could you
> > download that and try again?
> >
> > On Thu, Apr 19, 2018, 4:15 PM Jim Apple  wrote:
> >
> > > -1:
> > >
> > > /tmp/foo:$ sha512sum --check apache-impala-2.12.0.tar.gz.sha512
> > > sha512sum: /tmp/apache-impala-2.12.0.tar.gz: No such file or directory
> > > /tmp/apache-impala-2.12.0.tar.gz: FAILED open or read
> > > sha512sum: WARNING: 1 listed file could not be read
> > >
> > >
> > > On Thu, Apr 19, 2018 at 2:40 PM, Sailesh Mukil 
> > > wrote:
> > >
> > > > This is a vote to release Impala 2.12.0
> > > >
> > > > The artifacts for testing can be downloaded from:
> > > > https://dist.apache.org/repos/dist/dev/impala/2.12.0/RC1/
> > > > Git tag: 2.12.0-rc1
> > > > Tree hash: 07f18b8539a044c506bb3ff5a118f3b7b5ea1cf7
> > > >
> > > > Please vote +1 or -1. -1 votes should be accompanied by an
> explanation
> > of
> > > > the reason. Only PMC members have binding votes, but other
> > > > community members are encouraged to cast non-binding votes. This vote
> > > will
> > > > pass if there are 3 binding +1 votes and more binding +1 votes than
> -1
> > > > votes.
> > > >
> > > > This wiki page describes how to check the release before you vote:
> > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> > > >
> > > > The vote will be open until 3:00PM PST on Tuesday, April 24
> > > >
> > >
> >
>

Native toolchain branching

[Thomas Tauber-Marshall]

Apache Impala people,

(if you don't contribute to Cloudera's native-toolchain project, you can
stop reading)

I just wanted to let you know about some changes being made to
native-toolchain, a project Cloudera hosts to make it easier to bootstrap
Impala's dependencies: https://github.com/cloudera/native-toolchain

Cloudera is in the process of branching a new major version of our distro,
CDH6.0, while continuing to maintain the old major version, CDH5.x.

To support that, native-toolchain has also been branched - master now
corresponds to CDH6.0, and there's a new 5.x branch.

This means that for now, most changes to native-toolchain that go into
master will also need to be cherry-picked onto 5.x. Given the low frequency
of commits to native-toolchain, we will probably not automate this process.

Thanks,
Thomas Tauber-Marshall

Re: [ANNOUNCE] Apache Impala 2.11.0 release

[Thomas Tauber-Marshall]

On Thu, Jan 18, 2018 at 10:43 AM Lars Volker  wrote:

> Done: https://twitter.com/ApacheImpala/status/954061402070704129


Thanks!


>
>
> On Thu, Jan 18, 2018 at 10:31 AM, Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
> > -everyone but dev@
> >
> > Could someone with access to the Impala Twitter account (according to
> Henry
> > that should be anyone on the PMC) announce this on Twitter as well?
> Thanks
> >
> > On Thu, Jan 18, 2018 at 10:17 AM Thomas Tauber-Marshall <
> > tmarsh...@cloudera.com> wrote:
> >
> > > The Apache Impala team is pleased to announce the release of Impala
> > 2.11.0
> > >
> > > Impala is a high-performance distributed SQL engine.
> > >
> > > The release is available at: https://impala.apache.org/downloads.html
> > >
> > > Thanks,
> > >
> > > The Apache Impala team
> > >
> >
>

Re: [ANNOUNCE] Apache Impala 2.11.0 release

[Thomas Tauber-Marshall]

-everyone but dev@

Could someone with access to the Impala Twitter account (according to Henry
that should be anyone on the PMC) announce this on Twitter as well? Thanks

On Thu, Jan 18, 2018 at 10:17 AM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> The Apache Impala team is pleased to announce the release of Impala 2.11.0
>
> Impala is a high-performance distributed SQL engine.
>
> The release is available at: https://impala.apache.org/downloads.html
>
> Thanks,
>
> The Apache Impala team
>

[ANNOUNCE] Apache Impala 2.11.0 release

[Thomas Tauber-Marshall]

The Apache Impala team is pleased to announce the release of Impala 2.11.0

Impala is a high-performance distributed SQL engine.

The release is available at: https://impala.apache.org/downloads.html

Thanks,

The Apache Impala team

2.11 release progress

[Thomas Tauber-Marshall]

Just an update on the 2.11 release progress: 2.11 has effectively been
'released' now, as the artifacts are available on the website:
http://impala.apache.org/downloads.html

However, the docs are not quite ready yet. Once they are, hopefully in a
few days, I'll send out the formal ANNOUNCEMENT, here and elsewhere.

If you have any outstanding DOCS reviews for 2.11, please get them done in
a timely manner. Thanks!

Re: New Impala Contributors: IMPALA-6296

[Thomas Tauber-Marshall]

On Wed, Jan 3, 2018 at 10:06 AM Manaswini Maharana 
wrote:

> Thanks for the pointers, Thomas. I was able to run the Jenkin tests, but
> not sure how to manually add the links to Gerrit as recommended. Do I  add
> it through the reply text box as shown below?
>

Yes, you can just post it as a comment on the review using the "Reply"
button


>
> - Mansi
> ​
>
> On Tue, Jan 2, 2018 at 3:04 PM, Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
>> On Fri, Dec 29, 2017 at 11:12 AM Manaswini Maharana <
>> mmahar...@cloudera.com>
>> wrote:
>>
>> > I've pushed the initial changes to -
>> > https://gerrit.cloudera.org/#/c/8923/
>> >
>> > Steps that I've followed to make this contribution - As this is my
>> first,
>> > I want to elaborate a little bit to ensure I'm covering the bases right.
>> >
>> > 1. After the following the extensive contribution guide
>>
> > <
>> https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala
>> >,
>
>
>> > setup the development environment on machine with Ubuntu 16.04 OS, 15 GB
>> > memory, Intel® Core™ i7-8550U CPU @ 1.80GHz × 8  & 512 GB SSD
>> > 2. Modified the build process to avoiding building against -so
>> > (IMPALA-5365 & IMPALA-3926)
>> > 3. Reproduced the issue. Attached Impalad error log.
>> > 4. Put in the code fix - added a condition check to ensure
>> > 'emit_perf_map_' evaluates to true before the dependent
>> > object(perf_map_lock_) is asserted using DCHECK.
>> > 5. Unit tested the changes using Impala-shell. No daemon crash
>> encountered
>> > this time.
>> > 6. Loaded and ran the "be" unit test cases to ensure the change did not
>> > break other stuff. Attached the results.
>> >
>> >
>> > Questions I have:
>> > 1. Is it recommened to run more end-to-end unit tests or does it depend
>> on
>> > the intensity of the change ? Like here I've limited it to just running
>> be
>> > tests, is that ok?
>> >
>>
>> In general, you're expected to run any tests that may reasonably be
>> affected by the change, and of course the more testing the better. This
>> change affects basically any query that uses codegen, which is a
>> significant portion of our tests, so running the full suite would probably
>> be a good idea (eg bin/run-all-tests.sh)
>>
>> You may also want to try out the Jenkins jobs that we provide for testing
>> patches, as described here:
>>
>> https://cwiki.apache.org/confluence/display/IMPALA/Using+Gerrit+to+submit+and+review+patches#UsingGerrittosubmitandreviewpatches-Verifyingapatch(opentoallImpalacontributors)
>>
>>
>> > 2. As per Tim's comment, I should see *.asm files in codegen-modules
>> after
>> > the fix, but I don't see any. Am I missing anything ?
>> >
>>
>> You may need to manually create the directory
>> $IMPALA_HOME/codegen-modules/
>>
>>
>> >
>> > Thanks & appreciate your input,
>> > Mansi
>> >
>> >
>> > On Fri, Dec 29, 2017 at 12:43 PM, Manaswini Maharana <
>> > mmahar...@cloudera.com> wrote:
>> >
>> >> Thank you!
>> >>
>> >> On Fri, Dec 29, 2017 at 11:47 AM, Jim Apple 
>> wrote:
>> >>
>> >>> Done
>> >>> On Fri, Dec 29, 2017 at 7:12 AM, Manaswini Maharana
>> >>>  wrote:
>> >>> > username: mmaharana
>> >>> >
>> >>> > On Fri, Dec 29, 2017 at 10:07 AM, Jin Chul Kim 
>> >>> wrote:
>> >>> >
>> >>> >> Hi,
>> >>> >>
>> >>> >> I guess she doesn't have a privilege to change assignee. The
>> >>> permission
>> >>> >> should be provided by somebody who has an admin privilege.
>> >>> >> @Mansi, please share your username.
>> >>> >>
>> >>> >> Best regards,
>> >>> >> Jinchul
>> >>> >>
>> >>> >> 2017-12-30 0:02 GMT+09:00 Vincent Tran :
>> >>> >>
>> >>> >> > You should be able to just simply "assign to me" if you are
>> signed
>> >>> into
>> >>> >> ADD
>> >>> >> > Jira.
>> >>> >> >
>> >>> >> > On Dec 29, 2017 9:54

Re: New Impala Contributors: IMPALA-6296

[Thomas Tauber-Marshall]

On Fri, Dec 29, 2017 at 11:12 AM Manaswini Maharana 
wrote:

> I've pushed the initial changes to -
> https://gerrit.cloudera.org/#/c/8923/
>
> Steps that I've followed to make this contribution - As this is my first,
> I want to elaborate a little bit to ensure I'm covering the bases right.
>
> 1. After the following the extensive contribution guide
> ,
> setup the development environment on machine with Ubuntu 16.04 OS, 15 GB
> memory, Intel® Core™ i7-8550U CPU @ 1.80GHz × 8  & 512 GB SSD
> 2. Modified the build process to avoiding building against -so
> (IMPALA-5365 & IMPALA-3926)
> 3. Reproduced the issue. Attached Impalad error log.
> 4. Put in the code fix - added a condition check to ensure
> 'emit_perf_map_' evaluates to true before the dependent
> object(perf_map_lock_) is asserted using DCHECK.
> 5. Unit tested the changes using Impala-shell. No daemon crash encountered
> this time.
> 6. Loaded and ran the "be" unit test cases to ensure the change did not
> break other stuff. Attached the results.
>
>
> Questions I have:
> 1. Is it recommened to run more end-to-end unit tests or does it depend on
> the intensity of the change ? Like here I've limited it to just running be
> tests, is that ok?
>

In general, you're expected to run any tests that may reasonably be
affected by the change, and of course the more testing the better. This
change affects basically any query that uses codegen, which is a
significant portion of our tests, so running the full suite would probably
be a good idea (eg bin/run-all-tests.sh)

You may also want to try out the Jenkins jobs that we provide for testing
patches, as described here:
https://cwiki.apache.org/confluence/display/IMPALA/Using+Gerrit+to+submit+and+review+patches#UsingGerrittosubmitandreviewpatches-Verifyingapatch(opentoallImpalacontributors)


> 2. As per Tim's comment, I should see *.asm files in codegen-modules after
> the fix, but I don't see any. Am I missing anything ?
>

You may need to manually create the directory $IMPALA_HOME/codegen-modules/


>
> Thanks & appreciate your input,
> Mansi
>
>
> On Fri, Dec 29, 2017 at 12:43 PM, Manaswini Maharana <
> mmahar...@cloudera.com> wrote:
>
>> Thank you!
>>
>> On Fri, Dec 29, 2017 at 11:47 AM, Jim Apple  wrote:
>>
>>> Done
>>> On Fri, Dec 29, 2017 at 7:12 AM, Manaswini Maharana
>>>  wrote:
>>> > username: mmaharana
>>> >
>>> > On Fri, Dec 29, 2017 at 10:07 AM, Jin Chul Kim 
>>> wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> I guess she doesn't have a privilege to change assignee. The
>>> permission
>>> >> should be provided by somebody who has an admin privilege.
>>> >> @Mansi, please share your username.
>>> >>
>>> >> Best regards,
>>> >> Jinchul
>>> >>
>>> >> 2017-12-30 0:02 GMT+09:00 Vincent Tran :
>>> >>
>>> >> > You should be able to just simply "assign to me" if you are signed
>>> into
>>> >> ADD
>>> >> > Jira.
>>> >> >
>>> >> > On Dec 29, 2017 9:54 AM, "Manaswini Maharana" <
>>> mmahar...@cloudera.com>
>>> >> > wrote:
>>> >> >
>>> >> > > Good morning, Team - any update on this?
>>> >> > >
>>> >> > > - Mansi
>>> >> > >
>>> >> > >
>>> >> > > On Thu, Dec 28, 2017 at 12:32 AM, Manaswini Maharana <
>>> >> > > mmahar...@cloudera.com
>>> >> > > > wrote:
>>> >> > >
>>> >> > > > Hello team, I'd like to contribute to Impala and would like to
>>> start
>>> >> it
>>> >> > > > with this (IMPALA-6296) jira. Can I please have it assigned to
>>> me?
>>> >> > > >
>>> >> > > > My development environment is ready. I'm on ubuntu 16.04 so yes
>>> some
>>> >> > > > challenges to set up the environment especially with the
>>> >> > LD_LIBRARY_PATH
>>> >> > > > issues. So far I'm able to work past it and might ask for
>>> another
>>> >> pair
>>> >> > of
>>> >> > > > eyes to ensure it follows standard. The builds are going fine
>>> and
>>> >> runs
>>> >> > a
>>> >> > > > bit longer as I've disabled the -so. Apart from setup, I'm was
>>> able
>>> >> to
>>> >> > > > reproduce the issue and might have been ready with the fix
>>> which I'd
>>> >> > like
>>> >> > > > to discuss  once I'm officially assigned the jira.
>>> >> > > >
>>> >> > > > Thanks Tim, for the instructions, it was very helpful.
>>> >> > > >
>>> >> > > >
>>> >> > > > Thanks!
>>> >> > > > Mansi
>>> >> > > >
>>> >> > > >
>>> >> > > >
>>> >> > > > On Tue, Dec 12, 2017 at 8:24 PM, Tim Armstrong <
>>> >> > tarmstr...@cloudera.com>
>>> >> > > > wrote:
>>> >> > > >
>>> >> > > >> If you'd like to contribute a patch to Impala, but aren't sure
>>> what
>>> >> > > >> you want to work on, you can look at Impala's newbie issues:
>>> >> > > >> https://issues.apache.org/jira/issues/?filter=12341668. You
>>> can
>>> >> find
>>> >> > > >> detailed instructions on submitting patches at
>>> >> > > >> https://cwiki.apache.org/confluence/display/IMPALA/
>>> >> > > Contributing+to+Impala
>>> >> > > >> .
>>> >> > > >> This is a walkthrough of a ticket a new contributor could take
>>> on,
>>> >> > > >> with hopefully eno

Re: [RESULT] Vote on Impala 2.11.0 release candidate 1

[Thomas Tauber-Marshall]

Also, is there a PMC member that can help me with step #19 (publishing the
release artifacts) from
https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#

Thanks

On Thu, Dec 28, 2017 at 1:03 PM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

>
> https://lists.apache.org/thread.html/e757a6c29658af5e60f545621b6a3b2c6ca14adc39a3218698be7c77@%3Cdev.impala.apache.org%3E
>
> The vote has passed with the following tally:
>
> +1 (binding)
> - Bharath Vissapragada
> - Jim Apple
> - Lars Volker
> - Michael Brown
> - Taras Bobrovytsky
>
> -1 (binding) - None
> 0 - None
>
> Thanks everyone for testing and voting on the release.
>

[RESULT] Vote on Impala 2.11.0 release candidate 1

[Thomas Tauber-Marshall]

https://lists.apache.org/thread.html/e757a6c29658af5e60f545621b6a3b2c6ca14adc39a3218698be7c77@%3Cdev.impala.apache.org%3E

The vote has passed with the following tally:

+1 (binding)
- Bharath Vissapragada
- Jim Apple
- Lars Volker
- Michael Brown
- Taras Bobrovytsky

-1 (binding) - None
0 - None

Thanks everyone for testing and voting on the release.

[VOTE] 2.11.0 release candidate 1

[Thomas Tauber-Marshall]

This is a vote to release Impala 2.11.0

The artifacts for testing can be downloaded from:
https://dist.apache.org/repos/dist/dev/impala/2.11.0/RC1/
Git tag: 2.11.0-rc1
Tree hash: 9618d0f833cf826252d2c1b2a720eaabe1bef689

Please vote +1 or -1. -1 votes should be accompanied by an explanation of
the reason. Only PMC members have binding votes, but other
community members are encouraged to cast non-binding votes. This vote will
pass if there are 3 binding +1 votes and more binding +1 votes than -1
votes.

This wiki page describes how to check the release before you vote:
https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#

The vote will be open until 9:00PM PST on Friday, December 22

Re: [DISCUSS] 2.11.0 release

[Thomas Tauber-Marshall]

I don't appear to have the permission to create the branch:

Total 0 (delta 0), reused 0 (delta 0)
remote: You are not authorized to edit this repository.
remote:
To https://git-wip-us.apache.org/repos/asf/impala.git
 ! [remote rejected] branch-2.11.0 -> branch-2.11.0 (pre-receive hook
declined)
error: failed to push some refs to '
https://git-wip-us.apache.org/repos/asf/impala.git'

Does anyone know how I can get those permissions?

On Mon, Dec 11, 2017 at 2:45 PM Jim Apple  wrote:

> sgtm
>
> On Mon, Dec 11, 2017 at 1:49 PM, Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
> > Now that the issues that were brought up here have gone in, I propose
> that
> > we branch 2.11 at:
> >
> > commit b4cf5f2174b338c097e42d43c32b494a6e0b46c0
> > Author: Thomas Tauber-Marshall 
> > Date:   Mon Dec 11 10:02:45 2017 -0800
> >
> > IMPALA-6298: Skip test_profile_fragment_instances on local filesystem
> >
> > (minus IMPALA-5017)
> >
> > Of course, if there are additional fixes we think should be included,
> they
> > can be cherry-picked over.
> >
> > On Fri, Dec 8, 2017 at 12:30 PM Thomas Tauber-Marshall <
> > tmarsh...@cloudera.com> wrote:
> >
> > > So now that https://issues.apache.org/jira/browse/IMPALA-6292 has gone
> > > in, and with https://issues.apache.org/jira/browse/IMPALA-6286 in
> GVO, I
> > > think the last currently known blocker is
> > > https://gerrit.cloudera.org/#/c/8802/. Once that goes in I think we're
> > > good.
> > >
> > > On Thu, Dec 7, 2017 at 5:07 PM Tim Armstrong 
> > > wrote:
> > >
> > >> Actually it looks like we have a new blocker that Taras filed:
> > >> https://issues.apache.org/jira/browse/IMPALA-6292
> > >>
> > >> On Thu, Dec 7, 2017 at 10:03 AM, Tim Armstrong <
> tarmstr...@cloudera.com
> > >
> > >> wrote:
> > >>
> > >> > I think that makes sense. We'll have to go through the fix versions
> of
> > >> > recent JIRAs and make sure that they weren't set to 2.12 though.
> > >> >
> > >> > On Thu, Dec 7, 2017 at 9:33 AM, Thomas Tauber-Marshall <
> > >> > tmarsh...@cloudera.com> wrote:
> > >> >
> > >> >> Since the response from the community has been good, and now that
> all
> > >> of
> > >> >> the blocker JIRAs targeted for 2.11 have been closed, I propose
> that
> > we
> > >> >> cut
> > >> >> the release at:
> > >> >>
> > >> >> commit a4916e6d5f5f3542100af791534bfaf9ed544720
> > >> >> Author: Michael Ho 
> > >> >> Date:   Tue Dec 5 23:01:00 2017 -0800
> > >> >>
> > >> >> IMPALA-6281: Fix use-after-free in InitAuth()
> > >> >>
> > >> >> There are still a lot of open JIRAs
> > >> >> <https://issues.apache.org/jira/browse/IMPALA-6225?jql=proje
> > >> >> ct%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%22In%
> > >> >> 20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%
> > >> >> 22%20%3D%20%22Impala%202.11.0%22>
> > >> >> targeted at 2.11 at lower priorities, so it would be helpful if
> > people
> > >> >> could go through the ones assigned to them and make sure nothing
> > >> important
> > >> >> is being missed, otherwise we'll bulk update all of these to target
> > >> 2.12
> > >> >>
> > >> >> If there are no further concerns, I'll start testing at that
> commit,
> > >> and
> > >> >> if
> > >> >> all goes well create a release candidate and [VOTE] thread.
> > >> >>
> > >> >> On Thu, Nov 30, 2017 at 2:12 PM Matthew Jacobs  >
> > >> >> wrote:
> > >> >>
> > >> >> > +1
> > >> >> >
> > >> >> > Thanks, Thomas!
> > >> >> >
> > >> >> > On Thu, Nov 30, 2017 at 1:50 PM Michael Brown <
> mi...@cloudera.com>
> > >> >> wrote:
> > >> >> >
> > >> >> > > +1
> > >> >> > >
> > >> >> > > On Thu, Nov 30, 2017 at 12:46 PM, Thomas Tauber-Marshall <
> > >> >> > > tmarsh...@cloudera.com> wrote:
> > >> >> > >
> > >> >> > > > Folks,
> > >> >> > > >
> > >> >> > > > It has been over 2 months since we released Apache Impala
> > 2.10.0
> > >> and
> > >> >> > > there
> > >> >> > > > have been new feature improvements and a good number of bug
> > fixes
> > >> >> > checked
> > >> >> > > > in since then.
> > >> >> > > >
> > >> >> > > > I propose that we release 2.11.0 soon and I volunteer to be
> its
> > >> >> release
> > >> >> > > > manager. Please speak up and let the community know if anyone
> > has
> > >> >> any
> > >> >> > > > objections to this.
> > >> >> > > >
> > >> >> > > > Thanks,
> > >> >> > > > Thomas
> > >> >> > > >
> > >> >> > >
> > >> >> > --
> > >> >> > Sent from My iPhone
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] 2.11.0 release

[Thomas Tauber-Marshall]

Now that the issues that were brought up here have gone in, I propose that
we branch 2.11 at:

commit b4cf5f2174b338c097e42d43c32b494a6e0b46c0
Author: Thomas Tauber-Marshall 
Date:   Mon Dec 11 10:02:45 2017 -0800

IMPALA-6298: Skip test_profile_fragment_instances on local filesystem

(minus IMPALA-5017)

Of course, if there are additional fixes we think should be included, they
can be cherry-picked over.

On Fri, Dec 8, 2017 at 12:30 PM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> So now that https://issues.apache.org/jira/browse/IMPALA-6292 has gone
> in, and with https://issues.apache.org/jira/browse/IMPALA-6286 in GVO, I
> think the last currently known blocker is
> https://gerrit.cloudera.org/#/c/8802/. Once that goes in I think we're
> good.
>
> On Thu, Dec 7, 2017 at 5:07 PM Tim Armstrong 
> wrote:
>
>> Actually it looks like we have a new blocker that Taras filed:
>> https://issues.apache.org/jira/browse/IMPALA-6292
>>
>> On Thu, Dec 7, 2017 at 10:03 AM, Tim Armstrong 
>> wrote:
>>
>> > I think that makes sense. We'll have to go through the fix versions of
>> > recent JIRAs and make sure that they weren't set to 2.12 though.
>> >
>> > On Thu, Dec 7, 2017 at 9:33 AM, Thomas Tauber-Marshall <
>> > tmarsh...@cloudera.com> wrote:
>> >
>> >> Since the response from the community has been good, and now that all
>> of
>> >> the blocker JIRAs targeted for 2.11 have been closed, I propose that we
>> >> cut
>> >> the release at:
>> >>
>> >> commit a4916e6d5f5f3542100af791534bfaf9ed544720
>> >> Author: Michael Ho 
>> >> Date:   Tue Dec 5 23:01:00 2017 -0800
>> >>
>> >> IMPALA-6281: Fix use-after-free in InitAuth()
>> >>
>> >> There are still a lot of open JIRAs
>> >> <https://issues.apache.org/jira/browse/IMPALA-6225?jql=proje
>> >> ct%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%22In%
>> >> 20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%
>> >> 22%20%3D%20%22Impala%202.11.0%22>
>> >> targeted at 2.11 at lower priorities, so it would be helpful if people
>> >> could go through the ones assigned to them and make sure nothing
>> important
>> >> is being missed, otherwise we'll bulk update all of these to target
>> 2.12
>> >>
>> >> If there are no further concerns, I'll start testing at that commit,
>> and
>> >> if
>> >> all goes well create a release candidate and [VOTE] thread.
>> >>
>> >> On Thu, Nov 30, 2017 at 2:12 PM Matthew Jacobs 
>> >> wrote:
>> >>
>> >> > +1
>> >> >
>> >> > Thanks, Thomas!
>> >> >
>> >> > On Thu, Nov 30, 2017 at 1:50 PM Michael Brown 
>> >> wrote:
>> >> >
>> >> > > +1
>> >> > >
>> >> > > On Thu, Nov 30, 2017 at 12:46 PM, Thomas Tauber-Marshall <
>> >> > > tmarsh...@cloudera.com> wrote:
>> >> > >
>> >> > > > Folks,
>> >> > > >
>> >> > > > It has been over 2 months since we released Apache Impala 2.10.0
>> and
>> >> > > there
>> >> > > > have been new feature improvements and a good number of bug fixes
>> >> > checked
>> >> > > > in since then.
>> >> > > >
>> >> > > > I propose that we release 2.11.0 soon and I volunteer to be its
>> >> release
>> >> > > > manager. Please speak up and let the community know if anyone has
>> >> any
>> >> > > > objections to this.
>> >> > > >
>> >> > > > Thanks,
>> >> > > > Thomas
>> >> > > >
>> >> > >
>> >> > --
>> >> > Sent from My iPhone
>> >> >
>> >>
>> >
>> >
>>
>

Re: Re:Re: Build failure in TestAdmissionController

[Thomas Tauber-Marshall]

On Sun, Dec 10, 2017 at 2:38 PM Quanlong Huang 
wrote:

> Hi all,
>
>
> I've fixed all the test failures in my dev environment and here is the
> patch: https://gerrit.cloudera.org/#/c/8807/
> It has passed the gerrit-verify-dryrun-external test and here is the link:
> https://jenkins.impala.io/job/gerrit-verify-dryrun/1604/ (Don't know how
> to let Jenkins comment in gerrit after build succeed like you do ...)
>

Once you've gotten a +2, you'll have to ask a committer to run
gerrit-verify-dryrun, which is what talks to gerrit.


>
>
> Would you please have a look at it?
>
>
> Thanks,
> Quanlong
>
> At 2017-12-09 21:49:23, "Quanlong Huang"  wrote:
> >Thanks, Dimitris!
> >
> >
> >The cause is my problematic username and group name. There is a dot in
> them, which causes many tests to fail.
> >
> >
> >I think we should make less assumption about the username and group
> names. I create a JIRA (https://issues.apache.org/jira/browse/IMPALA-6301)
> for this and will submit a code review later.
> >
> >
> >Quanlong
> >
> >At 2017-12-09 08:11:36, "Dimitris Tsirogiannis" <
> dtsirogian...@cloudera.com> wrote:
> >>I think the problem is the '.' in what the test infrastructure uses as a
> >>group to assign roles which is your linux group 'quanlong.huang'. You can
> >>try to quote the group (not sure if that will work) or change your group.
> >>
> >>Dimitris
> >>
> >>On Fri, Dec 8, 2017 at 4:00 PM, Quanlong Huang 
> >>wrote:
> >>
> >>> Hi all,
> >>>
> >>>
> >>> I'm working on the ORC support feature (IMPALA-5717). However, I
> encounter
> >>> the following test errors not related to my patch:
> >>>
> >>>
> >>> ERROR at setup of TestGrantRevoke.test_role_update[exec_option:
> >>> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> >>> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_th
> >>> reshold': 0} | table_format: text/none]
> >>> authorization/test_grant_revoke.py:47: in setup_method
> >>> self.__test_cleanup()
> >>> authorization/test_grant_revoke.py:72: in __test_cleanup
> >>> self.client.execute("grant role grant_revoke_test_admin to group
> %s" %
> >>> group_name)
> >>> common/impala_connection.py:160: in execute
> >>> return self.__beeswax_client.execute(sql_stmt, user=user)
> >>> beeswax/impala_beeswax.py:173: in execute
> >>> handle = self.__execute_query(query_string.strip(), user=user)
> >>> beeswax/impala_beeswax.py:339: in __execute_query
> >>> handle = self.execute_query_async(query_string, user=user)
> >>> beeswax/impala_beeswax.py:335: in execute_query_async
> >>> return self.__do_rpc(lambda: self.imp_service.query(query,))
> >>> beeswax/impala_beeswax.py:460: in __do_rpc
> >>> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> >>> E   ImpalaBeeswaxException: ImpalaBeeswaxException:
> >>> EINNER EXCEPTION: 
> >>> EMESSAGE: AnalysisException: Syntax error in line 1:
> >>> E   grant role grant_revoke_test_admin to group quanlong.huang
> >>> E   ^
> >>> E   Encountered: .
> >>> E   Expected
> >>> E
> >>> E   CAUSED BY: Exception: Syntax error
> >>> ……
> >>> === FAILURES
> >>> ==
> >>> _ TestAdmissionController.test_set_request_pool
> >>> __
> >>> hs2/hs2_test_suite.py:48: in add_session
> >>> fn(self)
> >>> custom_cluster/test_admission_controller.py:212: in
> test_set_request_pool
> >>> self.__check_pool_rejected(client, pool, expected_error)
> >>> custom_cluster/test_admission_controller.py:144: in
> __check_pool_rejected
> >>> assert re.search(expected_error_re, str(e))
> >>> E   assert None
> >>> E+  where None = ("No mapping
> found
> >>> for request from user '\\w+' with requested pool ''",
> >>> "ImpalaBeeswaxException:\n INNER EXCEPTION:  'beeswaxd.ttypes.BeeswaxException'>\n
> >>> MESSAGE: No mapping found for request from user 'quanlong.huang' with
> >>> requested pool ''\n")
> >>> E+where  = re.search
> >>> E+and   "ImpalaBeeswaxException:\n INNER EXCEPTION:  >>> 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: No mapping found for
> >>> request from user 'quanlong.huang' with requested pool ''\n" =
> >>> str(ImpalaBeeswaxException())
> >>>
> >>>
> >>> What can I do to fix this?
> >>>
> >>>
> >>> Thanks,
> >>> Quanlong
>

Re: [DISCUSS] 2.11.0 release

[Thomas Tauber-Marshall]

So now that https://issues.apache.org/jira/browse/IMPALA-6292 has gone in,
and with https://issues.apache.org/jira/browse/IMPALA-6286 in GVO, I think
the last currently known blocker is https://gerrit.cloudera.org/#/c/8802/.
Once that goes in I think we're good.

On Thu, Dec 7, 2017 at 5:07 PM Tim Armstrong 
wrote:

> Actually it looks like we have a new blocker that Taras filed:
> https://issues.apache.org/jira/browse/IMPALA-6292
>
> On Thu, Dec 7, 2017 at 10:03 AM, Tim Armstrong 
> wrote:
>
> > I think that makes sense. We'll have to go through the fix versions of
> > recent JIRAs and make sure that they weren't set to 2.12 though.
> >
> > On Thu, Dec 7, 2017 at 9:33 AM, Thomas Tauber-Marshall <
> > tmarsh...@cloudera.com> wrote:
> >
> >> Since the response from the community has been good, and now that all of
> >> the blocker JIRAs targeted for 2.11 have been closed, I propose that we
> >> cut
> >> the release at:
> >>
> >> commit a4916e6d5f5f3542100af791534bfaf9ed544720
> >> Author: Michael Ho 
> >> Date:   Tue Dec 5 23:01:00 2017 -0800
> >>
> >> IMPALA-6281: Fix use-after-free in InitAuth()
> >>
> >> There are still a lot of open JIRAs
> >> <https://issues.apache.org/jira/browse/IMPALA-6225?jql=proje
> >> ct%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%22In%
> >> 20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%
> >> 22%20%3D%20%22Impala%202.11.0%22>
> >> targeted at 2.11 at lower priorities, so it would be helpful if people
> >> could go through the ones assigned to them and make sure nothing
> important
> >> is being missed, otherwise we'll bulk update all of these to target 2.12
> >>
> >> If there are no further concerns, I'll start testing at that commit, and
> >> if
> >> all goes well create a release candidate and [VOTE] thread.
> >>
> >> On Thu, Nov 30, 2017 at 2:12 PM Matthew Jacobs 
> >> wrote:
> >>
> >> > +1
> >> >
> >> > Thanks, Thomas!
> >> >
> >> > On Thu, Nov 30, 2017 at 1:50 PM Michael Brown 
> >> wrote:
> >> >
> >> > > +1
> >> > >
> >> > > On Thu, Nov 30, 2017 at 12:46 PM, Thomas Tauber-Marshall <
> >> > > tmarsh...@cloudera.com> wrote:
> >> > >
> >> > > > Folks,
> >> > > >
> >> > > > It has been over 2 months since we released Apache Impala 2.10.0
> and
> >> > > there
> >> > > > have been new feature improvements and a good number of bug fixes
> >> > checked
> >> > > > in since then.
> >> > > >
> >> > > > I propose that we release 2.11.0 soon and I volunteer to be its
> >> release
> >> > > > manager. Please speak up and let the community know if anyone has
> >> any
> >> > > > objections to this.
> >> > > >
> >> > > > Thanks,
> >> > > > Thomas
> >> > > >
> >> > >
> >> > --
> >> > Sent from My iPhone
> >> >
> >>
> >
> >
>

Re: [DISCUSS] 2.11.0 release

[Thomas Tauber-Marshall]

On Thu, Dec 7, 2017 at 10:37 AM Jim Apple  wrote:

> I think it would be great to get a fix for
> https://issues.apache.org/jira/browse/IMPALA-6285 in 2.11.0 if possible.
> It
> apparently could create a large performance boost.
>
> It's marked as a Blocker with affects version 2.11, but no target version.
> There are a few other tickets like this:
>
>
> https://issues.apache.org/jira/browse/IMPALA-3887?jql=project%20%3D%20IMPALA%20AND%20affectedVersion%20%3D%20%22Impala%202.11.0%22%20AND%20%22Target%20Version%22%20%3D%20EMPTY%20and%20resolution%20%3D%20EMPTY%20and%20Priority%20%3D%20Blocker%20ORDER%20BY%20priority%20DESC


Good point. Of the four JIRAs shown there, two are flaky tests that I don't
think are really blockers (IMPALA-6257
<https://issues.apache.org/jira/browse/IMPALA-6257> and IMPALA-3887
<https://issues.apache.org/jira/browse/IMPALA-3887>) and the other two (
IMPALA-6285 <https://issues.apache.org/jira/browse/IMPALA-6285> and
IMPALA-6081 <https://issues.apache.org/jira/browse/IMPALA-6081>) have
reviews out that have already been +2ed. So, it seems like a good idea to
wait for those two to go in.


>
>
> On Thu, Dec 7, 2017 at 9:33 AM, Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
> > Since the response from the community has been good, and now that all of
> > the blocker JIRAs targeted for 2.11 have been closed, I propose that we
> cut
> > the release at:
> >
> > commit a4916e6d5f5f3542100af791534bfaf9ed544720
> > Author: Michael Ho 
> > Date:   Tue Dec 5 23:01:00 2017 -0800
> >
> > IMPALA-6281: Fix use-after-free in InitAuth()
> >
> > There are still a lot of open JIRAs
> > <https://issues.apache.org/jira/browse/IMPALA-6225?jql=
> > project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%
> >
> 22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%22%20%3D%20%
> > 22Impala%202.11.0%22>
> > targeted at 2.11 at lower priorities, so it would be helpful if people
> > could go through the ones assigned to them and make sure nothing
> important
> > is being missed, otherwise we'll bulk update all of these to target 2.12
> >
> > If there are no further concerns, I'll start testing at that commit, and
> if
> > all goes well create a release candidate and [VOTE] thread.
> >
> > On Thu, Nov 30, 2017 at 2:12 PM Matthew Jacobs 
> wrote:
> >
> > > +1
> > >
> > > Thanks, Thomas!
> > >
> > > On Thu, Nov 30, 2017 at 1:50 PM Michael Brown 
> > wrote:
> > >
> > > > +1
> > > >
> > > > On Thu, Nov 30, 2017 at 12:46 PM, Thomas Tauber-Marshall <
> > > > tmarsh...@cloudera.com> wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > It has been over 2 months since we released Apache Impala 2.10.0
> and
> > > > there
> > > > > have been new feature improvements and a good number of bug fixes
> > > checked
> > > > > in since then.
> > > > >
> > > > > I propose that we release 2.11.0 soon and I volunteer to be its
> > release
> > > > > manager. Please speak up and let the community know if anyone has
> any
> > > > > objections to this.
> > > > >
> > > > > Thanks,
> > > > > Thomas
> > > > >
> > > >
> > > --
> > > Sent from My iPhone
> > >
> >
>

Re: [DISCUSS] 2.11.0 release

[Thomas Tauber-Marshall]

Since the response from the community has been good, and now that all of
the blocker JIRAs targeted for 2.11 have been closed, I propose that we cut
the release at:

commit a4916e6d5f5f3542100af791534bfaf9ed544720
Author: Michael Ho 
Date:   Tue Dec 5 23:01:00 2017 -0800

IMPALA-6281: Fix use-after-free in InitAuth()

There are still a lot of open JIRAs
<https://issues.apache.org/jira/browse/IMPALA-6225?jql=project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%22%20%3D%20%22Impala%202.11.0%22>
targeted at 2.11 at lower priorities, so it would be helpful if people
could go through the ones assigned to them and make sure nothing important
is being missed, otherwise we'll bulk update all of these to target 2.12

If there are no further concerns, I'll start testing at that commit, and if
all goes well create a release candidate and [VOTE] thread.

On Thu, Nov 30, 2017 at 2:12 PM Matthew Jacobs  wrote:

> +1
>
> Thanks, Thomas!
>
> On Thu, Nov 30, 2017 at 1:50 PM Michael Brown  wrote:
>
> > +1
> >
> > On Thu, Nov 30, 2017 at 12:46 PM, Thomas Tauber-Marshall <
> > tmarsh...@cloudera.com> wrote:
> >
> > > Folks,
> > >
> > > It has been over 2 months since we released Apache Impala 2.10.0 and
> > there
> > > have been new feature improvements and a good number of bug fixes
> checked
> > > in since then.
> > >
> > > I propose that we release 2.11.0 soon and I volunteer to be its release
> > > manager. Please speak up and let the community know if anyone has any
> > > objections to this.
> > >
> > > Thanks,
> > > Thomas
> > >
> >
> --
> Sent from My iPhone
>

[DISCUSS] 2.11.0 release

[Thomas Tauber-Marshall]

Folks,

It has been over 2 months since we released Apache Impala 2.10.0 and there
have been new feature improvements and a good number of bug fixes checked
in since then.

I propose that we release 2.11.0 soon and I volunteer to be its release
manager. Please speak up and let the community know if anyone has any
objections to this.

Thanks,
Thomas