Re: jenkins.impala.io maintenance
Jenkins has been updated. On Wed, Nov 8, 2017 at 3:13 PM Thomas Tauber-Marshall < tmarsh...@cloudera.com> wrote: > jenkins.impala.io need updates for some plugins to address a new security > advisory. > > It will be put into maintenance mode at 3:00pm PST so no new jobs can be > submitted. The upgrade will happen after all pending jobs complete. Once > the upgrade completes, another email will be sent out. > > Please speak up if you have any objection to the above. > > Thanks, > Thomas >
jenkins.impala.io maintenance
jenkins.impala.io need updates for some plugins to address a new security advisory. It will be put into maintenance mode at 3:00pm PST so no new jobs can be submitted. The upgrade will happen after all pending jobs complete. Once the upgrade completes, another email will be sent out. Please speak up if you have any objection to the above. Thanks, Thomas
Re: [VOTE] Graduate to a TLP
+1 On Tue, Oct 17, 2017 at 9:12 PM Bharath Vissapragada wrote: > +1 > > On Tue, Oct 17, 2017 at 7:10 PM, Mostafa Mokhtar > wrote: > > > +1 > > > > Thanks > > Mostafa > > > > > On Oct 17, 2017, at 7:09 PM, Brock Noland wrote: > > > > > > +1 > > > > > >> On Tue, Oct 17, 2017 at 9:07 PM, Lars Volker wrote: > > >> +1 > > >> > > >>> On Oct 17, 2017 19:07, "Jim Apple" wrote: > > >>> > > >>> Following our discussion > > >>> https://lists.apache.org/thread.html/2f5db4788aff9b0557354b9106c032 > > >>> 8a29c1f90c1a74a228163949d2@%3Cdev.impala.apache.org%3E > > >>> , I propose that we graduate to a TLP. According to > > >>> https://incubator.apache.org/guides/graduation.html# > > >>> community_graduation_vote > > >>> this is not required, and https://impala.apache.org/bylaws.html does > > not > > >>> say whose votes are "binding" in a graduation vote, so all community > > >>> members are welcome to vote. > > >>> > > >>> This will remain open 72 hours. I will be notifying general@incubator > > it > > >>> is > > >>> occurring. > > >>> > > >>> This is my +1. > > >>> > > >
Re: Broken build from Sentry
The problem has now been fixed on the Sentry side. I have a patch out to revert back to 5.13 for Sentry. For anyone who's curious: the problem turned out to be that Sentry wasn't shading all of their packages through maven, which resulted in dependency conflicts since we use a different version of thrift than they do. On Tue, Jun 20, 2017 at 11:44 AM Tim Armstrong wrote: > Yeah my change should build against either version by design. > > On 20 Jun. 2017 9:21 am, "Henry Robinson" wrote: > > > Yes, I did. AFAICT it worked fine. > > > > On 20 June 2017 at 09:19, Alexander Behm wrote: > > > > > Henry, did you try the revert on top of Tim's already-checked-in > change? > > > > > > On Tue, Jun 20, 2017 at 9:18 AM, Alexander Behm < > alex.b...@cloudera.com> > > > wrote: > > > > > > > Let's revert the version to buy us some time. That solution is a > > ticking > > > > time bomb though since that version will disappear soon. > > > > > > > > On Tue, Jun 20, 2017 at 8:56 AM, Henry Robinson > > > wrote: > > > > > > > >> I was able to run a build with EE and FE tests with Sentry reverted > to > > > >> 5.12 > > > >> - unless there are objections I'm going to post a patch to revert > the > > > >> version bump. > > > >> > > > >> On 20 June 2017 at 06:53, Thomas Tauber-Marshall < > > > tmarsh...@cloudera.com> > > > >> wrote: > > > >> > > > >> > So we've had a successful run of the nightlies now, and I've > > uploaded > > > >> the > > > >> > new jars to the s3 bucket, but Sentry still fails for some reason. > > > >> > > > > >> > I filed: https://issues.apache.org/jira/browse/IMPALA-5540 to > track > > > >> this > > > >> > > > > >> > On Tue, Jun 20, 2017 at 1:25 AM Alexander Kolbasov < > > > ak...@cloudera.com> > > > >> > wrote: > > > >> > > > > >> > > Note that Apache upstream story is more complicated - there was > a > > > >> change > > > >> > > done upstream that refactored a bunch of Sentry code that will > > cause > > > >> > > similar issue (I think it is SENTRY-1205). The change is present > > in > > > >> > Sentry > > > >> > > master but not in upstream sentry HA branch. > > > >> > > > > > >> > > On Mon, Jun 19, 2017 at 11:02 PM, Dimitris Tsirogiannis < > > > >> > > dtsirogian...@cloudera.com> wrote: > > > >> > > > > > >> > > > +Sasha, who I believe has more up-to-date information on this. > > > >> > > > > > > >> > > > On Mon, Jun 19, 2017 at 10:56 PM, Henry Robinson < > > > he...@apache.org> > > > >> > > wrote: > > > >> > > > > > > >> > > >> FWIW, I've been able to start Sentry by setting: > > > >> > > >> > > > >> > > >> export IMPALA_SENTRY_VERSION=1.5.1-cdh5.12.0-SNAPSHOT > > > >> > > >> > > > >> > > >> (i.e. rolling back to the previous version of Sentry). I > > haven't > > > >> yet > > > >> > > tried > > > >> > > >> to run tests - does anyone know an ETA for a fix coming out > of > > > >> > Cloudera > > > >> > > >> for > > > >> > > >> the 5.13-SNAPSHOT? If it might be a while, we should consider > > > >> > regressing > > > >> > > >> the Sentry version to unblock checkins. > > > >> > > >> > > > >> > > >> On 19 June 2017 at 15:31, Tim Armstrong < > > tarmstr...@cloudera.com > > > > > > > >> > > wrote: > > > >> > > >> > > > >> > > >> > It's unfortunately not that simple. The API change has been > > in > > > >> > Apache > > > >> > > >> > sentry > > > >> > > >> > > > > >> > > >> > So rolling back the API change temporarily solves the > problem > > > for > > > >> &
Re: Broken build from Sentry
So we've had a successful run of the nightlies now, and I've uploaded the new jars to the s3 bucket, but Sentry still fails for some reason. I filed: https://issues.apache.org/jira/browse/IMPALA-5540 to track this On Tue, Jun 20, 2017 at 1:25 AM Alexander Kolbasov wrote: > Note that Apache upstream story is more complicated - there was a change > done upstream that refactored a bunch of Sentry code that will cause > similar issue (I think it is SENTRY-1205). The change is present in Sentry > master but not in upstream sentry HA branch. > > On Mon, Jun 19, 2017 at 11:02 PM, Dimitris Tsirogiannis < > dtsirogian...@cloudera.com> wrote: > > > +Sasha, who I believe has more up-to-date information on this. > > > > On Mon, Jun 19, 2017 at 10:56 PM, Henry Robinson > wrote: > > > >> FWIW, I've been able to start Sentry by setting: > >> > >> export IMPALA_SENTRY_VERSION=1.5.1-cdh5.12.0-SNAPSHOT > >> > >> (i.e. rolling back to the previous version of Sentry). I haven't yet > tried > >> to run tests - does anyone know an ETA for a fix coming out of Cloudera > >> for > >> the 5.13-SNAPSHOT? If it might be a while, we should consider regressing > >> the Sentry version to unblock checkins. > >> > >> On 19 June 2017 at 15:31, Tim Armstrong > wrote: > >> > >> > It's unfortunately not that simple. The API change has been in Apache > >> > sentry > >> > > >> > So rolling back the API change temporarily solves the problem for > >> Cloudera, > >> > but we're going to have to deal with it at some point and get Impala > >> > building against both versions of the API. > >> > > >> > On Mon, Jun 19, 2017 at 2:55 PM, Thomas Tauber-Marshall < > >> > tmarsh...@cloudera.com> wrote: > >> > > >> > > Yes, the Sentry team has been contacted and they're going to be > >> rolling > >> > it > >> > > back. > >> > > > >> > > On Mon, Jun 19, 2017 at 4:53 PM Todd Lipcon > >> wrote: > >> > > > >> > > > Quick question from a bystander: it seems like Sentry committed an > >> > > > API-incompatible change. Instead of fixing on the Impala side, > >> should > >> > the > >> > > > Sentry project be notified that they may want to roll back such a > >> > change? > >> > > > It seems like an error on their part to do such a thing within a > >> minor > >> > > > version. > >> > > > > >> > > > On Mon, Jun 19, 2017 at 1:56 PM, Thomas Tauber-Marshall < > >> > > > tmarsh...@cloudera.com> wrote: > >> > > > > >> > > > > I'm working on getting the s3 jars updated, which presumably > will > >> fix > >> > > > that. > >> > > > > > >> > > > > The problem (to my understanding) is that the nightlies haven't > >> > passed > >> > > > > since the change went into Sentry and so the Jenkins job that > >> > normally > >> > > > > produces the new jars is still pulling in old bits. > >> > > > > > >> > > > > I've been talking with releng and they expect the new jars to be > >> > > > available > >> > > > > later today. > >> > > > > > >> > > > > On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong < > >> > tarmstr...@cloudera.com > >> > > > > >> > > > > wrote: > >> > > > > > >> > > > > > Looks like the build still breaks when starting up sentry > after > >> my > >> > > fix: > >> > > > > > > >> > > > > > > >> > > > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/ > >> > 1547/console > >> > > > > > > >> > > > > > *20:08:54* --> Starting the Sentry Policy Server*20:08:59* > >> Error > >> > in > >> > > > > > /home/ubuntu/Impala/testdata/bin/run-all.sh at line 58: > >> > > > > > $IMPALA_HOME/testdata/bin/run-sentry-service.sh > \*20:08:59* > + > >> > > > > > onexit*20:08:59* + df -m*20:08:59* Filesystem 1M-blocks > >> Used > >> > > > > > Available Use% Mount
Re: Broken build from Sentry
Yes, the Sentry team has been contacted and they're going to be rolling it back. On Mon, Jun 19, 2017 at 4:53 PM Todd Lipcon wrote: > Quick question from a bystander: it seems like Sentry committed an > API-incompatible change. Instead of fixing on the Impala side, should the > Sentry project be notified that they may want to roll back such a change? > It seems like an error on their part to do such a thing within a minor > version. > > On Mon, Jun 19, 2017 at 1:56 PM, Thomas Tauber-Marshall < > tmarsh...@cloudera.com> wrote: > > > I'm working on getting the s3 jars updated, which presumably will fix > that. > > > > The problem (to my understanding) is that the nightlies haven't passed > > since the change went into Sentry and so the Jenkins job that normally > > produces the new jars is still pulling in old bits. > > > > I've been talking with releng and they expect the new jars to be > available > > later today. > > > > On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong > > wrote: > > > > > Looks like the build still breaks when starting up sentry after my fix: > > > > > > > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1547/console > > > > > > *20:08:54* --> Starting the Sentry Policy Server*20:08:59* Error in > > > /home/ubuntu/Impala/testdata/bin/run-all.sh at line 58: > > > $IMPALA_HOME/testdata/bin/run-sentry-service.sh > \*20:08:59* + > > > onexit*20:08:59* + df -m*20:08:59* Filesystem 1M-blocks Used > > > Available Use% Mounted on*20:08:59* udev 15070 1 > > > 15070 1% /dev*20:08:59* tmpfs 3015 1 3015 > > > 1% /run*20:08:59* /dev/xvda1161129 22275132204 15% > > > /*20:08:59* none 1 0 1 0% > > > /sys/fs/cgroup*20:08:59* none 5 0 5 0% > > > /run/lock*20:08:59* none 15075 1 15075 1% > > > /run/shm*20:08:59* none 100 0 100 0% > > > /run/user*20:08:59* + free -m*20:08:59* total used > > > free sharedbuffers cached*20:08:59* Mem: > > > 30148 19597 10550 11 91 14323*20:08:59* > > > -/+ buffers/cache: 5182 24965*20:08:59* Swap:0 > > > 0 0*20:08:59* + uptime -p*20:08:59* up 45 > > > minutes*20:08:59* + rm -rf /home/ubuntu/Impala/logs_static*20:08:59* + > > > mkdir -p /home/ubuntu/Impala/logs_static*20:08:59* + cp -r -L > > > /home/ubuntu/Impala/logs /home/ubuntu/Impala/logs_static*20:08:59* > > > Build step 'Execute shell' marked build as failure*20:08:59* Set build > > > name.*20:08:59* New build name is '#1547 > > > refs/changes/22/7222/3'*20:08:59* Variable with name > > > 'BUILD_DISPLAY_NAME' already exists, current value: '#1547 > > > refs/changes/22/7222/3', new value: '#1547 > > > refs/changes/22/7222/3'*20:09:12* Archiving artifacts*20:09:21* > > > Finished: FAILURE > > > > > > > > > On Mon, Jun 19, 2017 at 12:23 PM, Tim Armstrong < > tarmstr...@cloudera.com > > > > > > wrote: > > > > > > > It's unclear if there will be incompatibility between the updated > > client > > > > and the version of sentry we use for the minicluster. I kicked off a > > test > > > > run to see if it works. > > > > > > > > On Mon, Jun 19, 2017 at 12:06 PM, Henry Robinson > > > wrote: > > > > > > > >> Presumably this will break GVO jobs as well - should we commit Tim's > > > patch > > > >> to get us moving again while Alex works on the root cause? > > > >> > > > >> On 19 June 2017 at 09:23, Alexander Behm > > > wrote: > > > >> > > > >> > Meanwhile, I'll work on fixing the root cause: > > > >> > https://issues.apache.org/jira/browse/IMPALA-5530 > > > >> > > > > >> > On Mon, Jun 19, 2017 at 9:20 AM, Tim Armstrong < > > > tarmstr...@cloudera.com > > > >> > > > > >> > wrote: > > > >> > > > > >> > > You may have noticed that Impala doesn't build this morning > > because > > > >> of a > > > >> > > sentry exception class no longer existing. I was able to unblock > > > >> myself > > > >> > > with this change, if you want to cherry-pick it: > > > >> > > https://gerrit.cloudera.org/#/c/7222/ > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > -- > Todd Lipcon > Software Engineer, Cloudera >
Re: Broken build from Sentry
I'm working on getting the s3 jars updated, which presumably will fix that. The problem (to my understanding) is that the nightlies haven't passed since the change went into Sentry and so the Jenkins job that normally produces the new jars is still pulling in old bits. I've been talking with releng and they expect the new jars to be available later today. On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong wrote: > Looks like the build still breaks when starting up sentry after my fix: > > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1547/console > > *20:08:54* --> Starting the Sentry Policy Server*20:08:59* Error in > /home/ubuntu/Impala/testdata/bin/run-all.sh at line 58: > $IMPALA_HOME/testdata/bin/run-sentry-service.sh > \*20:08:59* + > onexit*20:08:59* + df -m*20:08:59* Filesystem 1M-blocks Used > Available Use% Mounted on*20:08:59* udev 15070 1 > 15070 1% /dev*20:08:59* tmpfs 3015 1 3015 > 1% /run*20:08:59* /dev/xvda1161129 22275132204 15% > /*20:08:59* none 1 0 1 0% > /sys/fs/cgroup*20:08:59* none 5 0 5 0% > /run/lock*20:08:59* none 15075 1 15075 1% > /run/shm*20:08:59* none 100 0 100 0% > /run/user*20:08:59* + free -m*20:08:59* total used > free sharedbuffers cached*20:08:59* Mem: > 30148 19597 10550 11 91 14323*20:08:59* > -/+ buffers/cache: 5182 24965*20:08:59* Swap:0 > 0 0*20:08:59* + uptime -p*20:08:59* up 45 > minutes*20:08:59* + rm -rf /home/ubuntu/Impala/logs_static*20:08:59* + > mkdir -p /home/ubuntu/Impala/logs_static*20:08:59* + cp -r -L > /home/ubuntu/Impala/logs /home/ubuntu/Impala/logs_static*20:08:59* > Build step 'Execute shell' marked build as failure*20:08:59* Set build > name.*20:08:59* New build name is '#1547 > refs/changes/22/7222/3'*20:08:59* Variable with name > 'BUILD_DISPLAY_NAME' already exists, current value: '#1547 > refs/changes/22/7222/3', new value: '#1547 > refs/changes/22/7222/3'*20:09:12* Archiving artifacts*20:09:21* > Finished: FAILURE > > > On Mon, Jun 19, 2017 at 12:23 PM, Tim Armstrong > wrote: > > > It's unclear if there will be incompatibility between the updated client > > and the version of sentry we use for the minicluster. I kicked off a test > > run to see if it works. > > > > On Mon, Jun 19, 2017 at 12:06 PM, Henry Robinson > wrote: > > > >> Presumably this will break GVO jobs as well - should we commit Tim's > patch > >> to get us moving again while Alex works on the root cause? > >> > >> On 19 June 2017 at 09:23, Alexander Behm > wrote: > >> > >> > Meanwhile, I'll work on fixing the root cause: > >> > https://issues.apache.org/jira/browse/IMPALA-5530 > >> > > >> > On Mon, Jun 19, 2017 at 9:20 AM, Tim Armstrong < > tarmstr...@cloudera.com > >> > > >> > wrote: > >> > > >> > > You may have noticed that Impala doesn't build this morning because > >> of a > >> > > sentry exception class no longer existing. I was able to unblock > >> myself > >> > > with this change, if you want to cherry-pick it: > >> > > https://gerrit.cloudera.org/#/c/7222/ > >> > > > >> > > >> > > > > >
Partial Sort Design
I'm going to be working on: https://issues.apache.org/jira/browse/IMPALA-5498, adding support for partial sorts to Impala. I've written up a brief design doc outlining my thoughts, if anyone would like to look over it and make suggestions: https://docs.google.com/document/d/1g4eJqBaZa8taA1cSsP-TJ-Mp92u8MGdYuh6bf2R51hg/edit?usp=sharing
Hive Schema Upgrade Required
I have two patches going out soon:
https://gerrit.cloudera.org/#/c/7072/
https://gerrit.cloudera.org/#/c/7076/
which combined update Impala's Hive dependency to a new version with an
updated schema. After pulling in these changes, you should find that the
hive metastore will fail to come up.
To fix this without doing a full data reload, you can use the following
command:
${IMPALA_TOOLCHAIN}/cdh_components/hive-1.1.0-cdh5.13.0-SNAPSHOT/bin/schematool
-upgradeSchema -dbType {type}
where type is one of 'postgres' or 'mysql', depending on your setup.
You'll need to pull the latest version of Hive into your toolchain, such as
by running buildall.sh with DOWNLOAD_CDH_COMPONENTS=true
If you see:
Error: ERROR: column "SCHEMA_VERSION_V2" of relation "VERSION" does not
exist (state=42703,code=0)
in the schematool output, this is expected and not really an error.
There is a JIRA for automating this process in the future:
https://issues.apache.org/jira/browse/IMPALA-5430
Re: about broadcast join and hash shuffle join
There's actually a review out right now for changing the default join algorithm when stats are unavailable to partitioned: https://gerrit.cloudera.org/#/c/6803/ On Fri, May 5, 2017 at 4:44 AM yu feng wrote: > Hi All: > > I find impala choose join algorithm by comparing data transmission size > between broad cast and shuffle join while generating physical execution > plan. what I am confused is why impala choose broadcast as default > implement(such as table do not compute stats) ? > > In my experience, shuffle join maybe the better choice, and some of my > queries use broadcast join between two subquery with huge resultset and the > query costs has difference up to ten times (8s and 80s). > > I think user should always compute stats for every partition, do you guys > have some good suggestion about this. > > Thanks a lot >
Sort Expr Materialization
I'm working on a patch to materialize sort exprs, both for performance and to fix a crash, and because theres a number of issues to consider, I've put together a brief design doc, if anyone would like to look it over and make suggestions: https://docs.google.com/document/d/1tBTOzMqW53w1WSYMnqGGyPWVp1hgZGRyQVDmXsHXtTY/edit?usp=sharing Thanks, Thomas
Could a commiter please gvm this
https://gerrit.cloudera.org/#/c/5917/ Thanks, Thomas
Could a committer...
carry forward the +2 and submit a gvm: https://gerrit.cloudera.org/#/c/5389/ Thanks, Thomas
Re: Preferred syntax for warning about ignored Status returns
I'd vote for option 1, given that, as you say, this is closely related to the return type, which putting the macro with the type makes clear. Also looking at buffer-pool.h in the review, it doesn't look overly noisy to me. On Fri, Jan 6, 2017 at 2:59 PM Tim Armstrong wrote: > Hi All, > I wanted to poll the Impala community for opinions about style for > declaring functions where the caller is expected to do something with the > return value. > > Ideally we'd be able to declare Status with an attribute that made this > take effect globally, but unfortunately that's not available until C++17. > > So we need to annotate each Status-returning function. The two alternatives > we discussed on this CR (https://gerrit.cloudera.org/#/c/4878/) were: > > #1 - a special macro wrapping Status > > MUST_USE(Status) DoSomethingThatCanFail(int64_t foo, Bar* bar); > > Pros: > * Closely connected to the return type that it affects > * It's easier to search/replace Status with MUST_USE(Status) > > Cons: > * Could get visually noisy if we use it everywhere > > #2 - a macro that gets appended to the declaration: > > Status DoSomethingThatCanFail(int64_t foo, Bar* bar) WARN_UNUSED_RESULT; > > Pros: > * Macro is slightly > * Less visually noisy since it's at the end of the declaration > > What do people think? >
Re: Failed to build the Impala front-end
It looks like this might be related to a recent change that renamed the packages from com.cloudera.impala to org.apache.impala. Did you just fetch/pull in the patches for IMPALA-3786 before getting this error? If so, can you compile successfully without those changes? On Tue, Oct 4, 2016 at 4:28 AM Wang, Youwei A wrote: > Greetings, everyone. > I am trying to build the Impala front-end using mvn package > -Dmaven.test.skip=true . > However, I got the following error which just appeared today: > [ERROR] Failed to execute goal on project impala-frontend: Could not > resolve dependencies for project > org.apache.impala:impala-frontend:jar:0.1-SNAPSHOT: Failed to collect > dependencies at org.apache.impala:impala-data-source-api:jar:1.0-SNAPSHOT: > Failed to read artifact descriptor for > org.apache.impala:impala-data-source-api:jar:1.0-SNAPSHOT: Could not > transfer artifact org.apache.impala:impala-data-source-api:pom:1.0-SNAPSHOT > from/to cdh.rcs.releases.repo ( > https://repository.cloudera.com/content/groups/cdh-releases-rcs): Remote > host closed connection during handshake: SSL peer shut down incorrectly -> > [Help 1] > > I have tried another machine, and then I got: > [ERROR] Failed to execute goal on project impala-frontend: Could not > resolve dependencies for project > com.cloudera.impala:impala-frontend:jar:0.1-SNAPSHOT: Failed to collect > dependencies for [org.htrace:htrace-core:jar:3.0.4 (compile), > com.cloudera.impala:impala-data-source-api:jar:1.0-SNAPSHOT (compile), > org.apache.hadoop:hadoop-hdfs:jar:2.6.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hadoop:hadoop-aws:jar:2.6.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0-cdh5.10.0-SNAPSHOT > (compile), org.apache.hadoop:hadoop-yarn-api:jar:2.6.0-cdh5.10.0-SNAPSHOT > (compile), > org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.6.0-cdh5.10.0-SNAPSHOT > (compile), > org.apache.sentry:sentry-core-common:jar:1.5.1-cdh5.10.0-SNAPSHOT > (compile), > org.apache.sentry:sentry-core-model-db:jar:1.5.1-cdh5.10.0-SNAPSHOT > (compile), > org.apache.sentry:sentry-provider-common:jar:1.5.1-cdh5.10.0-SNAPSHOT > (compile), > org.apache.sentry:sentry-provider-file:jar:1.5.1-cdh5.10.0-SNAPSHOT > (compile), > org.apache.sentry:sentry-provider-cache:jar:1.5.1-cdh5.10.0-SNAPSHOT > (compile), > org.apache.sentry:sentry-policy-common:jar:1.5.1-cdh5.10.0-SNAPSHOT > (compile), org.apache.sentry:sentry-policy-db:jar:1.5.1-cdh5.10.0-SNAPSHOT > (compile), > org.apache.sentry:sentry-binding-hive:jar:1.5.1-cdh5.10.0-SNAPSHOT > (compile), > org.apache.sentry:sentry-provider-db:jar:sh:1.5.1-cdh5.10.0-SNAPSHOT > (compile), com.twitter:parquet-hadoop-bundle:jar:1.5.0-cdh5.10.0-SNAPSHOT > (compile), org.apache.hbase:hbase-client:jar:1.2.0-cdh5.10.0-SNAPSHOT > (compile), org.apache.hbase:hbase-common:jar:1.2.0-cdh5.10.0-SNAPSHOT > (compile), org.apache.hbase:hbase-protocol:jar:1.2.0-cdh5.10.0-SNAPSHOT > (compile), commons-lang:commons-lang:jar:2.6 (compile), > net.sourceforge.czt.dev:java-cup:jar:0.11-a-czt02-cdh (compile), > org.apache.thrift:libthrift:jar:0.9.0 (compile), > org.apache.httpcomponents:httpcore:jar:4.2.5 (compile), > org.apache.thrift:libfb303:jar:0.9.0 (compile), > org.apache.hive:hive-hbase-handler:jar:1.1.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hive:hive-service:jar:1.1.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hive:hive-jdbc:jar:1.1.0-cdh5.10.0-SNAPSHOT (test), > org.apache.hive:hive-metastore:jar:1.1.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hive:hive-common:jar:1.1.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hive:hive-serde:jar:1.1.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hive:hive-shims:jar:1.1.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.hive:hive-exec:jar:1.1.0-cdh5.10.0-SNAPSHOT (compile), > org.apache.kudu:kudu-client:jar:1.0.0-SNAPSHOT (compile), > com.stumbleupon:async:jar:1.3.1 (compile), > postgresql:postgresql:jar:9.0-801.jdbc4 (compile), > commons-dbcp:commons-dbcp:jar:1.4 (compile), javax.jdo:jdo-api:jar:3.0.1 > (compile), org.antlr:antlr-runtime:jar:3.3 (compile), > commons-cli:commons-cli:jar:1.2 (compile), > commons-codec:commons-codec:jar:1.9 (compile), > org.slf4j:slf4j-api:jar:1.7.5 (compile), org.slf4j:slf4j-log4j12:jar:1.7.5 > (compile), com.google.guava:guava:jar:11.0.2 (compile), > junit:junit:jar:4.12 (test), > com.googlecode.json-simple:json-simple:jar:1.1.1 (compile), > org.glassfish:javax.json:jar:1.0.2 (compile)]: Failed to read artifact > descriptor for > org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.10.0-SNAPSHOT: Could not > transfer artifact > org.apache.hadoop:hadoop-common:pom:2.6.0-cdh5.10.0-20161003.145949-61 > from/to cdh.snapshots.repo ( > https://repository.cloudera.com/content/repositories/snapshots): peer not > authenticated -> [Help 1] > > I am sure I have setup the proxy correctly since mvn packag
Re: Can you please send the output of these commands?
thomas@thomas-desktop:~$cat /etc/*release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS" NAME="Ubuntu" VERSION="14.04.4 LTS, Trusty Tahr" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 14.04.4 LTS" VERSION_ID="14.04" HOME_URL="http://www.ubuntu.com/"; SUPPORT_URL="http://help.ubuntu.com/"; BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"; thomas@thomas-desktop:~$uname -a Linux thomas-desktop 3.19.0-58-generic #64~14.04.1-Ubuntu SMP Fri Mar 18 19:05:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux thomas@thomas-desktop:~$cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq 390 thomas@thomas-desktop:~$head -30 /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz stepping: 3 microcode : 0x17 cpu MHz : 3400.000 cache size : 8192 KB physical id : 0 siblings: 8 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt bugs: bogomips: 6784.45 clflush size: 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 On Fri, Sep 23, 2016 at 10:41 AM, Tim Armstrong wrote: > tarmstrong@tarmstrong-box:~$ cat /etc/*release > DISTRIB_ID=Ubuntu > DISTRIB_RELEASE=15.10 > DISTRIB_CODENAME=wily > DISTRIB_DESCRIPTION="Ubuntu 15.10" > NAME="Ubuntu" > VERSION="15.10 (Wily Werewolf)" > ID=ubuntu > ID_LIKE=debian > PRETTY_NAME="Ubuntu 15.10" > VERSION_ID="15.10" > HOME_URL="http://www.ubuntu.com/"; > SUPPORT_URL="http://help.ubuntu.com/"; > BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"; > tarmstrong@tarmstrong-box:~$ uname -a > Linux tarmstrong-box 4.2.0-30-generic #36-Ubuntu SMP Fri Feb 26 00:58:07 > UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > tarmstrong@tarmstrong-box:~$ cat > /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq > 400 > tarmstrong@tarmstrong-box:~$ head -30 /proc/cpuinfo > processor: 0 > vendor_id: GenuineIntel > cpu family: 6 > model: 60 > model name: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz > stepping: 3 > microcode: 0x1c > cpu MHz: 3799.968 > cache size: 8192 KB > physical id: 0 > siblings: 8 > core id: 0 > cpu cores: 4 > apicid: 0 > initial apicid: 0 > fpu: yes > fpu_exception: yes > cpuid level: 13 > wp: yes > flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx > pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl > xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor > ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic > movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida > arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase > tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt > bugs: > bogomips: 7183.71 > clflush size: 64 > cache_alignment: 64 > address sizes: 39 bits physical, 48 bits virtual > power management: > > processor: 1 > vendor_id: GenuineIntel > cpu family: 6 > tarmstrong@tarmstrong-box:~$ > > > On Fri, Sep 23, 2016 at 10:39 AM, Sailesh Mukil > wrote: > > > sailesh@ubuntu:~$ cat /etc/*release > > DISTRIB_ID=Ubuntu > > DISTRIB_RELEASE=14.04 > > DISTRIB_CODENAME=trusty > > DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS" > > NAME="Ubuntu" > > VERSION="14.04.2 LTS, Trusty Tahr" > > ID=ubuntu > > ID_LIKE=debian > > PRETTY_NAME="Ubuntu 14.04.2 LTS" > > VERSION_ID="14.04" > > HOME_URL="http://www.ubuntu.com/"; > > SUPPORT_URL="http://help.ubuntu.com/"; > > BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"; > > > > sailesh@ubuntu:~$ uname -a > > Linux ubuntu 3.16.0-45-generic #60~14.04.1-Ubuntu SMP Fri Jul 24 21:16:23 > > UTC 2015 x86_64 x86_64 x86_64 GNU/Linux > > > > sailesh@ubuntu:~$ cat /sys/devices/system/cpu/cpu0/ > > cpufreq/cpuinfo_max_freq > > 400 > > > > sailesh@ubuntu:~$ head -30 /proc/cpuinfo > > processor : 0 > > vendor_id : GenuineIntel > > cpu family : 6 > > model : 60 > > model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz > > stepping : 3 > > microcode :
Re: [Impala-ASF-CR] IMPALA-3786: Replace "cloudera" with "apache" (part 2)
On Thu, Sep 1, 2016 at 6:58 PM Thomas Tauber-Marshall < tmarsh...@cloudera.com> wrote: > Current status: > > I can get the packaging build as far as successfully generating the > parcels and setting up the cm cluster. Example run: > http://golden.jenkins.cloudera.com/job/CDH5-Packaging-On-Demand/3029/ > > However, the cluster health checks fail due to: > https://jira.cloudera.com/browse/OPSAPS-36138 > > I have also manually run a cluster with the generated parcels and verified > that things seem to work, so I feel fairly confident about it: > http://tmarshall2-1.vpc.cloudera.com:7180/cmf/home > (I'll shut this down tomorrow morning before I leave) > Scratch that, I had the wrong parcels deployed. Sorry. The cluster I manually ran does not work, though the error it runs into is in Sqoop and doesn't look related to the package rename. > > I'm going on PTO tomorrow for a week, so I won't be working on this more > for now. I'm not sure how urgent it is, but of course someone can take over > from here if needed. > > On Thu, Sep 1, 2016 at 11:40 AM Thomas Tauber-Marshall (Code Review) < > ger...@cloudera.org> wrote: > >> Thomas Tauber-Marshall has posted comments on this change. >> >> Change subject: IMPALA-3786: Replace "cloudera" with "apache" (part 2) >> .. >> >> >> Patch Set 4: >> >> Currently, the on demand packaging build won't run because there are no >> sles64 nodes online. Do you know what to do about that? >> >> http://golden.jenkins.cloudera.com/job/CDH5-Packaging-On-Demand/3033 >> http://golden.jenkins.cloudera.com/job/generic-package-sles64-11-impala/ >> http://golden.jenkins.cloudera.com/label/static-sles64-11-impala/? >> >> -- >> To view, visit http://gerrit.cloudera.org:8080/3937 >> To unsubscribe, visit http://gerrit.cloudera.org:8080/settings >> >> Gerrit-MessageType: comment >> Gerrit-Change-Id: I0d35fa6602a7fc0c212b2ef5e2b3322b77dde7e2 >> Gerrit-PatchSet: 4 >> Gerrit-Project: Impala-ASF >> Gerrit-Branch: master >> Gerrit-Owner: Thomas Tauber-Marshall >> Gerrit-Reviewer: Jim Apple >> Gerrit-Reviewer: Thomas Tauber-Marshall >> Gerrit-HasComments: No >> >> -- >> You received this message because you are subscribed to the Google Groups >> "impala-cr" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to impala-cr+unsubscr...@cloudera.com. >> For more options, visit https://groups.google.com/a/cloudera.com/d/optout >> . >> >
Re: [Impala-ASF-CR] IMPALA-3786: Replace "cloudera" with "apache" (part 2)
Current status: I can get the packaging build as far as successfully generating the parcels and setting up the cm cluster. Example run: http://golden.jenkins.cloudera.com/job/CDH5-Packaging-On-Demand/3029/ However, the cluster health checks fail due to: https://jira.cloudera.com/browse/OPSAPS-36138 I have also manually run a cluster with the generated parcels and verified that things seem to work, so I feel fairly confident about it: http://tmarshall2-1.vpc.cloudera.com:7180/cmf/home (I'll shut this down tomorrow morning before I leave) I'm going on PTO tomorrow for a week, so I won't be working on this more for now. I'm not sure how urgent it is, but of course someone can take over from here if needed. On Thu, Sep 1, 2016 at 11:40 AM Thomas Tauber-Marshall (Code Review) < ger...@cloudera.org> wrote: > Thomas Tauber-Marshall has posted comments on this change. > > Change subject: IMPALA-3786: Replace "cloudera" with "apache" (part 2) > .. > > > Patch Set 4: > > Currently, the on demand packaging build won't run because there are no > sles64 nodes online. Do you know what to do about that? > > http://golden.jenkins.cloudera.com/job/CDH5-Packaging-On-Demand/3033 > http://golden.jenkins.cloudera.com/job/generic-package-sles64-11-impala/ > http://golden.jenkins.cloudera.com/label/static-sles64-11-impala/? > > -- > To view, visit http://gerrit.cloudera.org:8080/3937 > To unsubscribe, visit http://gerrit.cloudera.org:8080/settings > > Gerrit-MessageType: comment > Gerrit-Change-Id: I0d35fa6602a7fc0c212b2ef5e2b3322b77dde7e2 > Gerrit-PatchSet: 4 > Gerrit-Project: Impala-ASF > Gerrit-Branch: master > Gerrit-Owner: Thomas Tauber-Marshall > Gerrit-Reviewer: Jim Apple > Gerrit-Reviewer: Thomas Tauber-Marshall > Gerrit-HasComments: No > > -- > You received this message because you are subscribed to the Google Groups > "impala-cr" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to impala-cr+unsubscr...@cloudera.com. > For more options, visit https://groups.google.com/a/cloudera.com/d/optout. >
[Impala-CR] IMPALA-2878: Fix Base64Decode error and remove duplicate codes.
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-2878: Fix Base64Decode error and remove duplicate codes. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4209 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I0170a7d180ab048d0ff2196a24ddc53626aa7aab Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Yuanhao Luo
[Impala-CR] IMPALA-3567: Part 1: groundwork to make Join build sides DataSinks
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3567: Part 1: groundwork to make Join build sides DataSinks .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4212 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I9d7608181eeacfe706a09c1e153d0a3e1ee9b475 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Marcel Kornacker Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3881: Add DataTables 1.10.12 to www/
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3881: Add DataTables 1.10.12 to www/ .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4227 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Ic1001870a50fbf21efed29a446815436da1a36fa Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-2347: Reuse metastore client connections in Catalog
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-2347: Reuse metastore client connections in Catalog .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I517c0e1efef2584cd8d34017b33574f2ad69bd52 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3952: Clear scratch batch mem pool if Open() failed.
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3952: Clear scratch batch mem pool if Open() failed. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4236 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I55f32ed698a5b6fed8c28af1391aa07e1560e782 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-1619, IMPALA-3018: Address various small memory allocation related bugs
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-1619, IMPALA-3018: Address various small memory allocation related bugs .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4235 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I6eb9a4472a65cf68edb0323b13d745277ead2e1d Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3964: Fix crash when a count(*) is performed on a nested collection.
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3964: Fix crash when a count(*) is performed on a nested collection. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4237 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Id0955c85d18dfba4bd29a35ec95d0355da050607 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Christopher Channing Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3843: Update warning for non-SSSE3 CPUs
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3843: Update warning for non-SSSE3 CPUs .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4218 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Id242b310715c7abea091a35e6a89647bded26c4b Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Tomas Gavenciak
Sorry for the spam
For anyone wondering, I just pushed the wrong branch to gerrit, resulting in a few dozen reviews being created. I'm currently working on abandoning them all. Sorry for the spam. Thanks, Thomas
[Impala-CR] Fix stress test runner bug introduced by IMPALA-3969
Thomas Tauber-Marshall has abandoned this change. Change subject: Fix stress test runner bug introduced by IMPALA-3969 .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I86484bb7c92ae1069f6a07cf3ea5027740364150 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3992: bad shell error message when running nonexistent file
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3992: bad shell error message when running nonexistent file .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4240 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Iebcf1dc8a1a08b400a2c769a9cff38ea02c8e525 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3856,IMPALA-3871: Fix BinaryPredicate normalization for Kudu
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3856,IMPALA-3871: Fix BinaryPredicate normalization for Kudu .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4241 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Iae7612433a2e27f8887abe6624f9ee0f4867b934 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3376: Extra definition level when writing Parquet files
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3376: Extra definition level when writing Parquet files .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I20f25a90aa1ef74b4f00f38f832bc1c1853342c6 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3918: Remove Cloudera copyrights and add ASF license header
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3918: Remove Cloudera copyrights and add ASF license header .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4213 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I311e493bec7e63ea6dd7229140045d486540612a Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3924: Ubuntu16 support
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3924: Ubuntu16 support .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4216 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Iaa1520c1e4aa4175468ac342b14c1262fa745f7a Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3918: Remove Cloudera copyrights and add ASF license header
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3918: Remove Cloudera copyrights and add ASF license header .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4211 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I2e0bd8420945b953e1b806041bea4d72a3943d86 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3611: track unused Disk IO buffer memory
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3611: track unused Disk IO buffer memory .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I8777cf76f04d34a46f53d53005412e0f1d63b5b7 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3790: Fix admission control flaky stress test
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3790: Fix admission control flaky stress test .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4210 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Id62f7603f1174aa02469c6ca57513c3f1fa1e221 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3946: fix MemPool integrity issues with empty chunks
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3946: fix MemPool integrity issues with empty chunks .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4222 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I03ad12e5b2b63cbb55e5c52562416d73a4bd9346 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3764,3914: fuzz test HDFS scanners and fix parquet bugs found
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3764,3914: fuzz test HDFS scanners and fix parquet bugs found .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I50cf43195a7c582caa02c85ae400ea2256fa3a3b Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-2700: ASCII NUL characters are doubled on insert into text tables
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-2700: ASCII NUL characters are doubled on insert into text tables .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Ia30fa314d1ee1e99f9e7598466eb1570ca7940fc Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-1112: Remove some unncessary code from cross-compilation
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-1112: Remove some unncessary code from cross-compilation .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4217 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I543809c69da0b4085a0e299b91cd550b274c46af Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3965: TSSLSocketWithWildcardSAN.py not exported as part of impala-shell build lib
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3965: TSSLSocketWithWildcardSAN.py not exported as part of impala-shell build lib .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I5a7ab8c20c0b20c21b7f8d008e39c940419e3c4d Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3786: Remove "Cloudera" from impalad webpage title
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3786: Remove "Cloudera" from impalad webpage title .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Idc4d57792bcce65ad1756d018ed9f245ff732b37 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3957: Test failure in S3 build: TestLoadData.test load
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3957: Test failure in S3 build: TestLoadData.test_load .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4221 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I966a469e94099d3d971e470ae6e992386070c5e9 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3940: Fix getting column stats through views.
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3940: Fix getting column stats through views. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Ie3b62a5e7e7d0e84850749108c13991647cedce6 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3201: buffer pool header only
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3201: buffer pool header only .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4233 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Id771dea2eb4c1aa13c30d59e8b184a7d1bca8d34 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3923: fix overflow in BufferedTupleStream::GetRows()
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3923: fix overflow in BufferedTupleStream::GetRows() .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Iabb977014d855d6edb035bf2e9519ac8b3234767 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] Remove dead and untested code
Thomas Tauber-Marshall has abandoned this change. Change subject: Remove dead and untested code .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4232 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I49c27cbfef03ef97befa9a607b3d8d7ac6e22a43 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3972: Improve display of /varz page
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3972: Improve display of /varz page .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I5a0d47da7abf913918d5fba5c327e26b73d701d2 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3969: stress test: add option to set common query options
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3969: stress test: add option to set common query options .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Iada041aace60c218a12178d8f1b9a68ff29de72e Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3829: OpenSession() logs errors on valid configuration keys
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3829: OpenSession() logs errors on valid configuration keys .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4226 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Ifa9b823abc39ba9809a35a6f0844fa3436f1e025 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3936: BufferedBlockMgr fixes for Pin() while write in flight.
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3936: BufferedBlockMgr fixes for Pin() while write in flight. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4230 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I4be4fad8e6f2303db19ea1e2bd0f13523781ae8e Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3953: Fixes for KuduScanNode BE test failure
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3953: Fixes for KuduScanNode BE test failure .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I16be206c60a692d2a26d719de8cc45e859b06e97 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3470: DecompressorTest is flaky.
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3470: DecompressorTest is flaky. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I7ebaa403abf45e31f38d6cf4e557d6274d877a8a Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Juan Yu Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3063: Separate join inversion from join ordering.
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3063: Separate join inversion from join ordering. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4239 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: If86db7753fc585bb4c69612745ec010327a4 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3918: remove Cloudera copyright from the shell welcome message
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3918: remove Cloudera copyright from the shell welcome message .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4238 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I3b3dcad8997e5b58b4ffda42fc95e3dba1e8a641 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3954: Add unique database to scanner test
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3954: Add unique_database to scanner test .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4242 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Ic48a4bac3df6a40cb5cb10c6f1c42583952c6c86 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3650: DISTRIBUTE BY required for managed Kudu tables
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3650: DISTRIBUTE BY required for managed Kudu tables .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4244 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Ieb15110b10b28ef6dd8ec136c2522b5f44dca43e Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3090: always log memory limit errors
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3090: always log memory limit errors .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4245 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: If5ec5572b0e26898da352b7e6b11eb01c6edb2e5 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] Add functional and targeted perf tests for joins with empty builds
Thomas Tauber-Marshall has abandoned this change. Change subject: Add functional and targeted perf tests for joins with empty builds .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I0a68cafec32011a47c569b254979601237e7f2a5 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3828: Enable inversion for inner joins.
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3828: Enable inversion for inner joins. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4246 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I8ba903f1df2446350cca7e71fdb13f550bf9de72 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3962: Clean up scratch tuple batch on scan failures
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3962: Clean up scratch tuple batch on scan failures .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4248 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: If1e27a1517d09ccaabdae1492b7e1fbf661ae3e5 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4249 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I2546bc68ba136b2713d744c1b920878606a2217b Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3662: Don't double allocate tuples buffer in parquet scanner
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3662: Don't double allocate tuples buffer in parquet scanner .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4250 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I826061a2be10fd0528ca4dd1e97146e3cb983370 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3981: Fix crash when accessing statestored / catalogd /memz page
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3981: Fix crash when accessing statestored / catalogd /memz page .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: If59b10f20044d1a468f27810a3029fe18fb19f29 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Kathy Sun Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3996: Migrate to updated Kudu insert string API
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3996: Migrate to updated Kudu insert string API .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I910c24724d0bc887b2d4a3e62ecdf72420a76f6f Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-4004: Don't access nested types in test failpoints.py
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-4004: Don't access nested types in test_failpoints.py .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4253 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Idc5e67b9748a13fcd76ea5fe140e2e6b18e809b7 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3988: Only use first 96 bits of query id
Thomas Tauber-Marshall has abandoned this change. Change subject: IMPALA-3988: Only use first 96 bits of query id .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/4254 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Ia56a03ed9a1d7e77c72b66a01cd48c5b6bf3624f Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Marcel Kornacker Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3090: always log memory limit errors
Hello Internal Jenkins, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4245 to review the following change. Change subject: IMPALA-3090: always log memory limit errors .. IMPALA-3090: always log memory limit errors Consistently log memory limit errors so that the error message contains a dump of the query memory trackers at the time that the memory limit was hit (instead of after the fact when the query is already partially cleaned up). Testing: Exhaustive build passed. Ran local stress test for a bit. Change-Id: If5ec5572b0e26898da352b7e6b11eb01c6edb2e5 Reviewed-on: http://gerrit.cloudera.org:8080/4049 Reviewed-by: Tim Armstrong Tested-by: Internal Jenkins (cherry picked from commit 7eb30309f3847f416f204bd5f7d6925102e94b67) --- M be/src/exec/analytic-eval-node.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/runtime/buffered-block-mgr.cc M be/src/runtime/mem-tracker.h M be/src/runtime/plan-fragment-executor.cc M be/src/runtime/row-batch.cc M be/src/runtime/runtime-state.h M be/src/runtime/sorter.cc 8 files changed, 38 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/45/4245/1 -- To view, visit http://gerrit.cloudera.org:8080/4245 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If5ec5572b0e26898da352b7e6b11eb01c6edb2e5 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3965: TSSLSocketWithWildcardSAN.py not exported as part of impala-shell build lib
Hello Internal Jenkins, Sailesh Mukil, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4214 to review the following change. Change subject: IMPALA-3965: TSSLSocketWithWildcardSAN.py not exported as part of impala-shell build lib .. IMPALA-3965: TSSLSocketWithWildcardSAN.py not exported as part of impala-shell build lib TSSLSocketWithWildcardSAN.py was recently added to the impala-shell as a part of IMPALA-3159. However, it was not exported as a part of the shell tarball. This change adds the file to the tarball. Change-Id: I5a7ab8c20c0b20c21b7f8d008e39c940419e3c4d Reviewed-on: http://gerrit.cloudera.org:8080/3872 Reviewed-by: Sailesh Mukil Tested-by: Internal Jenkins (cherry picked from commit 0d689d362624a7a6b588f1c0eaa4242684868a6d) --- M shell/make_shell_tarball.sh 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/14/4214/1 -- To view, visit http://gerrit.cloudera.org:8080/4214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5a7ab8c20c0b20c21b7f8d008e39c940419e3c4d Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3470: DecompressorTest is flaky.
Hello Juan Yu, Internal Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4234 to review the following change. Change subject: IMPALA-3470: DecompressorTest is flaky. .. IMPALA-3470: DecompressorTest is flaky. Make sure the random number is greater than 0 and stream is indeed truncated. Change-Id: I7ebaa403abf45e31f38d6cf4e557d6274d877a8a Reviewed-on: http://gerrit.cloudera.org:8080/3954 Reviewed-by: Juan Yu Tested-by: Internal Jenkins (cherry picked from commit 3307acfef084ec46820dc2c8c90395908fea7ec1) --- M be/src/util/decompress-test.cc 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/34/4234/1 -- To view, visit http://gerrit.cloudera.org:8080/4234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7ebaa403abf45e31f38d6cf4e557d6274d877a8a Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Juan Yu Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3662: Don't double allocate tuples buffer in parquet scanner
Hello Michael Ho, Internal Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4250 to review the following change. Change subject: IMPALA-3662: Don't double allocate tuples buffer in parquet scanner .. IMPALA-3662: Don't double allocate tuples buffer in parquet scanner HdfsScanner::StartNewRowBatch() is called once per row batch by the parquet scanner to allocate a new row batch and tuple buffer. Similarly, a scratch batch is created for each row batch in HdfsParquetScanner::AssembleRows() which also contains the tuple buffer. In reality, only the tuple buffer in the scratch batch is used. So, the tuple buffer allocated by HdfsScanner::StartNewRowBatch() is unused memory for the parquet scanner. This change fixes the problem above by implementing HdfsParquetScanner::StartNewRowBatch() which creates a new row batch without allocating the tuple buffer. With this patch, the memory consumption when materializing very wide tuples is reduced by half. Change-Id: I826061a2be10fd0528ca4dd1e97146e3cb983370 Reviewed-on: http://gerrit.cloudera.org:8080/4064 Reviewed-by: Michael Ho Tested-by: Internal Jenkins (cherry picked from commit 1522da3510a36635e3fc694b26211554fcd2793a) --- M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-parquet-scanner.h M be/src/exec/hdfs-scanner.h 3 files changed, 14 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/50/4250/1 -- To view, visit http://gerrit.cloudera.org:8080/4250 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I826061a2be10fd0528ca4dd1e97146e3cb983370 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: anujphadke
[Impala-CR] Add functional and targeted perf tests for joins with empty builds
Hello Internal Jenkins, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4247 to review the following change. Change subject: Add functional and targeted perf tests for joins with empty builds .. Add functional and targeted perf tests for joins with empty builds I wrote these tests for my IMPALA-3987 patch, but other issues block that optimisations. These tests exercise an interesting corner case so I split them out into a separate patch. The functional tests exercise every join mode for nested loop join and hash join with an empty build side. The perf test exercises hash join with an empty build side. Testing: Made sure the tests passed with both partitioned and non-partitioned hash join implementations. Ran the targeted perf query through the single node perf run script to make sure it worked. Change-Id: I0a68cafec32011a47c569b254979601237e7f2a5 Reviewed-on: http://gerrit.cloudera.org:8080/4051 Reviewed-by: Tim Armstrong Tested-by: Internal Jenkins (cherry picked from commit f613dcd02da989204026999b26247a024fb199ab) --- A testdata/workloads/functional-query/queries/QueryTest/empty-build-joins.test M testdata/workloads/functional-query/queries/QueryTest/single-node-nlj.test A testdata/workloads/targeted-perf/queries/primitive_empty_build_join_1.test M tests/query_test/test_join_queries.py 4 files changed, 262 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/47/4247/1 -- To view, visit http://gerrit.cloudera.org:8080/4247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0a68cafec32011a47c569b254979601237e7f2a5 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3940: Fix getting column stats through views.
Hello Internal Jenkins, Alex Behm, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4223 to review the following change. Change subject: IMPALA-3940: Fix getting column stats through views. .. IMPALA-3940: Fix getting column stats through views. The bug: During join ordering we rely on the column stats of join predicates for estimating the join cardinality. We have code that tries to find the stats of a column through views but there was a bug in identifying slots that belong to base table scans. The bug lead us to incorrectly accept slots of view references which do not have stats. This patch fixes the above issue and adds new test infrastructure for creating test-local views. It adds a TPCH-equivalent database that contains views of the form "select * from tpch_basetbl" for all TPCH tables and add tests the plans of all TPCH queries on the view database. Change-Id: Ie3b62a5e7e7d0e84850749108c13991647cedce6 Reviewed-on: http://gerrit.cloudera.org:8080/3865 Reviewed-by: Alex Behm Tested-by: Internal Jenkins (cherry picked from commit 286da59219f322ce99563537214e3bb30c0fa8c5) --- M fe/src/main/java/com/cloudera/impala/analysis/SlotDescriptor.java M fe/src/main/java/com/cloudera/impala/catalog/View.java M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/com/cloudera/impala/analysis/AnalyzerTest.java M fe/src/test/java/com/cloudera/impala/analysis/ToSqlTest.java A fe/src/test/java/com/cloudera/impala/common/FrontendTestBase.java M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java M fe/src/test/java/com/cloudera/impala/planner/PlannerTestBase.java M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test 13 files changed, 2,158 insertions(+), 588 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/23/4223/1 -- To view, visit http://gerrit.cloudera.org:8080/4223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie3b62a5e7e7d0e84850749108c13991647cedce6 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3954: Add unique database to scanner test
Hello Michael Brown, Internal Jenkins, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4242 to review the following change. Change subject: IMPALA-3954: Add unique_database to scanner test .. IMPALA-3954: Add unique_database to scanner test Without a unique_database hdfs-text-scan-with-header.test will create a table "functional.mixed" and parallel executions of the test will interfere with each other. This change adds a unique_database to the test call to prevent this. Change-Id: Ic48a4bac3df6a40cb5cb10c6f1c42583952c6c86 Reviewed-on: http://gerrit.cloudera.org:8080/4045 Reviewed-by: Michael Brown Reviewed-by: Tim Armstrong Tested-by: Internal Jenkins (cherry picked from commit 4d03ed40c6e9cce30f09ab111c4077c7c6329233) --- M tests/query_test/test_scanners.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/42/4242/1 -- To view, visit http://gerrit.cloudera.org:8080/4242 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic48a4bac3df6a40cb5cb10c6f1c42583952c6c86 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail
Hello Internal Jenkins, Dan Hecht, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4249 to review the following change. Change subject: IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail .. IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail HdfsTableSink::Close() makes calls to functions that can fail with a Status. However, since the function has a void return type, these error statuses are just logged and we cannot take any action according to the type of error. This patch moves the closing of the partition file into the FlushFinal function from Close(), so that in case of an error on closing the file, the error is propagated up and some action can be taken. We try and close all the partition files in the map in Close() as well because if a query is cancelled, FlushFinal will not be called and we would end up leaking some file descriptors. Also fixed some long lines in this patch. Change-Id: I2546bc68ba136b2713d744c1b920878606a2217b Reviewed-on: http://gerrit.cloudera.org:8080/4018 Reviewed-by: Dan Hecht Tested-by: Internal Jenkins (cherry picked from commit 02608f89f5ee3385ef63533eb012cb75690d0b5d) --- M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h 2 files changed, 15 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/49/4249/1 -- To view, visit http://gerrit.cloudera.org:8080/4249 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2546bc68ba136b2713d744c1b920878606a2217b Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3957: Test failure in S3 build: TestLoadData.test load
Hello Internal Jenkins, Sailesh Mukil, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4221 to review the following change. Change subject: IMPALA-3957: Test failure in S3 build: TestLoadData.test_load .. IMPALA-3957: Test failure in S3 build: TestLoadData.test_load The test_load() test failed with an error which said that the number of of files in the destination was wrong. This could probably be because the filesystem_client.copy() (for S3) in the setup_method() silently failed without copying one of the files as a one off error. I'm not sure why S3 failed to do the copy, but this patch adds an assert after the copy to make sure that if it the s3_client.copy() fails to do the copy, it will assert instead of continuing with the rest of the tests. Change-Id: I966a469e94099d3d971e470ae6e992386070c5e9 Reviewed-on: http://gerrit.cloudera.org:8080/3881 Reviewed-by: Sailesh Mukil Tested-by: Internal Jenkins (cherry picked from commit ac4f22b1b0a68600884358d373fffbcd817f22b8) --- M tests/util/s3_util.py 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/21/4221/1 -- To view, visit http://gerrit.cloudera.org:8080/4221 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I966a469e94099d3d971e470ae6e992386070c5e9 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] IMPALA-3843: Update warning for non-SSSE3 CPUs
Hello Internal Jenkins, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4218 to review the following change. Change subject: IMPALA-3843: Update warning for non-SSSE3 CPUs .. IMPALA-3843: Update warning for non-SSSE3 CPUs As described in IMPALA-3843, the old message for non-SSSE3 CPUs implies that Impala will exit without SSSE3. This is no longer the case after IMPALA-1399 and IMPALA-1646 have been resolved. The new message merely warns that there might be problems (and SSSE3 might be the culprit). Change-Id: Id242b310715c7abea091a35e6a89647bded26c4b Reviewed-on: http://gerrit.cloudera.org:8080/3884 Reviewed-by: Tim Armstrong Tested-by: Internal Jenkins (cherry picked from commit b6934f0fcd4cafc737ee44d9676657406511c162) --- M be/src/util/cpu-info.cc 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/18/4218/1 -- To view, visit http://gerrit.cloudera.org:8080/4218 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id242b310715c7abea091a35e6a89647bded26c4b Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Tomas Gavenciak
[Impala-CR] IMPALA-3964: Fix crash when a count(*) is performed on a nested collection.
Hello Michael Ho, Internal Jenkins, Dan Hecht, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4237 to review the following change. Change subject: IMPALA-3964: Fix crash when a count(*) is performed on a nested collection. .. IMPALA-3964: Fix crash when a count(*) is performed on a nested collection. The Bug: Prior to this patch, a DCHECK was used to verify that the underlying memory pool for the scratch batch was empty in a count based scenario. For IMPALA-3964 (where a count(*) is performed on a nested collection), if a Parquet column chunk is compressed, upon reading each new data page it would be decompressed and eventually placed in to the underlying scratch batch memory pool causing the aforementioned DCHECK to fail. This was not picked up in the test suite as the TPCH nested Parquet data is not compressed. The Fix: Removed the erroneous DCHECK. Added logic to determine if any memory in the scratch batch needs to be freed (due to the transfer that occurs from the decompressed data pool), if so, it will be done. Augmented the load_nested.py script to snappy compress each of the tables within the 'tpch_nested_parquet' database. This is consistent with how the flat TPCH Parquet data set is stored. Regarding test coverage, there are already a number of tests that will perform nested collection counts against the tables in the 'tpch_nested_parquet' database. For uncompressed nested Parquet, the 'test_nested_types.py' test suite leverages the 'ComplexTypesTbl' table to provide good coverage. Change-Id: Id0955c85d18dfba4bd29a35ec95d0355da050607 Reviewed-on: http://gerrit.cloudera.org:8080/3940 Reviewed-by: Michael Ho Reviewed-by: Dan Hecht Tested-by: Internal Jenkins (cherry picked from commit 90a6b3206e0b522650bfe21c5754c15d009f708c) --- M be/src/exec/hdfs-parquet-scanner.cc M testdata/bin/load_nested.py 2 files changed, 7 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/37/4237/1 -- To view, visit http://gerrit.cloudera.org:8080/4237 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id0955c85d18dfba4bd29a35ec95d0355da050607 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Christopher Channing Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3790: Fix admission control flaky stress test
Hello Internal Jenkins, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4210 to review the following change. Change subject: IMPALA-3790: Fix admission control flaky stress test .. IMPALA-3790: Fix admission control flaky stress test In addition to a previous change which extended the admission control test timeouts for code coverage jobs, the tests with high concurrency are still experiencing timeouts in the admission control queues (which is different from the timeouts that were set on the test cases). Rather than extend the timeouts on the queues as well (which would increase the already ridiculously long test time ~2hrs), this limits the number of concurrent queries that are submitted with code coverage. Change-Id: Id62f7603f1174aa02469c6ca57513c3f1fa1e221 Reviewed-on: http://gerrit.cloudera.org:8080/3861 Reviewed-by: Tim Armstrong Tested-by: Internal Jenkins (cherry picked from commit 7237241087dcee4ff2367d56c166257be859dea9) --- M tests/custom_cluster/test_admission_controller.py 1 file changed, 12 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/10/4210/1 -- To view, visit http://gerrit.cloudera.org:8080/4210 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id62f7603f1174aa02469c6ca57513c3f1fa1e221 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3567: Part 1: groundwork to make Join build sides DataSinks
none / none | 1.09 | 1.10| -0.92% | 2.26% | 2.22%| 1 | 10| | TPCH_NESTED(_20) | TPCH-Q13 | parquet / none / none | 3.75 | 3.78| -0.94% | 2.04% | 2.86%| 1 | 10| | TPCH_NESTED(_20) | TPCH-Q20 | parquet / none / none | 4.33 | 4.37| -1.10% | 3.00% | 2.43%| 1 | 10| | TPCH_NESTED(_20) | TPCH-Q2 | parquet / none / none | 2.39 | 2.42| -1.38% | 1.54% | 1.30%| 1 | 10| | TPCH_NESTED(_20) | TPCH-Q11 | parquet / none / none | 1.43 | 1.46| -1.78% | 2.05% | 2.77%| 1 | 10| | TPCH_NESTED(_20) | TPCH-Q6 | parquet / none / none | 2.29 | 2.33| -1.79% | 0.56% | 1.23%| 1 | 10| | TPCH_NESTED(_20) | TPCH-Q15 | parquet / none / none | 5.04 | 5.13| -1.84% | 0.61% | 2.01%| 1 | 10| | TPCH_NESTED(_20) | TPCH-Q1 | parquet / none / none | 5.98 | 6.12| -2.30% | 1.84% | 3.19%| 1 | 10| +--+--+---++-++---++-+---+ Change-Id: I9d7608181eeacfe706a09c1e153d0a3e1ee9b475 Reviewed-on: http://gerrit.cloudera.org:8080/3842 Reviewed-by: Tim Armstrong Reviewed-by: Marcel Kornacker Tested-by: Internal Jenkins (cherry picked from commit 3e2411f3078314c03bbe9c9b225770aa1580fdc4) --- M be/src/exec/CMakeLists.txt M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/hash-join-node.cc M be/src/exec/hash-join-node.h M be/src/exec/hbase-table-sink.cc M be/src/exec/hbase-table-sink.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/kudu-table-sink-test.cc M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h A be/src/exec/nested-loop-join-builder.cc A be/src/exec/nested-loop-join-builder.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/nested-loop-join-node.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/row-batch-cache.h M be/src/runtime/data-stream-sender.cc M be/src/runtime/data-stream-sender.h M be/src/runtime/data-stream-test.cc M be/src/runtime/plan-fragment-executor.cc M be/src/runtime/plan-fragment-executor.h M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h M be/src/util/stopwatch.h 29 files changed, 718 insertions(+), 461 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/12/4212/1 -- To view, visit http://gerrit.cloudera.org:8080/4212 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9d7608181eeacfe706a09c1e153d0a3e1ee9b475 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Marcel Kornacker Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3996: Migrate to updated Kudu insert string API
Hello Matthew Jacobs, Internal Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4252 to review the following change. Change subject: IMPALA-3996: Migrate to updated Kudu insert string API .. IMPALA-3996: Migrate to updated Kudu insert string API In Kudu 0.10 the Insert API changed for Strings and Binary values to avoid ambiguity around memory copying. They changed SetString() to copy the value given to it to avoid misuse, and added a SetStringNoCopy() with the old behavior. We upgraded to 0.10 which has some perf impact but is still correct. This changes the code to use the NoCopy call which was the behavior we had previously. See Kudu commit: https://github.com/apache/kudu/commit/48766a4ce17d422ced9a6ec78c9a9969ac44d8c9 Change-Id: I910c24724d0bc887b2d4a3e62ecdf72420a76f6f Reviewed-on: http://gerrit.cloudera.org:8080/4055 Reviewed-by: Matthew Jacobs Tested-by: Internal Jenkins (cherry picked from commit 6b5f9cf0f53d8a9b35182af81ffecb1351e8d477) --- M be/src/exec/kudu-table-sink.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/52/4252/1 -- To view, visit http://gerrit.cloudera.org:8080/4252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I910c24724d0bc887b2d4a3e62ecdf72420a76f6f Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3650: DISTRIBUTE BY required for managed Kudu tables
Hello Matthew Jacobs, Internal Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4244 to review the following change. Change subject: IMPALA-3650: DISTRIBUTE BY required for managed Kudu tables .. IMPALA-3650: DISTRIBUTE BY required for managed Kudu tables As of Kudu 0.9, DISTRIBUTE BY is now required when creating a new Kudu table. Create table analysis, data loading, and tests are updated to reflect this. This also bumps the Kudu version to 0.10.0. Change-Id: Ieb15110b10b28ef6dd8ec136c2522b5f44dca43e Reviewed-on: http://gerrit.cloudera.org:8080/3987 Reviewed-by: Matthew Jacobs Tested-by: Internal Jenkins (cherry picked from commit d113205ceea7d9134d2e4def9ddecdf942dd4c3e) --- M be/src/exec/kudu-testutil.h M bin/impala-config.sh M fe/src/main/java/com/cloudera/impala/analysis/CreateTableStmt.java M fe/src/main/java/com/cloudera/impala/catalog/delegates/KuduDdlDelegate.java M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeDDLTest.java M testdata/bin/generate-schema-statements.py M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-query/queries/QueryTest/create_kudu.test M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu-show-create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_crud.test M tests/query_test/test_kudu.py 13 files changed, 93 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/44/4244/1 -- To view, visit http://gerrit.cloudera.org:8080/4244 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ieb15110b10b28ef6dd8ec136c2522b5f44dca43e Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3981: Fix crash when accessing statestored / catalogd /memz page
Hello Internal Jenkins, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4251 to review the following change. Change subject: IMPALA-3981: Fix crash when accessing statestored / catalogd /memz page .. IMPALA-3981: Fix crash when accessing statestored / catalogd /memz page The /memz page tried to add JVM metrics even when they didn't exist for all daemons, not just Impala. This led to a crash when they tried to access ExecEnv::GetInstance() without an initialised ExecEnv at statestored and catalogd To fix, changed the memz handler method to take an optional metric group, provided by the caller. memz handler will check the existence of the metric group. Used C++11 lambdas rather than boost::bind to help simplify the code. Testing: Ran locally and looked at impalad/memz, statestored/memz and catalogd/memz Add a test file test_web_pages.py to test sending request to /memz on impalad / statestored / catalogd Change-Id: If59b10f20044d1a468f27810a3029fe18fb19f29 Reviewed-on: http://gerrit.cloudera.org:8080/3998 Reviewed-by: Tim Armstrong Tested-by: Internal Jenkins (cherry picked from commit 1fc487c7dfab862ca88a8f6ae7790e3b004756a8) --- M be/src/catalog/catalogd-main.cc M be/src/runtime/data-stream-mgr.cc M be/src/runtime/exec-env.cc M be/src/scheduling/simple-scheduler.cc M be/src/service/impala-server.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestored-main.cc M be/src/util/default-path-handlers.cc M be/src/util/default-path-handlers.h M be/src/util/memory-metrics.cc M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h A tests/webserver/test_web_pages.py M www/memz.tmpl 15 files changed, 96 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/51/4251/1 -- To view, visit http://gerrit.cloudera.org:8080/4251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If59b10f20044d1a468f27810a3029fe18fb19f29 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Kathy Sun Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3972: Improve display of /varz page
Hello Henry Robinson, Internal Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4228 to review the following change. Change subject: IMPALA-3972: Improve display of /varz page .. IMPALA-3972: Improve display of /varz page gflags has an API that returns metadata for all command-line flags. This patch changes /varz to use that to generate a table view, where rows are highlighted if the flag has changed from its default value. Change-Id: I5a0d47da7abf913918d5fba5c327e26b73d701d2 Reviewed-on: http://gerrit.cloudera.org:8080/3941 Reviewed-by: Henry Robinson Tested-by: Internal Jenkins (cherry picked from commit fe97579fe805b34bdccf4569ac626d11801a4823) --- M be/src/util/default-path-handlers.cc M www/common-header.tmpl A www/flags.tmpl 3 files changed, 96 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/28/4228/1 -- To view, visit http://gerrit.cloudera.org:8080/4228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5a0d47da7abf913918d5fba5c327e26b73d701d2 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-CR] Fix stress test runner bug introduced by IMPALA-3969
Hello Michael Brown, Internal Jenkins, Dan Hecht, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4243 to review the following change. Change subject: Fix stress test runner bug introduced by IMPALA-3969 .. Fix stress test runner bug introduced by IMPALA-3969 In some cases if common_query_options was not specified, the stress test runner could fail during the mem_limit binary search phase. Change-Id: I86484bb7c92ae1069f6a07cf3ea5027740364150 Reviewed-on: http://gerrit.cloudera.org:8080/4019 Reviewed-by: Michael Brown Reviewed-by: Dan Hecht Tested-by: Internal Jenkins (cherry picked from commit 0849827147b1b4d415b32c3a569946820347aa98) --- M tests/stress/concurrent_select.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/43/4243/1 -- To view, visit http://gerrit.cloudera.org:8080/4243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I86484bb7c92ae1069f6a07cf3ea5027740364150 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3992: bad shell error message when running nonexistent file
Hello Henry Robinson, Internal Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4240 to review the following change. Change subject: IMPALA-3992: bad shell error message when running nonexistent file .. IMPALA-3992: bad shell error message when running nonexistent file Fix the error handling code and add a test. Change-Id: Iebcf1dc8a1a08b400a2c769a9cff38ea02c8e525 Reviewed-on: http://gerrit.cloudera.org:8080/4022 Reviewed-by: Henry Robinson Tested-by: Internal Jenkins (cherry picked from commit 50e21247d60b9d33cb2601126a1af9231fc7e43b) --- M shell/impala_shell.py M tests/shell/test_shell_commandline.py 2 files changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/40/4240/1 -- To view, visit http://gerrit.cloudera.org:8080/4240 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iebcf1dc8a1a08b400a2c769a9cff38ea02c8e525 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] Remove dead and untested code
Hello Internal Jenkins, Dan Hecht, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4232 to review the following change. Change subject: Remove dead and untested code .. Remove dead and untested code Remove some code that the code coverage build revealed was untested. It is all dead, aside from SetErrorMsg(), which only had one callsite. Change-Id: I49c27cbfef03ef97befa9a607b3d8d7ac6e22a43 Reviewed-on: http://gerrit.cloudera.org:8080/3989 Reviewed-by: Dan Hecht Tested-by: Internal Jenkins (cherry picked from commit 5ec76c618f50405c154ebe5587b6be5d4f8d234d) --- M be/src/common/status.h M be/src/exprs/scalar-fn-call.cc M be/src/runtime/coordinator.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/raw-value.cc M be/src/runtime/raw-value.h M be/src/service/query-exec-state.h 8 files changed, 2 insertions(+), 220 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/32/4232/1 -- To view, visit http://gerrit.cloudera.org:8080/4232 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I49c27cbfef03ef97befa9a607b3d8d7ac6e22a43 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-4004: Don't access nested types in test failpoints.py
Hello Internal Jenkins, Alex Behm, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4253 to review the following change. Change subject: IMPALA-4004: Don't access nested types in test_failpoints.py .. IMPALA-4004: Don't access nested types in test_failpoints.py As part of fixing IMPALA-3692, the query in test_failpoints.py was updated to have a predicate on a string column in a parquet table. The update to the query was based on the failing query in the bug which happens to access nested columns. Apparently, this doesn't quite work with the legacy join and agg. This change fixes the test to also work with legacy join and agg. With the debug actions added in the fix of IMPALA-3692, it's not necessary to access nested types to reproduce the problem as long as there is a predicate on a string column. Change-Id: Idc5e67b9748a13fcd76ea5fe140e2e6b18e809b7 Reviewed-on: http://gerrit.cloudera.org:8080/4074 Reviewed-by: Alex Behm Tested-by: Internal Jenkins (cherry picked from commit b251fb0615d2bcdaade93a5d939c7ab6bc64960c) --- M tests/failure/test_failpoints.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/53/4253/1 -- To view, visit http://gerrit.cloudera.org:8080/4253 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idc5e67b9748a13fcd76ea5fe140e2e6b18e809b7 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3988: Only use first 96 bits of query id
Hello Marcel Kornacker, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4254 to review the following change. Change subject: IMPALA-3988: Only use first 96 bits of query id .. IMPALA-3988: Only use first 96 bits of query id This adds utility functions in uid-util.h to create query and instance ids and convert between them. It also adapts SimpleScheduler to utilize those functions when creating the instance id (TPlanFragmentInstanceCtx.fragment_instance_id). Change-Id: Ia56a03ed9a1d7e77c72b66a01cd48c5b6bf3624f Reviewed-on: http://gerrit.cloudera.org:8080/4065 Reviewed-by: Marcel Kornacker Tested-by: Marcel Kornacker (cherry picked from commit b69e469e99e4b0c08bc4718eb55b041cf7c11ffe) --- M be/src/scheduling/simple-scheduler.cc M be/src/service/impala-server.cc M be/src/util/CMakeLists.txt A be/src/util/uid-util-test.cc M be/src/util/uid-util.h M common/thrift/ImpalaInternalService.thrift 6 files changed, 96 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/54/4254/1 -- To view, visit http://gerrit.cloudera.org:8080/4254 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia56a03ed9a1d7e77c72b66a01cd48c5b6bf3624f Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Marcel Kornacker Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3828: Enable inversion for inner joins.
Hello Internal Jenkins, Alex Behm, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4246 to review the following change. Change subject: IMPALA-3828: Enable inversion for inner joins. .. IMPALA-3828: Enable inversion for inner joins. Testing: Ran the FE planner tests. Examined all the changed plans to verify that the changes are benefitial according to our cardinality estimates. Still need to do a real perf run. Change-Id: I8ba903f1df2446350cca7e71fdb13f550bf9de72 Reviewed-on: http://gerrit.cloudera.org:8080/4035 Reviewed-by: Alex Behm Tested-by: Internal Jenkins (cherry picked from commit 1bbd667fd3bb647eed93ff74a9206ad403c1578b) --- M fe/src/main/java/com/cloudera/impala/planner/Planner.java M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/order.test M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test 12 files changed, 1,824 insertions(+), 1,840 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/46/4246/1 -- To view, visit http://gerrit.cloudera.org:8080/4246 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8ba903f1df2446350cca7e71fdb13f550bf9de72 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-1619, IMPALA-3018: Address various small memory allocation related bugs
Hello Internal Jenkins, Dan Hecht, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4235 to review the following change. Change subject: IMPALA-1619, IMPALA-3018: Address various small memory allocation related bugs .. IMPALA-1619, IMPALA-3018: Address various small memory allocation related bugs This patch addresses a potential overflow in calculation FreePool::Rellocate() and its handling of zero-length allocations. This patch also adds code to gracefully handle malloc() failures when initializing/resizing hash tables. Change-Id: I6eb9a4472a65cf68edb0323b13d745277ead2e1d Reviewed-on: http://gerrit.cloudera.org:8080/3807 Reviewed-by: Tim Armstrong Reviewed-by: Dan Hecht Tested-by: Internal Jenkins (cherry picked from commit 5a55ba76108443fb7a136d6a48fb95840ab7e5e6) --- M be/src/exec/hash-table.cc M be/src/runtime/free-pool-test.cc M be/src/runtime/free-pool.h 3 files changed, 16 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/35/4235/1 -- To view, visit http://gerrit.cloudera.org:8080/4235 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6eb9a4472a65cf68edb0323b13d745277ead2e1d Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3856,IMPALA-3871: Fix BinaryPredicate normalization for Kudu
Hello Matthew Jacobs, Internal Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4241 to review the following change. Change subject: IMPALA-3856,IMPALA-3871: Fix BinaryPredicate normalization for Kudu .. IMPALA-3856,IMPALA-3871: Fix BinaryPredicate normalization for Kudu Change-Id: Iae7612433a2e27f8887abe6624f9ee0f4867b934 Reviewed-on: http://gerrit.cloudera.org:8080/3986 Reviewed-by: Matthew Jacobs Tested-by: Internal Jenkins (cherry picked from commit 0983da92bad9407c864f884973f9530e0059036d) --- M fe/src/main/java/com/cloudera/impala/analysis/BinaryPredicate.java M fe/src/main/java/com/cloudera/impala/analysis/Expr.java M fe/src/main/java/com/cloudera/impala/analysis/LiteralExpr.java M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test 4 files changed, 54 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/41/4241/1 -- To view, visit http://gerrit.cloudera.org:8080/4241 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iae7612433a2e27f8887abe6624f9ee0f4867b934 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3952: Clear scratch batch mem pool if Open() failed.
Hello Internal Jenkins, Alex Behm, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4236 to review the following change. Change subject: IMPALA-3952: Clear scratch batch mem pool if Open() failed. .. IMPALA-3952: Clear scratch batch mem pool if Open() failed. We used to be able to hit a DCHECK in HdfsParquetScanner::Close() when using the legacy aggs/joins if HdfsParquetScanner::Open() failed. With the legacy aggs/joins the tuple ptrs of the scratch batch are allocated from the scratch batch's mem pool, and if Open() failed we never freed or transferred the scratch batch's mem pool. Testing: I tested this patch together with the fix for IMPALA-3964 on core/hdfs with the legacy aggs and joins enabled. Change-Id: I55f32ed698a5b6fed8c28af1391aa07e1560e782 Reviewed-on: http://gerrit.cloudera.org:8080/3953 Reviewed-by: Alex Behm Tested-by: Internal Jenkins (cherry picked from commit 3a630a5d6871db6afe9ca343aea80b14c6525f3d) --- M be/src/exec/hdfs-parquet-scanner.cc 1 file changed, 7 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/36/4236/1 -- To view, visit http://gerrit.cloudera.org:8080/4236 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I55f32ed698a5b6fed8c28af1391aa07e1560e782 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3923: fix overflow in BufferedTupleStream::GetRows()
Hello Internal Jenkins, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4220 to review the following change. Change subject: IMPALA-3923: fix overflow in BufferedTupleStream::GetRows() .. IMPALA-3923: fix overflow in BufferedTupleStream::GetRows() Testing: Added a targeted unit test. Change-Id: Iabb977014d855d6edb035bf2e9519ac8b3234767 Reviewed-on: http://gerrit.cloudera.org:8080/3878 Reviewed-by: Tim Armstrong Tested-by: Internal Jenkins (cherry picked from commit e6702c67309cae4b46da6de935bcf873800a2f3c) --- M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h 3 files changed, 25 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/20/4220/1 -- To view, visit http://gerrit.cloudera.org:8080/4220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iabb977014d855d6edb035bf2e9519ac8b3234767 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3918: Remove Cloudera copyrights and add ASF license header
Hello Internal Jenkins, Dan Hecht, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4211 to review the following change. Change subject: IMPALA-3918: Remove Cloudera copyrights and add ASF license header .. IMPALA-3918: Remove Cloudera copyrights and add ASF license header For files that have a Cloudera copyright (and no other copyright notice), make changes to follow the ASF source file header policy here: http://www.apache.org/legal/src-headers.html#headers Specifically: 1) Remove the Cloudera copyright. 2) Modify NOTICE.txt according to http://www.apache.org/legal/src-headers.html#notice to follow that format and add a line for Cloudera. 3) Replace or add the existing ASF license text with the one given on the website. Much of this change was automatically generated via: git grep -li 'Copyright.*Cloudera' > modified_files.txt cat modified_files.txt | xargs perl -n -i -e 'print unless m#Copyright.*Cloudera#i;' cat modified_files_txt | xargs fix_apache_license.py [1] Some manual fixups were performed following those steps, especially when license text was completely missing from the file. [1] https://gist.github.com/anonymous/ff71292094362fc5c594 with minor modification to ORIG_LICENSE to match Impala's license text. Change-Id: I2e0bd8420945b953e1b806041bea4d72a3943d86 Reviewed-on: http://gerrit.cloudera.org:8080/3779 Reviewed-by: Dan Hecht Tested-by: Internal Jenkins (cherry picked from commit ffa7829b70933932d1ba5dfad9b6dc18d04e10e0) --- M CMakeLists.txt M NOTICE.txt M be/CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/CMakeLists.txt M be/src/benchmarks/atod-benchmark.cc M be/src/benchmarks/atof-benchmark.cc M be/src/benchmarks/atoi-benchmark.cc M be/src/benchmarks/bitmap-benchmark.cc M be/src/benchmarks/bloom-filter-benchmark.cc M be/src/benchmarks/hash-benchmark.cc M be/src/benchmarks/in-predicate-benchmark.cc M be/src/benchmarks/int-hash-benchmark.cc M be/src/benchmarks/lock-benchmark.cc M be/src/benchmarks/multiint-benchmark.cc M be/src/benchmarks/network-perf-benchmark.cc M be/src/benchmarks/overflow-benchmark.cc M be/src/benchmarks/parse-timestamp-benchmark.cc M be/src/benchmarks/rle-benchmark.cc M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/benchmarks/status-benchmark.cc M be/src/benchmarks/string-benchmark.cc M be/src/benchmarks/string-compare-benchmark.cc M be/src/benchmarks/string-search-benchmark.cc M be/src/benchmarks/thread-create-benchmark.cc M be/src/benchmarks/tuple-layout-benchmark.cc M be/src/catalog/CMakeLists.txt M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog-util.cc M be/src/catalog/catalog-util.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/catalog/catalogd-main.cc M be/src/codegen/CMakeLists.txt M be/src/codegen/codegen-anyval-ir.cc M be/src/codegen/codegen-anyval.cc M be/src/codegen/codegen-anyval.h M be/src/codegen/codegen-symbol-emitter.cc M be/src/codegen/codegen-symbol-emitter.h M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir-data.h M be/src/codegen/impala-ir.cc M be/src/codegen/impala-ir.h M be/src/codegen/instruction-counter-test.cc M be/src/codegen/instruction-counter.cc M be/src/codegen/instruction-counter.h M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/codegen/mcjit-mem-mgr.h M be/src/common/CMakeLists.txt M be/src/common/atomic-test.cc M be/src/common/atomic.h M be/src/common/compiler-util.h M be/src/common/global-flags.cc M be/src/common/global-types.h M be/src/common/hdfs.h M be/src/common/init.cc M be/src/common/init.h M be/src/common/logging.cc M be/src/common/logging.h M be/src/common/names.h M be/src/common/object-pool.h M be/src/common/status.cc M be/src/common/status.h M be/src/common/version.h M be/src/exec/CMakeLists.txt M be/src/exec/aggregation-node-ir.cc M be/src/exec/aggregation-node.cc M be/src/exec/aggregation-node.h M be/src/exec/analytic-eval-node.cc M be/src/exec/analytic-eval-node.h M be/src/exec/base-sequence-scanner.cc M be/src/exec/base-sequence-scanner.h M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/delimited-text-parser-test.cc M be/src/exec/delimited-text-parser.cc M be/src/exec/delimited-text-parser.h M be/src/exec/delimited-text-parser.inline.h M be/src/exec/empty-set-node.cc M be/src/exec/empty-set-node.h M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/external-data-source-executor.cc M be/src/exec/external-data-source-executor.h M be/src/exec/filter-context.cc M be/src/exec/filter-context.h M be/src
[Impala-CR] IMPALA-3962: Clean up scratch tuple batch on scan failures
Hello Michael Ho, Internal Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4248 to review the following change. Change subject: IMPALA-3962: Clean up scratch tuple batch on scan failures .. IMPALA-3962: Clean up scratch tuple batch on scan failures The parquet scanner doesn't clean up 'scratch_batch_' properly which causes it to process a partially filled scratch_batch_ if any of the column reader fails. This change cleans up the scratch batch if any of the parquet column readers fails. The clean up involves freeing the mem_pool of scratch_batch_ and setting number of tuples in scratch_batch_ to 0. This change also extends debug action to emulate the behavior of exceeding the query's memory limit. Change-Id: If1e27a1517d09ccaabdae1492b7e1fbf661ae3e5 Reviewed-on: http://gerrit.cloudera.org:8080/3991 Reviewed-by: Michael Ho Tested-by: Internal Jenkins (cherry picked from commit 2aa86309d15cba8a965c83d81308e32df899ec8a) --- M be/src/exec/exec-node.cc M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M common/thrift/PlanNodes.thrift M tests/failure/test_failpoints.py 10 files changed, 76 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/48/4248/1 -- To view, visit http://gerrit.cloudera.org:8080/4248 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If1e27a1517d09ccaabdae1492b7e1fbf661ae3e5 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3969: stress test: add option to set common query options
Hello David Knupp, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4225 to review the following change. Change subject: IMPALA-3969: stress test: add option to set common query options .. IMPALA-3969: stress test: add option to set common query options It can be useful for debugging purposes to run the stress test with custom query options, for example with codegen disabled. This patch adds a command line option to the stress test entry point that allows a caller to set query options. To reduce support as new options are chosen, we allow freeform, space-delimited option=value arguments as the option's value, like: --common_query_options option1=value1 ... optionN=valueN This means we don't do much validation that these options and values are well-formed. Callers must take care to type the correct options and values. Testing: I ran concurrent_select.py by hand against an Impala cluster with codegen both enabled and disabled, ala --common_query_options DISABLE_CODEGEN=true Both the log written by concurrent_select.py and the Impala logs on the cluster indicated DISABLE_CODEGEN was being set as I directed. I also did negative testing for a few bad --common_query_options values. Either concurrent_select.py catches the error, or when the first query runs, the error is reported. Change-Id: Iada041aace60c218a12178d8f1b9a68ff29de72e Reviewed-on: http://gerrit.cloudera.org:8080/3887 Reviewed-by: David Knupp Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong (cherry picked from commit 9162d5d0544bf080da0a86331e9f77fc56803678) --- M tests/stress/concurrent_select.py 1 file changed, 49 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/25/4225/1 -- To view, visit http://gerrit.cloudera.org:8080/4225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iada041aace60c218a12178d8f1b9a68ff29de72e Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3063: Separate join inversion from join ordering.
Hello Internal Jenkins, Alex Behm, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4239 to review the following change. Change subject: IMPALA-3063: Separate join inversion from join ordering. .. IMPALA-3063: Separate join inversion from join ordering. Before this change joins were inverted while doing join ordering. That approach was unnecessarily complex because it required modifying the global analysis state for correct conjunct placement, etc. However, join inversion is independent of join ordering, and the existing approach could lead to generating invalid plans with distributed non-equi right outer/semi joins, which we cannot execute in the backend. After this change joins are inverted in a separate pass over the single-node plan. This simplifies the inversion logic and allows us to avoid generating those invalid plans. Note that this change is not only a separation of functionality for the following reasons: 1. Our join cardinality estimation is not symmetric, i.e., A JOIN B may not give the same estimate as B JOIN A due to our FK/PK detection heuristic. In the context of this patch this means that an inverted join may have a different cardinality estimate, so plans may change depending on whether the inversion is done during join ordering of after. 2. We currently only invert outer/semi/anti joins based on the rhs table ref join op. In this patch I want to preserve the existing behavior as much as possible, but when doing the join ordering in a separate pass we may see a join opn in a JoinNode that is different from the rhs table ref. So in some situations the inversion behavior based on the join op could be different and there are some examples in this patch. This patch also moves the logic of converting hash joins to nested-loop joins into a separate pass over the single-node plan. Change-Id: If86db7753fc585bb4c69612745ec010327a4 Reviewed-on: http://gerrit.cloudera.org:8080/3846 Reviewed-by: Alex Behm Tested-by: Internal Jenkins (cherry picked from commit 532b1fe1186725b8e81fff93b59fc7cebf563c8b) --- M be/src/exec/blocking-join-node.h M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java M fe/src/main/java/com/cloudera/impala/analysis/BinaryPredicate.java M fe/src/main/java/com/cloudera/impala/analysis/SelectList.java M fe/src/main/java/com/cloudera/impala/analysis/TableRef.java M fe/src/main/java/com/cloudera/impala/planner/AggregationNode.java M fe/src/main/java/com/cloudera/impala/planner/AnalyticEvalNode.java M fe/src/main/java/com/cloudera/impala/planner/DistributedPlanner.java M fe/src/main/java/com/cloudera/impala/planner/ExchangeNode.java M fe/src/main/java/com/cloudera/impala/planner/HashJoinNode.java M fe/src/main/java/com/cloudera/impala/planner/HdfsScanNode.java M fe/src/main/java/com/cloudera/impala/planner/JoinNode.java M fe/src/main/java/com/cloudera/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/com/cloudera/impala/planner/PlanNode.java M fe/src/main/java/com/cloudera/impala/planner/Planner.java M fe/src/main/java/com/cloudera/impala/planner/SelectNode.java M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java M fe/src/main/java/com/cloudera/impala/planner/SingularRowSrcNode.java M fe/src/main/java/com/cloudera/impala/planner/SortNode.java M fe/src/main/java/com/cloudera/impala/planner/SubplanNode.java M fe/src/main/java/com/cloudera/impala/planner/UnionNode.java M fe/src/main/java/com/cloudera/impala/planner/UnnestNode.java M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test 29 files changed, 661 insertions(+), 363 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/39/4239/1 -- To view, visit http://gerrit.cloudera.org:8080/4239 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If86db7753fc585bb4c69612745ec010327a4 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: anujphadke
[Impala-CR] IMPALA-3953: Fixes for KuduScanNode BE test failure
Hello Internal Jenkins, Dan Hecht, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4231 to review the following change. Change subject: IMPALA-3953: Fixes for KuduScanNode BE test failure .. IMPALA-3953: Fixes for KuduScanNode BE test failure After a previous fix for IMPALA-3857, KuduScanNodeTest TestLimitsAreEnforced (BE test) occasionally throws when a scanner thread takes a lock_ that isn't valid, crashing the process. It looks like the issue is likely that TestScanEmptyString isn't closing its KuduScanNode, and a lingering ScannerThread may end up touching invalid memory later. This fixes the test case and also: 1) Adds some missing synchronization in KuduScanNode which was found in the process of investigating this issue. 2) Adds a DCHECK on ~KuduScanNode() to ensure it was closed. This was tested by running KuduScanNodeTest in a loop for 5 hours. Without the fix, the failure was produced within several hours. Change-Id: I16be206c60a692d2a26d719de8cc45e859b06e97 Reviewed-on: http://gerrit.cloudera.org:8080/3888 Reviewed-by: Tim Armstrong Reviewed-by: Dan Hecht Tested-by: Internal Jenkins (cherry picked from commit 6fc399ebc435121cdb7865ff4987aca1c95af5fc) --- M be/src/exec/kudu-scan-node-test.cc M be/src/exec/kudu-scan-node.cc 2 files changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/4231/1 -- To view, visit http://gerrit.cloudera.org:8080/4231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I16be206c60a692d2a26d719de8cc45e859b06e97 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-CR] IMPALA-3376: Extra definition level when writing Parquet files
Hello Matthew Jacobs, Internal Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/4207 to review the following change. Change subject: IMPALA-3376: Extra definition level when writing Parquet files .. IMPALA-3376: Extra definition level when writing Parquet files Currently, when writing a new value to a parquet file, we write the definition level before checking if there's enough space on the current page for the value. If there isn't, we create a new page and rewrite the definition level to it, but this leaves the definition level for that value still written to the old page. To fix this, we should make sure that we have enough space to write both the definition level and the value before writing either. This patch also modifies the parquet-reader tool, which reads parquet files and performs minimal sanity checking on their metadata, to check for extra definition levels, and adds a test that runs the tool automatically. Change-Id: I20f25a90aa1ef74b4f00f38f832bc1c1853342c6 Reviewed-on: http://gerrit.cloudera.org:8080/3835 Reviewed-by: Thomas Tauber-Marshall Tested-by: Internal Jenkins Reviewed-by: Matthew Jacobs (cherry picked from commit 45d059855b4afc2d5730c5e7ecaed7141993f354) --- M be/src/exec/hdfs-parquet-table-writer.cc M be/src/util/parquet-reader.cc M be/src/util/rle-encoding.h M tests/query_test/test_insert_parquet.py 4 files changed, 137 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/07/4207/1 -- To view, visit http://gerrit.cloudera.org:8080/4207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I20f25a90aa1ef74b4f00f38f832bc1c1853342c6 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Sailesh Mukil
