date:20210218

[Impala-ASF-CR] IMPALA-10467: Implement ds theta union() function

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17048 )

Change subject: IMPALA-10467: Implement ds_theta_union() function
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/17048
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I91baf58c76eb43748acd5245047edac8c66761b2
Gerrit-Change-Number: 17048
Gerrit-PatchSet: 3
Gerrit-Owner: Fucun Chu 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 19 Feb 2021 07:50:08 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10467: Implement ds theta union() function

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17048 )

Change subject: IMPALA-10467: Implement ds_theta_union() function
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6902/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/17048
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I91baf58c76eb43748acd5245047edac8c66761b2
Gerrit-Change-Number: 17048
Gerrit-PatchSet: 3
Gerrit-Owner: Fucun Chu 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 19 Feb 2021 07:50:09 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10467: Implement ds theta union() function

2021-02-18 Thread Gabor Kaszab (Code Review)

Gabor Kaszab has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17048 )

Change subject: IMPALA-10467: Implement ds_theta_union() function
..


Patch Set 2: Code-Review+2

Thanks for implementing this! It seems that adding new and new DataSketches 
functionality is sometimes more copy-paste and names rewrite than actually 
implementing something new :)


--
To view, visit http://gerrit.cloudera.org:8080/17048
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I91baf58c76eb43748acd5245047edac8c66761b2
Gerrit-Change-Number: 17048
Gerrit-PatchSet: 2
Gerrit-Owner: Fucun Chu 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 19 Feb 2021 07:49:34 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10467: Implement ds theta union() function

2021-02-18 Thread Fucun Chu (Code Review)

Fucun Chu has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/17048 )

Change subject: IMPALA-10467: Implement ds_theta_union() function
..

IMPALA-10467: Implement ds_theta_union() function

This function receives a set of serialized Apache DataSketches Theta
sketches produced by ds_theta_sketch() and merges them into a single
sketch.

An example usage is to create a sketch for each partition of a table,
write these sketches to a separate table and based on which partition
the user is interested of the relevant sketches can be union-ed
together to get an estimate. E.g.:
  SELECT
  ds_theta_estimate(ds_theta_union(sketch_col))
  FROM sketch_tbl
  WHERE partition_col=1 OR partition_col=5;

Testing:
  - Apart from the automated tests I added to this patch I also
tested ds_theta_union() on a bigger dataset to check that
serialization, deserialization and merging steps work well. I
took TPCH25.linelitem, created a number of sketches with grouping
by l_shipdate and called ds_theta_union() on those sketches

Change-Id: I91baf58c76eb43748acd5245047edac8c66761b2
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/theta_sketches_from_impala.parquet
M testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test
M tests/query_test/test_datasketches.py
7 files changed, 152 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/17048/2
--
To view, visit http://gerrit.cloudera.org:8080/17048
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I91baf58c76eb43748acd5245047edac8c66761b2
Gerrit-Change-Number: 17048
Gerrit-PatchSet: 2
Gerrit-Owner: Fucun Chu 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 19 Feb 2021 06:30:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..

IMPALA-10519: Allow setting of num_reactors for KuduClient

The KuduClient allows setting a number of reactor threads, which are
used for sending rpcs to Kudu such as for scans. The defaults is 4
threads, which may be too few for many use cases, since there is a
single KuduClient per impalad.

This patch adds a flag --kudu_client_num_reactor_threads which allows
users to set the number of threads. The default for the flag is 4,
which reflects the current default within Kudu, to avoid possible
regressions.

Some follow up work could be to do performance experiments and set
the default empirically.

Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Reviewed-on: http://gerrit.cloudera.org:8080/17086
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M be/src/exec/kudu-util.cc
1 file changed, 10 insertions(+), 0 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 5
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall

[Impala-ASF-CR] IMPALA-10516: Bump up the versions of jackson databind and slf4j

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17085 )

Change subject: IMPALA-10516: Bump up the versions of jackson databind and slf4j
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/17085
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie7b84a90fec955dbaebd36b63294229b05eb00d8
Gerrit-Change-Number: 17085
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Comment-Date: Fri, 19 Feb 2021 01:33:58 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10516: Bump up the versions of jackson databind and slf4j

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/17085 )

Change subject: IMPALA-10516: Bump up the versions of jackson databind and slf4j
..

IMPALA-10516: Bump up the versions of jackson databind and slf4j

A flaw was found in FasterXML Jackson Databind, where it did not have
entity expansion secured properly.

This patch bumps up jackson databind to 2.10.5.1. It also changes slf4j
to 1.7.30.

Testing:
 - Built Impala on local machine as clean build. Verified that new
   versions of jar files jackson-databind-2.10.5.1.jar,
   slf4j-api-1.7.30.jar, and slf4j-log4j12-1.7.30.jar were built in
   fe/target/build-classpath.txt.

Change-Id: Ie7b84a90fec955dbaebd36b63294229b05eb00d8
Reviewed-on: http://gerrit.cloudera.org:8080/17085
Reviewed-by: Joe McDonnell 
Tested-by: Impala Public Jenkins 
---
M java/pom.xml
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Joe McDonnell: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/17085
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie7b84a90fec955dbaebd36b63294229b05eb00d8
Gerrit-Change-Number: 17085
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8167/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 19 Feb 2021 00:46:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 19 Feb 2021 00:37:27 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6901/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 19 Feb 2021 00:37:28 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Bikramjeet Vig (Code Review)

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 19 Feb 2021 00:35:02 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Thomas Tauber-Marshall (Code Review)

Thomas Tauber-Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17086/2/be/src/exec/kudu-util.cc
File be/src/exec/kudu-util.cc:

http://gerrit.cloudera.org:8080/#/c/17086/2/be/src/exec/kudu-util.cc@48
PS2, Line 48: 4,
> nit: maybe use the default 4 to be consistent with current setting, if we a
Done



--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 19 Feb 2021 00:27:27 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Thomas Tauber-Marshall (Code Review)

Hello Hao Hao, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17086

to look at the new patch set (#3).

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..

IMPALA-10519: Allow setting of num_reactors for KuduClient

The KuduClient allows setting a number of reactor threads, which are
used for sending rpcs to Kudu such as for scans. The defaults is 4
threads, which may be too few for many use cases, since there is a
single KuduClient per impalad.

This patch adds a flag --kudu_client_num_reactor_threads which allows
users to set the number of threads. The default for the flag is 4,
which reflects the current default within Kudu, to avoid possible
regressions.

Some follow up work could be to do performance experiments and set
the default empirically.

Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
---
M be/src/exec/kudu-util.cc
1 file changed, 10 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/17086/3
--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Hao Hao (Code Review)

Hao Hao has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..


Patch Set 2: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17086/2/be/src/exec/kudu-util.cc
File be/src/exec/kudu-util.cc:

http://gerrit.cloudera.org:8080/#/c/17086/2/be/src/exec/kudu-util.cc@48
PS2, Line 48: 16
nit: maybe use the default 4 to be consistent with current setting, if we are 
not sure what a good default will be (as it may vary depends on the workload.)



--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 19 Feb 2021 00:21:55 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] [WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max filters

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17075 )

Change subject: [WIP] IMPALA-10494: Making use of the min/max column stats to 
improve min/max filters
..


Patch Set 4:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8166/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/17075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
Gerrit-Change-Number: 17075
Gerrit-PatchSet: 4
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Comment-Date: Thu, 18 Feb 2021 23:15:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..

IMPALA-10161: User LDAP Search bind support

This change adds user search bind support next to simple bind that can
be configured with LDAP filters. The group check was done with LDAP
search earlier, this change adds the possibility to configure it with
Hadoop library like options, which is the LDAP filter with optional
patterns. The '{0}' will be replaced with the user name while the
'{1}' pattern will be replaced with the user dn.

The following new flags have been added:
 --ldap_search_bind_authentication: a flag to change between simple and
search bind
 --ldap_user_search_basedn: the base dn for the LDAP subtree to search
 --ldap_group_search_basedn: the base dn for the LDAP subtree to search

Tested:
 - Custom cluster tests have been added

Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Reviewed-on: http://gerrit.cloudera.org:8080/17047
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M be/src/rpc/authentication.cc
M be/src/util/CMakeLists.txt
A be/src/util/ldap-search-bind.cc
A be/src/util/ldap-search-bind.h
A be/src/util/ldap-simple-bind.cc
A be/src/util/ldap-simple-bind.h
M be/src/util/ldap-util.cc
M be/src/util/ldap-util.h
M be/src/util/webserver.cc
M fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java
A 
fe/src/test/java/org/apache/impala/customcluster/LdapSearchBindImpalaShellTest.java
A 
fe/src/test/java/org/apache/impala/customcluster/LdapSimpleBindImpalaShellTest.java
M fe/src/test/java/org/apache/impala/testutil/LdapUtil.java
M fe/src/test/resources/users.ldif
14 files changed, 905 insertions(+), 336 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 6
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..


Patch Set 5: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 5
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Thu, 18 Feb 2021 23:12:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9867: Add Support for Spilling to S3: Milestone 1

2021-02-18 Thread Thomas Tauber-Marshall (Code Review)

Thomas Tauber-Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16318 )

Change subject: IMPALA-9867: Add Support for Spilling to S3: Milestone 1
..


Patch Set 36: Code-Review+2

Obviously there's a bunch of follow up work here, but I think it looks great as 
a first milestone.


--
To view, visit http://gerrit.cloudera.org:8080/16318
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I419b1d5dbbfe35334d9f964c4b65e553579fdc89
Gerrit-Change-Number: 16318
Gerrit-PatchSet: 36
Gerrit-Owner: Yida Wu 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Thu, 18 Feb 2021 23:07:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max filters

2021-02-18 Thread Qifan Chen (Code Review)

Qifan Chen has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/17075 )

Change subject: [WIP] IMPALA-10494: Making use of the min/max column stats to 
improve min/max filters
..

[WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max 
filters

This patch adds the functionality to compute the minimal and
the maximal value for a column of type integer, double, timestamp
and date during column statistics computation and to store the stats
in HMS. These two new stats are used by the back end to discard
min/max filters whose coverage are too close to the actual range.

Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/incr-stats-util.cc
M be/src/exec/incr-stats-util.h
M be/src/service/hs2-util.cc
M be/src/service/hs2-util.h
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
10 files changed, 453 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/17075/4
--
To view, visit http://gerrit.cloudera.org:8080/17075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
Gerrit-Change-Number: 17075
Gerrit-PatchSet: 4
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen

[Impala-ASF-CR] [WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max filters

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17075 )

Change subject: [WIP] IMPALA-10494: Making use of the min/max column stats to 
improve min/max filters
..


Patch Set 3:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8165/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/17075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
Gerrit-Change-Number: 17075
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Comment-Date: Thu, 18 Feb 2021 22:09:33 +
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max filters

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17075 )

Change subject: [WIP] IMPALA-10494: Making use of the min/max column stats to 
improve min/max filters
..


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17075/3/fe/src/main/java/org/apache/impala/catalog/Column.java
File fe/src/main/java/org/apache/impala/catalog/Column.java:

http://gerrit.cloudera.org:8080/#/c/17075/3/fe/src/main/java/org/apache/impala/catalog/Column.java@73
PS3, Line 73:LOG.error("Column::updateStats(ColumnStatisticsData): " + 
name_ + " #distinct=" + stats_.getNumDistinctValues());
line too long (116 > 90)


http://gerrit.cloudera.org:8080/#/c/17075/3/fe/src/main/java/org/apache/impala/catalog/Column.java@82
PS3, Line 82:LOG.error("Column::updateStats(TColumnStats): " + name_ + " 
#distinct=" + stats_.getNumDistinctValues());
line too long (108 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/17075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
Gerrit-Change-Number: 17075
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Comment-Date: Thu, 18 Feb 2021 21:59:49 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] [WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max filters

2021-02-18 Thread Qifan Chen (Code Review)

Qifan Chen has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/17075 )

Change subject: [WIP] IMPALA-10494: Making use of the min/max column stats to 
improve min/max filters
..

[WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max 
filters

This patch adds the functionality to compute the minimal and
the maximal value for a column of type integer, double, timestamp
and date during column statistics computation and to store the stats
in HMS. These two new stats are used by the back end to discard
min/max filters whose coverage are too close to the actual range.

Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/incr-stats-util.cc
M be/src/exec/incr-stats-util.h
M be/src/service/hs2-util.cc
M be/src/service/hs2-util.h
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
11 files changed, 430 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/17075/3
--
To view, visit http://gerrit.cloudera.org:8080/17075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
Gerrit-Change-Number: 17075
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8164/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 21:35:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17086 )

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8163/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 21:28:12 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Thomas Tauber-Marshall (Code Review)

Hello Hao Hao, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17086

to look at the new patch set (#2).

Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..

IMPALA-10519: Allow setting of num_reactors for KuduClient

The KuduClient allows setting a number of reactor threads, which are
used for sending rpcs to Kudu such as for scans. The defaults is 4
threads, which may be too few for many use cases, since there is a
single KuduClient per impalad.

This patch adds a flag --kudu_client_num_reactor_threads which allows
users to set the number of threads. The default for the flag is 16,
which is consistent with the number of threads used in other IO
scenarios such as --num_s3_io_threads and --num_ozone_io_threads.

Some follow up work could be to do performance experiments and set
the default empirically.

Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
---
M be/src/exec/kudu-util.cc
1 file changed, 10 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/17086/2
--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-10519: Allow setting of num reactors for KuduClient

2021-02-18 Thread Thomas Tauber-Marshall (Code Review)

Thomas Tauber-Marshall has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/17086


Change subject: IMPALA-10519: Allow setting of num_reactors for KuduClient
..

IMPALA-10519: Allow setting of num_reactors for KuduClient

The KuduClient allows setting a number of reactor threads, which are
used for sending rpcs to Kudu such as for scans. The defaults is 4
threads, which may be too few for many use cases, since there is a
single KuduClient per impalad.

This patch adds a flag --kudu_client_num_reactor_threads which allows
users to set the number of threads. The default for the flag is 16,
which is consistent with the number of threads used in other IO
scenarios such as --num_s3_io_threads and --num_ozone_io_threads.

Some follow up work could be to do performance experiments and set
the default empirically.

Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
---
M be/src/exec/kudu-util.cc
1 file changed, 5 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/17086/1
--
To view, visit http://gerrit.cloudera.org:8080/17086
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
Gerrit-Change-Number: 17086
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Tauber-Marshall

[Impala-ASF-CR] IMPALA-10516: Bump up the versions of jackson databind and slf4j

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17085 )

Change subject: IMPALA-10516: Bump up the versions of jackson databind and slf4j
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8162/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17085
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie7b84a90fec955dbaebd36b63294229b05eb00d8
Gerrit-Change-Number: 17085
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Comment-Date: Thu, 18 Feb 2021 20:03:27 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10516: Bump up the versions of jackson databind and slf4j

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17085 )

Change subject: IMPALA-10516: Bump up the versions of jackson databind and slf4j
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6900/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/17085
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie7b84a90fec955dbaebd36b63294229b05eb00d8
Gerrit-Change-Number: 17085
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Comment-Date: Thu, 18 Feb 2021 19:49:59 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10516: Bump up the versions of jackson databind and slf4j

2021-02-18 Thread Joe McDonnell (Code Review)

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17085 )

Change subject: IMPALA-10516: Bump up the versions of jackson databind and slf4j
..


Patch Set 1: Code-Review+2

This looks good to me.


--
To view, visit http://gerrit.cloudera.org:8080/17085
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie7b84a90fec955dbaebd36b63294229b05eb00d8
Gerrit-Change-Number: 17085
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Comment-Date: Thu, 18 Feb 2021 19:49:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10516: Bump up the versions of jackson databind and slf4j

2021-02-18 Thread Wenzhe Zhou (Code Review)

Wenzhe Zhou has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/17085


Change subject: IMPALA-10516: Bump up the versions of jackson databind and slf4j
..

IMPALA-10516: Bump up the versions of jackson databind and slf4j

A flaw was found in FasterXML Jackson Databind, where it did not have
entity expansion secured properly.

This patch bumps up jackson databind to 2.10.5.1. It also changes slf4j
to 1.7.30.

Testing:
 - Built Impala on local machine as clean build. Verified that new
   versions of jar files jackson-databind-2.10.5.1.jar,
   slf4j-api-1.7.30.jar, and slf4j-log4j12-1.7.30.jar were built in
   fe/target/build-classpath.txt.

Change-Id: Ie7b84a90fec955dbaebd36b63294229b05eb00d8
---
M java/pom.xml
1 file changed, 2 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/17085/1
--
To view, visit http://gerrit.cloudera.org:8080/17085
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie7b84a90fec955dbaebd36b63294229b05eb00d8
Gerrit-Change-Number: 17085
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Joe McDonnell

[Impala-ASF-CR] Bump slf4j version from 1.7.25 to 1.7.30

2021-02-18 Thread Csaba Ringhofer (Code Review)

Csaba Ringhofer has abandoned this change. ( 
http://gerrit.cloudera.org:8080/17082 )

Change subject: Bump slf4j version from 1.7.25 to 1.7.30
..


Abandoned

Abandoning as others started to work on the same thing: IMPALA-10516
--
To view, visit http://gerrit.cloudera.org:8080/17082
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I963ef457d0469fe8ff021e551a7abd9f0b4faf0c
Gerrit-Change-Number: 17082
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] Bump slf4j version from 1.7.25 to 1.7.30

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17082 )

Change subject: Bump slf4j version from 1.7.25 to 1.7.30
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8161/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17082
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I963ef457d0469fe8ff021e551a7abd9f0b4faf0c
Gerrit-Change-Number: 17082
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 19:02:59 +
Gerrit-HasComments: No

[Impala-ASF-CR] Bump slf4j version from 1.7.25 to 1.7.30

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17082 )

Change subject: Bump slf4j version from 1.7.25 to 1.7.30
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6899/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/17082
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I963ef457d0469fe8ff021e551a7abd9f0b4faf0c
Gerrit-Change-Number: 17082
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 18:43:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] Bump slf4j version from 1.7.25 to 1.7.30

2021-02-18 Thread Csaba Ringhofer (Code Review)

Csaba Ringhofer has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/17082


Change subject: Bump slf4j version from 1.7.25 to 1.7.30
..

Bump slf4j version from 1.7.25 to 1.7.30

Change-Id: I963ef457d0469fe8ff021e551a7abd9f0b4faf0c
---
M java/pom.xml
1 file changed, 1 insertion(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/17082/1
--
To view, visit http://gerrit.cloudera.org:8080/17082
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I963ef457d0469fe8ff021e551a7abd9f0b4faf0c
Gerrit-Change-Number: 17082
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer

[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables

2021-02-18 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17081 )

Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write 
id for ACID tables
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17081/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/17081/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3187
PS1, Line 3187: long txnId = MetastoreShim.openTransaction(msClient);
are we opening a transaction post adding the partitions? Shouldn't the flow be, 
open transaction, allocate write id add partition, commit transaction?



--
To view, visit http://gerrit.cloudera.org:8080/17081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd
Gerrit-Change-Number: 17081
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Thu, 18 Feb 2021 18:39:04 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10515: Refactoring to support external frontend.

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17080 )

Change subject: IMPALA-10515: Refactoring to support external frontend.
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8160/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17080
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70035ec082be09a9583275191248d085414b27dd
Gerrit-Change-Number: 17080
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Thu, 18 Feb 2021 18:19:54 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10515: Refactoring to support external frontend.

2021-02-18 Thread Aman Sinha (Code Review)

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17080

to look at the new patch set (#3).

Change subject: IMPALA-10515: Refactoring to support external frontend.
..

IMPALA-10515: Refactoring to support external frontend.

This patch does some minimal refactoring of the frontend
code to allow an external frontend to leverage certain apis
and class variables. A couple of new methods are also added.
The changes include:
 - modifying a few class methods to be public static
 - changing visibility of a few private member variables to
   protected for use by a derived class
 - adding an alternate PlannerContext constructor
 - passing an additional initialization parameter to BackendConfig
 - minimal refactor of selectivity calculation for predicates
 - refactor of count(*) optimization methods
 - small changes to the validatePlan and useParallelPlan
 - new clone method for functions and one for null literals

Testing:
 - Ran PlannerTest and TpcdsPlannerTest
 - Ran e2e TPCH tests for sanity check
 - Since these are refactoring changes, no new tests are added in
   this patch but once other supporting changes are done, I
   anticipate adding sanity tests for external FE in a future commit.

Change-Id: I70035ec082be09a9583275191248d085414b27dd
---
M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
20 files changed, 126 insertions(+), 59 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/17080/3
--
To view, visit http://gerrit.cloudera.org:8080/17080
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I70035ec082be09a9583275191248d085414b27dd
Gerrit-Change-Number: 17080
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8159/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 4
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Thu, 18 Feb 2021 17:47:56 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..


Patch Set 5: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 5
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Thu, 18 Feb 2021 17:31:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6898/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 5
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Thu, 18 Feb 2021 17:31:22 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Csaba Ringhofer (Code Review)

Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 4
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Thu, 18 Feb 2021 17:30:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Tamas Mate (Code Review)

Tamas Mate has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..


Patch Set 3:

(1 comment)

Thank you for the reviews Csaba, Thomas.
Updated the change.

http://gerrit.cloudera.org:8080/#/c/17047/3/be/src/util/ldap-search-bind.cc
File be/src/util/ldap-search-bind.cc:

http://gerrit.cloudera.org:8080/#/c/17047/3/be/src/util/ldap-search-bind.cc@112
PS3, Line 112:   VLOG(2) << "LDAP bind successful"
> We write successful even if success is false.
Done



--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Thu, 18 Feb 2021 17:29:17 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Tamas Mate (Code Review)

Tamas Mate has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..

IMPALA-10161: User LDAP Search bind support

This change adds user search bind support next to simple bind that can
be configured with LDAP filters. The group check was done with LDAP
search earlier, this change adds the possibility to configure it with
Hadoop library like options, which is the LDAP filter with optional
patterns. The '{0}' will be replaced with the user name while the
'{1}' pattern will be replaced with the user dn.

The following new flags have been added:
 --ldap_search_bind_authentication: a flag to change between simple and
search bind
 --ldap_user_search_basedn: the base dn for the LDAP subtree to search
 --ldap_group_search_basedn: the base dn for the LDAP subtree to search

Tested:
 - Custom cluster tests have been added

Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
---
M be/src/rpc/authentication.cc
M be/src/util/CMakeLists.txt
A be/src/util/ldap-search-bind.cc
A be/src/util/ldap-search-bind.h
A be/src/util/ldap-simple-bind.cc
A be/src/util/ldap-simple-bind.h
M be/src/util/ldap-util.cc
M be/src/util/ldap-util.h
M be/src/util/webserver.cc
M fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java
A 
fe/src/test/java/org/apache/impala/customcluster/LdapSearchBindImpalaShellTest.java
A 
fe/src/test/java/org/apache/impala/customcluster/LdapSimpleBindImpalaShellTest.java
M fe/src/test/java/org/apache/impala/testutil/LdapUtil.java
M fe/src/test/resources/users.ldif
14 files changed, 905 insertions(+), 336 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/17047/4
--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 4
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Csaba Ringhofer (Code Review)

Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..


Patch Set 3: Code-Review+1

(1 comment)

I only found a small issue, can upgrade to +2 once it is resolved.

http://gerrit.cloudera.org:8080/#/c/17047/3/be/src/util/ldap-search-bind.cc
File be/src/util/ldap-search-bind.cc:

http://gerrit.cloudera.org:8080/#/c/17047/3/be/src/util/ldap-search-bind.cc@112
PS3, Line 112:   VLOG(2) << "LDAP bind successful"
We write successful even if success is false.



--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Thu, 18 Feb 2021 17:09:31 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables

2021-02-18 Thread Csaba Ringhofer (Code Review)

Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17081 )

Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write 
id for ACID tables
..


Patch Set 1: Code-Review+1

(3 comments)

I am ok with raising this to +2 if this is critical, but I would prefer to 
think about this topic more deeply.

http://gerrit.cloudera.org:8080/#/c/17081/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17081/1//COMMIT_MSG@9
PS1, Line 9: should bump the write id
Don't we also need a lock in this case? For example currently this can happen 
at the same time as a TRUNCATE.


http://gerrit.cloudera.org:8080/#/c/17081/1//COMMIT_MSG@9
PS1, Line 9: ADD PARTITION
Don't we need to do the same in DROP PARTITION?


http://gerrit.cloudera.org:8080/#/c/17081/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/17081/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3188
PS1, Line 3188: allocateTableWriteId
What happens if allocateTableWriteId throws a TransactionException?



--
To view, visit http://gerrit.cloudera.org:8080/17081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd
Gerrit-Change-Number: 17081
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 16:16:20 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10325: Parquet scan should use min/max statistics to skip pages based on equi-join predicate

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/16720 )

Change subject: IMPALA-10325: Parquet scan should use min/max statistics to 
skip pages based on equi-join predicate
..

IMPALA-10325: Parquet scan should use min/max statistics to skip pages based on 
equi-join predicate

This patch adds a new class of predicates called overlap predicates
to aid in the acceptance or rejection of a row group, a page, or a
row in a Parquet table, utilizing the minimal and the maximal values
gathered from an equi hash join and the Parquet column index stats.
When a row group or page is rejected, all contained rows within are
rejected all together.

For example in the following query, the min and max in the overlap
predicate are computed from the join column from table 'b', and
are compared against the min/max of each row group or page at the
scan node for 'a'.

  select straight_join count(*)
  from lineitem a join [SHUFFLE] lineitem b
  where a.l_shipdate = b.l_receiptdate
  and b.l_commitdate = "1992-01-31";

An overlap predicate associated with the column type B in hash
table and scan column type A will be formed when both A and B are
of or can be converted to as:
  1. booleans;
  2. integers (tinyint, smallint, int, or bigint);
  3. approximate numeric (float or double);
  4. decimals with the same precision and scale;
  5. strings;
  6. date; or
  7. timestamps.

The overlap predicate is implemented as a min/max filter and can be
observed in the explain output of a query.

A new query option 'minmax_filter_threshold' is provided to control
the new feature. Setting it to 0.0 disables the feature. Setting it
to a value > 0.0 but less than 1.0 provides a threshold. An overlap
predicate will be evaluated against a row group and possibly the
containing pages/rows, as long as its overlap ratio is less than the
threshold. The overlap ratio is the common area of the row group
and the filter, divided by the area of the row group.

A second query option, minmax_filtering_level, is provided to
specify the filtering scope:
  1. ROW_GROUP: the overlap is only tested for row groups;
  2. PAGE: the overlap is tested for both row groups and pages;
  3. ROW: the overlap is for row groups, pages and rows.

Two new run-time profile counters are added to report the number of
row groups or pages filtered out via the overlap predicates
respectively:
  1. NumRuntimeFilteredRowGroups
  2. NumRuntimeFilteredPages

Two new column "Min value" and "Max value" are added to the
"Filter routing table" and "Final filter table" in profile to
display the min and the max values for a min/max filter.

Testing:
1. Unit tested on various column types with TPCH and TPCDS tables.
   Benefits were significant when the join column on the outer table
   is sorted and there exist many row groups or pages no overlapping
   with the min/max filters;
2. Added following new tests:
a) In overlap_min_max_filters.test to demonstrate the number of
   filtered out pages and row groups with the two new profile
   counters;
b) In runtime-filter-propagation.test to demonstrate that the
   overlap predicates work with different column types;
3. Core testing;
4. Performance measurement: the overal improvement with 3TB scale
   TPCDS is at 1.45% with the filter threshold at 0.5 and filtering
   level at ROW_GROUP. Good improvement (over 10%) are seen with
   query 16, 25, 62, 83, 94 and 99, due to the join column
   ship_date_sk being strongly correlated to the partition column
   sold_date_sk.

To do in follow-up JIRAs:
1. Improve filtering efficiency;
2. Apply the overlap predicate on partition columns;
3. IR code-gen for various MinMaxFilter::EvalOverlap methods.
4. Address the current limitation that the "Min value" and
   "Max value" columns may be empty for LOCAL filters.

Change-Id: I379405ee75b14929df7d6b5d20dabc6f51375691
Reviewed-on: http://gerrit.cloudera.org:8080/16720
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M be/src/exec/exec-node.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scanner-ir.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.h
M be/src/exec/parquet/parquet-column-stats.cc
M be/src/exec/parquet/parquet-column-stats.h
M be/src/exec/partitioned-hash-join-builder.cc
M be/src/exec/scan-node.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/date-value.cc
M be/src/runtime/date-value.h
M be/src/runtime/raw-value.h
M be/src/runtime/runtime-filter-ir.cc
M be/src/runtime/string-value-test.cc
M be/src/runtime/string-value.cc
M be/src/runtime/string-value.h
M be/src/runtime/timestamp-value.cc
M be/src/runtime/timestamp-value.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/debug-util.cc
M

[Impala-ASF-CR] IMPALA-10325: Parquet scan should use min/max statistics to skip pages based on equi-join predicate

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16720 )

Change subject: IMPALA-10325: Parquet scan should use min/max statistics to 
skip pages based on equi-join predicate
..


Patch Set 71: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/16720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I379405ee75b14929df7d6b5d20dabc6f51375691
Gerrit-Change-Number: 16720
Gerrit-PatchSet: 71
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Thu, 18 Feb 2021 14:31:50 +
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max filters

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17075 )

Change subject: [WIP] IMPALA-10494: Making use of the min/max column stats to 
improve min/max filters
..


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8158/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/17075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
Gerrit-Change-Number: 17075
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 14:25:07 +
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max filters

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17075 )

Change subject: [WIP] IMPALA-10494: Making use of the min/max column stats to 
improve min/max filters
..


Patch Set 2:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/17075/2/be/src/exec/incr-stats-util.cc
File be/src/exec/incr-stats-util.cc:

http://gerrit.cloudera.org:8080/#/c/17075/2/be/src/exec/incr-stats-util.cc@123
PS2, Line 123:   low_value.bool_val &= value.bool_val;
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17075/2/be/src/exec/incr-stats-util.cc@150
PS2, Line 150:   high_value.bool_val ^= value.bool_val;
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17075/2/fe/src/main/java/org/apache/impala/catalog/ColumnStats.java
File fe/src/main/java/org/apache/impala/catalog/ColumnStats.java:

http://gerrit.cloudera.org:8080/#/c/17075/2/fe/src/main/java/org/apache/impala/catalog/ColumnStats.java@293
PS2, Line 293:   Preconditions.checkState(false, "Unsupported type 
encountered in setLowValue()");
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17075/2/fe/src/main/java/org/apache/impala/catalog/ColumnStats.java@318
PS2, Line 318:   Preconditions.checkState(false, "Unsupported type 
encountered in setLowValue()");
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17075/2/fe/src/main/java/org/apache/impala/catalog/ColumnStats.java@350
PS2, Line 350: LOG.error("ColumnStats::update() with Hive 
ColumnStatisticsData!" + ", coltype=" + colType.toSql());
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17075/2/fe/src/main/java/org/apache/impala/catalog/ColumnStats.java@354
PS2, Line 354: /// called.
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17075/2/fe/src/main/java/org/apache/impala/catalog/ColumnStats.java@355
PS2, Line 355: ///
line has trailing whitespace



--
To view, visit http://gerrit.cloudera.org:8080/17075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
Gerrit-Change-Number: 17075
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 14:15:15 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] [WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max filters

2021-02-18 Thread Qifan Chen (Code Review)

Qifan Chen has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/17075


Change subject: [WIP] IMPALA-10494: Making use of the min/max column stats to 
improve min/max filters
..

[WIP] IMPALA-10494: Making use of the min/max column stats to improve min/max 
filters

This patch adds the functionality to compute the minimal and
the maximal value for a column during column statistics computation
and to store the stats in HMS. These two new stats are used by the
back end to discard min/max filters whose coverage are too close to
the actual range.

Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/incr-stats-util.cc
M be/src/exec/incr-stats-util.h
M be/src/service/hs2-util.cc
M be/src/service/hs2-util.h
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
10 files changed, 391 insertions(+), 20 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/17075/2
--
To view, visit http://gerrit.cloudera.org:8080/17075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df
Gerrit-Change-Number: 17075
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen

[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17081 )

Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write 
id for ACID tables
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8157/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd
Gerrit-Change-Number: 17081
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 13:14:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables

2021-02-18 Thread Zoltan Borok-Nagy (Code Review)

Zoltan Borok-Nagy has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/17081


Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write 
id for ACID tables
..

IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables

ALTER TABLE ADD PARTITION should bump the write id for ACID tables.
Both for INSERT-only and full ACID tables.

Testing:
 * added e2e test

Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/query_test/test_acid.py
2 files changed, 43 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/17081/1
--
To view, visit http://gerrit.cloudera.org:8080/17081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd
Gerrit-Change-Number: 17081
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8156/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Thu, 18 Feb 2021 09:52:42 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Tamas Mate (Code Review)

Tamas Mate has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..


Patch Set 3:

Patch set 3 resolves the merge conflict.
Also, added a small refactor, removed the use_ldap variable and changed it to 
the FLAGS_enable_ldap_auth in AuthManager::Init().


--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Thu, 18 Feb 2021 09:36:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10161: User LDAP Search bind support

2021-02-18 Thread Tamas Mate (Code Review)

Tamas Mate has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/17047 )

Change subject: IMPALA-10161: User LDAP Search bind support
..

IMPALA-10161: User LDAP Search bind support

This change adds user search bind support next to simple bind that can
be configured with LDAP filters. The group check was done with LDAP
search earlier, this change adds the possibility to configure it with
Hadoop library like options, which is the LDAP filter with optional
patterns. The '{0}' will be replaced with the user name while the
'{1}' pattern will be replaced with the user dn.

The following new flags have been added:
 --ldap_search_bind_authentication: a flag to change between simple and
search bind
 --ldap_user_search_basedn: the base dn for the LDAP subtree to search
 --ldap_group_search_basedn: the base dn for the LDAP subtree to search

Tested:
 - Custom cluster tests have been added

Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
---
M be/src/rpc/authentication.cc
M be/src/util/CMakeLists.txt
A be/src/util/ldap-search-bind.cc
A be/src/util/ldap-search-bind.h
A be/src/util/ldap-simple-bind.cc
A be/src/util/ldap-simple-bind.h
M be/src/util/ldap-util.cc
M be/src/util/ldap-util.h
M be/src/util/webserver.cc
M fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java
A 
fe/src/test/java/org/apache/impala/customcluster/LdapSearchBindImpalaShellTest.java
A 
fe/src/test/java/org/apache/impala/customcluster/LdapSimpleBindImpalaShellTest.java
M fe/src/test/java/org/apache/impala/testutil/LdapUtil.java
M fe/src/test/resources/users.ldif
14 files changed, 903 insertions(+), 336 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/17047/3
--
To view, visit http://gerrit.cloudera.org:8080/17047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I978744ad05d9ef408328d1e4dd2d18c329f4d3b7
Gerrit-Change-Number: 17047
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Thomas Tauber-Marshall

[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17077 )

Change subject: IMPALA-10509: Add tool to visualize Impala query plan from 
profile
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: David Rorke 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Thu, 18 Feb 2021 08:49:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/17077 )

Change subject: IMPALA-10509: Add tool to visualize Impala query plan from 
profile
..

IMPALA-10509: Add tool to visualize Impala query plan from profile

Impala query plan can be huge and complex. It becomes hard to analyze
this complex query plan, its execution summary, and the runtime filter
interactions from the query profile. We need a tool to help visualize
this query plan from a plain text query profile.

This commit adds experimental python3 script plan-graph.py to parse the
text profile into a GraphViz DOT format. The output of this script can
then be converted into an image using dot program or immediately
visualized using program such as ZGRViewer.

The script can produce a graph in the following verbosity levels:
* 0: minimal graph, containing the concise query plan, time spent on
 each plan node, and the number of rows produced.
* 1: level 0 graph plus additional runtime filter vertices.
* 2: level 1 graph plus additional description in query plan vertices
 such as the plan node parameter, selective predicates, group by
 clause, runtime filter info, etc. The runtime filter vertices are
 also arranged to a lower position than scan nodes that should
 be finished before the runtime filter is published. This is
 currently the default verbosity level.
* 3: level 2 graph, but with the scanner to runtime filter dependency
 edges explicitly drawn.

Testing:
- Manually test the script against query profiles from TPC-DS runs and
  verify that it produces a valid graph visualization.

Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Reviewed-on: http://gerrit.cloudera.org:8080/17077
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
A bin/diagnostics/experimental/plan-graph.py
1 file changed, 633 insertions(+), 0 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: David Rorke 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto

[Impala-ASF-CR] IMPALA-10325: Parquet scan should use min/max statistics to skip pages based on equi-join predicate

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16720 )

Change subject: IMPALA-10325: Parquet scan should use min/max statistics to 
skip pages based on equi-join predicate
..


Patch Set 71:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6897/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I379405ee75b14929df7d6b5d20dabc6f51375691
Gerrit-Change-Number: 16720
Gerrit-PatchSet: 71
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Thu, 18 Feb 2021 08:46:38 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10325: Parquet scan should use min/max statistics to skip pages based on equi-join predicate

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16720 )

Change subject: IMPALA-10325: Parquet scan should use min/max statistics to 
skip pages based on equi-join predicate
..


Patch Set 71: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I379405ee75b14929df7d6b5d20dabc6f51375691
Gerrit-Change-Number: 16720
Gerrit-PatchSet: 71
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Thu, 18 Feb 2021 08:46:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10325: Parquet scan should use min/max statistics to skip pages based on equi-join predicate

2021-02-18 Thread Zoltan Borok-Nagy (Code Review)

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16720 )

Change subject: IMPALA-10325: Parquet scan should use min/max statistics to 
skip pages based on equi-join predicate
..


Patch Set 70: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I379405ee75b14929df7d6b5d20dabc6f51375691
Gerrit-Change-Number: 16720
Gerrit-PatchSet: 70
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Thu, 18 Feb 2021 08:46:19 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-xxxxx: [WIP] Refactoring to support external frontend.

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17080 )

Change subject: IMPALA-x: [WIP] Refactoring to support external frontend.
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8155/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17080
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70035ec082be09a9583275191248d085414b27dd
Gerrit-Change-Number: 17080
Gerrit-PatchSet: 2
Gerrit-Owner: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 08:26:15 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-xxxxx: [WIP] Refactoring to support external frontend.

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17080 )

Change subject: IMPALA-x: [WIP] Refactoring to support external frontend.
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8154/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17080
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70035ec082be09a9583275191248d085414b27dd
Gerrit-Change-Number: 17080
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 08:12:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-xxxxx: [WIP] Refactoring to support external frontend.

2021-02-18 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17080 )

Change subject: IMPALA-x: [WIP] Refactoring to support external frontend.
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17080/2/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:

http://gerrit.cloudera.org:8080/#/c/17080/2/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@391
PS2, Line 391: root.getOutputSmap(), null, limit, analyticNodeSort, 
planCtx.getRootAnalyzer());
line too long (92 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/17080
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70035ec082be09a9583275191248d085414b27dd
Gerrit-Change-Number: 17080
Gerrit-PatchSet: 2
Gerrit-Owner: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 18 Feb 2021 08:06:26 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-xxxxx: [WIP] Refactoring to support external frontend.

2021-02-18 Thread Aman Sinha (Code Review)

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17080

to look at the new patch set (#2).

Change subject: IMPALA-x: [WIP] Refactoring to support external frontend.
..

IMPALA-x: [WIP] Refactoring to support external frontend.

This patch does some minimal refactoring of the frontend
code to allow an external frontend to leverage certain apis
and class variables. A couple of new methods are also added.
The changes include:
 - modifying a few class methods to be public static
 - changing visibility of a few private member variables to
   protected for use by a derived class
 - adding an alternate PlannerContext constructor
 - passing an additional initialization parameter to BackendConfig
 - minimal refactor of selectivity calculation for predicates
 - refactor of count(*) optimization methods
 - small changes to the validatePlan and useParallelPlan
 - new clone method for functions and one for null literals

Testing:
 - Ran PlannerTest
 - Since these are refactoring changes, no new tests are added in
   this patch but once other supporting changes are done, I
   anticipate adding sanity tests for external FE in a future commit.

Change-Id: I70035ec082be09a9583275191248d085414b27dd
---
M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
20 files changed, 125 insertions(+), 59 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/17080/2
--
To view, visit http://gerrit.cloudera.org:8080/17080
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I70035ec082be09a9583275191248d085414b27dd
Gerrit-Change-Number: 17080
Gerrit-PatchSet: 2
Gerrit-Owner: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins

65 matches

Mail list logo