date:20191126

Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14641 )

Change subject: IMPALA-8138: Reintroduce rpc debugging options
..


Patch Set 7: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/14641
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9c047ebce6d32c5ae461f70279391fa2df4c2029
Gerrit-Change-Number: 14641
Gerrit-PatchSet: 7
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:55:58 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options

Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14641 )

Change subject: IMPALA-8138: Reintroduce rpc debugging options
..


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/14641/7/be/src/util/debug-util.h
File be/src/util/debug-util.h:

http://gerrit.cloudera.org:8080/#/c/14641/7/be/src/util/debug-util.h@146
PS7, Line 146: std::vector args)
const-reference



--
To view, visit http://gerrit.cloudera.org:8080/14641
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9c047ebce6d32c5ae461f70279391fa2df4c2029
Gerrit-Change-Number: 14641
Gerrit-PatchSet: 5
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:55:44 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4080: [Part 1] Move static state from ExecNode into a PlanNode

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14764 )

Change subject: IMPALA-4080: [Part 1] Move static state from ExecNode into a 
PlanNode
..


Patch Set 5:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/5154/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/14764
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472
Gerrit-Change-Number: 14764
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:40:49 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with python 2.7.9+

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14802 )

Change subject: IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with 
python 2.7.9+
..


Patch Set 2: Verified+1

Build Successful

https://jenkins.impala.io/job/gerrit-docs-auto-test/541/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/14802
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I681bb7d9900f00402259758d579f396da9cd007d
Gerrit-Change-Number: 14802
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:36:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with python 2.7.9+

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14802 )

Change subject: IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with 
python 2.7.9+
..


Patch Set 1: Verified+1

Build Successful

https://jenkins.impala.io/job/gerrit-docs-auto-test/540/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/14802
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I681bb7d9900f00402259758d579f396da9cd007d
Gerrit-Change-Number: 14802
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:34:23 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9092: Add support for creating external Kudu table

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14750 )

Change subject: IMPALA-9092: Add support for creating external Kudu table
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5153/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14750
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I76f81d41db0cf2269ee1b365857164a43677e14d
Gerrit-Change-Number: 14750
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Andrew Wong 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:22:07 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4080: [Part 1] Move static state from ExecNode into a PlanNode

2019-11-26 Thread Bikramjeet Vig (Code Review)

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14764 )

Change subject: IMPALA-4080: [Part 1] Move static state from ExecNode into a 
PlanNode
..


Patch Set 5:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/aggregation-node-base.h
File be/src/exec/aggregation-node-base.h:

http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/aggregation-node-base.h@50
PS4, Line 50:   /// If true, the input to this node should be passed into each 
Aggregator in 'aggs_'.
> line too long (109 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/14764/3/be/src/exec/aggregation-node-base.h
File be/src/exec/aggregation-node-base.h:

http://gerrit.cloudera.org:8080/#/c/14764/3/be/src/exec/aggregation-node-base.h@43
PS3, Line 43: const PlanNode& pnode
> This is going to be an annoying nitpick, but I think it should be a const r
Done


http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:

http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/exec-node.h@356
PS4, Line 356:   /// state is move there and accessed directly.
> line too long (93 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/exec-node.cc
File be/src/exec/exec-node.cc:

http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/exec-node.cc@217
PS4, Line 217: row_descriptor_(*(pnode.row_descriptor_)),
> line too long (160 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/partitioned-hash-join-node.cc
File be/src/exec/partitioned-hash-join-node.cc:

http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/partitioned-hash-join-node.cc@82
PS4, Line 82: Status PartitionedHashJoinPlanNode::CreateNode(
> line too long (92 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:

http://gerrit.cloudera.org:8080/#/c/14764/4/be/src/exec/union-node.cc@73
PS4, Line 73: 
first_materialized_child_idx_(pnode.tnode_->union_node.first_materialized_child_idx),
> line too long (92 > 90)
Done



--
To view, visit http://gerrit.cloudera.org:8080/14764
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472
Gerrit-Change-Number: 14764
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:21:43 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4080: [Part 1] Move static state from ExecNode into a PlanNode

2019-11-26 Thread Bikramjeet Vig (Code Review)

Hello Michael Ho, Daniel Becker, Tim Armstrong, Csaba Ringhofer, Impala Public 
Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/14764

to look at the new patch set (#5).

Change subject: IMPALA-4080: [Part 1] Move static state from ExecNode into a 
PlanNode
..

IMPALA-4080: [Part 1] Move static state from ExecNode into a PlanNode

This patch adds a new class called PlanNode which contains a subset
of the static state  of their corresponding ExecNode, of which there
is one instance per fragment. ExecNode contains the runtime state
and there can be up to MT_DOP instances of it per fragment.

It also adds a similar class called AggregatorConfig which contains
static state initialized from the thrift aggregator struct and is
passed as an input to the Aggregator class's constructor.

Eventually all static state including codegened function pointers
would be moved to the PlanNodes.

Testing:
TODO: run exhaustive tests

Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472
---
M be/src/exec/aggregation-node-base.cc
M be/src/exec/aggregation-node-base.h
M be/src/exec/aggregation-node.cc
M be/src/exec/aggregation-node.h
M be/src/exec/aggregator.cc
M be/src/exec/aggregator.h
M be/src/exec/analytic-eval-node.cc
M be/src/exec/analytic-eval-node.h
M be/src/exec/blocking-join-node.cc
M be/src/exec/blocking-join-node.h
M be/src/exec/cardinality-check-node.cc
M be/src/exec/cardinality-check-node.h
M be/src/exec/data-source-scan-node.cc
M be/src/exec/data-source-scan-node.h
M be/src/exec/empty-set-node.cc
M be/src/exec/empty-set-node.h
M be/src/exec/exchange-node.cc
M be/src/exec/exchange-node.h
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/exec/grouping-aggregator.cc
M be/src/exec/grouping-aggregator.h
M be/src/exec/hbase-scan-node.cc
M be/src/exec/hbase-scan-node.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node-mt.h
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scan-node.h
M be/src/exec/kudu-scan-node-base.cc
M be/src/exec/kudu-scan-node-base.h
M be/src/exec/kudu-scan-node-mt.cc
M be/src/exec/kudu-scan-node-mt.h
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-scan-node.h
M be/src/exec/nested-loop-join-node.cc
M be/src/exec/nested-loop-join-node.h
M be/src/exec/non-grouping-aggregator.cc
M be/src/exec/non-grouping-aggregator.h
M be/src/exec/partial-sort-node.cc
M be/src/exec/partial-sort-node.h
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M be/src/exec/scan-node.cc
M be/src/exec/scan-node.h
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M be/src/exec/singular-row-src-node.cc
M be/src/exec/singular-row-src-node.h
M be/src/exec/sort-node.cc
M be/src/exec/sort-node.h
M be/src/exec/streaming-aggregation-node.cc
M be/src/exec/streaming-aggregation-node.h
M be/src/exec/subplan-node.cc
M be/src/exec/subplan-node.h
M be/src/exec/topn-node.cc
M be/src/exec/topn-node.h
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M be/src/exec/unnest-node.cc
M be/src/exec/unnest-node.h
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/fragment-instance-state.h
64 files changed, 1,324 insertions(+), 698 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/14764/5
--
To view, visit http://gerrit.cloudera.org:8080/14764
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472
Gerrit-Change-Number: 14764
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with python 2.7.9+

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/14802

to look at the new patch set (#2).

Change subject: IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with 
python 2.7.9+
..

IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with python 2.7.9+

Change-Id: I681bb7d9900f00402259758d579f396da9cd007d
---
M docs/topics/impala_ssl.xml
1 file changed, 63 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/14802/2
--
To view, visit http://gerrit.cloudera.org:8080/14802
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I681bb7d9900f00402259758d579f396da9cd007d
Gerrit-Change-Number: 14802
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with python 2.7.9+

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14802 )

Change subject: IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with 
python 2.7.9+
..


Patch Set 2:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/541/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/14802
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I681bb7d9900f00402259758d579f396da9cd007d
Gerrit-Change-Number: 14802
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:13:02 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with python 2.7.9+

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14802 )

Change subject: IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with 
python 2.7.9+
..


Patch Set 1:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/540/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/14802
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I681bb7d9900f00402259758d579f396da9cd007d
Gerrit-Change-Number: 14802
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:10:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with python 2.7.9+

Alex Rodoni has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/14802


Change subject: IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with 
python 2.7.9+
..

IMPALA-9198: [DOCS] impala-shell works with tlsv1.2 only with python 2.7.9+

Change-Id: I681bb7d9900f00402259758d579f396da9cd007d
---
M docs/topics/impala_ssl.xml
1 file changed, 31 insertions(+), 40 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/14802/1
--
To view, visit http://gerrit.cloudera.org:8080/14802
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I681bb7d9900f00402259758d579f396da9cd007d
Gerrit-Change-Number: 14802
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni

[Impala-ASF-CR] IMPALA-9181: Serialize TQueryCtx once per query

Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14777 )

Change subject: IMPALA-9181: Serialize TQueryCtx once per query
..


Patch Set 3: Code-Review+2

(2 comments)

http://gerrit.cloudera.org:8080/#/c/14777/3/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:

http://gerrit.cloudera.org:8080/#/c/14777/3/be/src/runtime/coordinator-backend-state.cc@215
PS3, Line 215:   unique_ptr sidecar_buf = 
make_unique();
 :   sidecar_buf->assign_copy(serialized_buf, serialized_len);
 :   unique_ptr rpc_sidecar = 
RpcSidecar::FromFaststring(move(sidecar_buf));
Not this change but this can in theory be converted to sidecar using Slice too. 
That said, we may better hold off from doing so now until we convert 
ExecQueryFInstance() RPC to asynchronous as the memory management is slightly 
different in that case.

May be a TODO for now is sufficient.


http://gerrit.cloudera.org:8080/#/c/14777/3/be/src/service/control-service.cc
File be/src/service/control-service.cc:

http://gerrit.cloudera.org:8080/#/c/14777/3/be/src/service/control-service.cc@120
PS3, Line 120: Status GetSidecar(int sidecar_idx, RpcContext* rpc_context, T* 
thrift_obj) {
static



--
To view, visit http://gerrit.cloudera.org:8080/14777
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
Gerrit-Change-Number: 14777
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Wed, 27 Nov 2019 00:09:52 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9092: Add support for creating external Kudu table

2019-11-26 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has uploaded a new patch set (#6). ( 
http://gerrit.cloudera.org:8080/14750 )

Change subject: IMPALA-9092: Add support for creating external Kudu table
..

IMPALA-9092: Add support for creating external Kudu table

In HMS-3 the translation layer converts a managed kudu table into a external 
kudu table
and adds additional table property 'external.table.purge' to 'true'. This means 
any
installation which is using HMS-3 (or a Hive version which has HIVE-22158) will 
always
create Kudu tables as external tables. This is problematic since the the output 
of show
create table will now be different and may confuse the users.

In order to improve the user experience of such synchronized tables (external 
tables with
external.table.purge property set to true), this patch adds support in Impala 
to create
external Kudu tables. Previous versions of Impala disallowed creating a 
external Kudu
table if the Kudu table did not exist. After this patch, Impala will check if 
the Kudu
table exists and if it does not it will create a Kudu table based on the schema 
provided
in the create table statement. The command will error out if the Kudu table 
already
exists. However, this applies to only the synchronized tables. Previous way to 
create a
pure external table behaves the same.

Following syntax of creating a synchronized table is now allowed:

CREATE EXTERNAL TABLE foo (
  id int PRIMARY KEY,
  name string)
PARTITION BY HASH PARTITIONS 8
STORED AS KUDU
TBLPROPERTIES ('external.table.purge'='true')

The syntax is very similar to creating a managed table, except for the EXTERNAL 
keyword
and additional table property. A synchronized table will behave similar to 
managed Kudu
tables (drops and renames are allowed). The output of show create table on a 
synchronized
table will display the full column and partition spec similar to the managed 
tables.

Testing:
1. After the CDP version bump all of the existing Kudu tables now create 
synchronized
tables so there is good coverage there.
2. Added additional tests which create synchronized tables and compares the 
show create
table output.
3. Ran exhaustive tests with both CDP and CDH builds.

Change-Id: I76f81d41db0cf2269ee1b365857164a43677e14d
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/common/skip.py
M tests/metadata/test_show_create_table.py
M tests/query_test/test_kudu.py
15 files changed, 792 insertions(+), 269 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/14750/6
--
To view, visit http://gerrit.cloudera.org:8080/14750
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I76f81d41db0cf2269ee1b365857164a43677e14d
Gerrit-Change-Number: 14750
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Andrew Wong 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-9092: Add support for creating external Kudu table

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14750 )

Change subject: IMPALA-9092: Add support for creating external Kudu table
..


Patch Set 6:

(33 comments)

http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java:

http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@71
PS6, Line 71: super.AnalysisError(appendSynchronizedTblProps(stmt, 
isExternalPurgeTbl), expectedError);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@102
PS6, Line 102: "range(x, y) (partition value = (1+1, 2+2), 
partition value = ((1+1+1)+1, 10), " +
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@221
PS6, Line 221: "(PARTITION VALUE = 'abc')' is not a key column. 
Only key columns can be used "
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@277
PS6, Line 277: "partition by range(a, b) (partition (0, 0) < values 
<= (1, 1)) stored as kudu",
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@286
PS6, Line 286: "partitioning columns: (1 vs 2). Range partition: 
'PARTITION 0 < VALUES <= 1'",
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
File fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java:

http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@270
PS6, Line 270:   "  id INT NOT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@271
PS6, Line 271:   "  bool_col BOOLEAN NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@272
PS6, Line 272:   "  tinyint_col TINYINT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (102 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@273
PS6, Line 273:   "  smallint_col SMALLINT NULL ENCODING 
AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@274
PS6, Line 274:   "  int_col INT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@275
PS6, Line 275:   "  bigint_col BIGINT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@276
PS6, Line 276:   "  float_col FLOAT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@277
PS6, Line 277:   "  double_col DOUBLE NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@278
PS6, Line 278:   "  date_string_col STRING NULL ENCODING 
AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@279
PS6, Line 279:   "  string_col STRING NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@280
PS6, Line 280:   "  timestamp_col TIMESTAMP NULL ENCODING 
AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@281
PS6, Line 281:   "  year INT NULL ENCODING AUTO_ENCODING 
COMPRESS

[Impala-ASF-CR] IMPALA-9092: Add support for creating external Kudu table

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14750 )

Change subject: IMPALA-9092: Add support for creating external Kudu table
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5152/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14750
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I76f81d41db0cf2269ee1b365857164a43677e14d
Gerrit-Change-Number: 14750
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Andrew Wong 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 26 Nov 2019 23:12:14 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8607: [DOCS] Additional copy edits in impala shell options.xml

Alex Rodoni has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/14800 )

Change subject: IMPALA-8607: [DOCS] Additional copy edits in 
impala_shell_options.xml
..

IMPALA-8607: [DOCS] Additional copy edits in impala_shell_options.xml

Change-Id: If9b74d3833ba5ca19da9eedf4def17eb03e1b03c
Reviewed-on: http://gerrit.cloudera.org:8080/14800
Reviewed-by: Alex Rodoni 
Tested-by: Impala Public Jenkins 
---
M docs/topics/impala_shell_options.xml
1 file changed, 25 insertions(+), 21 deletions(-)

Approvals:
  Alex Rodoni: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/14800
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If9b74d3833ba5ca19da9eedf4def17eb03e1b03c
Gerrit-Change-Number: 14800
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-9181: Serialize TQueryCtx once per query

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14777 )

Change subject: IMPALA-9181: Serialize TQueryCtx once per query
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5151/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14777
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
Gerrit-Change-Number: 14777
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Tue, 26 Nov 2019 22:49:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/14059 )

Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2
..

IMPALA-8863: Add support to run tests over HTTP/HS2

This change adds support to run backend tests over HTTP using a new
version of Impyla (0.16.1). It also adds a test that exercises
authentication over HTTP.

Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018
Reviewed-on: http://gerrit.cloudera.org:8080/14059
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M be/src/exec/blocking-plan-root-sink.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-server.h
M bin/start-impala-cluster.py
M fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java
A fe/src/test/java/org/apache/impala/customcluster/LdapImpylaHttpTest.java
A fe/src/test/java/org/apache/impala/customcluster/RunShellCommand.java
M infra/python/deps/compiled-requirements.txt
M tests/common/impala_connection.py
M tests/common/impala_test_suite.py
M tests/common/test_dimensions.py
M tests/custom_cluster/test_client_ssl.py
M tests/custom_cluster/test_hs2.py
M tests/custom_cluster/test_session_expiration.py
M tests/custom_cluster/test_shell_interactive.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/hs2/test_hs2.py
M tests/hs2/test_json_endpoints.py
M tests/query_test/test_cast_with_format.py
M tests/query_test/test_chars.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
A tests/util/run_impyla_http_query.py
26 files changed, 529 insertions(+), 243 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/14059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018
Gerrit-Change-Number: 14059
Gerrit-PatchSet: 37
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14059 )

Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2
..


Patch Set 36: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/14059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018
Gerrit-Change-Number: 14059
Gerrit-PatchSet: 36
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 26 Nov 2019 22:46:39 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9092: Add support for creating external Kudu table

2019-11-26 Thread Hao Hao (Code Review)

Hao Hao has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14750 )

Change subject: IMPALA-9092: Add support for creating external Kudu table
..


Patch Set 5: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/14750
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I76f81d41db0cf2269ee1b365857164a43677e14d
Gerrit-Change-Number: 14750
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Andrew Wong 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 26 Nov 2019 22:42:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9181: Serialize TQueryCtx once per query

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14777 )

Change subject: IMPALA-9181: Serialize TQueryCtx once per query
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5150/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14777
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
Gerrit-Change-Number: 14777
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Tue, 26 Nov 2019 22:42:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9181: Serialize TQueryCtx once per query

2019-11-26 Thread Sahil Takiar (Code Review)

Sahil Takiar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14777 )

Change subject: IMPALA-9181: Serialize TQueryCtx once per query
..


Patch Set 3: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/14777
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
Gerrit-Change-Number: 14777
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Tue, 26 Nov 2019 22:36:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9181: Serialize TQueryCtx once per query

2019-11-26 Thread Sahil Takiar (Code Review)

Sahil Takiar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14777 )

Change subject: IMPALA-9181: Serialize TQueryCtx once per query
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/14777/1/be/src/runtime/coordinator-backend-state.h
File be/src/runtime/coordinator-backend-state.h:

http://gerrit.cloudera.org:8080/#/c/14777/1/be/src/runtime/coordinator-backend-state.h@85
PS1, Line 85: const kudu::Slice& serialized_query_ctx
> That makes the memory management more complicated, since we don't want the
No, I think it should be fine as is.



--
To view, visit http://gerrit.cloudera.org:8080/14777
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
Gerrit-Change-Number: 14777
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Tue, 26 Nov 2019 22:36:24 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9092: Add support for creating external Kudu table

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14750 )

Change subject: IMPALA-9092: Add support for creating external Kudu table
..


Patch Set 5:

(33 comments)

http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java:

http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@71
PS5, Line 71: super.AnalysisError(appendSynchronizedTblProps(stmt, 
isExternalPurgeTbl), expectedError);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@102
PS5, Line 102: "range(x, y) (partition value = (1+1, 2+2), 
partition value = ((1+1+1)+1, 10), " +
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@221
PS5, Line 221: "(PARTITION VALUE = 'abc')' is not a key column. 
Only key columns can be used "
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@277
PS5, Line 277: "partition by range(a, b) (partition (0, 0) < values 
<= (1, 1)) stored as kudu",
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@286
PS5, Line 286: "partitioning columns: (1 vs 2). Range partition: 
'PARTITION 0 < VALUES <= 1'",
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
File fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java:

http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@270
PS5, Line 270:   "  id INT NOT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@271
PS5, Line 271:   "  bool_col BOOLEAN NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@272
PS5, Line 272:   "  tinyint_col TINYINT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (102 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@273
PS5, Line 273:   "  smallint_col SMALLINT NULL ENCODING 
AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@274
PS5, Line 274:   "  int_col INT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@275
PS5, Line 275:   "  bigint_col BIGINT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@276
PS5, Line 276:   "  float_col FLOAT NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@277
PS5, Line 277:   "  double_col DOUBLE NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@278
PS5, Line 278:   "  date_string_col STRING NULL ENCODING 
AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@279
PS5, Line 279:   "  string_col STRING NULL ENCODING AUTO_ENCODING 
COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@280
PS5, Line 280:   "  timestamp_col TIMESTAMP NULL ENCODING 
AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,\n" +
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/14750/5/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@281
PS5, Line 281:   "  year INT NULL ENCODING AUTO_ENCODING 
COMPRESS

[Impala-ASF-CR] IMPALA-9092: Add support for creating external Kudu table

2019-11-26 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/14750 )

Change subject: IMPALA-9092: Add support for creating external Kudu table
..

IMPALA-9092: Add support for creating external Kudu table

In HMS-3 the translation layer converts a managed kudu table into a external 
kudu table
and adds additional table property 'external.table.purge' to 'true'. This means 
any
installation which is using HMS-3 (or a Hive version which has HIVE-22158) will 
always
create Kudu tables as external tables. This is problematic since the the output 
of show
create table will now be different and may confuse the users.

In order to improve the user experience of such synchronized tables (external 
tables with
external.table.purge property set to true), this patch adds support in Impala 
to create
external Kudu tables. Previous versions of Impala disallowed creating a 
external Kudu
table if the Kudu table did not exist. After this patch, Impala will check if 
the Kudu
table exists and if it does not it will create a Kudu table based on the schema 
provided
in the create table statement. The command will error out if the Kudu table 
already
exists. However, this applies to only the synchronized tables. Previous way to 
create a
pure external table behaves the same.

Following syntax of creating a synchronized table is now allowed:

CREATE EXTERNAL TABLE foo (
  id int PRIMARY KEY,
  name string)
PARTITION BY HASH PARTITIONS 8
STORED AS KUDU
TBLPROPERTIES ('external.table.purge'='true')

The syntax is very similar to creating a managed table, except for the EXTERNAL 
keyword
and additional table property. A synchronized table will behave similar to 
managed Kudu
tables (drops and renames are allowed). The output of show create table on a 
synchronized
table will display the full column and partition spec similar to the managed 
tables.

Testing:
1. After the CDP version bump all of the existing Kudu tables now create 
synchronized
tables so there is good coverage there.
2. Added additional tests which create synchronized tables and compares the 
show create
table output.
3. Ran exhaustive tests with both CDP and CDH builds.

Change-Id: I76f81d41db0cf2269ee1b365857164a43677e14d
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/common/skip.py
M tests/metadata/test_show_create_table.py
M tests/query_test/test_kudu.py
15 files changed, 795 insertions(+), 269 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/14750/5
--
To view, visit http://gerrit.cloudera.org:8080/14750
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I76f81d41db0cf2269ee1b365857164a43677e14d
Gerrit-Change-Number: 14750
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Andrew Wong 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-8607: [DOCS] Additional copy edits in impala shell options.xml

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14800 )

Change subject: IMPALA-8607: [DOCS] Additional copy edits in 
impala_shell_options.xml
..


Patch Set 1: Verified+1

Build Successful 

https://jenkins.impala.io/job/gerrit-docs-auto-test/539/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/14800
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If9b74d3833ba5ca19da9eedf4def17eb03e1b03c
Gerrit-Change-Number: 14800
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 26 Nov 2019 22:15:39 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9181: Serialize TQueryCtx once per query

2019-11-26 Thread Thomas Tauber-Marshall (Code Review)

Hello Michael Ho, Sahil Takiar, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/14777

to look at the new patch set (#3).

Change subject: IMPALA-9181: Serialize TQueryCtx once per query
..

IMPALA-9181: Serialize TQueryCtx once per query

When issuing Exec() rpcs to backends, we currently serialize the
TQueryCtx once per backend. This is inefficient as the TQueryCtx is
the same for all backends and really only needs to be serialized once.

Serializing the TQueryCtx can be expensive as it contains both the
full text of the original query and the descriptor table, which can be
quite large. In a synthetic dataset I tested with, scanning a table
with 100k partitions leads to a descriptor table size of ~20MB.

This patch serializes the TQueryCtx in the coordinator and then passes
it to each BackendState when calling Exec().

Followup work might consider if we really need all of the info in the
TQueryCtx to be distributed to all backends.

Testing:
- Passed full run of existing tests.
- Single node perf run showed no significant change.

Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-exec-mgr.cc
M be/src/runtime/query-exec-mgr.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/test-env.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M common/protobuf/control_service.proto
M common/thrift/ImpalaInternalService.thrift
13 files changed, 138 insertions(+), 108 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/14777/3
--
To view, visit http://gerrit.cloudera.org:8080/14777
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
Gerrit-Change-Number: 14777
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall

[Impala-ASF-CR] 1;10;0cIMPALA-9181: Serialize TQueryCtx once per query

2019-11-26 Thread Thomas Tauber-Marshall (Code Review)

Hello Michael Ho, Sahil Takiar, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/14777

to look at the new patch set (#2).

Change subject: 1;10;0cIMPALA-9181: Serialize TQueryCtx once per query
..

1;10;0cIMPALA-9181: Serialize TQueryCtx once per query

When issuing Exec() rpcs to backends, we currently serialize the
TQueryCtx once per backend. This is inefficient as the TQueryCtx is
the same for all backends and really only needs to be serialized once.

Serializing the TQueryCtx can be expensive as it contains both the
full text of the original query and the descriptor table, which can be
quite large. In a synthetic dataset I tested with, scanning a table
with 100k partitions leads to a descriptor table size of ~20MB.

This patch serializes the TQueryCtx in the coordinator and then passes
it to each BackendState when calling Exec().

Followup work might consider if we really need all of the info in the
TQueryCtx to be distributed to all backends.

Testing:
- Passed full run of existing tests.
- Single node perf run showed no significant change.

Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-exec-mgr.cc
M be/src/runtime/query-exec-mgr.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/test-env.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M common/protobuf/control_service.proto
M common/thrift/ImpalaInternalService.thrift
13 files changed, 138 insertions(+), 108 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/14777/2
--
To view, visit http://gerrit.cloudera.org:8080/14777
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
Gerrit-Change-Number: 14777
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar

[Impala-ASF-CR] IMPALA-8607: [DOCS] Additional copy edits in impala shell options.xml

Alex Rodoni has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14800 )

Change subject: IMPALA-8607: [DOCS] Additional copy edits in 
impala_shell_options.xml
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/14800
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If9b74d3833ba5ca19da9eedf4def17eb03e1b03c
Gerrit-Change-Number: 14800
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:59:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] 1;10;0cIMPALA-9181: Serialize TQueryCtx once per query

2019-11-26 Thread Thomas Tauber-Marshall (Code Review)

Thomas Tauber-Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14777 )

Change subject: 1;10;0cIMPALA-9181: Serialize TQueryCtx once per query
..


Patch Set 2:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/14777/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/14777/1//COMMIT_MSG@10
PS1, Line 10: TQueryCtx
> why is serializing this per fragment expensive, is it typically a very larg
Done


http://gerrit.cloudera.org:8080/#/c/14777/1/be/src/runtime/coordinator-backend-state.h
File be/src/runtime/coordinator-backend-state.h:

http://gerrit.cloudera.org:8080/#/c/14777/1/be/src/runtime/coordinator-backend-state.h@85
PS1, Line 85: const kudu::Slice& serialized_query_ctx
> nit: pass via constructor instead? since that is where query_ctx is passed
That makes the memory management more complicated, since we don't want the 
serialized_query_ctx memory to be used except for when Exec() is running.

The only thing we use the query_ctx from the constructor for is to get the 
TQueryOptions, so we could just pass that in instead of the TQueryCtx if you 
think that makes it clearer what's going on.


http://gerrit.cloudera.org:8080/#/c/14777/1/be/src/runtime/query-exec-mgr.cc
File be/src/runtime/query-exec-mgr.cc:

http://gerrit.cloudera.org:8080/#/c/14777/1/be/src/runtime/query-exec-mgr.cc@50
PS1, Line 50:  bool dummy;
:   QueryState* qs =
:   GetOrCreateQueryState(query_ctx, 
request->per_backend_mem_limit(), &dummy);
> not your code, but is it possible for the QueryState to already exist, or c
Yes, its possible for it to already exist if this is the coordinator, as the 
coordinator calls CreateQueryState() in Coordinator::Exec().


http://gerrit.cloudera.org:8080/#/c/14777/1/be/src/service/control-service.cc
File be/src/service/control-service.cc:

http://gerrit.cloudera.org:8080/#/c/14777/1/be/src/service/control-service.cc@118
PS1, Line 118: // 'thrift_obj'.
> nit: add some method code comments
Done


http://gerrit.cloudera.org:8080/#/c/14777/1/be/src/service/control-service.cc@135
PS1, Line 135: e sidecars. The QueryState will mak
> probably could do this if you make query_ctx a unique_ptr and just transfer
Yeah, I think I would prefer to leave this as a followup task rather than 
making this patch more complicated.


http://gerrit.cloudera.org:8080/#/c/14777/1/common/thrift/ImpalaInternalService.thrift
File common/thrift/ImpalaInternalService.thrift:

http://gerrit.cloudera.org:8080/#/c/14777/1/common/thrift/ImpalaInternalService.thrift@658
PS1, Line 658: er than fully coverti
> could you add some more docs to this? it's not that clear to me what this e
Done



--
To view, visit http://gerrit.cloudera.org:8080/14777
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6a4dd302fd5602ec2775492a041ddd51e7d7a6c6
Gerrit-Change-Number: 14777
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Tue, 26 Nov 2019 22:00:00 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8607: [DOCS] Additional copy edits in impala shell options.xml

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14800 )

Change subject: IMPALA-8607: [DOCS] Additional copy edits in 
impala_shell_options.xml
..


Patch Set 1:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/539/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/14800
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If9b74d3833ba5ca19da9eedf4def17eb03e1b03c
Gerrit-Change-Number: 14800
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:52:02 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8607: [DOCS] Additional copy edits in impala shell options.xml

Alex Rodoni has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/14800


Change subject: IMPALA-8607: [DOCS] Additional copy edits in 
impala_shell_options.xml
..

IMPALA-8607: [DOCS] Additional copy edits in impala_shell_options.xml

Change-Id: If9b74d3833ba5ca19da9eedf4def17eb03e1b03c
---
M docs/topics/impala_shell_options.xml
1 file changed, 25 insertions(+), 21 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/14800/1
--
To view, visit http://gerrit.cloudera.org:8080/14800
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If9b74d3833ba5ca19da9eedf4def17eb03e1b03c
Gerrit-Change-Number: 14800
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni

[Impala-ASF-CR] IMPALA-9126: part 2: no hash join probe structures in build

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14716 )

Change subject: IMPALA-9126: part 2: no hash join probe structures in build
..


Patch Set 11:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5149/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14716
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0065f7f44f44f02b7616b1f694178ca42341c42d
Gerrit-Change-Number: 14716
Gerrit-PatchSet: 11
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:47:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9182: Print the socket address of the client closing a session or cancelling a query from the WebUI

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14782 )

Change subject: IMPALA-9182: Print the socket address of the client closing a 
session or cancelling a query from the WebUI
..


Patch Set 7: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/14782
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
Gerrit-Change-Number: 14782
Gerrit-PatchSet: 7
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vincent Tran 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:33:53 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9182: Print the socket address of the client closing a session or cancelling a query from the WebUI

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14782 )

Change subject: IMPALA-9182: Print the socket address of the client closing a 
session or cancelling a query from the WebUI
..


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5301/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/14782
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
Gerrit-Change-Number: 14782
Gerrit-PatchSet: 7
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vincent Tran 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:33:54 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9182: Print the socket address of the client closing a session or cancelling a query from the WebUI

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14782 )

Change subject: IMPALA-9182: Print the socket address of the client closing a 
session or cancelling a query from the WebUI
..


Patch Set 6: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/14782
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
Gerrit-Change-Number: 14782
Gerrit-PatchSet: 6
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vincent Tran 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:33:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9137, IMPALA-9138: Mark failed RPCs as retryable and update blacklist

Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14677 )

Change subject: IMPALA-9137, IMPALA-9138: Mark failed RPCs as retryable and 
update blacklist
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/14677/3/common/thrift/Status.thrift
File common/thrift/Status.thrift:

http://gerrit.cloudera.org:8080/#/c/14677/3/common/thrift/Status.thrift@48
PS3, Line 48: 4: optional TRPCErrorMessage rpc_msg
> yeah, the point about constructing new Status objects from existing ones is
Discussed with Sahil offline about it. Summary below:

- StatusAuxInfo may be a slightly better fit for future extensibility
- Most control services are initiated from the coordinator so not much need for 
elaborate propagation of the aux info
- For the most part, the propagation of aux info only needs to happen for 
fragment instance's execution. In which case. we can consider dumping the aux 
info in RuntimeState or some per-fragment instance state and collect them when 
reporting status.



--
To view, visit http://gerrit.cloudera.org:8080/14677
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I733cca13847fde43c8ea2ae574d3ae04bd06419c
Gerrit-Change-Number: 14677
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:29:08 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7550: Add documentation to profile counters

Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/14776 )

Change subject: IMPALA-7550: Add documentation to profile counters
..

Patch Set 1:

(10 comments)

I like the direction this is going.

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/exec/hdfs-scan-node-base.h
File be/src/exec/hdfs-scan-node-base.h:

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/exec/hdfs-scan-node-base.h@525
PS1, Line 525: /// Disk accessed bitmap
> I am not too sure about the following two counters. Wondering if we should
These are weird cause they're not actually included in the profile directly -
they're used for other purposes. I'd leave them for now.

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/exec/hdfs-scan-node-base.cc@127
PS1, Line 127:
PROFILE_DEFINE_SUMMARY_STATS_COUNTER(ParquetUncompressedBytesReadPerColumn,
STABLE,
> This seems redundant, any advice to modify it will be appreciated.
I'm OK with the redundant text, but it might be reasonable to use a template
string and substitute in "compressed/uncompressed".

I've seen attempts to avoid this by constructing the strings with Substitute(),
e.g. num_io_threads_per_rotational_disk_help_msg in disk-io-mgr.cc. This is OK
if you're just substituting things into a fixed template but gets confusing if
you're piecing together more complex strings..

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/exec/hdfs-scan-node-base.cc@139
PS1, Line 139: PROFILE_DEFINE_COUNTER(DataCacheHitCount, STABLE, HIGH,
TUnit::UNIT,
> Did not find the explanation for the following data cache counters. Might n
Maybe Joe McDonnell can provide some text here?

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/exec/scan-node.cc
File be/src/exec/scan-node.cc:

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/exec/scan-node.cc@82
PS1, Line 82: PROFILE_DEFINE_TIMER(RowBatchQueueGetWaitTime, UNSTABLE, LOW,
"Wall clock time that the "
> Hi, Tim
We could probably define an enum or something like that. I'd be ok with just
including it in the description text, I think.

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/exec/scan-node.cc@95
PS1, Line 95: // PROFILE_BYTES_READ_COUNTER. We can replace them with
PROFILE_BytesRead.name() where
I think doing this makes sense - that would force all counters to be defined in
this consistent way.

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/runtime/coordinator-backend-state.cc@60
PS1, Line 60: // TODO: the counters are inside the impala namespace, so we need
to declare them there,
This approach of converting the namespaces where needed seems fine to me. The
code is inconsistent about whether to use "using namespace impala" or
"namespace impala {" within .cc files. I prefer the second, personally, but
there's no strong reason to do one or the other.

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/util/runtime-profile-counters.h
File be/src/util/runtime-profile-counters.h:

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/util/runtime-profile-counters.h@154
PS1, Line 154: STABLE,
nit: formatting is weird, I'd expect the enum values to only be indented two
spaces.

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/util/runtime-profile-counters.h@164
PS1, Line 164: enum class Significance {
> I add this Significance to measure if the counter is important to a user. M
I think we should probably combine them, because they're pretty correlated -
UNSTABLE and DEBUG would only really make sense as low level counters.

*Maybe* we could split stable into stable_high_level and stable_low_level or
something like that, in which case stable_high_level would be things that are
interesting to users and stable_low_level would be mainly exposed for analysis
by tools.

For me it's important that there are fairly clear criteria for developers to
decide how to classify counters. I think with 3 x 5 options it gets too hard to
decide how to classify each counter.

http://gerrit.cloudera.org:8080/#/c/14776/1/be/src/util/runtime-profile-counters.h@189
PS1, Line 189: // TODO
Are you planning to do this?

http://gerrit.cloudera.org:8080/#/c/14776/1/www/profile_docs.tmpl
File www/profile_docs.tmpl:

http://gerrit.cloudera.org:8080/#/c/14776/1/www/profile_docs.tmpl@1
PS1, Line 1:

[Impala-ASF-CR] IMPALA-9126: part 2: no hash join probe structures in build

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14716 )

Change subject: IMPALA-9126: part 2: no hash join probe structures in build
..


Patch Set 12: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/14716
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0065f7f44f44f02b7616b1f694178ca42341c42d
Gerrit-Change-Number: 14716
Gerrit-PatchSet: 12
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:04:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9126: part 2: no hash join probe structures in build

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14716 )

Change subject: IMPALA-9126: part 2: no hash join probe structures in build
..


Patch Set 12:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5300/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/14716
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0065f7f44f44f02b7616b1f694178ca42341c42d
Gerrit-Change-Number: 14716
Gerrit-PatchSet: 12
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:04:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9126: part 2: no hash join probe structures in build

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14716 )

Change subject: IMPALA-9126: part 2: no hash join probe structures in build
..


Patch Set 11: Code-Review+2

(2 comments)

carry

http://gerrit.cloudera.org:8080/#/c/14716/10/be/src/exec/partitioned-hash-join-builder.h
File be/src/exec/partitioned-hash-join-builder.h:

http://gerrit.cloudera.org:8080/#/c/14716/10/be/src/exec/partitioned-hash-join-builder.h@91
PS10, Line 91:  RuntimeState* state,
 :   BufferPool::ClientH
> nit: formatting (this can be moved up to the above line)
Done


http://gerrit.cloudera.org:8080/#/c/14716/10/be/src/runtime/bufferpool/buffer-pool.cc
File be/src/runtime/bufferpool/buffer-pool.cc:

http://gerrit.cloudera.org:8080/#/c/14716/10/be/src/runtime/bufferpool/buffer-pool.cc@383
PS10, Line 383:
> Maybe DCHECK(tracker_.get() == nullptr) to ensure that Close() was called i
Done



--
To view, visit http://gerrit.cloudera.org:8080/14716
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0065f7f44f44f02b7616b1f694178ca42341c42d
Gerrit-Change-Number: 14716
Gerrit-PatchSet: 11
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 26 Nov 2019 21:03:51 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9126: part 2: no hash join probe structures in build

Hello Michael Ho, Thomas Tauber-Marshall, Csaba Ringhofer, Impala Public 
Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/14716

to look at the new patch set (#11).

Change subject: IMPALA-9126: part 2: no hash join probe structures in build
..

IMPALA-9126: part 2: no hash join probe structures in build

This is actually independent of part 1, and can
be merged ahead of it if needed.

This cleans a up a bit of tech debt, where the hash
join builder allocated probe-side streams. This was
implemented before we had reliable memory reservations.
Now we can simply transfer reservation.

The reason things are this way is because the separation
of PhjBuilder from PartitionedHashJoinNode (IMPALA-3567)
happened before we switched to the new BufferPool
(IMPALA-4674). It wasn't possible to reliably
transfer reservations, instead the workaround of
allocating and transferring probe streams was
necessary.

After this change, PartitionedHashJoinBuilder does
not explicitly touch any probe-side data structures.
There is still some implicit sharing of things like
the buffer pool client, which is expected as long
as the builder belongs to the ExecNode.

Testing:
Ran exhaustive tests. We should already have adequate coverage for
spilling and non-spilling hash joins.

Change-Id: I0065f7f44f44f02b7616b1f694178ca42341c42d
---
M be/src/exec/partitioned-hash-join-builder.cc
M be/src/exec/partitioned-hash-join-builder.h
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
6 files changed, 189 insertions(+), 184 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/14716/11
--
To view, visit http://gerrit.cloudera.org:8080/14716
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0065f7f44f44f02b7616b1f694178ca42341c42d
Gerrit-Change-Number: 14716
Gerrit-PatchSet: 11
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-9101: Add support for detecting self-events on partition events

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14799 )

Change subject: IMPALA-9101: Add support for detecting self-events on partition 
events
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5148/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14799
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b4148f6be0f9f946c8ad8f314d64b095731744c
Gerrit-Change-Number: 14799
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Tue, 26 Nov 2019 20:51:49 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9101: Add support for detecting self-events on partition events

2019-11-26 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/14799


Change subject: IMPALA-9101: Add support for detecting self-events on partition 
events
..

IMPALA-9101: Add support for detecting self-events on partition events

This commit redoes some of the self-event detection logic, specifically for the 
partition
events. Before the patch, the self-event identifiers for a partition were 
stored at a
table level when generating the partition events. This was problematic since 
unlike
ADD_PARTITION and DROP_PARTITION event, ALTER_PARTITION event is generated one 
per
partition. Due to this if there are multiple ALTER_PARTITION events generated, 
only the
first event is identified as a self-event and the reset of the events are 
processed. This
patch fixes this by adding the self-event identifiers to each partition so that 
when the
event is later received, each ALTER_PARTITION uses the state stored in 
HdfsPartition to
evaluate the self-events. The patch makes sure that the event processor takes a 
table lock
during self-event evaluation to avoid races with other parts of the code which 
try to
modify the table at the same time.

Additionally, this patch also changes the event processor to refresh a loaded 
table
(incomplete tables are not refreshed) when a ALTER_TABLE event is received 
instead of
invalidating the table. This makes the events processor consistent with respect 
to all the
other event types. In future, we should add a flag to choose the behavior 
preference
(prefer invalidate or refresh).

Also, this patch fixes the following related issues:
1. Self-event logic was not triggered for alter database events when user 
modifies the
comment on the database.
2. In case of queries like "alter table add if not exists partition...", the 
partition is
not added since its pre-existing. The self-event identifiers should not be 
added in such
cases since no event is expected from such queries.
3. Changed wait_for_event_processing test util method in EventProcessorUtils to 
use a more
deterministic way to determine if the catalog updates have propogated to 
impalad instead
of waiting for a random duration of time. This also speeds up the event 
processing tests
significantly.

Testing Done:
1. Added a e2e self-events test which runs multiple impala queries and makes 
sure that the
event is skips processing.
2. Ran MetastoreEventsProcessorTest
3. [TODO] Run core tests on CDH and CDP builds.

Change-Id: I9b4148f6be0f9f946c8ad8f314d64b095731744c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/SelfEventContext.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_event_processing.py
M tests/util/event_processor_utils.py
13 files changed, 983 insertions(+), 593 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/14799/2
--
To view, visit http://gerrit.cloudera.org:8080/14799
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9b4148f6be0f9f946c8ad8f314d64b095731744c
Gerrit-Change-Number: 14799
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Quanlong Huang

[Impala-ASF-CR] IMPALA-9101: Add support for detecting self-events on partition events

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14799 )

Change subject: IMPALA-9101: Add support for detecting self-events on partition 
events
..


Patch Set 2:

(10 comments)

http://gerrit.cloudera.org:8080/#/c/14799/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:

http://gerrit.cloudera.org:8080/#/c/14799/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@892
PS2, Line 892: 
.parseLong(hmsParameters_.get(MetastoreEventPropertyKey.CATALOG_VERSION.getKey(;
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/14799/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/14799/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@765
PS2, Line 765: // now that HMS alter operation has succeeded, add this 
version to list of inflight
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/14799/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3852
PS2, Line 3852:   
addToInflightVersionsOfPartition(msPartition.getParameters(), hdfsPartition);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/14799/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/14799/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@2666
PS2, Line 2666:   private void alterTableComputeStats(String tblName, 
List> partValsList) throws ImpalaException {
line too long (111 > 90)


http://gerrit.cloudera.org:8080/#/c/14799/2/tests/custom_cluster/test_event_processing.py
File tests/custom_cluster/test_event_processing.py:

http://gerrit.cloudera.org:8080/#/c/14799/2/tests/custom_cluster/test_event_processing.py@251
PS2, Line 251: [
flake8: E131 continuation line unaligned for hanging indent


http://gerrit.cloudera.org:8080/#/c/14799/2/tests/custom_cluster/test_event_processing.py@290
PS2, Line 290: .
flake8: E131 continuation line unaligned for hanging indent


http://gerrit.cloudera.org:8080/#/c/14799/2/tests/custom_cluster/test_event_processing.py@323
PS2, Line 323: d
flake8: E501 line too long (99 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/14799/2/tests/custom_cluster/test_event_processing.py@338
PS2, Line 338: .
flake8: E131 continuation line unaligned for hanging indent


http://gerrit.cloudera.org:8080/#/c/14799/2/tests/custom_cluster/test_event_processing.py@408
PS2, Line 408: 
flake8: W292 no newline at end of file


http://gerrit.cloudera.org:8080/#/c/14799/2/tests/util/event_processor_utils.py
File tests/util/event_processor_utils.py:

http://gerrit.cloudera.org:8080/#/c/14799/2/tests/util/event_processor_utils.py@43
PS2, Line 43: )
flake8: E501 line too long (91 > 90 characters)



--
To view, visit http://gerrit.cloudera.org:8080/14799
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b4148f6be0f9f946c8ad8f314d64b095731744c
Gerrit-Change-Number: 14799
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Tue, 26 Nov 2019 20:09:12 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9126: part 2: no hash join probe structures in build

2019-11-26 Thread Thomas Tauber-Marshall (Code Review)

Thomas Tauber-Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14716 )

Change subject: IMPALA-9126: part 2: no hash join probe structures in build
..


Patch Set 10: Code-Review+2

(2 comments)

http://gerrit.cloudera.org:8080/#/c/14716/10/be/src/exec/partitioned-hash-join-builder.h
File be/src/exec/partitioned-hash-join-builder.h:

http://gerrit.cloudera.org:8080/#/c/14716/10/be/src/exec/partitioned-hash-join-builder.h@91
PS10, Line 91:
 :   RuntimeState* state
nit: formatting (this can be moved up to the above line)


http://gerrit.cloudera.org:8080/#/c/14716/10/be/src/runtime/bufferpool/buffer-pool.cc
File be/src/runtime/bufferpool/buffer-pool.cc:

http://gerrit.cloudera.org:8080/#/c/14716/10/be/src/runtime/bufferpool/buffer-pool.cc@383
PS10, Line 383: {}
Maybe DCHECK(tracker_.get() == nullptr) to ensure that Close() was called if 
Init() was



--
To view, visit http://gerrit.cloudera.org:8080/14716
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0065f7f44f44f02b7616b1f694178ca42341c42d
Gerrit-Change-Number: 14716
Gerrit-PatchSet: 10
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 26 Nov 2019 19:58:32 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9182: Print the socket address of the client closing a session or cancelling a query from the WebUI

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14782 )

Change subject: IMPALA-9182: Print the socket address of the client closing a 
session or cancelling a query from the WebUI
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5147/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14782
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
Gerrit-Change-Number: 14782
Gerrit-PatchSet: 5
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vincent Tran 
Gerrit-Comment-Date: Tue, 26 Nov 2019 19:37:19 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9182: Print the socket address of the client closing a session or cancelling a query from the WebUI

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14782 )

Change subject: IMPALA-9182: Print the socket address of the client closing a 
session or cancelling a query from the WebUI
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5146/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14782
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
Gerrit-Change-Number: 14782
Gerrit-PatchSet: 6
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vincent Tran 
Gerrit-Comment-Date: Tue, 26 Nov 2019 19:36:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6660: Change -0/+0 floating point to compare as equal in hash table

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/14588 )

Change subject: IMPALA-6660: Change -0/+0 floating point to compare as equal in 
hash table
..

IMPALA-6660: Change -0/+0 floating point to compare as equal in hash table

Currently -0.0/+0.0 values are hashed to different values due to
their different binary representation, while -0.0==+0.0 is true in
C++. This caused them to be distinct values in hash maps despite
being treated as equal in comparisons.

This commit fixes the hashing of -0.0/+0.0, thus changing the
behaviour of hash joins and aggregations (since aggregations
follow the behaviour of the join). That way, the canonical form for
-0/+0 is changed to +0.

Tests:
 - Added e2e tests for aggregation (group by and distinct) and
   join queries with -0.0 and +0.0 present.

Change-Id: I6bb1a817c81c452d041238c19cb6c9f602a5d565
Reviewed-on: http://gerrit.cloudera.org:8080/14588
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M be/src/codegen/codegen-anyval.cc
M be/src/codegen/codegen-anyval.h
M be/src/exec/hash-table.cc
M be/src/runtime/raw-value.cc
M be/src/runtime/raw-value.h
M be/src/runtime/raw-value.inline.h
M testdata/workloads/functional-query/queries/QueryTest/aggregation.test
M testdata/workloads/functional-query/queries/QueryTest/joins.test
8 files changed, 100 insertions(+), 14 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/14588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6bb1a817c81c452d041238c19cb6c9f602a5d565
Gerrit-Change-Number: 14588
Gerrit-PatchSet: 11
Gerrit-Owner: Norbert Luksa 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Norbert Luksa 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy

[Impala-ASF-CR] IMPALA-6660: Change -0/+0 floating point to compare as equal in hash table

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14588 )

Change subject: IMPALA-6660: Change -0/+0 floating point to compare as equal in 
hash table
..


Patch Set 10: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/14588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6bb1a817c81c452d041238c19cb6c9f602a5d565
Gerrit-Change-Number: 14588
Gerrit-PatchSet: 10
Gerrit-Owner: Norbert Luksa 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Norbert Luksa 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Tue, 26 Nov 2019 19:14:23 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8607: [DOCS] Document the new global level .impalarc file

Alex Rodoni has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/14779 )

Change subject: IMPALA-8607: [DOCS] Document the new global level .impalarc file
..

IMPALA-8607: [DOCS] Document the new global level .impalarc file

Change-Id: I90195efceca8a03403f9a674f689648a04ab495d
Reviewed-on: http://gerrit.cloudera.org:8080/14779
Reviewed-by: Bikramjeet Vig 
Tested-by: Impala Public Jenkins 
---
M docs/topics/impala_shell_options.xml
1 file changed, 77 insertions(+), 74 deletions(-)

Approvals:
  Bikramjeet Vig: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/14779
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I90195efceca8a03403f9a674f689648a04ab495d
Gerrit-Change-Number: 14779
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Ethan Xue 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-9182: Print the socket address of the client closing a session or cancelling a query from the WebUI

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14782 )

Change subject: IMPALA-9182: Print the socket address of the client closing a 
session or cancelling a query from the WebUI
..


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/14782/5/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:

http://gerrit.cloudera.org:8080/#/c/14782/5/be/src/service/impala-http-handler.cc@239
PS5, Line 239:" $0", req.source_socket));
tab used for whitespace



--
To view, visit http://gerrit.cloudera.org:8080/14782
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
Gerrit-Change-Number: 14782
Gerrit-PatchSet: 5
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vincent Tran 
Gerrit-Comment-Date: Tue, 26 Nov 2019 18:53:20 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9182: Print the socket address of the client closing a session or cancelling a query from the WebUI

2019-11-26 Thread Vincent Tran (Code Review)

Vincent Tran has uploaded a new patch set (#6). ( 
http://gerrit.cloudera.org:8080/14782 )

Change subject: IMPALA-9182: Print the socket address of the client closing a 
session or cancelling a query from the WebUI
..

IMPALA-9182: Print the socket address of the client closing a session or 
cancelling a query from the WebUI

This change appends the socket address (HOST:PORT) of the client
who made the request to close a session or cancel a query from
the coordinator's debug WebUI.

Existing statuses:
"Cancelled from Impala's debug web interface"
"Session closed from Impala's debug web interface"

New statuses:
"Cancelled from Impala's debug web interface by client at
 :"
"Session closed from Impala's debug web interface by client
 at :"

Testing:
-Verified visually that the status message is printed in the impalad
 log with the socket address when one cancels a query or closes a session.
-Added a new e2e test to verify that the new status gets printed in
 runtime profile and coordinator log when a query is cancelled in this
 way.

Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
---
M be/src/kudu/util/web_callback_registry.h
M be/src/service/impala-http-handler.cc
M be/src/util/webserver.cc
M tests/webserver/test_web_pages.py
4 files changed, 33 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/14782/6
--
To view, visit http://gerrit.cloudera.org:8080/14782
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
Gerrit-Change-Number: 14782
Gerrit-PatchSet: 6
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vincent Tran

[Impala-ASF-CR] IMPALA-9182: Print the socket address of the client closing a session or cancelling a query from the WebUI

2019-11-26 Thread Vincent Tran (Code Review)

Vincent Tran has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/14782 )

Change subject: IMPALA-9182: Print the socket address of the client closing a 
session or cancelling a query from the WebUI
..

IMPALA-9182: Print the socket address of the client closing a session or 
cancelling a query from the WebUI

This change appends the socket address (HOST:PORT) of the client
who made the request to close a session or cancel a query from
the coordinator's debug WebUI.

Existing statuses:
"Cancelled from Impala's debug web interface"
"Session closed from Impala's debug web interface"

New statuses:
"Cancelled from Impala's debug web interface by client at
 :"
"Session closed from Impala's debug web interface by client
 at :"

Testing:
-Verified visually that the status message is printed in the impalad
 log with the socket address when one cancels a query or closes a session.
-Added a new e2e test to verify that the new status gets printed in
 runtime profile and coordinator log when a query is cancelled in this
 way.

Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
---
M be/src/kudu/util/web_callback_registry.h
M be/src/service/impala-http-handler.cc
M be/src/util/webserver.cc
M tests/webserver/test_web_pages.py
4 files changed, 33 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/14782/5
--
To view, visit http://gerrit.cloudera.org:8080/14782
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icf74ad06ce1c40fab4ce37de6b7ca78e3e520b43
Gerrit-Change-Number: 14782
Gerrit-PatchSet: 5
Gerrit-Owner: Vincent Tran 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vincent Tran

[Impala-ASF-CR] IMPALA-8607: [DOCS] Document the new global level .impalarc file

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14779 )

Change subject: IMPALA-8607: [DOCS] Document the new global level .impalarc file
..


Patch Set 3: Verified+1

Build Successful

https://jenkins.impala.io/job/gerrit-docs-auto-test/538/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/14779
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90195efceca8a03403f9a674f689648a04ab495d
Gerrit-Change-Number: 14779
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Ethan Xue 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 26 Nov 2019 18:49:10 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8607: [DOCS] Document the new global level .impalarc file

2019-11-26 Thread Bikramjeet Vig (Code Review)

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14779 )

Change subject: IMPALA-8607: [DOCS] Document the new global level .impalarc file
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/14779
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90195efceca8a03403f9a674f689648a04ab495d
Gerrit-Change-Number: 14779
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Ethan Xue 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 26 Nov 2019 18:37:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8607: [DOCS] Document the new global level .impalarc file

Alex Rodoni has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14779 )

Change subject: IMPALA-8607: [DOCS] Document the new global level .impalarc file
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/14779/2/docs/topics/impala_shell_options.xml
File docs/topics/impala_shell_options.xml:

http://gerrit.cloudera.org:8080/#/c/14779/2/docs/topics/impala_shell_options.xml@599
PS2, Line 599: /etc/.impalarc
> I just noticed that the default path for global config does not have the pr
Done



--
To view, visit http://gerrit.cloudera.org:8080/14779
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90195efceca8a03403f9a674f689648a04ab495d
Gerrit-Change-Number: 14779
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Ethan Xue 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 26 Nov 2019 18:26:34 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8607: [DOCS] Document the new global level .impalarc file

2019-11-26 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14779 )

Change subject: IMPALA-8607: [DOCS] Document the new global level .impalarc file
..


Patch Set 3:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/538/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/14779
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90195efceca8a03403f9a674f689648a04ab495d
Gerrit-Change-Number: 14779
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Ethan Xue 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 26 Nov 2019 18:26:30 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8607: [DOCS] Document the new global level .impalarc file