[Impala-ASF-CR] IMPALA-8272: Fix test catalog tablesfilesusage failing

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12649 )

Change subject: IMPALA-8272: Fix test_catalog_tablesfilesusage failing
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2326/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12649
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I41cfa59882edafcd5e61d2e119cd8e8bff08e544
Gerrit-Change-Number: 12649
Gerrit-PatchSet: 1
Gerrit-Owner: Yongzhi Chen 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Sat, 02 Mar 2019 05:15:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8272: Fix test catalog tablesfilesusage failing

2019-03-01 Thread Yongzhi Chen (Code Review) 
Yongzhi Chen has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/12649


Change subject: IMPALA-8272: Fix test_catalog_tablesfilesusage failing
..

IMPALA-8272: Fix test_catalog_tablesfilesusage failing

The test can run in any context, do not make any assumption.

Change-Id: I41cfa59882edafcd5e61d2e119cd8e8bff08e544
---
M tests/webserver/test_web_pages.py
1 file changed, 1 insertion(+), 6 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/12649/1
--
To view, visit http://gerrit.cloudera.org:8080/12649
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I41cfa59882edafcd5e61d2e119cd8e8bff08e544
Gerrit-Change-Number: 12649
Gerrit-PatchSet: 1
Gerrit-Owner: Yongzhi Chen

[Impala-ASF-CR] IMPALA-8273 : Change metastore configuration template to not exclude impala specific parameters

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12646 )

Change subject: IMPALA-8273 : Change metastore configuration template to not 
exclude impala specific parameters
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2325/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12646
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic367b1d06459295c7721cdb20ac7faed9a2aa4c9
Gerrit-Change-Number: 12646
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Sat, 02 Mar 2019 03:32:15 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8266 : Event filtering logic may not filter all the events

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12641 )

Change subject: IMPALA-8266 : Event filtering logic may not filter all the 
events
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2324/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12641
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaeaa26017ee223cca18344e5e1d6ace87200fd9c
Gerrit-Change-Number: 12641
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 02 Mar 2019 03:21:29 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8273 : Change metastore configuration template to not exclude impala specific parameters

2019-03-01 Thread Vihang Karajgaonkar (Code Review) 
Vihang Karajgaonkar has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/12646


Change subject: IMPALA-8273 : Change metastore configuration template to not 
exclude impala specific parameters
..

IMPALA-8273 : Change metastore configuration template to not exclude impala 
specific parameters

When HMS configuration key 
"hive.metastore.notification.parameters.exclude.patterns"
is set to "^impala" it excludes all the parameter key/values from the table
and partition which start with "impala". This can cause problems for
event processing since it relies on these property keys (for instance, 
impala.disableHmsSync).

Testing done:
Confirmed existing test pass.
Ran jenkins job which runs all the fe tests where the events disable flag tests 
were
failing from MetastoreEventsProcessingTest class. Job currently running.
Will update the message once the result comes back.

Change-Id: Ic367b1d06459295c7721cdb20ac7faed9a2aa4c9
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
3 files changed, 18 insertions(+), 4 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/12646/1
--
To view, visit http://gerrit.cloudera.org:8080/12646
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic367b1d06459295c7721cdb20ac7faed9a2aa4c9
Gerrit-Change-Number: 12646
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-8266 : Event filtering logic may not filter all the events

2019-03-01 Thread Vihang Karajgaonkar (Code Review) 
Vihang Karajgaonkar has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/12641 )

Change subject: IMPALA-8266 : Event filtering logic may not filter all the 
events
..

IMPALA-8266 : Event filtering logic may not filter all the events

This patch fixes a bug in event filtering logic. The bug shows up when
atleast one event is filtered out and then a inverse event immediately
shows up after a create_table or create_database event. For example,
consider a event stream has following sequence create_database, create_table,
drop_table, drop_database. In such a case only the first create_database
gets filtered out instead of both the create_database and create_table
event. This leads to a exception while processing create_table since the
database creation is skipped.

Testing done:
1. Adds additional cases in the existing test which generates such
sequence of events.

Change-Id: Iaeaa26017ee223cca18344e5e1d6ace87200fd9c
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
2 files changed, 73 insertions(+), 35 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/12641/2
--
To view, visit http://gerrit.cloudera.org:8080/12641
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaeaa26017ee223cca18344e5e1d6ace87200fd9c
Gerrit-Change-Number: 12641
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12624 )

Change subject: IMPALA-8256: Better error message for 
ImpalaServicePool::RejectTooBusy()
..


Patch Set 4: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3857/


--
To view, visit http://gerrit.cloudera.org:8080/12624
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98
Gerrit-Change-Number: 12624
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Sat, 02 Mar 2019 00:16:53 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues

2019-03-01 Thread Michael Ho (Code Review) 
Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12631 )

Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues
..


Patch Set 4:

(2 comments)

LGTM. Please fix the other two usages of "host"

http://gerrit.cloudera.org:8080/#/c/12631/4/docs/topics/impala_known_issues.xml
File docs/topics/impala_known_issues.xml:

http://gerrit.cloudera.org:8080/#/c/12631/4/docs/topics/impala_known_issues.xml@555
PS4, Line 555: host
Impalad


http://gerrit.cloudera.org:8080/#/c/12631/4/docs/topics/impala_known_issues.xml@555
PS4, Line 555: host
Impalad



--
To view, visit http://gerrit.cloudera.org:8080/12631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5
Gerrit-Change-Number: 12631
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Pooja Nilangekar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 23:32:26 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads

2019-03-01 Thread Michael Ho (Code Review) 
Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12579 )

Change subject: IMPALA-7800: Reject new connections after --fe_service_threads
..


Patch Set 5:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/12579/5/be/src/rpc/TAcceptQueueServer.cpp
File be/src/rpc/TAcceptQueueServer.cpp:

http://gerrit.cloudera.org:8080/#/c/12579/5/be/src/rpc/TAcceptQueueServer.cpp@198
PS5, Line 198: if (entry->expiration_time_ != 0)
 :   wait_time = entry->expiration_time_ - 
MonotonicMillis();
styling nit:

  if (entry->expiration_time_ != 0) {
   wait_time = entry->expiration_time_ - MonotonicMillis();
  }


http://gerrit.cloudera.org:8080/#/c/12579/5/be/src/rpc/TAcceptQueueServer.cpp@200
PS5, Line 200: LOG(INFO) << "All " << maxTasks_ << " server threads are 
in use. "
 :   << "Waiting for " << wait_time << " msecs.";
Does it make sense to log only if we are timing out a request below ?


http://gerrit.cloudera.org:8080/#/c/12579/5/be/src/rpc/TAcceptQueueServer.cpp@263
PS5, Line 263:   MonotonicMillis() + FLAGS_accepted_cnxn_timeout * 
MILLIS_PER_SEC;
nit: indent 4


http://gerrit.cloudera.org:8080/#/c/12579/5/be/src/rpc/TAcceptQueueServer.cpp@267
PS5, Line 267: connection_setup_pool.Offer(
In theory, this could block too but it's still counted towards the 
expiration_time_. So, is it possible for the code to reach line 199 above and 
compute a negative wait_time value ?


http://gerrit.cloudera.org:8080/#/c/12579/5/tests/custom_cluster/test_frontend_connection_limit.py
File tests/custom_cluster/test_frontend_connection_limit.py:

http://gerrit.cloudera.org:8080/#/c/12579/5/tests/custom_cluster/test_frontend_connection_limit.py@1
PS5, Line 1:
nit: blank line



--
To view, visit http://gerrit.cloudera.org:8080/12579
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64
Gerrit-Change-Number: 12579
Gerrit-PatchSet: 5
Gerrit-Owner: Zoram Thanga 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Fri, 01 Mar 2019 23:30:56 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8153: [DOCS] Admission Control page added to web ui

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12645 )

Change subject: IMPALA-8153: [DOCS] Admission Control page added to web ui
..


Patch Set 1: Verified+1

Build Successful

https://jenkins.impala.io/job/gerrit-docs-auto-test/241/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/12645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifa4a23d5dc3be678080c565a311d0ee9660a6b19
Gerrit-Change-Number: 12645
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 23:27:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Fredy Wijaya (Code Review) 
Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..


Patch Set 10:

I created a follow-up JIRA: https://issues.apache.org/jira/browse/IMPALA-8271 
to fix update the rest of Thrift enums in query-options.cc


--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 10
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 23:11:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8153: [DOCS] Admission Control page added to web ui

2019-03-01 Thread Alex Rodoni (Code Review) 
Alex Rodoni has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/12645


Change subject: IMPALA-8153: [DOCS] Admission Control page added to web ui
..

IMPALA-8153: [DOCS] Admission Control page added to web ui

Change-Id: Ifa4a23d5dc3be678080c565a311d0ee9660a6b19
---
M docs/topics/impala_webui.xml
1 file changed, 83 insertions(+), 47 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/12645/1
--
To view, visit http://gerrit.cloudera.org:8080/12645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifa4a23d5dc3be678080c565a311d0ee9660a6b19
Gerrit-Change-Number: 12645
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni

[Impala-ASF-CR] IMPALA-8153: [DOCS] Admission Control page added to web ui

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12645 )

Change subject: IMPALA-8153: [DOCS] Admission Control page added to web ui
..


Patch Set 1:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/241/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/12645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifa4a23d5dc3be678080c565a311d0ee9660a6b19
Gerrit-Change-Number: 12645
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 01 Mar 2019 23:08:03 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Csaba Ringhofer (Code Review) 
Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..


Patch Set 10: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc@264
PS8, Line 264: query_options->__set_hbase_caching(atoi(value.c_str()));
 : break;
 :   case TImpalaQueryOptions::HBASE_CACHE_BLOCKS:
 : query_options->__set_hbase_cache_blocks(
 : iequals(value, "true") || iequals(value, "1"));
> Good idea. I can make the whole thing in a single pass.
Thanks, I think it became more readable.



--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 10
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 23:01:56 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8003: Improve logging for diskspace errors from dataload

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12399 )

Change subject: IMPALA-8003: Improve logging for diskspace errors from dataload
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2323/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12399
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8766d819c68345cac7d0a32258415ef1a188f53
Gerrit-Change-Number: 12399
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Comment-Date: Fri, 01 Mar 2019 22:39:53 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..


Patch Set 10:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2322/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 10
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 22:35:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8261: Enhance create-test-configuration.sh to not fail when FE has not been built

2019-03-01 Thread Fredy Wijaya (Code Review) 
Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12630 )

Change subject: IMPALA-8261: Enhance create-test-configuration.sh to not fail 
when FE has not been built
..


Patch Set 3:

> Patch Set 3: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3856/

Error due to flaky test not related to this patch: 
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/4727/

Will restart the GVO until the build is more stable.


--
To view, visit http://gerrit.cloudera.org:8080/12630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6536dcffc1124e79c1ed111ad92d257493cc8feb
Gerrit-Change-Number: 12630
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 22:04:40 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8261: Enhance create-test-configuration.sh to not fail when FE has not been built

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12630 )

Change subject: IMPALA-8261: Enhance create-test-configuration.sh to not fail 
when FE has not been built
..


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3856/


--
To view, visit http://gerrit.cloudera.org:8080/12630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6536dcffc1124e79c1ed111ad92d257493cc8feb
Gerrit-Change-Number: 12630
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 22:01:28 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8003: Improve logging for diskspace errors from dataload

2019-03-01 Thread Joe McDonnell (Code Review) 
Hello Laszlo Gaal, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12399

to look at the new patch set (#2).

Change subject: IMPALA-8003: Improve logging for diskspace errors from dataload
..

IMPALA-8003: Improve logging for diskspace errors from dataload

When dataload runs on a machine without enough diskspace, it
can fail with arcane errors that don't mentioned disk space.
This checks the HDFS NameNode log for common errors that indicate
disk space issues ("Failed to place enough replicas" and
"NOT_ENOUGH_STORAGE_SPACE") and prints a better error message.

Change-Id: If8766d819c68345cac7d0a32258415ef1a188f53
---
M testdata/bin/create-load-data.sh
1 file changed, 32 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/12399/2
--
To view, visit http://gerrit.cloudera.org:8080/12399
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8766d819c68345cac7d0a32258415ef1a188f53
Gerrit-Change-Number: 12399
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Laszlo Gaal

[Impala-ASF-CR] IMPALA-8003: Improve logging for diskspace errors from dataload

2019-03-01 Thread Joe McDonnell (Code Review) 
Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12399 )

Change subject: IMPALA-8003: Improve logging for diskspace errors from dataload
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12399/1/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:

http://gerrit.cloudera.org:8080/#/c/12399/1/testdata/bin/create-load-data.sh@134
PS1, Line 134: echo "Diskspace at start of dataload:"
 : df "${IMPALA_HOME}"
> Do you think it would be worth adding a comparison and a warning here if fr
Added a warning.



--
To view, visit http://gerrit.cloudera.org:8080/12399
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8766d819c68345cac7d0a32258415ef1a188f53
Gerrit-Change-Number: 12399
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Comment-Date: Fri, 01 Mar 2019 21:59:18 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates

2019-03-01 Thread Bharath Vissapragada (Code Review) 
Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12591 )

Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates
..


Patch Set 10:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/12591/10/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/12591/10/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@500
PS10, Line 500: // Get a new catalog version to assign to the table being 
altered.
  : long newCatalogVersion = 
catalog_.incrementAndGetCatalogVersion();
Can you move this below the tryLock(tbl)? It takes the versionLock_ for you. 
Also move L523 (to unlock) closer to this, so that you don't hold the version 
lock for longer.  (See the pattern elsewhere in this file).


http://gerrit.cloudera.org:8080/#/c/12591/10/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1974
PS10, Line 1974: catalog_.getCatalogVersion())
This looks incorrect? The new table create below has a different version.


http://gerrit.cloudera.org:8080/#/c/12591/10/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3312
PS10, Line 3312: newCatalogVersion
Now that I think about this, I think newCatalogVersion variable doesn't make 
much sense here.

Can we refactor the code better without passing around these 
newCatalogVersions? I think it is super confusing to those who read the code.



--
To view, visit http://gerrit.cloudera.org:8080/12591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353
Gerrit-Change-Number: 12591
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Krishna 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 21:57:54 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Fredy Wijaya (Code Review) 
Fredy Wijaya has uploaded a new patch set (#10). ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..

IMPALA-8254: Fix error when running compute stats with compression_codec set

This patch fixes an error when running compute stats with
compression_codec set. This patch also updates missing compression
codecs from the compression_codec query option.

Testing:
- Updated BE query-options-test
- Added test_compute.stats.py
- Ran all tests in test_compute.stats.py
- Ran all tests in test_set.py

Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/metadata/test_compute_stats.py
4 files changed, 44 insertions(+), 19 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/12635/10
--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 10
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12591 )

Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates
..


Patch Set 9:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3855/


--
To view, visit http://gerrit.cloudera.org:8080/12591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353
Gerrit-Change-Number: 12591
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Krishna 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 21:38:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12631 )

Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues
..


Patch Set 4: Verified+1

Build Successful

https://jenkins.impala.io/job/gerrit-docs-auto-test/240/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/12631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5
Gerrit-Change-Number: 12631
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Pooja Nilangekar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 20:20:52 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12591 )

Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates
..


Patch Set 10:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2321/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353
Gerrit-Change-Number: 12591
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Krishna 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 20:18:29 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12631 )

Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues
..


Patch Set 4:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/240/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/12631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5
Gerrit-Change-Number: 12631
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Pooja Nilangekar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 20:18:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues

2019-03-01 Thread Alex Rodoni (Code Review) 
Alex Rodoni has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12631 )

Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues
..


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12631/1/docs/topics/impala_known_issues.xml
File docs/topics/impala_known_issues.xml:

http://gerrit.cloudera.org:8080/#/c/12631/1/docs/topics/impala_known_issues.xml@557
PS1, Line 557: h
> It should be in <> like filename and hostname:port
Done


http://gerrit.cloudera.org:8080/#/c/12631/3/docs/topics/impala_known_issues.xml
File docs/topics/impala_known_issues.xml:

http://gerrit.cloudera.org:8080/#/c/12631/3/docs/topics/impala_known_issues.xml@565
PS3, Line 565: host
> Impalad
Done



--
To view, visit http://gerrit.cloudera.org:8080/12631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5
Gerrit-Change-Number: 12631
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Pooja Nilangekar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 20:18:19 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues

2019-03-01 Thread Alex Rodoni (Code Review) 
Hello Michael Ho, Pooja Nilangekar, Joe McDonnell, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12631

to look at the new patch set (#4).

Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues
..

[DOCS] Add HADOOP-15720 to the list of Known Issues

Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5
---
M docs/topics/impala_known_issues.xml
1 file changed, 19 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/12631/4
--
To view, visit http://gerrit.cloudera.org:8080/12631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5
Gerrit-Change-Number: 12631
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Pooja Nilangekar

[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates

2019-03-01 Thread Tim Armstrong (Code Review) 
Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12591 )

Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates
..


Patch Set 10: Code-Review-2

I'm pretty sure the failures in 
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/4718/ *are* related to 
the patch, many look metadata related, e.g. 
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/4718/testReport/junit/authorization.test_authorization/TestAuthorization/test_invalidate_metadata_sentry_unavailable/

Hypothetically if tests were so flaky that we did have 38 test failures on a 
typical run then we should stop merging large changes and start fixing the 
flaky tests.

In either case we should not merge this until we understand what happened with 
the tests.


--
To view, visit http://gerrit.cloudera.org:8080/12591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353
Gerrit-Change-Number: 12591
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Krishna 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 20:11:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12624 )

Change subject: IMPALA-8256: Better error message for 
ImpalaServicePool::RejectTooBusy()
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12624
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98
Gerrit-Change-Number: 12624
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Fri, 01 Mar 2019 19:57:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12624 )

Change subject: IMPALA-8256: Better error message for 
ImpalaServicePool::RejectTooBusy()
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3857/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/12624
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98
Gerrit-Change-Number: 12624
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Fri, 01 Mar 2019 19:57:32 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates

2019-03-01 Thread Vihang Karajgaonkar (Code Review) 
Vihang Karajgaonkar has uploaded a new patch set (#10). ( 
http://gerrit.cloudera.org:8080/12591 )

Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates
..

IMPALA-7972 Detect self-events to avoid unnecessary invalidates

This patch adds support to detect self-generated events from catalog.
This is used to avoid unnecessary invalidates to the tables from such
self-events. Currently, alter_table, alter_partition, add_partition and
drop_partition event types can invalidate the table metadata.

Originally, we planned to have a global version number support from
metastore (see HIVE-21115). But since that is still not complete, we
rely on a combination of other identifiers to determine if a event is
self-generated or not. These self-event identifiers consists of values
from the table/partition parameters. A catalog service uuid
and the catalog version number. The uuid is generated for each
catalogservice when it comes up and it adds it to the table/partition
parameters with the key "impala.CatalogServiceId". The catalog version
number is added with the key "impala.CatalogVersion".

When catalog executes a DDL operation it appends the current catalog
version to the list of version numbers for the in-flight events for the
table. Events processor clears this version when the corresponding
version number identified by serviceId is received in the event. This is
needed since it is possible that a external non-Impala system which
generates the event presents the same serviceId and version number later
on. The algorithm to detect a self-event is as below.

1. Add the service id and expected catalog version to table/partition
parameters when executing the DDL operation. When the HMS operation is
successful, add the version number to the list of version for in-flight
events at table level.
2. When the event is received, the first time you see the combination of
serviceId and version number, event processor clears the version number
from table's list and determines the event as self-generated (and hence
ignored)
3. If the event data presents a unknown serviceId or if the version
number is not present in the list of in-flight versions, event is not a
self-event and needs to be processed.

In order to limit the total memory footprint, only 10 version numbers
are stored at the table. Since the event processor is expected to poll
every few seconds this should be a reasonable bound which satisfies most
use-cases. Otherwise, event processor may wrongly process a self-event
to invalidate the table. In such a case, its a performance penalty not a
correctness issue.

In case of drop_partition event, the partition object is not available
in the event. Hence we cannot determine if its a self-event. In such
cases currently we always issue a invalidate command. This is a known
limitation and will be improved in IMPALA-7973

Patch adds new tests to trigger alter table/partition DDLs from impala
and makes sure that the table is not invalidated.

Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
6 files changed, 1,191 insertions(+), 211 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/12591/10
--
To view, visit http://gerrit.cloudera.org:8080/12591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353
Gerrit-Change-Number: 12591
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Krishna 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-8249: End-to-end test framework doesn't read aggregated counters properly

2019-03-01 Thread Joe McDonnell (Code Review) 
Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12589 )

Change subject: IMPALA-8249: End-to-end test framework doesn't read aggregated 
counters properly
..


Patch Set 4: Code-Review+2

(1 comment)

Thanks for fixing this.

I'm going to file a separate JIRA for us to think about implementing this using 
the Thrift profile. It might be a bit easier than parsing the text.

http://gerrit.cloudera.org:8080/#/c/12589/4/tests/common/test_result_verifier.py
File tests/common/test_result_verifier.py:

http://gerrit.cloudera.org:8080/#/c/12589/4/tests/common/test_result_verifier.py@522
PS4, Line 522: It expects the profile to be written in
 :   verbose mode.
Nit: Small clarification.
"It expects the profile to write counters in verbose mode."



--
To view, visit http://gerrit.cloudera.org:8080/12589
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a6fc0d3f7cbaa87aa848cdafffad21fb1514930
Gerrit-Change-Number: 12589
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 01 Mar 2019 19:25:24 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()

2019-03-01 Thread Lars Volker (Code Review) 
Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12624 )

Change subject: IMPALA-8256: Better error message for 
ImpalaServicePool::RejectTooBusy()
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12624
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98
Gerrit-Change-Number: 12624
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Fri, 01 Mar 2019 19:05:16 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues

2019-03-01 Thread Michael Ho (Code Review) 
Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12631 )

Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12631/3/docs/topics/impala_known_issues.xml
File docs/topics/impala_known_issues.xml:

http://gerrit.cloudera.org:8080/#/c/12631/3/docs/topics/impala_known_issues.xml@565
PS3, Line 565: host
Impalad



--
To view, visit http://gerrit.cloudera.org:8080/12631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5
Gerrit-Change-Number: 12631
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Pooja Nilangekar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 18:54:58 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()

2019-03-01 Thread Michael Ho (Code Review) 
Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12624 )

Change subject: IMPALA-8256: Better error message for 
ImpalaServicePool::RejectTooBusy()
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12624/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12624/3//COMMIT_MSG@18
PS3, Line 18: test_rpc_timeout.py
> Is that file missing from the change then?
There is no change in that test file. A pre-existing test already exercises 
that path.



--
To view, visit http://gerrit.cloudera.org:8080/12624
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98
Gerrit-Change-Number: 12624
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Fri, 01 Mar 2019 18:59:35 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()

2019-03-01 Thread Lars Volker (Code Review) 
Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12624 )

Change subject: IMPALA-8256: Better error message for 
ImpalaServicePool::RejectTooBusy()
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12624/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12624/3//COMMIT_MSG@18
PS3, Line 18: test_rpc_timeout.py
Is that file missing from the change then?



--
To view, visit http://gerrit.cloudera.org:8080/12624
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98
Gerrit-Change-Number: 12624
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Fri, 01 Mar 2019 18:55:38 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7916: Remove support for authorization policy file flag

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12637 )

Change subject: IMPALA-7916: Remove support for authorization policy file flag
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2320/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12637
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
Gerrit-Change-Number: 12637
Gerrit-PatchSet: 5
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 01 Mar 2019 18:37:22 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7916: Remove support for authorization policy file flag

2019-03-01 Thread Fredy Wijaya (Code Review) 
Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12637 )

Change subject: IMPALA-7916: Remove support for authorization policy file flag
..


Patch Set 5:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12637/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12637/5//COMMIT_MSG@7
PS5, Line 7: flag
   :
nit: remove


http://gerrit.cloudera.org:8080/#/c/12637/5//COMMIT_MSG@9
PS5, Line 9: This patch removes support for the flag 
--authorization_policy_file.
   : The flag can still be passed to the backend but a warning message 
will
   : be logged in the frontend that the policy file has no effect on
   : authorization.
mention that the tests that rely on sentry authorization policy file have been 
rewritten to use sentry service instead.


http://gerrit.cloudera.org:8080/#/c/12637/5/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:

http://gerrit.cloudera.org:8080/#/c/12637/5/fe/src/main/java/org/apache/impala/service/JniFrontend.java@711
PS5, Line 711: if (BackendConfig.INSTANCE.isAuthorizationFileSet()) {
 :   LOG.warn("authorization_policy_file flag is no longer 
supported. The " +
 :   "flag is being ignored.");
 : }
We should handle it in the backend instead. See: 
https://github.com/apache/impala/blob/master/be/src/common/global-flags.cc#L299-L334.
 In other words, we need to remove the authorization_file property from Thrift.


http://gerrit.cloudera.org:8080/#/c/12637/4/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:

http://gerrit.cloudera.org:8080/#/c/12637/4/tests/authorization/test_authorization.py@42
PS4, Line 42: from tests.util.filesystem_utils import WAREHOUSE
> flake8: F401 'tests.util.calculation_util.get_random_id' imported but unuse
Done



--
To view, visit http://gerrit.cloudera.org:8080/12637
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
Gerrit-Change-Number: 12637
Gerrit-PatchSet: 5
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 01 Mar 2019 18:30:40 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..


Patch Set 9:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2319/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 18:27:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7916: Remove support for authorization policy file flag

2019-03-01 Thread Austin Nobis (Code Review) 
Austin Nobis has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/12637 )

Change subject: IMPALA-7916: Remove support for authorization policy file flag
..

IMPALA-7916: Remove support for authorization policy file flag

This patch removes support for the flag --authorization_policy_file.
The flag can still be passed to the backend but a warning message will
be logged in the frontend that the policy file has no effect on
authorization.

Testing:
- Ran all FE tests
- Ran all E2E tests

Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
---
M fe/src/main/java/org/apache/impala/analysis/AuthorizationStmt.java
M 
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthProvider.java
M 
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java
M 
fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/service/CustomClusterGroupMapper.java
D fe/src/test/resources/authz-policy.ini.template
M testdata/bin/create-load-data.sh
M tests/authorization/test_authorization.py
M tests/conftest.py
16 files changed, 411 insertions(+), 990 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12637/5
--
To view, visit http://gerrit.cloudera.org:8080/12637
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
Gerrit-Change-Number: 12637
Gerrit-PatchSet: 5
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

2019-03-01 Thread Paul Rogers (Code Review) 
Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
..


Abandoned

Can live without it.
--
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 6
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Philip Zeyliger

[Impala-ASF-CR] IMPALA-8106: Cleanup of literal expression nodes

2019-03-01 Thread Paul Rogers (Code Review) 
Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/12265 )

Change subject: IMPALA-8106: Cleanup of literal expression nodes
..


Abandoned

Not a priority at the moment.
--
To view, visit http://gerrit.cloudera.org:8080/12265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I740f877964d5bc677fef98f1ccfa5be32cc8f7ac
Gerrit-Change-Number: 12265
Gerrit-PatchSet: 14
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[Impala-ASF-CR] IMPALA-8156: Add format options to the EXPLAIN statement

2019-03-01 Thread Paul Rogers (Code Review) 
Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/12340 )

Change subject: IMPALA-8156: Add format options to the EXPLAIN statement
..


Abandoned

Not a priority at the moment.
--
To view, visit http://gerrit.cloudera.org:8080/12340
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I84a8bc4fbff52b70a747f3a3c08abe6973e37fc1
Gerrit-Change-Number: 12340
Gerrit-PatchSet: 1
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues

2019-03-01 Thread Pooja Nilangekar (Code Review) 
Pooja Nilangekar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12631 )

Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12631/1/docs/topics/impala_known_issues.xml
File docs/topics/impala_known_issues.xml:

http://gerrit.cloudera.org:8080/#/c/12631/1/docs/topics/impala_known_issues.xml@557
PS1, Line 557: h
> Should this be in <> like filename and hostname:port? Or will it appear lit
It should be in <> like filename and hostname:port



--
To view, visit http://gerrit.cloudera.org:8080/12631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5
Gerrit-Change-Number: 12631
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Pooja Nilangekar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 18:00:29 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8041, Part 2: Refactor SELECT list

2019-03-01 Thread Paul Rogers (Code Review) 
Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/12144 )

Change subject: IMPALA-8041, Part 2: Refactor SELECT list
..


Abandoned

Not a priority at the moment.
--
To view, visit http://gerrit.cloudera.org:8080/12144
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I5e173dad717e739c097f1f0cd86a0352648ff886
Gerrit-Change-Number: 12144
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[Impala-ASF-CR] IMPALA-8041, Part 1: Move rewrite rules into expr nodes

2019-03-01 Thread Paul Rogers (Code Review) 
Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/12143 )

Change subject: IMPALA-8041, Part 1: Move rewrite rules into expr nodes
..


Abandoned

Not a priority at the moment.
--
To view, visit http://gerrit.cloudera.org:8080/12143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ifda7082abea4f6448e6cfef0d9ef7b8565d73fce
Gerrit-Change-Number: 12143
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[Impala-ASF-CR] IMPALA-8261: Enhance create-test-configuration.sh to not fail when FE has not been built

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12630 )

Change subject: IMPALA-8261: Enhance create-test-configuration.sh to not fail 
when FE has not been built
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3856/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/12630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6536dcffc1124e79c1ed111ad92d257493cc8feb
Gerrit-Change-Number: 12630
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:43:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7916: Remove support for authorization policy file flag

2019-03-01 Thread Austin Nobis (Code Review) 
Austin Nobis has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12637 )

Change subject: IMPALA-7916: Remove support for authorization policy file flag
..


Patch Set 4:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:

http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@51
PS4, Line 51: import org.apache.impala.thrift.*;
> nit: don't use wildcard imports
Done


http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@54
PS4, Line 54: import org.junit.*;
> same thing here
Done


http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@68
PS4, Line 68:  private static final SentryAuthorizationConfig authzConfig =
:   
SentryAuthorizationConfig.createHadoopGroupAuthConfig("server1",
:   System.getenv("IMPALA_HOME") + 
"/fe/src/test/resources/sentry-site.xml");
:   private static final ImpaladTestCatalog authzCatalog_ =
:   new ImpaladTestCatalog(authzConfig);
:   private static final Frontend authzFe_ = new 
Frontend(authzConfig, authzCatalog_);
> static final variables should use upper case names
Done


http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@78
PS4, Line 78: analysisContext_
> can we rename this to authzCtx to differentiate between normal ctx vs authz
Done


http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@80
PS4, Line 80:   setupImpalaCatalog(authzCatalog_);
: } catch(ImpalaException e) {
:   throw new RuntimeException(e);
: }
> We can just declare the ImpalaException in the constructor.
Done


http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@183
PS4, Line 183: List dbs = 
authzFe_.getDbs(PatternMatcher.createHivePatternMatcher("*"),
> line too long (92 > 90)
Done



--
To view, visit http://gerrit.cloudera.org:8080/12637
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
Gerrit-Change-Number: 12637
Gerrit-PatchSet: 4
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:53:12 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8261: Enhance create-test-configuration.sh to not fail when FE has not been built

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12630 )

Change subject: IMPALA-8261: Enhance create-test-configuration.sh to not fail 
when FE has not been built
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6536dcffc1124e79c1ed111ad92d257493cc8feb
Gerrit-Change-Number: 12630
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:43:00 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Fredy Wijaya (Code Review) 
Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..


Patch Set 9:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12635/7/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/12635/7/be/src/service/query-options.cc@178
PS7, Line 178:
> Can you add a few tests to verify case insensitivity?
Done


http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc@176
PS8, Line 176: s GetThriftEnum
> Other functions in the class use camel case.
Done


http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc@188
PS8, Line 188: lue = static_cast same as line 176
Done


http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc@264
PS8, Line 264: query_options->__set_hbase_caching(atoi(value.c_str()));
 : break;
 :   case TImpalaQueryOptions::HBASE_CACHE_BLOCKS:
 : query_options->__set_hbase_cache_blocks(
 : iequals(value, "true") || iequals(value, "1"));
> optional: get_thrift_enum could have a signature similar to ParseMemValue()
Good idea. I can make the whole thing in a single pass.



--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:42:25 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Fredy Wijaya (Code Review) 
Fredy Wijaya has uploaded a new patch set (#9). ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..

IMPALA-8254: Fix error when running compute stats with compression_codec set

This patch fixes an error when running compute stats with
compression_codec set. This patch also updates missing compression
codecs from the compression_codec query option.

Testing:
- Updated BE query-options-test
- Added test_compute.stats.py
- Ran all tests in test_compute.stats.py
- Ran all tests in test_set.py

Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/metadata/test_compute_stats.py
4 files changed, 44 insertions(+), 19 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/12635/9
--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 9
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-8261: Enhance create-test-configuration.sh to not fail when FE has not been built

2019-03-01 Thread David Knupp (Code Review) 
David Knupp has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12630 )

Change subject: IMPALA-8261: Enhance create-test-configuration.sh to not fail 
when FE has not been built
..


Patch Set 2: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6536dcffc1124e79c1ed111ad92d257493cc8feb
Gerrit-Change-Number: 12630
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:41:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12591 )

Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates
..


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3855/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/12591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353
Gerrit-Change-Number: 12591
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Krishna 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:30:12 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7916: Remove support for authorization policy file flag

2019-03-01 Thread Fredy Wijaya (Code Review) 
Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12637 )

Change subject: IMPALA-7916: Remove support for authorization policy file flag
..


Patch Set 4:

(5 comments)

Thanks! Overall it looks good now, just couple nits.

http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:

http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@51
PS4, Line 51: import org.apache.impala.thrift.*;
nit: don't use wildcard imports


http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@54
PS4, Line 54: import org.junit.*;
same thing here


http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@68
PS4, Line 68:  private static final SentryAuthorizationConfig authzConfig =
:   
SentryAuthorizationConfig.createHadoopGroupAuthConfig("server1",
:   System.getenv("IMPALA_HOME") + 
"/fe/src/test/resources/sentry-site.xml");
:   private static final ImpaladTestCatalog authzCatalog_ =
:   new ImpaladTestCatalog(authzConfig);
:   private static final Frontend authzFe_ = new 
Frontend(authzConfig, authzCatalog_);
static final variables should use upper case names


http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@78
PS4, Line 78: analysisContext_
can we rename this to authzCtx to differentiate between normal ctx vs authz ctx?


http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@80
PS4, Line 80:   setupImpalaCatalog(authzCatalog_);
: } catch(ImpalaException e) {
:   throw new RuntimeException(e);
: }
We can just declare the ImpalaException in the constructor.



--
To view, visit http://gerrit.cloudera.org:8080/12637
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
Gerrit-Change-Number: 12637
Gerrit-PatchSet: 4
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:37:13 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC

2019-03-01 Thread Zoltan Borok-Nagy (Code Review) 
Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12168 )

Change subject: IMPALA-6503: Support reading complex types from ORC
..


Patch Set 18:

(14 comments)

Found some style-violations. Besides those the reader and scanner code looks 
good to me.
On Monday I'll try to take a deeper look at the OrcSchemaResolver. Thanks for 
the google doc and slides that explain the details!

http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h
File be/src/exec/hdfs-orc-scanner.h:

http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h@178
PS18, Line 178:   std::unordered_set 
missing_field_slots_;
Please add comment to this member.

Nit: please insert blank lines after member declarations.


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc
File be/src/exec/orc-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@287
PS18, Line 287: continue
nit: please put it into braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@320
PS18, Line 320: RETURN_IF_ERROR
nit: braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@398
PS18, Line 398:   CreateChildForSlot(node, child_slot);
nit: put into braces.


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@411
PS18, Line 411: CreateChildForSlot
nit: braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@451
PS18, Line 451: RETURN_IF_ERROR(child->ReadValue(array_start_ + array_idx_, 
tuple, pool));
nit: put it into braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@467
PS18, Line 467: RETURN_IF_ERROR
nit: braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@502
PS18, Line 502: (field == SchemaPathConstants::MAP_KEY) 
key_readers_.push_back(child);
  :   else value_readers_.push_back(chil
nit: the whole 'if' statement doesn't fit into one line, so please put it into 
braces.


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@512
PS18, Line 512:   CreateChildForSlot(node, child_slot);
nit: braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@527
PS18, Line 527: y_selected) key_readers_.push_back(child);
  : else value_readers_
nit: braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@539
PS18, Line 539: CreateChildForSlot
nit: braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@540
PS18, Line 540: CreateChildForSlot
nit: braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@579
PS18, Line 579: RETURN_IF_ERROR(child->ReadValue(array_offset_, tuple, 
pool));
nit: braces


http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@590
PS18, Line 590: RETURN_IF_ERROR(child->ReadValue(offset + tuple_idx, tuple, 
pool));
nit: braces



--
To view, visit http://gerrit.cloudera.org:8080/12168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790
Gerrit-Change-Number: 12168
Gerrit-PatchSet: 18
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:26:05 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7916: Remove support for authorization policy file flag

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12637 )

Change subject: IMPALA-7916: Remove support for authorization policy file flag
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2318/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12637
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
Gerrit-Change-Number: 12637
Gerrit-PatchSet: 4
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:09:19 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Csaba Ringhofer (Code Review) 
Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..


Patch Set 8:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12635/7/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/12635/7/be/src/service/query-options.cc@178
PS7, Line 178: es) {
> Oops. Done.
Can you add a few tests to verify case insensitivity?


http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc@176
PS8, Line 176: get_thrift_enum
Other functions in the class use camel case.


http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc@188
PS8, Line 188: get_thrift_enum_values
same as line 176


http://gerrit.cloudera.org:8080/#/c/12635/8/be/src/service/query-options.cc@264
PS8, Line 264: if (!get_thrift_enum(value,
 : _THdfsCompression_VALUES_TO_NAMES, _type)) {
 :   return Status(Substitute("Invalid compression codec: 
'$0'. Valid compression "
 :   "codecs are $1.", value,
 :   
get_thrift_enum_values(_THdfsCompression_VALUES_TO_NAMES)));
optional: get_thrift_enum could have a signature similar to ParseMemValue() to 
simplify its usage. I think it is ok to always write "Valid values are: ..."



--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 17:07:53 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()

2019-03-01 Thread Andrew Sherman (Code Review) 
Andrew Sherman has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12624 )

Change subject: IMPALA-8256: Better error message for 
ImpalaServicePool::RejectTooBusy()
..


Patch Set 3: Code-Review+1

LGTM


--
To view, visit http://gerrit.cloudera.org:8080/12624
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98
Gerrit-Change-Number: 12624
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Fri, 01 Mar 2019 16:53:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..


Patch Set 8:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2317/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 16:52:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7916: Remove support for authorization policy file flag

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12637 )

Change subject: IMPALA-7916: Remove support for authorization policy file flag
..


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:

http://gerrit.cloudera.org:8080/#/c/12637/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@183
PS4, Line 183: List dbs = 
authzFe_.getDbs(PatternMatcher.createHivePatternMatcher("*"),
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/12637/4/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:

http://gerrit.cloudera.org:8080/#/c/12637/4/tests/authorization/test_authorization.py@42
PS4, Line 42: from tests.util.calculation_util import get_random_id
flake8: F401 'tests.util.calculation_util.get_random_id' imported but unused



--
To view, visit http://gerrit.cloudera.org:8080/12637
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
Gerrit-Change-Number: 12637
Gerrit-PatchSet: 4
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 01 Mar 2019 16:26:55 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7916: Remove support for authorization policy file flag

2019-03-01 Thread Austin Nobis (Code Review) 
Austin Nobis has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/12637 )

Change subject: IMPALA-7916: Remove support for authorization policy file flag
..

IMPALA-7916: Remove support for authorization policy file flag

This patch removes support for the flag --authorization_policy_file.
The flag can still be passed to the backend but a warning message will
be logged in the frontend that the policy file has no effect on
authorization.

Testing:
- Ran all FE tests
- Ran all E2E tests

Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
---
M fe/src/main/java/org/apache/impala/analysis/AuthorizationStmt.java
M 
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthProvider.java
M 
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java
M 
fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/service/CustomClusterGroupMapper.java
D fe/src/test/resources/authz-policy.ini.template
M testdata/bin/create-load-data.sh
M tests/authorization/test_authorization.py
M tests/conftest.py
16 files changed, 406 insertions(+), 992 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12637/4
--
To view, visit http://gerrit.cloudera.org:8080/12637
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd
Gerrit-Change-Number: 12637
Gerrit-PatchSet: 4
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Fredy Wijaya (Code Review) 
Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..


Patch Set 8:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/12635/7/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/12635/7/be/src/service/query-options.cc@175
PS7, Line 175: template
 : static bool get_thrift_enum(const string& value,
> Most of this could be expressed with a template function that gets the enum
Done


http://gerrit.cloudera.org:8080/#/c/12635/7/be/src/service/query-options.cc@178
PS7, Line 178: es) {
> The original comparison was case insensitive, which is expected for query o
Oops. Done.


http://gerrit.cloudera.org:8080/#/c/12635/7/be/src/service/query-options.cc@186
PS7, Line 186:
> I would prefer to mention "thrift" and "enum" in the name.
Done



--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 01 Mar 2019 16:09:05 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set

2019-03-01 Thread Fredy Wijaya (Code Review) 
Fredy Wijaya has uploaded a new patch set (#8). ( 
http://gerrit.cloudera.org:8080/12635 )

Change subject: IMPALA-8254: Fix error when running compute stats with 
compression_codec set
..

IMPALA-8254: Fix error when running compute stats with compression_codec set

This patch fixes an error when running compute stats with
compression_codec set. This patch also updates missing compression
codecs from the compression_codec query option.

Testing:
- Updated BE query-options-test
- Added test_compute.stats.py
- Ran all tests in test_compute.stats.py
- Ran all tests in test_set.py

Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/metadata/test_compute_stats.py
4 files changed, 52 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/12635/8
--
To view, visit http://gerrit.cloudera.org:8080/12635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2
Gerrit-Change-Number: 12635
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet

2019-03-01 Thread Zoltan Borok-Nagy (Code Review) 
Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12247 )

Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet
..


Patch Set 9: Code-Review+1

(5 comments)

http://gerrit.cloudera.org:8080/#/c/12247/8//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12247/8//COMMIT_MSG@39
PS8, Line 39: without conversion to UTC
> The main problem with normalizing to UTC from Impala's standpoint is the pe
OK, thanks for the explanation for both of you!


http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/hdfs-parquet-table-writer.cc
File be/src/exec/parquet/hdfs-parquet-table-writer.cc:

http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/hdfs-parquet-table-writer.cc@579
PS8, Line 579: result_
> result_ shouldn't be a local variable, as we return a pointer to it. This i
Ah, I completely missed that.


http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/parquet-metadata-utils.cc
File be/src/exec/parquet/parquet-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/parquet-metadata-utils.cc@142
PS8, Line 142: /// converted_type is not set because Impala always writes 
timestamps without UTC
> That's true, but it has a metadata field to tell the caller the desired sem
Sorry, I was really thinking about Hive and Spark.

But it doesn't really matter, since now we have this 'isAdjustedToUTC' field. 
And Zoltan also said in one of his comments that "'pure' TIMESTAMP and 
TIMESTAMP WITHOUT TIME ZONE shall not be normalized to UTC"


http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/runtime/timestamp-value.inline.h
File be/src/runtime/timestamp-value.inline.h:

http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/runtime/timestamp-value.inline.h@154
PS8, Line 154: // TODO: consider optimizing this (IMPALA-8268)
 :   kudu::int128_t nanos128 =
 : static_cast(unixtime_seconds) * NANOS_PER_SEC
 : + time_.fractional_seconds();
 :
 :   if (nanos128 <  std::numeric_limits::min()
> I created a ticket for benchmarking and optimizing these new functions: IMP
OK, thanks!


http://gerrit.cloudera.org:8080/#/c/12247/8/testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
File 
testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test:

http://gerrit.cloudera.org:8080/#/c/12247/8/testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test@102
PS8, Line 102:  QUERY
 : create table int96_nanos (ts timestamp) stored as parquet;
 : 
 :  QUERY
 : # Insert edge values as "normal" int96 timestamps that can 
represent all values.
 : set parquet_timestamp_type=INT96_NANOS;
 : insert into int96_nanos values
> Thanks for the tip!
You're welcome!



--
To view, visit http://gerrit.cloudera.org:8080/12247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f
Gerrit-Change-Number: 12247
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: Zoltan Ivanfi 
Gerrit-Comment-Date: Fri, 01 Mar 2019 15:43:45 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet

2019-03-01 Thread Zoltan Ivanfi (Code Review) 
Zoltan Ivanfi has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12247 )

Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet
..


Patch Set 8:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/parquet-metadata-utils.cc
File be/src/exec/parquet/parquet-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/parquet-metadata-utils.cc@142
PS8, Line 142: /// converted_type is not set because Impala always writes 
timestamps without UTC
> As far as I know Parquet-MR does not do any timezone conversion and leaves
That's true, but it has a metadata field to tell the caller the desired 
semantics, which can be either UTC-normalized or not.



--
To view, visit http://gerrit.cloudera.org:8080/12247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f
Gerrit-Change-Number: 12247
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: Zoltan Ivanfi 
Gerrit-Comment-Date: Fri, 01 Mar 2019 14:11:43 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet

2019-03-01 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12247 )

Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet
..


Patch Set 9:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2316/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f
Gerrit-Change-Number: 12247
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: Zoltan Ivanfi 
Gerrit-Comment-Date: Fri, 01 Mar 2019 13:46:34 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet

2019-03-01 Thread Csaba Ringhofer (Code Review) 
Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12247 )

Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet
..


Patch Set 8:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/12247/8//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12247/8//COMMIT_MSG@39
PS8, Line 39: without conversion to UTC
> No, "pure" TIMESTAMP and TIMESTAMP WITHOUT TIME ZONE shall not be normalize
The main problem with normalizing to UTC from Impala's standpoint is the 
performance cost of the UTC->localtime conversion when reading it back. Doing 
the timezone conversion for all timestamps in all rows is very costly compared 
to other tasks during Parquet scanning.


http://gerrit.cloudera.org:8080/#/c/12247/8//COMMIT_MSG@39
PS8, Line 39: tha
> nit: the
Done


http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/hdfs-parquet-table-writer.cc
File be/src/exec/parquet/hdfs-parquet-table-writer.cc:

http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/hdfs-parquet-table-writer.cc@579
PS8, Line 579: result_
> What about deleting the member 'result_', and only have it here as a local
result_ shouldn't be a local variable, as we return a pointer to it. This is 
the expected interface by BaseColumnWriter::AppendRow() - it has no template 
for the type of the column, so it handles values with a void* that points to 
the value that should be inserted to the current row, and is expected to live 
until we step to the next row.


http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/parquet-metadata-utils.h
File be/src/exec/parquet/parquet-metadata-utils.h:

http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/parquet-metadata-utils.h@60
PS8, Line 60: Return
> nit: Returns
Done


http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/parquet-metadata-utils.cc
File be/src/exec/parquet/parquet-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/exec/parquet/parquet-metadata-utils.cc@142
PS8, Line 142: /// converted_type is not set because Impala always writes 
timestamps without UTC
> Parquet supports both UTC-normalized and timezone-agnostic timestamps, aka
As far as I know Parquet-MR does not do any timezone conversion and leaves this 
task to the caller, e.g. Hive.


http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/runtime/timestamp-value.inline.h
File be/src/runtime/timestamp-value.inline.h:

http://gerrit.cloudera.org:8080/#/c/12247/8/be/src/runtime/timestamp-value.inline.h@154
PS8, Line 154: kudu::int128_t nanos128 =
 : static_cast(unixtime_seconds) * NANOS_PER_SEC
 : + time_.fractional_seconds();
 :
 :   if (nanos128 <  std::numeric_limits::min()
 :   || nanos128 >  std::numeric_limits::max()) return 
false;
> I think we can still avoid using int128_t.
I created a ticket for benchmarking and optimizing these new functions: 
IMPALA-8268  Performance is not too important at the moment as 
parquet_timestamp_type is a development query option and should be mainly used 
to test whether the new timestamps can be read by other Hadoop components.

I think that the most costly things in this function are 
time_.fractional_seconds() and UtcToUnixTime()'s time_.total_seconds(), as 
these need int64 integer division. These could be avoided by not using 
UtcToUnixTime and converting to nanoseconds from day_ + time_ directly, but 
this has to be done carefully near the edge values.


http://gerrit.cloudera.org:8080/#/c/12247/8/testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
File 
testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test:

http://gerrit.cloudera.org:8080/#/c/12247/8/testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test@102
PS8, Line 102:  QUERY
 : create table int96_nanos (ts timestamp) stored as parquet;
 : 
 :  QUERY
 : # Insert edge values as "normal" int96 timestamps that can 
represent all values.
 : set parquet_timestamp_type=INT96_NANOS;
 : insert into int96_nanos values
> nit: you dont't need to start a new QUERY block for each query when you don
Thanks for the tip!



--
To view, visit http://gerrit.cloudera.org:8080/12247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f
Gerrit-Change-Number: 12247
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: Zoltan Ivanfi 
Gerrit-Comment-Date: Fri, 01 Mar 2019 13:35:32 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet

2019-03-01 Thread Csaba Ringhofer (Code Review) 
Hello Zoltan Borok-Nagy, Zoltan Ivanfi, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12247

to look at the new patch set (#9).

Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet
..

IMPALA-5051: Add INT64 timestamp write support in Parquet

Add query option "parquet_timestamp_type" that chooses the
Parquet type used when writing TIMESTAMP columns. This is an
experimental feature at the moment, because these types are not
widely adopted in other Hadoop components yet. For this reason
the query option is added as "development" level, and the default
behavior is not changed.

The following options can be used:
INT96_NANOS (default):
  This is the same as the old behavior, can represent any
  timestamp that can be handled by Impala.
INT64_MILLIS, INT64_MICROS:
  Can encode the whole [1400..1) range handled by Impala
  at the cost of reduced precision. Values are rounded towards
  minus infinity during writing.
INT64_NANOS:
  Can encode a reduced range without losing nanosecond precision:
  [1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807]
  Values outside this range are converted to NULLs without warning.

The change was done completely in the backend and all TIMESTAMP
columns are written using the type set in the query option.
An alternative design would have been to implement some parts
in the fronted by adding TIMESTAMP->BIGINT conversion functions
to the query plan, which would make it easier to add the possibility
of per-column setting in the future. I choose the current design
because it seemed much simpler and there are no clear plans for the
per-column setting. Most of the code will be still useful if we
decide to go the other way in the future.

All types are written without conversion to UTC (the way Impala
always wrote timestamps), and this information is expressed in the
new Parquet logical types by setting isAdjustedToUTC to false. The
old logical type (converted_type) is net set, because old readers do
not read isAdjustedToUTC, and assume that TIMESTAMP_MILLIS and
TIMESTAMP_MICROS are written in UTC. These readers can still read
int64 timestamp columns as INT_64.

Testing:
- added unit tests for new TimestampValue->int64 functions
- add EE tests for checking values / min-max stats / metadata
  written for int64 Parquet timestamps
- ran core tests

Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f
---
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/hdfs-parquet-table-writer.h
M be/src/exec/parquet/parquet-common.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.h
M be/src/runtime/timestamp-value.inline.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/debug-util.cc
M be/src/util/debug-util.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M 
testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
M tests/query_test/test_insert_parquet.py
M tests/util/get_parquet_metadata.py
18 files changed, 537 insertions(+), 69 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/12247/9
--
To view, visit http://gerrit.cloudera.org:8080/12247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f
Gerrit-Change-Number: 12247
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: Zoltan Ivanfi