[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/10813 ) Change subject: IMPALA-7163: Implement a state machine for the QueryState class .. Patch Set 13: (7 comments) http://gerrit.cloudera.org:8080/#/c/10813/13//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10813/13//COMMIT_MSG@28 PS13, Line 28: The fragment instances update the query wide query status if an error is hit nit: a few long lines here http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.h File be/src/runtime/fragment-instance-state.h: http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.h@79 PS13, Line 79: hould WaitForPrepare nit: update comment http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc File be/src/runtime/fragment-instance-state.cc: http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc@86 PS13, Line 86: if (!status.ok()) { : goto done; : } nit: one line http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc@98 PS13, Line 98: if (!status.ok()) { : goto done; : } nit: one line http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/query-state.cc File be/src/runtime/query-state.cc: http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/query-state.cc@217 PS13, Line 217: The method comment mentions that : "A state transition happens if the current state is a non-terminal state" maybe add if(IsTerminalState(old_state)) return query_status_; or add a dcheck for that otherwise we can potentially set an error/cancel state after BackendExecState::FINISHED http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/query-state.cc@390 PS13, Line 390: fis_map_.emplace(fis->instance_id(), fis); any specific reason why you moved this before you actually start up the thread? If you keep it here, then maybe update the comment above it http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/query-state.cc@435 PS13, Line 435: ReportExecStatusAux(true, thread_create_status, nullptr, true); looks like we can now return a done exec status before the started fragments complete prepare phase. Is that expected? or should we wait for prepare here. the commit message says that: "The PREPARING state blocks regardless of whether a fragment instance hit an error or not, until all the fragment instances have completed successfully or unsuccessfully" we should probably call wait for prepare before calling UpdateBackendExecState() -- To view, visit http://gerrit.cloudera.org:8080/10813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Gerrit-Change-Number: 10813 Gerrit-PatchSet: 13 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Sat, 04 Aug 2018 03:45:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7387: Set correct MIME type for JSON webpages
Anonymous Coward #168 has posted comments on this change. ( http://gerrit.cloudera.org:8080/0 ) Change subject: IMPALA-7387: Set correct MIME type for JSON webpages .. Patch Set 9: (1 comment) Patch set 11 http://gerrit.cloudera.org:8080/#/c/0/9/be/src/util/webserver.cc File be/src/util/webserver.cc: http://gerrit.cloudera.org:8080/#/c/0/9/be/src/util/webserver.cc@151 PS9, Line 151: \ > I don't think that is a standard. Relevant discussion here [1] which refers please : https://blog.csdn.net/gufachongyang02/article/details/21627539 -- To view, visit http://gerrit.cloudera.org:8080/0 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ad94343151730851a4ad01a1f4b9326a36a37ea Gerrit-Change-Number: 0 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 04 Aug 2018 02:40:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6789: disable impersonation in hive in minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonation in hive in minicluster .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/188/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 02:32:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonation in hive in minicluster
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9 to look at the new patch set (#3). Change subject: IMPALA-6789: disable impersonation in hive in minicluster .. IMPALA-6789: disable impersonation in hive in minicluster Due to bug of HADOOP-7050, users with dots in their username can't launch HiveServer2 in minicluster. To work arround this, we can set hive.server2.enable.doAs to false to disable impersonation in hive. Impala's authorization only depends on Sentry, so we can disable this without breaking any tests. This patch also quotes the group name in AuthorizationStmtTest#testShow to avoid syntax errors when group name contains dots. Test: * Build succeed with username quanlong.huang locally Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/resources/mysql-hive-site.xml.template M fe/src/test/resources/postgresql-hive-site.xml.template 3 files changed, 16 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/9/3 -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG@7 PS2, Line 7: impersonate > nit: impersonation Done http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG@11 PS2, Line 11: impersonate > same Done http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@963 PS1, Line 963: `%s` > Ah ok. Is this the only test that is broken due to this? Wondering if there Yes. Currently, it's the only place that needs fixing. I build successfully in our internal Jenkins. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 01:58:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/7 ) Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2920/ -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 04 Aug 2018 01:43:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 2: Code-Review+1 (3 comments) Patch lgtm. I'll let Phil take a pass since he helped you root cause this. http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG@7 PS2, Line 7: impersonate nit: impersonation http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG@11 PS2, Line 11: impersonate same http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@963 PS1, Line 963: `%s` > Sorry that I only mentioned this in the commit message. My username is quan Ah ok. Is this the only test that is broken due to this? Wondering if there are other places that need some fixing. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 01:34:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/187/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 01:30:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7396: unblock testing with --thread creation fault injection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11123 ) Change subject: IMPALA-7396: unblock testing with --thread_creation_fault_injection .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/186/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc9da645c449a266e3a54d0e35b18a2823019cb7 Gerrit-Change-Number: 11123 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 04 Aug 2018 01:00:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9 to look at the new patch set (#2). Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. IMPALA-6789: disable impersonate in hive in minicluster Due to bug of HADOOP-7050, users with dots in their username can't launch HiveServer2 in minicluster. To work arround this, we can set hive.server2.enable.doAs to false to disable impersonate in hive. Impala's authorization only depends on Sentry, so we can disable this without breaking any tests. This patch also quotes the group name in AuthorizationStmtTest#testShow to avoid syntax errors when group name contains dots. Test: * Build succeed with username quanlong.huang locally Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/resources/mysql-hive-site.xml.template M fe/src/test/resources/postgresql-hive-site.xml.template 3 files changed, 17 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/9/2 -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@963 PS1, Line 963: `%s` > Why this? Do you have some usernames that conflict with keywords or somethi Sorry that I only mentioned this in the commit message. My username is quanlong.huang which will have syntax error if the name is not quoted. The DOT in the name is the cause. http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template File fe/src/test/resources/postgresql-hive-site.xml.template: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template@145 PS1, Line 145: : hive.server2.enable.impersonation : Enable user impersonation for HiveServer2 : true : > This looks the same. Is it deprecated in favor of doAs? If so, this should Yes, according to https://github.com/cloudera/hive/commit/1cac04b0e739019b9da99fe916e2fe5497c4abb8 , this is the old name of hive.server2.enable.doAs http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template@198 PS1, Line 198: : hive.server2.enable.doAs : false : > Can you add a comment as to why we are explicitly disabling this. Done -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 00:58:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7396: unblock testing with --thread creation fault injection
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11123 Change subject: IMPALA-7396: unblock testing with --thread_creation_fault_injection .. IMPALA-7396: unblock testing with --thread_creation_fault_injection These are minimal fixes to allow testing new patches with the fault injection. ClientRequestState: fixing this seems somewhat involved, so disabled the fault injection for now to unblock further testing. HdfsScanNode: thread_state_.DecrementNumActive() was called on the error path before the count had actually been incremented. Testing: Ran some of the end-to-end tests for 20-30 minutes and confirmed that no Impalads crashed. start-impala-cluster.py \ --impalad_args=--thread_creation_fault_injection=true impala-py.test tests/query_test/*.py -n2 --verbose Change-Id: Idc9da645c449a266e3a54d0e35b18a2823019cb7 --- M be/src/exec/hdfs-scan-node.cc M be/src/service/client-request-state.cc 2 files changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/11123/1 -- To view, visit http://gerrit.cloudera.org:8080/11123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idc9da645c449a266e3a54d0e35b18a2823019cb7 Gerrit-Change-Number: 11123 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7387: Set correct MIME type for JSON webpages
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/0 ) Change subject: IMPALA-7387: Set correct MIME type for JSON webpages .. Patch Set 10: 21:54:06 [INFO] BUILD FAILURE 21:54:06 [ERROR] Failed to execute goal on project impala-frontend: Could not resolve dependencies for project org.apache.impala:impala-frontend:jar:0.1-SNAPSHOT: Could not transfer artifact org.apache.sentry:sentry-provider-db:jar:2.0.0-cdh6.x-20180731.151011-502571 from/to impala.cdh.repo (https://native-toolchain.s3.amazonaws.com/build/cdh_components/506967/maven): Access denied to: https://native-toolchain.s3.amazonaws.com/build/cdh_components/506967/maven/org/apache/sentry/sentry-provider-db/2.0.0-cdh6.x-SNAPSHOT/sentry-provider-db-2.0.0-cdh6.x-20180731.151011-502571.jar , ReasonPhrase:Forbidden. -> [Help 1] 21:54:06 [ERROR] 21:54:06 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 21:54:06 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 21:54:06 [ERROR] -- To view, visit http://gerrit.cloudera.org:8080/0 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ad94343151730851a4ad01a1f4b9326a36a37ea Gerrit-Change-Number: 0 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 04 Aug 2018 00:17:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@963 PS1, Line 963: `%s` Why this? Do you have some usernames that conflict with keywords or something? http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template File fe/src/test/resources/postgresql-hive-site.xml.template: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template@145 PS1, Line 145: : hive.server2.enable.impersonation : Enable user impersonation for HiveServer2 : true : This looks the same. Is it deprecated in favor of doAs? If so, this should probably be deleted. http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template@198 PS1, Line 198: : hive.server2.enable.doAs : false : Can you add a comment as to why we are explicitly disabling this. Also place it in the Kerberos section? -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 04 Aug 2018 00:15:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7387: Set correct MIME type for JSON webpages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/0 ) Change subject: IMPALA-7387: Set correct MIME type for JSON webpages .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2919/ -- To view, visit http://gerrit.cloudera.org:8080/0 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ad94343151730851a4ad01a1f4b9326a36a37ea Gerrit-Change-Number: 0 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 04 Aug 2018 00:15:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/185/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 8 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Sat, 04 Aug 2018 00:03:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2918/ -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 23:52:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/183/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 03 Aug 2018 23:46:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/184/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 7 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 03 Aug 2018 23:46:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11120 ) Change subject: IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement .. IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Reviewed-on: http://gerrit.cloudera.org:8080/11120 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M docs/topics/impala_truncate_table.xml 1 file changed, 51 insertions(+), 82 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Gerrit-Change-Number: 11120 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11120 ) Change subject: IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Gerrit-Change-Number: 11120 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 23:32:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Hello Lars Volker, Zoram Thanga, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10847 to look at the new patch set (#8). Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. IMPALA-6271: Impala daemon should log a message when it's being shut down Currently Impalad does not log any message when SIGTERM is sent to impalad to terminate or to do a graceful shut down. This change logs a message when SIGTERM is received by impalad/catalogd/statestored. This logging will assist in debugging the issues seen in the field where impalad was not gracefully shut down (some other signal was generated that led to impalad/catalogd/statestored crash). Testing: --- a) Used kill to send signals to impalad/catalogd/statestored `kill -s SIGTERM ` and see the log message is being logged in impalad/catalogd/statestored.INFO. b) Ran test_breakpad.py to check that existing breakpad functionalities are not affected. c) Ran exhaustive tests without failure. d) Added new test in test_breakpad.py to handle SIGTERM for impalad/statestored/catalogd. Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 --- M be/src/common/init.cc M be/src/util/minidump.cc M tests/custom_cluster/test_breakpad.py 3 files changed, 50 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/10847/8 -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 8 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/182/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 23:31:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Hello Lars Volker, Zoram Thanga, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10847 to look at the new patch set (#7). Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. IMPALA-6271: Impala daemon should log a message when it's being shut down Currently Impalad does not log any message when SIGTERM is sent to impalad to terminate or to do a graceful shut down. This change logs a message when SIGTERM is received by impalad/catalogd/statestored. This logging will assist in debugging the issues seen in the field where impalad was not gracefully shut down (some other signal was generated that led to impalad/catalogd/statestored crash). Testing: --- a) Used kill to send signals to impalad/catalogd/statestored `kill -s SIGTERM ` and see the log message is being logged in impalad/catalogd/statestored.INFO. b) Ran test_breakpad.py to check that existing breakpad functionalities are not affected. c) Ran exhaustive tests without failure. d) Added new test in test_breakpad.py to handle SIGTERM for impalad/statestored/catalogd. Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 --- M be/src/common/init.cc M be/src/statestore/statestored-main.cc M be/src/util/minidump.cc M tests/custom_cluster/test_breakpad.py 4 files changed, 50 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/10847/7 -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 7 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 6: (9 comments) http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc File be/src/catalog/catalogd-main.cc: http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@62 PS6, Line 62: HandlerTerm > "HandleSigTerm"? Done http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@63 PS6, Line 63: LOG(INFO) << "Caught SIGTERM daemon going down" << endl; > nit: please format this more grammatically. For example "Caught SIGTERM. Da Changed the log message and removed the endl , looks like the messages are being flushed perhaps by atexit hook as you say. Since the testcase check for the message "Caught SIGTERM" which seems to be working fine. http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@64 PS6, Line 64: exit(0); > Does this change the behavior? I.e., would we have exited with 0 before, to This exit will ensure a normal process termination, if I don't use it the process will not be terminated. http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@120 PS6, Line 120: // We want to log a message when catalogd is > nit: wrap comments at 90 chars, here and elsewhere. Done http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@125 PS6, Line 125: sigaction(SIGTERM, &action, nullptr); > why not set this earlier? seems like there could be a relatively long start Done http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/service/impalad-main.cc File be/src/service/impalad-main.cc: http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/service/impalad-main.cc@60 PS6, Line 60: // doing an exit. > This code is now copy-pasted to several files. Can you dedup it? Removed the code block and moved it to the common handler http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/service/impalad-main.cc@104 PS6, Line 104: // We want to log a message when impalad is > same here Done http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/util/minidump.cc File be/src/util/minidump.cc: http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/util/minidump.cc@101 PS6, Line 101: LOG(INFO) << "Signal received SIGUSR1" << endl; > The other messages are "Caught SIGTERM daemon going down". Can you make the Done http://gerrit.cloudera.org:8080/#/c/10847/6/tests/custom_cluster/test_breakpad.py File tests/custom_cluster/test_breakpad.py: http://gerrit.cloudera.org:8080/#/c/10847/6/tests/custom_cluster/test_breakpad.py@239 PS6, Line 239: self.assert_impalad_log_contains('INFO', 'Caught SIGTERM daemon going down', > Is there a way to also check the exit code of the processes? Checked the code for exit of the process, could not find the function where the exit of the process is checked. -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 6 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 03 Aug 2018 23:29:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11120 ) Change subject: IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/27/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Gerrit-Change-Number: 11120 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 23:22:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/181/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 23:19:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11120 ) Change subject: IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/27/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Gerrit-Change-Number: 11120 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 03 Aug 2018 23:14:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11120 Change subject: IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement .. IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 --- M docs/topics/impala_truncate_table.xml 1 file changed, 51 insertions(+), 82 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/11120/1 -- To view, visit http://gerrit.cloudera.org:8080/11120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Gerrit-Change-Number: 11120 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9 Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. IMPALA-6789: disable impersonate in hive in minicluster Due to bug of HADOOP-7050, users with dots in their username can't launch HiveServer2 in minicluster. To work arround this, we can set hive.server2.enable.doAs to false to disable impersonate in hive. Impala's authorization only depends on Sentry, so we can disable this without breaking any tests. This patch also quotes the group name in AuthorizationStmtTest#testShow to avoid syntax errors when group name contains dots. Test: * Build succeed with username quanlong.huang locally Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/resources/mysql-hive-site.xml.template M fe/src/test/resources/postgresql-hive-site.xml.template 3 files changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/9/1 -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8 to look at the new patch set (#2). Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. IMPALA-5031: Fix undefined behavior: ptr overflow In expr.add, the standard says: When an expression that has integral type is added to or subtracted from a pointer, the result has the type of the pointer operand. ... If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined. In the end-to-end tests this is triggered, and the interesting part of the backtrace is: include/c++/4.9.2/bits/stl_iterator.h:782:45: runtime error: pointer index expression with base 0x overflowed to 0xffe8 #0 __normal_iterator>::operator+(long) const stl_iterator.h:782:45 #1 MemPool::AcquireData(MemPool*, bool) runtime/mem-pool.cc:190:62 #2 RowBatch::TransferResourceOwnership(RowBatch*) runtime/row-batch.cc:444:26 #3 RowBatch::AcquireState(RowBatch*) runtime/row-batch.cc:505:8 #4 HdfsScanNode::GetNextInternal(RuntimeState*, RowBatch*, bool*) exec/hdfs-scan-node.cc:105:16 #5 HdfsScanNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/hdfs-scan-node.cc:81:19 #6 StreamingAggregationNode::GetRowsStreaming(RuntimeState*, RowBatch*) exec/streaming-aggregation-node.cc:116:51 #7 StreamingAggregationNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/streaming-aggregation-node.cc:92:41 Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 --- M be/src/runtime/mem-pool.cc 1 file changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8/2 -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7354: planner test resource estimates for more workloads
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11087 ) Change subject: IMPALA-7354: planner test resource estimates for more workloads .. Patch Set 5: I'll also let Bikram take a look before merging -- To view, visit http://gerrit.cloudera.org:8080/11087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9f836699200ea87fb03bf36abad0e23949ac26 Gerrit-Change-Number: 11087 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 22:53:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7354: planner test resource estimates for more workloads
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11087 ) Change subject: IMPALA-7354: planner test resource estimates for more workloads .. Patch Set 5: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java@273 PS5, Line 273: printBytes > My thought was that the reservation is a precise number (if the query gets not too strong an opinion on this one. just noted that in the commit message, part of this change was motivated by user perception so wanted to see more about the difference between these too. I'm fine with keeping it this way. -- To view, visit http://gerrit.cloudera.org:8080/11087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9f836699200ea87fb03bf36abad0e23949ac26 Gerrit-Change-Number: 11087 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 22:51:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/8/1/be/src/runtime/mem-pool.cc File be/src/runtime/mem-pool.cc: http://gerrit.cloudera.org:8080/#/c/8/1/be/src/runtime/mem-pool.cc@190 PS1, Line 190: vector::iterator insert_chunk = chunks_.begin() + (current_chunk_idx_ + 1); I had to stare at this for a minute. Is the bug in the case when current_chunk_idx_ is -1? I always found the current_chunk_idx_ logic here confusing... The new code seems fine, might be worth dropping a comment to explain why the parens are significant. -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 22:50:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 22:50:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7354: planner test resource estimates for more workloads
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11087 ) Change subject: IMPALA-7354: planner test resource estimates for more workloads .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java@273 PS5, Line 273: printBytes > just wondering why this one is not changed as well? My thought was that the reservation is a precise number (if the query gets any less than this, it will fail) but the estimate is an imprecise number. I was thinking that the extra precision lets planner tests catch smaller errors in the calculations. On the other hand, sub-megabyte precision isn't informative for users and we still expose and verify it at higher explain levels at plan and fragment levels: https://github.com/apache/impala/blob/master/testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test#L8 So I could go either way. Given that it jumped out at you, maybe users will find the inconsistency confusing. What do you think? -- To view, visit http://gerrit.cloudera.org:8080/11087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9f836699200ea87fb03bf36abad0e23949ac26 Gerrit-Change-Number: 11087 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 22:46:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8 Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. IMPALA-5031: Fix undefined behavior: ptr overflow In expr.add, the standard says: When an expression that has integral type is added to or subtracted from a pointer, the result has the type of the pointer operand. ... If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined. In the end-to-end tests this is triggered, and the interesting part of the backtrace is: include/c++/4.9.2/bits/stl_iterator.h:782:45: runtime error: pointer index expression with base 0x overflowed to 0xffe8 #0 __normal_iterator>::operator+(long) const stl_iterator.h:782:45 #1 MemPool::AcquireData(MemPool*, bool) runtime/mem-pool.cc:190:62 #2 RowBatch::TransferResourceOwnership(RowBatch*) runtime/row-batch.cc:444:26 #3 RowBatch::AcquireState(RowBatch*) runtime/row-batch.cc:505:8 #4 HdfsScanNode::GetNextInternal(RuntimeState*, RowBatch*, bool*) exec/hdfs-scan-node.cc:105:16 #5 HdfsScanNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/hdfs-scan-node.cc:81:19 #6 StreamingAggregationNode::GetRowsStreaming(RuntimeState*, RowBatch*) exec/streaming-aggregation-node.cc:116:51 #7 StreamingAggregationNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/streaming-aggregation-node.cc:92:41 Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 --- M be/src/runtime/mem-pool.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8/1 -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/7 ) Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/180/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 22:35:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/7 ) Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2920/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 22:30:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7354: planner test resource estimates for more workloads
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11087 ) Change subject: IMPALA-7354: planner test resource estimates for more workloads .. Patch Set 5: (2 comments) looks fine... I have one clarifying question. http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java@273 PS5, Line 273: printBytes just wondering why this one is not changed as well? http://gerrit.cloudera.org:8080/#/c/11087/5/testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test File testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test: http://gerrit.cloudera.org:8080/#/c/11087/5/testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test@25 PS5, Line 25: 66.00MB Threads=2 : Per-Host Resource Estimates: Memory=744MB this seems odd: decimal places for memory reservation but not for estimates. -- To view, visit http://gerrit.cloudera.org:8080/11087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9f836699200ea87fb03bf36abad0e23949ac26 Gerrit-Change-Number: 11087 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 22:24:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/7 ) Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 22:06:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/7 Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. IMPALA-7394: Don't print stack trace in ExpireSessions() When sessions expire, ExpireSessions() may create excessive amount of log spew due to printing of the stack traces esp when there are a massive amount of sessions expiring at the same time. This stack crawling also unnecessarily extends the critical section under the session_state_map_lock_, increasing the contention for that lock. This change converts the use of Status() to Status::Expected() to remove the unnecessary stack crawl. Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa --- M be/src/service/impala-server.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/7/1 -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] IMPALA-7390: Pin rpc-mgr-kerberized-test to localhost.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/3 ) Change subject: IMPALA-7390: Pin rpc-mgr-kerberized-test to localhost. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/3/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/3/1//COMMIT_MSG@11 PS1, Line 11: failing due to DNS resolution reasons > Does it mean in general, to run Impala with docker, one has to specify the test-with-docker creates an /etc/hosts file that looks like: 127.0.0.1 localhost That's happening because of a workaround: # Update /etc/hosts to remove the entry for the unique docker hostname, # and instead point it to 127.0.0.1. Otherwise, HttpFS returns Location: # redirects to said hostname, but the relevant datanode isn't listening # on the wildcard address. sed -e /$(hostname)/d /etc/hosts -e /127.0.0.1/s,localhost,"localhost $(hostname)," \ > /tmp/hosts Kerberos decides the hostname is dockercontainerhostname, but the IP is 127.0.0.1, and then when it looks up the reverse, it finds localhost. Interestingly, if I swap the order in /etc/hosts, this test passes, but a different test fails. (thrift-server-test, if I recall correctly.) All that to say, this is quite specific to the way test-with-docker is working (and the way we configure our minicluster). A user genuinely using docker probably would be doing different hostname anyway. The workaround above exists because tests that use webhdfs (e.g., metadata.test_ddl.TestDdlStatements.test_drop_table_with_purge) end up getting the wrong addresses. I'm going to take a little bit more of a look about why WebHDFS is different in this docker environment than not, but I don't want to go down too deep a rabbit hole there. I'll report back if I figure it out, and otherwise I'll just update the comments here a bit. -- To view, visit http://gerrit.cloudera.org:8080/3 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91003cbc86177feb7f99563f61297f7da7fabab4 Gerrit-Change-Number: 3 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 03 Aug 2018 21:14:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7387: Set correct MIME type for JSON webpages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/0 ) Change subject: IMPALA-7387: Set correct MIME type for JSON webpages .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2919/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/0 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ad94343151730851a4ad01a1f4b9326a36a37ea Gerrit-Change-Number: 0 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 21:01:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7387: Set correct MIME type for JSON webpages
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/0 ) Change subject: IMPALA-7387: Set correct MIME type for JSON webpages .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/0/9/be/src/util/webserver.cc File be/src/util/webserver.cc: http://gerrit.cloudera.org:8080/#/c/0/9/be/src/util/webserver.cc@151 PS9, Line 151: \ > Compatible Chinese should add charset=utf-8 I don't think that is a standard. Relevant discussion here [1] which refers to [2] that says "Note: No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients." Also, I tried running a sample query in chinese charset and I can see it fine in my firefox browser without UTF-8 charset in the HTTP header. Here is a screenshot [3] [1] https://github.com/request/request/issues/383 [2] http://www.iana.org/assignments/media-types/application/json [3] https://pasteboard.co/HxvPg1X.png -- To view, visit http://gerrit.cloudera.org:8080/0 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ad94343151730851a4ad01a1f4b9326a36a37ea Gerrit-Change-Number: 0 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 21:00:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11005 ) Change subject: [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/179/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I669db217f86db5ff2802335e7b1ae8027ea7161c Gerrit-Change-Number: 11005 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 20:48:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 4: clang-tidy failed doing maven compilation related to Sentry. [ERROR] location: class org.apache.impala.util.SentryPolicyService.SentryServiceClient [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/util/SentryUtil.java:[25,52] package org.apache.sentry.provider.db.service.thrift does not exist [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/util/SentryUtil.java:[26,52] package org.apache.sentry.provider.db.service.thrift does not exist [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/util/SentryUtil.java:[50,37] cannot find symbol [ERROR] symbol: class SentryPolicyServiceClient [ERROR] location: class org.apache.impala.util.SentryUtil [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/util/SentryUtil.java:[50,14] cannot find symbol [ERROR] symbol: class TSentryRole [ERROR] location: class org.apache.impala.util.SentryUtil [ERROR] -> [Help 1] [ERROR] This can't possibly have anything to do with that, so I'm re-running. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 20:40:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 5: Code-Review+2 Carry. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 5 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 20:40:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 20:33:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2918/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 20:33:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2917/ -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 20:28:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries
Pooja Nilangekar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/11005 ) Change subject: [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries .. [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries This change ensures that coordinator::UpdateFilter is executed only for queries which are in the EXECUTING state. Additionally, it also guarantees that coordinator::ReleaseExecResources is executed only when no other thread is executing UpdateFilter. Testing: Ran all back-end tests. [TODO] Add a test to verify that this change guarantees that UpdateFilter and ReleaseExecResources never execute concurrently. Change-Id: I669db217f86db5ff2802335e7b1ae8027ea7161c --- M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h 2 files changed, 45 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/11005/7 -- To view, visit http://gerrit.cloudera.org:8080/11005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I669db217f86db5ff2802335e7b1ae8027ea7161c Gerrit-Change-Number: 11005 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/11005 ) Change subject: [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.h File be/src/runtime/coordinator.h: http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.h@301 PS6, Line 301: /// Synchronizes updates to the filter_routing_table_. : SpinLock filter_update_lock_; : : /// Protects filter_routing_table_. : /// Usage pattern: : /// 1. To update filter_routing_table_: Acquire shared access on filter_lock_ and : ///upgrade to exclusive access by subsequently acquiring filter_update_lock_. : /// 2. To read, initialize/destroy filter_routing_table: Directly acquire exclusive : ///access on filter_lock_. : boost::shared_mutex filter_lock_; : > maybe write a consolidated comment to better explain the need for two locks Done http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.h@334 PS6, Line 334: : /// Called when the query is done executing due to reaching EOS or client : /// cancellation. If 'exec_state_' != EXECUTING, does nothing. > nit: "Caller must have exclusive access to filter_lock_" Done http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.cc@265 PS6, Line 265: lock_guard lock(filter_lock_); : for (const FragmentExecParams& fragment_params: schedule_.fragment_exec_params()) { : int num_instances = fragment_params.instance_exec_par > fyi: this method is called in exec before any fragments are started. Also, Make sense. I have removed the comment. I have kept the lock_guard for the sake of keeping a consistent code flow across all accesses. Acquiring the lock should not be expensive if there is no contention on it. Or would you recommend entirely removing it form here? -- To view, visit http://gerrit.cloudera.org:8080/11005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I669db217f86db5ff2802335e7b1ae8027ea7161c Gerrit-Change-Number: 11005 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 20:24:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2916/ -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 20:23:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 19:47:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. IMPALA-7377: Update Sentry for the object ownership feature Update CDH_BUILD_NUMBER to 506967, especially for the new Sentry update with the object owership feature (IMPALA-7075). This patch updates the Sentry package names and Sentry Maven dependencies. Testing: - Ran core tests Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Reviewed-on: http://gerrit.cloudera.org:8080/11094 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/util/SentryPolicyService.java M fe/src/main/java/org/apache/impala/util/SentryProxy.java M fe/src/main/java/org/apache/impala/util/SentryUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java 6 files changed, 38 insertions(+), 20 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/178/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 5 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 03 Aug 2018 19:23:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10813 ) Change subject: IMPALA-7163: Implement a state machine for the QueryState class .. Patch Set 12: (3 comments) http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc File be/src/runtime/fragment-instance-state.cc: http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc@107 PS13, Line 107: yS == Apparently WAITING_FOR_CODEGEN < WAITING_FOR_PREPARE http://gerrit.cloudera.org:8080/#/c/10813/12/be/src/runtime/query-state.h File be/src/runtime/query-state.h: http://gerrit.cloudera.org:8080/#/c/10813/12/be/src/runtime/query-state.h@244 PS12, Line 244: Status WaitForFinish(); > We call it towards the end of StartFInstances(). We update the query state Oh right. Missed that. http://gerrit.cloudera.org:8080/#/c/10813/12/tests/failure/test_failpoints.py File tests/failure/test_failpoints.py: http://gerrit.cloudera.org:8080/#/c/10813/12/tests/failure/test_failpoints.py@162 PS12, Line 162: i = 0 > When I remove this, it gives me this error: It should work with for i in range(50): -- To view, visit http://gerrit.cloudera.org:8080/10813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Gerrit-Change-Number: 10813 Gerrit-PatchSet: 12 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 03 Aug 2018 19:10:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 5: (12 comments) http://gerrit.cloudera.org:8080/#/c/11052/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11052/4//COMMIT_MSG@9 PS4, Line 9: After this patch, the statestore keeps track of the last successful : heartbeat timestamp which is exposed as a subscriber metric on the : statestore debug page. Also adds a monitoring thread that : periodically checks the last heartbeat timestamp for all the : subscribers and l > After this patch, the statestore keeps track of the last successful heartbe Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h File be/src/statestore/statestore.h: http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@418 PS4, Line 418: ms); > milliseconds or ms Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@442 PS4, Line 442: /// The timestamp of the last successful heartbeat in milliseconds. A timestamp much : /// older than the heartbeat frequency implies an unr > The timestamp of the last successful heartbeat in milliseconds. Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@444 PS4, Line 444: 4_t last_heartbe > heartbeat frequency Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@532 PS4, Line 532: :unique_ptr heartbeat_monitoring_thread_; > Thread that monitors the heartbeats of all subscribers. Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@684 PS4, Line 684: rtbeat_monitoring_frequency_ms mi > the heartbeats of all subscribers every Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@686 PS4, Line 686: > that Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@686 PS4, Line 686: : [[noreturn]] void MonitorSub > subscriber's Id. Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.cc File be/src/statestore/statestore.cc: http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.cc@424 PS4, Line 424: DCHECK(metrics != NULL); : metrics_ = metrics; : num_subscribers_metric_ = metrics->AddGauge(STATESTORE_LIVE_SUBSCRIBERS, 0); > move this to Init() and return status Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.cc@1003 PS4, Line 1003: s, > $0 Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.cc@1004 PS4, Line 1004: > nit: add a space after comma Done http://gerrit.cloudera.org:8080/#/c/11052/4/www/statestore_subscribers.tmpl File www/statestore_subscribers.tmpl: http://gerrit.cloudera.org:8080/#/c/11052/4/www/statestore_subscribers.tmpl@31 PS4, Line 31: Last heartbeat timestamp > nit: add " (ms)" The timestamp is printed in the following format: -mm-dd hh:mm:ss.ms -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 5 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 03 Aug 2018 18:47:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Pooja Nilangekar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. IMPALA-6644: Add last heartbeat timestamp into Statestore metric After this patch, the statestore keeps track of the last successful heartbeat timestamp which is exposed as a subscriber metric on the statestore debug page. Also adds a monitoring thread that periodically checks the last heartbeat timestamp for all the subscribers and logs the IDs of those that have not been updated since the last periodic check. Testing: Manually inspected the Web UI and statestore logs to verify that the last heartbeat timestamp for each subscriber is updated periodically. Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M www/statestore_subscribers.tmpl 3 files changed, 67 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/11052/5 -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 5 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11064 ) Change subject: IMPALA-7362: Add query option to set timezone .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Gerrit-Change-Number: 11064 Gerrit-PatchSet: 9 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:45:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11064 ) Change subject: IMPALA-7362: Add query option to set timezone .. IMPALA-7362: Add query option to set timezone This change adds a new query option "timezone" which defines the timezone used for utc<->local conversions. The main goal is to simplify testing, but I think that some users may also find it useful so it is added as a "general" query option. Examples: set timezone=UTC; set timezone="Europe/Budapest" The timezones are validated, but as query options are not sent to the coordinator immediately, the error checking will only happen when running a query. Leading/trailing " and 'characters are stripped because the / character cannot be entered unquoted in some contexts. Currently the timezone has effect in the following cases: -function now() -conversions between unix time and timestamp if flag use_local_tz_for_unix_timestamp_conversions is true -reading parquet timestamps written by Hive if flag convert_legacy_hive_parquet_utc_timestamps is true In the near future Parquet timestamps's isAdjustedToUTC property will be supported, which will decide whether to do utc->local conversion on a per file+column basis. This conversion will be also affected. Testing: - Extended test_local_tz_conversion.py to actually test utc<->local conversion. Until now the effect of flag use_local_tz_for_unix_timestamp_conversions was practically untested. - Added a shell test to check that the default of the query option is the system's timezone. - Added a shell test to check timezone validation. Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Reviewed-on: http://gerrit.cloudera.org:8080/11064 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A testdata/workloads/functional-query/queries/QueryTest/local-timestamp-functions.test M tests/custom_cluster/test_local_tz_conversion.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py 9 files changed, 159 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Gerrit-Change-Number: 11064 Gerrit-PatchSet: 10 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7217: Incorrect UPDATE/DELETE authorization privilege
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10966 ) Change subject: IMPALA-7217: Incorrect UPDATE/DELETE authorization privilege .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/10966/4/fe/src/main/java/org/apache/impala/analysis/FromClause.java File fe/src/main/java/org/apache/impala/analysis/FromClause.java: http://gerrit.cloudera.org:8080/#/c/10966/4/fe/src/main/java/org/apache/impala/analysis/FromClause.java@70 PS4, Line 70: registers clearer: Analyze this statement after registering the specified privileges. follow-up question here: would a map of table-refs to privs be useful for cases such as the delete test? the map can be used as an override, e.g., 'privilege' is applied unless there's an override. however, given the usage in ModifyStmt, perhaps targeting the specific table in the operation as suggested seems easier/clearer. http://gerrit.cloudera.org:8080/#/c/10966/4/fe/src/main/java/org/apache/impala/analysis/TableRef.java File fe/src/main/java/org/apache/impala/analysis/TableRef.java: http://gerrit.cloudera.org:8080/#/c/10966/4/fe/src/main/java/org/apache/impala/analysis/TableRef.java@629 PS4, Line 629: public void setPrivilege(Privilege privilege) { this.priv_ = privilege; } allowed to be null? -- To view, visit http://gerrit.cloudera.org:8080/10966 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69d451f727a7df6c41166a15cf1ed6f5334dc739 Gerrit-Change-Number: 10966 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 17:34:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4690: [DOCS] More content for CONV()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11075 ) Change subject: IMPALA-4690: [DOCS] More content for CONV() .. Patch Set 4: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/26/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 Gerrit-Change-Number: 11075 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Fri, 03 Aug 2018 17:27:19 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11080 ) Change subject: [DOCS] Dedicated coordinators and executors explained .. Patch Set 4: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/25/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:20:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2917/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 17:18:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 17:18:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 17:15:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4690: [DOCS] More content for CONV()
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/11075 ) Change subject: IMPALA-4690: [DOCS] More content for CONV() .. Patch Set 4: > I think you may have forgotten to upload PS4. Oops, sorry! -- To view, visit http://gerrit.cloudera.org:8080/11075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 Gerrit-Change-Number: 11075 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Fri, 03 Aug 2018 17:07:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4690: [DOCS] More content for CONV()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11075 ) Change subject: IMPALA-4690: [DOCS] More content for CONV() .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/26/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 Gerrit-Change-Number: 11075 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Fri, 03 Aug 2018 17:07:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4690: [DOCS] More content for CONV()
Hello Jim Apple, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11075 to look at the new patch set (#4). Change subject: IMPALA-4690: [DOCS] More content for CONV() .. IMPALA-4690: [DOCS] More content for CONV() Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 --- M docs/topics/impala_math_functions.xml 1 file changed, 54 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/11075/4 -- To view, visit http://gerrit.cloudera.org:8080/11075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 Gerrit-Change-Number: 11075 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple
[native-toolchain-CR] IMPALA-7364: Add RapidJson 1.1.0
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11092 ) Change subject: IMPALA-7364: Add RapidJson 1.1.0 .. Patch Set 3: It should be uploaded. LMK if you have any issues with it. -- To view, visit http://gerrit.cloudera.org:8080/11092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie98462fde43c97101e95d0761be1594bc2636859 Gerrit-Change-Number: 11092 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/11080 ) Change subject: [DOCS] Dedicated coordinators and executors explained .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11080/3/docs/impala_keydefs.ditamap File docs/impala_keydefs.ditamap: http://gerrit.cloudera.org:8080/#/c/11080/3/docs/impala_keydefs.ditamap@10960 PS3, Line 10960: impala_dedicated_coordinatori.xml > A typo here which caused the merge to fail. Done -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:03:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11080 ) Change subject: [DOCS] Dedicated coordinators and executors explained .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/25/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:02:20 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Hello Greg Rahn, Fredy Wijaya, Alan Choi, Philip Zeyliger, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11080 to look at the new patch set (#4). Change subject: [DOCS] Dedicated coordinators and executors explained .. [DOCS] Dedicated coordinators and executors explained Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 --- M docs/impala.ditamap M docs/impala_keydefs.ditamap A docs/topics/impala_dedicated_coordinator.xml M docs/topics/impala_scalability.xml 4 files changed, 631 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/11080/4 -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 17:00:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2916/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 17:00:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6034: Add scanned bytes limits per query
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11081 ) Change subject: IMPALA-6034: Add scanned bytes limits per query .. Patch Set 1: Zoltan, would be good if you could review this one given you've been working on the I/O layer. I think Bikram wanted some more eyes on it. -- To view, visit http://gerrit.cloudera.org:8080/11081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e85f80b70b3fce47e637e9322ed0316ee84f6a9 Gerrit-Change-Number: 11081 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 16:59:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 3: Code-Review+2 Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:57:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/177/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:56:34 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11080 ) Change subject: [DOCS] Dedicated coordinators and executors explained .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11080/3/docs/impala_keydefs.ditamap File docs/impala_keydefs.ditamap: http://gerrit.cloudera.org:8080/#/c/11080/3/docs/impala_keydefs.ditamap@10960 PS3, Line 10960: impala_dedicated_coordinatori.xml A typo here which caused the merge to fail. [gen-list] [DOTX008E][ERROR] File 'file:/home/ubuntu/Impala/docs/topics/impala_dedicated_coordinatori.xml' does not exist or cannot be loaded. -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 16:49:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7390: Pin rpc-mgr-kerberized-test to localhost.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/3 ) Change subject: IMPALA-7390: Pin rpc-mgr-kerberized-test to localhost. .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/3/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/3/1//COMMIT_MSG@11 PS1, Line 11: failing due to DNS resolution reasons Does it mean in general, to run Impala with docker, one has to specify the hostname ? Is there anything we can do at this line (https://github.com/apache/impala/blob/master/be/src/common/init.cc#L191) to fix the issue ? http://gerrit.cloudera.org:8080/#/c/3/1/be/src/rpc/rpc-mgr-kerberized-test.cc File be/src/rpc/rpc-mgr-kerberized-test.cc: http://gerrit.cloudera.org:8080/#/c/3/1/be/src/rpc/rpc-mgr-kerberized-test.cc@86 PS1, Line 86: FLAGS_hostname = "localhost"; nit: may help to add a comment on why InitCommonRuntime() doesn't set this up correctly. -- To view, visit http://gerrit.cloudera.org:8080/3 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91003cbc86177feb7f99563f61297f7da7fabab4 Gerrit-Change-Number: 3 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 03 Aug 2018 16:48:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4690: [DOCS] More content for CONV()
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11075 ) Change subject: IMPALA-4690: [DOCS] More content for CONV() .. Patch Set 3: I think you may have forgotten to upload PS4. -- To view, visit http://gerrit.cloudera.org:8080/11075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 Gerrit-Change-Number: 11075 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Fri, 03 Aug 2018 16:46:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7203. Support UDFs in LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11053 ) Change subject: IMPALA-7203. Support UDFs in LocalCatalog .. Patch Set 3: (6 comments) http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java File fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java: http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@75 PS3, Line 75: If the value is 'null', indicates that the function exists but :* has not been loaded yet. seems to have been replaced by the bit in FunctionOverloads that specifies if loading is needed. perhaps stale? http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@190 PS3, Line 190: private void loadFunctionNames() { lemme see if I've got this correct. each query has its own LocalCatalog, therefore its own LocalDb. the provider has a lifetime that spans all LocalCatalogs. As a result, any query that uses any udf will wind up fetching all function names. a caching provider can help with this. http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java: http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@64 PS3, Line 64: Load Retrieve? Unclear if this has a side-effect. http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@69 PS3, Line 69: from the HMS remove since its specific for the provider implementation? http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@71 PS3, Line 71: org.apache.hadoop.hive.metastore.api. remove the fully qualified name for consistency with the other api's here. http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/util/FunctionUtils.java File fe/src/main/java/org/apache/impala/util/FunctionUtils.java: http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/util/FunctionUtils.java@84 PS3, Line 84: // TODO(todd): cache these jars based on the mtime and file ID of the : // remote JAR? Can we share a cache with the backend? > agreed, we should re-use FeSupport CacheJar and related methods. however, a clarification the code that was refactored seems to be used on the catalogd end. the localdb otoh lives in the impalad. in that case, there's a lib-cache available so we should use it so that we avoid copying jars around multiple times to the same host when functions are used in the frontend and the backend. this redundancy will only be there when an impalad is both a coordinator and an executor. while this should be less likely, the combo is still supported. -- To view, visit http://gerrit.cloudera.org:8080/11053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6130d07b9c641525382a618a9f8da048c7ae75ed Gerrit-Change-Number: 11053 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 16:41:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 3: Code-Review+1 Thanks! I'd like someone who's more of an expert on docker to give this a final +2. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:40:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:22:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2915/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:22:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11106/2/docker/test-with-docker.py File docker/test-with-docker.py: http://gerrit.cloudera.org:8080/#/c/11106/2/docker/test-with-docker.py@506 PS2, Line 506: # Ubuntu16.04 machines use an absolute symlink here, : # e.g., /etc/localtime -> /usr/share/zoneinfo/America/Los_Angeles : # but CentOS7 uses a relative one, e.g., /etc/localtime -> ../usr/share/zoneinfo/UTC, : # so we use "in" rather than "startswith" > Thaks for the explanation. Yep, you're right. I was still thinking I was using readlink here. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:20:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Hello Laszlo Gaal, Attila Jeges, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11106 to look at the new patch set (#3). Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. IMPALA-7385: Fix test-with-docker errors having to do with time zones. ExprTest.TimestampFunctions, query_test.test_scanners.TestOrc.test_type_conversions, and query_test.test_queries.TestHdfsQueries.test_hdfs_scan_node were all failing when using test-with-docker with mismatched dates. As it turns out, there is code that calls readlink(/etc/localtime) and parses the output to identify the current timezone name. This is described in localtime(5) on Ubuntu16: It should be an absolute or relative symbolic link pointing to /usr/share/zoneinfo/, followed by a timezone identifier such as "Europe/Berlin" or "Etc/UTC". ... Because the timezone identifier is extracted from the symlink target name of /etc/localtime, this file may not be a normal file or hardlink." To honor this requirement, and to make the tests pass, I re-jiggered how I pass the time zone information from the host into the container. The previously failing tests now pass. Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f --- M docker/entrypoint.sh M docker/test-with-docker.py 2 files changed, 17 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/11106/3 -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:19:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7096: restore scanner thread memory heuristics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11103 ) Change subject: IMPALA-7096: restore scanner thread memory heuristics .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/176/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9907fa8c4d2b0b85f67f4f160899c1c258ad82b Gerrit-Change-Number: 11103 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 16:17:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7096: restore scanner thread memory heuristics
Hello Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11103 to look at the new patch set (#5). Change subject: IMPALA-7096: restore scanner thread memory heuristics .. IMPALA-7096: restore scanner thread memory heuristics This restores some of the heuristics removed in IMPALA-4835 that can help scans from hitting OOM conditions. The heuristics are implemented at the query level rather than in each scan node in isolation. Introduce a ScannerMemLimiter class that belongs to the QueryState that tracks the amount of memory estimated to be consumed for all scanner threads running for the query on the current backend. Also check soft memory limits to see if scanner threads should be started or the current scanner thread should stop. The long-term plan is to switch to the MT scan node implementations. When that happens this code can be removed. In the meantime this code is imperfect but will help avoid OOM in many scenarios. Testing: Added regression tests for HDFS and Kudu where we previously could run out of memory with a low mem_limit. Change-Id: Ib9907fa8c4d2b0b85f67f4f160899c1c258ad82b --- M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scan-node.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/runtime/CMakeLists.txt M be/src/runtime/query-state.cc M be/src/runtime/query-state.h A be/src/runtime/scanner-mem-limiter.cc A be/src/runtime/scanner-mem-limiter.h A testdata/workloads/functional-query/queries/QueryTest/hdfs-scanner-thread-mem-scaling.test A testdata/workloads/functional-query/queries/QueryTest/kudu-scan-mem-usage.test M tests/query_test/test_mem_usage_scaling.py 15 files changed, 439 insertions(+), 49 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/11103/5 -- To view, visit http://gerrit.cloudera.org:8080/11103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib9907fa8c4d2b0b85f67f4f160899c1c258ad82b Gerrit-Change-Number: 11103 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7096: restore scanner thread memory heuristics
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11103 ) Change subject: IMPALA-7096: restore scanner thread memory heuristics .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/11103/4/be/src/exec/hdfs-scan-node.cc File be/src/exec/hdfs-scan-node.cc: http://gerrit.cloudera.org:8080/#/c/11103/4/be/src/exec/hdfs-scan-node.cc@59 PS4, Line 59: const int SCANNER_THREAD_MEM_USAGE = 32 * 1024 * 1024; Make configurable? http://gerrit.cloudera.org:8080/#/c/11103/4/be/src/runtime/scanner-mem-limiter.h File be/src/runtime/scanner-mem-limiter.h: http://gerrit.cloudera.org:8080/#/c/11103/4/be/src/runtime/scanner-mem-limiter.h@26 PS4, Line 26: #include "common/atomic.h" Not needed -- To view, visit http://gerrit.cloudera.org:8080/11103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9907fa8c4d2b0b85f67f4f160899c1c258ad82b Gerrit-Change-Number: 11103 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 15:43:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 ) Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/175/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 15:27:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7258: Support querying HBase tables in LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11079 ) Change subject: IMPALA-7258: Support querying HBase tables in LocalCatalog .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cc94bc38861443de5a375b7e63d29215e0ca899 Gerrit-Change-Number: 11079 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 15:15:20 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 ) Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. Patch Set 2: (4 comments) "It seems that parquet.thrift isn't just updated to contain the timestamp logical type, but a couple of unrelated things are added as well." Yes, I have updated it to a newer version from https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift which contains several unrelated changes. I have modified the commit message to reflect this more clearly. http://gerrit.cloudera.org:8080/#/c/11057/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11057/1//COMMIT_MSG@19 PS1, Line 19: > nit: 'a solution' Done http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-readers.cc File be/src/exec/parquet-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-readers.cc@1657 PS1, Line 1657: Create > Here you use the word "Create", while for TYPE_DECIMAL we use the word "Get Thanks for noticing the inconsistency - I prefer "Create". http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-readers.cc@1690 PS1, Line 1690: null > nit: Please use nullptr. Done http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-stats.inline.h File be/src/exec/parquet-column-stats.inline.h: http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-stats.inline.h@158 PS1, Line 158: data + size, size, result) == -1) { : return false; > nit: too much indentation Done -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 14:57:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 ) Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/11057/2/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/11057/2/tests/query_test/test_scanners.py@751 PS2, Line 751: " flake8: E128 continuation line under-indented for visual indent http://gerrit.cloudera.org:8080/#/c/11057/2/tests/query_test/test_scanners.py@762 PS2, Line 762: " flake8: E128 continuation line under-indented for visual indent -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 14:53:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Hello Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11057 to look at the new patch set (#2). Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet Changes: - parquet.thrift is updated to a newer version which contains the timestamp logical type. - INT64 columns with converted types TIMESTAMP_MILLIS and TIMESTAMP_MICROS can be read as TIMESTAMP. - If the logical type is timestamp, then the type will contain the information whether the UTC->local conversion is necessary. This feature is only supported for the new timestamp types, so INT96 timestamps must still use flag convert_legacy_hive_parquet_utc_timestamps. TODOs: - the tests only work if local time is CET - a solution would be to create a custom cluster test which sets the local time zone, but I would prefer to solve this without restarting the cluster - it would be better to test with more parquet files (for example test both plain and dictionary encoding). - CREATE TABLE LIKE PARQUET should be also tested with the new types Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 --- M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M be/src/exec/parquet-column-stats.inline.h M be/src/exec/parquet-common.h M be/src/exec/parquet-metadata-utils.cc M common/thrift/parquet.thrift M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java M testdata/data/README A testdata/data/int64_timestamp_plain.parquet A testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test M tests/query_test/test_scanners.py 11 files changed, 399 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11057/2 -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11064 ) Change subject: IMPALA-7362: Add query option to set timezone .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Gerrit-Change-Number: 11064 Gerrit-PatchSet: 9 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 14:31:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11064 ) Change subject: IMPALA-7362: Add query option to set timezone .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2914/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Gerrit-Change-Number: 11064 Gerrit-PatchSet: 9 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 14:31:42 + Gerrit-HasComments: No