[Impala-ASF-CR] IMPALA-7340. Only send necessary fields in THdfsPartition
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11026 ) Change subject: IMPALA-7340. Only send necessary fields in THdfsPartition .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97e8402efdfdeea06463bb71a40ebb6abd1f11f0 Gerrit-Change-Number: 11026 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 05:38:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7296: bytes limit for row batch queue
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10977 ) Change subject: IMPALA-7296: bytes limit for row batch queue .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h File be/src/util/blocking-queue.h: http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@117 PS5, Line 117: ElemBytesFn()(*out); > It's a bit unfortunate that each element goes through two indirect function Actually, this may not actually be an indirect call after all, depending on how the compiler ends up generating the code. May be safe to double check that's actually the case with the compiled code. -- To view, visit http://gerrit.cloudera.org:8080/10977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa06d1d8da2a6d101efda08f620c0bf84a71e681 Gerrit-Change-Number: 10977 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 25 Jul 2018 05:37:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6335. Allow most shell tests to run in parallel
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11045 ) Change subject: IMPALA-6335. Allow most shell tests to run in parallel .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2863/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 Gerrit-Change-Number: 11045 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Comment-Date: Wed, 25 Jul 2018 05:22:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 05:22:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7342: Add initial support for user-level permissions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11039 ) Change subject: IMPALA-7342: Add initial support for user-level permissions .. Patch Set 8: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/52/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I07e0d46d2e50d35bd64ee573b5aa4b779eb9e62f Gerrit-Change-Number: 11039 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Jul 2018 05:20:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7342: Add initial support for user-level permissions
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/11039 ) Change subject: IMPALA-7342: Add initial support for user-level permissions .. IMPALA-7342: Add initial support for user-level permissions This patch refactors the authorization code in prepartion to add initial support for for user-level permissions (IMPALA-6794) and object ownership (IMPALA-7075). It introduces the notion of Principal to store either role or user. The authorization tests are updated to run the tests with user and role permissions. Testing: - Update authorization tests - Ran core tests Change-Id: I07e0d46d2e50d35bd64ee573b5aa4b779eb9e62f --- M be/src/catalog/catalog-util.cc M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/CreateDropRoleStmt.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ShowGrantRoleStmt.java M fe/src/main/java/org/apache/impala/catalog/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/Principal.java R fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java D fe/src/main/java/org/apache/impala/catalog/Role.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/util/SentryPolicyService.java M fe/src/main/java/org/apache/impala/util/SentryProxy.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java 20 files changed, 885 insertions(+), 553 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11039/8 -- To view, visit http://gerrit.cloudera.org:8080/11039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I07e0d46d2e50d35bd64ee573b5aa4b779eb9e62f Gerrit-Change-Number: 11039 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7257. Support Kudu tables in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10912 ) Change subject: IMPALA-7257. Support Kudu tables in LocalCatalog .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2862/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd Gerrit-Change-Number: 10912 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 05:19:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10914 ) Change subject: IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ae47a0b58022ed77abe51d2596c2b1d0111fae3 Gerrit-Change-Number: 10914 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 05:19:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: right-shifting too far
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11047 ) Change subject: IMPALA-5031: Fix undefined behavior: right-shifting too far .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/50/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5be92912198af2a5e00053ceb9a4fab43ca6bb8 Gerrit-Change-Number: 11047 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Jul 2018 04:51:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7342: Add initial support for user-level permissions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11039 ) Change subject: IMPALA-7342: Add initial support for user-level permissions .. Patch Set 7: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/51/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I07e0d46d2e50d35bd64ee573b5aa4b779eb9e62f Gerrit-Change-Number: 11039 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Jul 2018 04:26:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7342: Add initial support for user-level permissions
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11039 Change subject: IMPALA-7342: Add initial support for user-level permissions .. IMPALA-7342: Add initial support for user-level permissions This patch refactors the authorization code in prepartion to add initial support for for user-level permissions (IMPALA-6794) and object ownership (IMPALA-7075). It introduces the notion of Principal to store either role or user. The authorization tests are updated to run the tests with user and role permissions. Testing: - Update authorization tests - Ran core tests Change-Id: I07e0d46d2e50d35bd64ee573b5aa4b779eb9e62f --- M be/src/catalog/catalog-util.cc M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/CreateDropRoleStmt.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ShowGrantRoleStmt.java M fe/src/main/java/org/apache/impala/catalog/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/Principal.java R fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java D fe/src/main/java/org/apache/impala/catalog/Role.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/util/SentryPolicyService.java M fe/src/main/java/org/apache/impala/util/SentryProxy.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java 20 files changed, 886 insertions(+), 553 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11039/7 -- To view, visit http://gerrit.cloudera.org:8080/11039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I07e0d46d2e50d35bd64ee573b5aa4b779eb9e62f Gerrit-Change-Number: 11039 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: right-shifting too far
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11047 to look at the new patch set (#2). Change subject: IMPALA-5031: Fix undefined behavior: right-shifting too far .. IMPALA-5031: Fix undefined behavior: right-shifting too far In expr.shift, the C++ standard says of right shifts: The behavior is undefined if the right operand is negative, or greater than or equal to the length in bits of the promoted left operand. In HdfsAvroScannerTest.DecimalTest, this is triggered, and the interesting part of the backtrace is: exec/hdfs-avro-scanner-ir.cc:272:18: runtime error: shift exponent 32 is too large for 32-bit type 'int32_t' (aka 'int') #0 0x1786f65 in HdfsAvroScanner::ReadAvroDecimal(int, unsigned char**, unsigned char*, bool, void*, MemPool*) exec/hdfs-avro-scanner-ir.cc:272:18 #1 0x1617778 in void HdfsAvroScannerTest::TestReadAvroType, bool (HdfsAvroScanner::*)(int, unsigned char**, unsigned char*, bool, void*, MemPool*), unsigned long>(bool (HdfsAvroScanner::*)(int, unsigned char**, unsigned char*, bool, void*, MemPool*), unsigned long, unsigned char*, long, DecimalValue, int, TErrorCode::type) exec/hdfs-avro-scanner-test.cc:88:20 #2 0x1605705 in void HdfsAvroScannerTest::TestReadAvroDecimal(unsigned char*, long, DecimalValue, int, TErrorCode::type) exec/hdfs-avro-scanner-test.cc:184:5 Change-Id: Ic5be92912198af2a5e00053ceb9a4fab43ca6bb8 --- M be/src/exec/hdfs-avro-scanner-ir.cc 1 file changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/11047/2 -- To view, visit http://gerrit.cloudera.org:8080/11047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic5be92912198af2a5e00053ceb9a4fab43ca6bb8 Gerrit-Change-Number: 11047 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: right-shifting too far
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11047 ) Change subject: IMPALA-5031: Fix undefined behavior: right-shifting too far .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/50/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5be92912198af2a5e00053ceb9a4fab43ca6bb8 Gerrit-Change-Number: 11047 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Jul 2018 04:19:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: right-shifting too far
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11047 ) Change subject: IMPALA-5031: Fix undefined behavior: right-shifting too far .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/49/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5be92912198af2a5e00053ceb9a4fab43ca6bb8 Gerrit-Change-Number: 11047 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Jul 2018 04:16:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: right-shifting too far
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11047 ) Change subject: IMPALA-5031: Fix undefined behavior: right-shifting too far .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/49/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5be92912198af2a5e00053ceb9a4fab43ca6bb8 Gerrit-Change-Number: 11047 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Jul 2018 03:58:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: right-shifting too far
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11047 Change subject: IMPALA-5031: Fix undefined behavior: right-shifting too far .. IMPALA-5031: Fix undefined behavior: right-shifting too far In expr.shift, the C++ standard says of right shifts: The behavior is undefined if the right operand is negative, or greater than or equal to the length in bits of the promoted left operand. In HdfsAvroScannerTest.DecimalTest, this is triggered, and the interesting part of the backtrace is: exec/hdfs-avro-scanner-ir.cc:272:18: runtime error: shift exponent 32 is too large for 32-bit type 'int32_t' (aka 'int') #0 0x1786f65 in HdfsAvroScanner::ReadAvroDecimal(int, unsigned char**, unsigned char*, bool, void*, MemPool*) exec/hdfs-avro-scanner-ir.cc:272:18 #1 0x1617778 in void HdfsAvroScannerTest::TestReadAvroType, bool (HdfsAvroScanner::*)(int, unsigned char**, unsigned char*, bool, void*, MemPool*), unsigned long>(bool (HdfsAvroScanner::*)(int, unsigned char**, unsigned char*, bool, void*, MemPool*), unsigned long, unsigned char*, long, DecimalValue, int, TErrorCode::type) exec/hdfs-avro-scanner-test.cc:88:20 #2 0x1605705 in void HdfsAvroScannerTest::TestReadAvroDecimal(unsigned char*, long, DecimalValue, int, TErrorCode::type) exec/hdfs-avro-scanner-test.cc:184:5 Change-Id: Ic5be92912198af2a5e00053ceb9a4fab43ca6bb8 --- M be/src/exec/hdfs-avro-scanner-ir.cc 1 file changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/11047/1 -- To view, visit http://gerrit.cloudera.org:8080/11047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic5be92912198af2a5e00053ceb9a4fab43ca6bb8 Gerrit-Change-Number: 11047 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: memset NULL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11042 ) Change subject: IMPALA-5031: Fix undefined behavior: memset NULL .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2861/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I804f642f4be3b74c24f871f656c5147ee226d2c8 Gerrit-Change-Number: 11042 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 25 Jul 2018 03:29:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: memset NULL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11042 ) Change subject: IMPALA-5031: Fix undefined behavior: memset NULL .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I804f642f4be3b74c24f871f656c5147ee226d2c8 Gerrit-Change-Number: 11042 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 25 Jul 2018 03:29:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7257. Support Kudu tables in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10912 ) Change subject: IMPALA-7257. Support Kudu tables in LocalCatalog .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/48/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd Gerrit-Change-Number: 10912 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 02:27:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6335. Allow most shell tests to run in parallel
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11045 ) Change subject: IMPALA-6335. Allow most shell tests to run in parallel .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/47/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 Gerrit-Change-Number: 11045 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Comment-Date: Wed, 25 Jul 2018 02:17:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6335. Allow most shell tests to run in parallel
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11045 ) Change subject: IMPALA-6335. Allow most shell tests to run in parallel .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/46/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 Gerrit-Change-Number: 11045 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Comment-Date: Wed, 25 Jul 2018 02:08:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2860/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 02:07:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7257. Support Kudu tables in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10912 ) Change subject: IMPALA-7257. Support Kudu tables in LocalCatalog .. Patch Set 6: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/48/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/10912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd Gerrit-Change-Number: 10912 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 02:06:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7257. Support Kudu tables in LocalCatalog
Hello Bharath Vissapragada, Tianyi Wang, Vuk Ercegovac, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10912 to look at the new patch set (#6). Change subject: IMPALA-7257. Support Kudu tables in LocalCatalog .. IMPALA-7257. Support Kudu tables in LocalCatalog This adds support for querying Kudu tables when --use_local_catalog is enabled. With this change, most of the kudu e2e tests pass. Those that don't are related to missing support for CREATE TABLE AS SELECT, which currently has some downcasts to 'Db' causing issues. That will be addressed in a separate patch along with fixing CTAS for FS-backed tables. Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd --- M fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddReplaceColsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/KuduPartitionExpr.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java A fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java A fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalView.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 33 files changed, 697 insertions(+), 343 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/10912/6 -- To view, visit http://gerrit.cloudera.org:8080/10912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd Gerrit-Change-Number: 10912 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10914 ) Change subject: IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2859/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ae47a0b58022ed77abe51d2596c2b1d0111fae3 Gerrit-Change-Number: 10914 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 02:06:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7296: bytes limit for row batch queue
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10977 ) Change subject: IMPALA-7296: bytes limit for row batch queue .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h File be/src/util/blocking-queue.h: http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@142 PS5, Line 142: int64_t val_bytes = ElemBytesFn()(val); this can go up outside the lock, right? http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@170 PS5, Line 170: int64_t val_bytes = ElemBytesFn()(val); same, can compute outside the lock http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@249 PS5, Line 249: get_bytes_dequeued_.Load(); : if (dequeued == 0) return false; : put_bytes_enqueued_ -= dequeued; : get_bytes_dequeued_.Add(-dequeued); could you just Exchange(0) rather than loading and decrementing? seems like it's one atomic op instead of two and equally correct -- To view, visit http://gerrit.cloudera.org:8080/10977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa06d1d8da2a6d101efda08f620c0bf84a71e681 Gerrit-Change-Number: 10977 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 25 Jul 2018 02:03:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/10855 )
Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
..
Patch Set 4:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/10855/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10855/4//COMMIT_MSG@16
PS4, Line 16: This patch also introduces a new service pool for all query
execution
: control related RPCs in the future so that control commands from
: coordinators aren't blocked by long-running DataStream services'
RPCs.
do we want to consider the ability to put these on separate TCP connections
altogether? If they're latency-critical it might be worth doing it so that a
control-plane message can jump ahead a 300MB row batch message in the case of
big batches.
I think doing so wouldn't be too tough -- just need to change the connection
lookup code in KRPC to add another field to the equality check beyond just
remote address (eg an opaque "connection class" field or somesuch)
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/dml-exec-state.cc
File be/src/runtime/dml-exec-state.cc:
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/dml-exec-state.cc@404
PS4, Line 404: void DmlExecState::ToProto(InsertExecStatusPB* dml_status) {
do you want to clear dml_status first just in case? Similarly for other ToProto
functions
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/dml-exec-state.cc@455
PS4, Line 455:if (!kudu_stats->has_num_row_errors()) {
: kudu_stats->set_num_row_errors(0);
: }
I don't think this is necessary- it's not illegal to access an "unset" field.
It just returns the default value. In the case that the default value is
unspecified, it's what you'd expect (ie 0 for ints)
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/query-state.cc@268
PS4, Line 268: rpc_controller.AddOutboundSidecar(move(sidecar), _idx).ok
should this be a CHECK_OK?
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/query-state.cc@273
PS4, Line 273: "final"
need a space here
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/query-state.cc@274
PS4, Line 274: ERROR
should this be a DFATAL? do you expect this to ever actually happen?
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/query-state.cc@293
PS4, Line 293: kudu::Status rpc_status = proxy->ReportExecStatus(req, ,
_controller);
do you want any timeout on this RPC?
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/service/control-service.cc
File be/src/service/control-service.cc:
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/service/control-service.cc@72
PS4, Line 72: // TODO: implement something more efficient here, we're
currently
: // acquiring/releasing the map lock and doing a map lookup for
: // every report (assign each query a local int32_t id and use
that to index into a
: // vector of ClientRequestStates, w/o lookup or locking?)
seems like an easy fix here is to use a RWLock since we expect that queries
start and stop much less frequently than status is reported, right?
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/service/control-service.cc@87
PS4, Line 87: mem_tracker_->Release(rpc_context->GetTransferSize());
: rpc_context->RespondSuccess();
is everything in this function exception-safe? eg I thought Thrift could
occasionally throw exceptions, but maybe your wrappers are already covering for
that. If not, you may want to use something like a SCOPED_CLEANUP to ensure
that the RPC is responded to and the memory is released.
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/service/control-service.cc@104
PS4, Line 104: if (LIKELY(sidecar_status.ok())) {
do you want to warn or even CHECK on this?
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/util/container-util.h
File be/src/util/container-util.h:
http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/util/container-util.h@147
PS4, Line 147: void MergeMapValues(const google::protobuf::Map& src,
If you templatized this on the map type instead of the K and V types, you could
probably just use 'auto' below, and then avoid having to include the protobuf
headers in this generic util file. It would also likely become generic enough
to use with any associative container.
--
To view, visit http://gerrit.cloudera.org:8080/10855
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe
Gerrit-Change-Number: 10855
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Ho
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-5607: Part 1 [DOCS] Return type changes for EXTRACT and DATE PART
Jinchul Kim has posted comments on this change. ( http://gerrit.cloudera.org:8080/11044 ) Change subject: IMPALA-5607: Part 1 [DOCS] Return type changes for EXTRACT and DATE_PART .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/11044 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a7b1c2f984aac1d6d84480523d07a4125de1f52 Gerrit-Change-Number: 11044 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 25 Jul 2018 01:42:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7296: bytes limit for row batch queue
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/10977 )
Change subject: IMPALA-7296: bytes limit for row batch queue
..
Patch Set 5:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue-test.cc
File be/src/util/blocking-queue-test.cc:
http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue-test.cc@97
PS5, Line 97: LONG_TIMEOUT_MICROS);
Does it make the test timing sensitive ? Should we use a larger timeout ?
http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h
File be/src/util/blocking-queue.h:
http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@117
PS5, Line 117: ElemBytesFn()(*out);
It's a bit unfortunate that each element goes through two indirect function
calls (one in BlockingPut() and one in BlockingGet()) before it's consumed.
Is it a reasonable alternative to modify BlockingPut() to take an extra
parameter for the element size (with default value set to 0 for callers which
don't care about size limit of the queue) and store the size for each entry as
part of the queue ? I think it's reasonable to cap each entry to no more than
2GB so we can represent the size as int32_t. Would that be too much space
overhead in the queue ?
http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@152
PS5, Line 152: if (val_bytes > 0)
Not sure why the need for if() here. I can only assume this is to avoid
accessing put_bytes_enqueued_.
Given the access to total_put_wait_time_ above, put_bytes_enqueued_ is most
likely in the cache already. This complication is most likely not warranted
given the speculative execution of modern CPU. If anything, this just adds
unnecessary pressure to the branch predicator.
http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@185
PS5, Line 185: if (val_bytes > 0)
See comments above.
http://gerrit.cloudera.org:8080/#/c/10977/5/be/src/util/blocking-queue.h@237
PS5, Line 237: bool HasCapacityInternal(const
boost::unique_lock& lock, int64_t val_bytes) {
nit: long line
--
To view, visit http://gerrit.cloudera.org:8080/10977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa06d1d8da2a6d101efda08f620c0bf84a71e681
Gerrit-Change-Number: 10977
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 25 Jul 2018 01:41:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6335. Allow most shell tests to run in parallel
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11045 ) Change subject: IMPALA-6335. Allow most shell tests to run in parallel .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/47/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 Gerrit-Change-Number: 11045 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Comment-Date: Wed, 25 Jul 2018 01:34:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6335. Allow most shell tests to run in parallel
Hello Pooja Nilangekar, Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11045 to look at the new patch set (#2). Change subject: IMPALA-6335. Allow most shell tests to run in parallel .. IMPALA-6335. Allow most shell tests to run in parallel This adds an IMPALA_HISTFILE environment variable (and --history_file argument) to the shell which overrides the default location of ~/.impalahistory for the shell history. The shell tests now override this variable so that they can run in parallel without stomping on each other's history. This also fixes a couple flaky test which were previously missing the "execute_serially" annotation -- that annotation is no longer needed after this fix. A couple of the tests still need to be executed serially because they look at metrics such as the number of executed or running queries, and those metrics are unstable if other tests run in parallel. I tested this by running: ./bin/impala-py.test tests/shell/test_shell_interactive.py \ -m 'not execute_serially \ -n 80 \ --random ... several times in a row on an 88-core box. Prior to the change, several would fail each time. Now they pass. Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 --- M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 6 files changed, 21 insertions(+), 64 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/11045/2 -- To view, visit http://gerrit.cloudera.org:8080/11045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 Gerrit-Change-Number: 11045 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar
[Impala-ASF-CR] IMPALA-6335. Allow most shell tests to run in parallel
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11045 ) Change subject: IMPALA-6335. Allow most shell tests to run in parallel .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/46/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 Gerrit-Change-Number: 11045 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Jul 2018 01:32:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6335. Allow most shell tests to run in parallel
Todd Lipcon has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11045 Change subject: IMPALA-6335. Allow most shell tests to run in parallel .. IMPALA-6335. Allow most shell tests to run in parallel This adds an IMPALA_HISTFILE environment variable (and --history_file argument) to the shell which overrides the default location of ~/.impalahistory for the shell history. The shell tests now override this variable so that they can run in parallel without stomping on each other's history. This also fixes a couple flaky test which were previously missing the "execute_serially" annotation -- that annotation is no longer needed after this fix. A couple of the tests still need to be executed serially because they look at metrics such as the number of executed or running queries, and those metrics are unstable if other tests run in parallel. I tested this by running: ./bin/impala-py.test tests/shell/test_shell_interactive.py \ -m 'not execute_serially \ -n 80 \ --random ... several times in a row on an 88-core box. Prior to the change, several would fail each time. Now they pass. Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 --- M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 6 files changed, 20 insertions(+), 63 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/11045/1 -- To view, visit http://gerrit.cloudera.org:8080/11045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 Gerrit-Change-Number: 11045 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon
[Impala-ASF-CR] IMPALA-5607: Part 1 [DOCS] Return type changes for EXTRACT and DATE PART
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11044 ) Change subject: IMPALA-5607: Part 1 [DOCS] Return type changes for EXTRACT and DATE_PART .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/6/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11044 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a7b1c2f984aac1d6d84480523d07a4125de1f52 Gerrit-Change-Number: 11044 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 25 Jul 2018 01:28:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5607: Part 1 [DOCS] Return type changes for EXTRACT and DATE PART
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11044 ) Change subject: IMPALA-5607: Part 1 [DOCS] Return type changes for EXTRACT and DATE_PART .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/6/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11044 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a7b1c2f984aac1d6d84480523d07a4125de1f52 Gerrit-Change-Number: 11044 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Jul 2018 01:23:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5607: Part 1 [DOCS] Return type changes for EXTRACT and DATE PART
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11044 Change subject: IMPALA-5607: Part 1 [DOCS] Return type changes for EXTRACT and DATE_PART .. IMPALA-5607: Part 1 [DOCS] Return type changes for EXTRACT and DATE_PART Change-Id: I0a7b1c2f984aac1d6d84480523d07a4125de1f52 --- M docs/topics/impala_datetime_functions.xml 1 file changed, 14 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/11044/1 -- To view, visit http://gerrit.cloudera.org:8080/11044 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0a7b1c2f984aac1d6d84480523d07a4125de1f52 Gerrit-Change-Number: 11044 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10855 ) Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/45/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Wed, 25 Jul 2018 00:54:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10855 ) Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/query-state.cc File be/src/runtime/query-state.cc: http://gerrit.cloudera.org:8080/#/c/10855/4/be/src/runtime/query-state.cc@289 PS4, Line 289: rpc_mgr->GetProxy clang-tidy failure: Missing status check. -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Wed, 25 Jul 2018 00:43:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7218: [DOCS] Support column list in ALTER VIEW
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11043 ) Change subject: IMPALA-7218: [DOCS] Support column list in ALTER VIEW .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/5/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19e5cf97302a46738fd832344415fb7ad4ca0e41 Gerrit-Change-Number: 11043 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Comment-Date: Wed, 25 Jul 2018 00:35:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10855 ) Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/44/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Wed, 25 Jul 2018 00:33:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7218: [DOCS] Support column list in ALTER VIEW
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11043 ) Change subject: IMPALA-7218: [DOCS] Support column list in ALTER VIEW .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/5/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19e5cf97302a46738fd832344415fb7ad4ca0e41 Gerrit-Change-Number: 11043 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Jul 2018 00:23:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7218: [DOCS] Support column list in ALTER VIEW
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11043 Change subject: IMPALA-7218: [DOCS] Support column list in ALTER VIEW .. IMPALA-7218: [DOCS] Support column list in ALTER VIEW Change-Id: I19e5cf97302a46738fd832344415fb7ad4ca0e41 --- M docs/topics/impala_alter_view.xml 1 file changed, 47 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/11043/1 -- To view, visit http://gerrit.cloudera.org:8080/11043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I19e5cf97302a46738fd832344415fb7ad4ca0e41 Gerrit-Change-Number: 11043 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Hello Sailesh Mukil, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10855 to look at the new patch set (#4). Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC .. IMPALA-7213: Port ReportExecStatus() RPC to use KRPC This change converts ReportExecStatus() RPC from thrift based RPC to KRPC. This is done in part of the preparation for fixing IMPALA-2990 as we can take advantage of TCP connection multiplexing in KRPC to avoid overwhelming the coordinator with too many connections by reducing the number of TCP connection to one for each executor. This patch also introduces a new service pool for all query execution control related RPCs in the future so that control commands from coordinators aren't blocked by long-running DataStream services' RPCs. The majority of this patch is mechanical conversion of some Thrift structures used in ReportExecStatus() RPC to Protobuf. Note that the runtime profile is still retained as a Thrift structure as Impala clients will still fetch query profiles using Thrift RPCs. This also avoids duplicating the serialization implementation in both Thrift and Protobuf for the runtime profile. The Thrift runtime profiles are serialized and sent as a sidecar in ReportExecStatus() RPC. Change-Id: I7638583b433dcac066b87198e448743d90415ebe --- M be/src/benchmarks/expr-benchmark.cc M be/src/catalog/catalog-util.cc M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/hbase-table-sink.cc M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-parquet-table-writer.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-writer.cc M be/src/exec/hdfs-table-writer.h M be/src/rpc/jni-thrift-util.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.h M be/src/runtime/backend-client.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/dml-exec-state.cc M be/src/runtime/dml-exec-state.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/test-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/scheduler-test-util.cc M be/src/service/CMakeLists.txt M be/src/service/client-request-state.cc M be/src/service/client-request-state.h A be/src/service/control-service.cc A be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h A be/src/util/error-util-internal.h M be/src/util/error-util-test.cc M be/src/util/error-util.cc M be/src/util/error-util.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/uid-util.h M bin/impala-config.sh M common/protobuf/CMakeLists.txt M common/protobuf/common.proto A common/protobuf/control_service.proto M common/protobuf/data_stream_service.proto M common/protobuf/row_batch.proto M common/protobuf/rpc_test.proto M common/thrift/ImpalaInternalService.thrift 58 files changed, 1,019 insertions(+), 616 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/10855/4 -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10855 ) Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/45/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Wed, 25 Jul 2018 00:16:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10855 ) Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC .. Patch Set 3: Still working on some targeted BE tests for some error cases but please feel free to go ahead to do another pass. -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Wed, 25 Jul 2018 00:01:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10855 ) Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/44/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Tue, 24 Jul 2018 23:58:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Hello Sailesh Mukil, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10855 to look at the new patch set (#3). Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC .. IMPALA-7213: Port ReportExecStatus() RPC to use KRPC This change converts ReportExecStatus() RPC from thrift based RPC to KRPC. This is done in part of the preparation for fixing IMPALA-2990 as we can take advantage of TCP connection multiplexing in KRPC to avoid overwhelming the coordinator with too many connections by reducing the number of TCP connection to one for each executor. This patch also introduces a new service pool for all query execution control related RPCs in the future so that control commands from coordinators aren't blocked by long-running DataStream services' RPCs. The majority of this patch is mechanical conversion of some Thrift structures used in ReportExecStatus() RPC to Protobuf. Note that the runtime profile is still retained as a Thrift structure as Impala clients will still fetch query profiles using Thrift RPCs. This also avoids duplicating the serialization implementation in both Thrift and Protobuf for the runtime profile. The Thrift runtime profiles are serialized and sent as a sidecar in ReportExecStatus() RPC. Change-Id: I7638583b433dcac066b87198e448743d90415ebe --- M be/src/benchmarks/expr-benchmark.cc M be/src/catalog/catalog-util.cc M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/hbase-table-sink.cc M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-parquet-table-writer.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-writer.cc M be/src/exec/hdfs-table-writer.h M be/src/rpc/jni-thrift-util.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.h M be/src/runtime/backend-client.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/data-stream-test.cc M be/src/runtime/dml-exec-state.cc M be/src/runtime/dml-exec-state.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/test-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/scheduler-test-util.cc M be/src/service/CMakeLists.txt M be/src/service/client-request-state.cc M be/src/service/client-request-state.h A be/src/service/control-service.cc A be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h A be/src/util/error-util-internal.h M be/src/util/error-util-test.cc M be/src/util/error-util.cc M be/src/util/error-util.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/uid-util.h M bin/impala-config.sh M common/protobuf/CMakeLists.txt M common/protobuf/common.proto A common/protobuf/control_service.proto M common/protobuf/data_stream_service.proto M common/protobuf/row_batch.proto M common/protobuf/rpc_test.proto M common/thrift/ImpalaInternalService.thrift 59 files changed, 1,017 insertions(+), 617 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/10855/3 -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/10855 )
Change subject: IMPALA-7213: Port ReportExecStatus() RPC to use KRPC
..
Patch Set 2:
(36 comments)
http://gerrit.cloudera.org:8080/#/c/10855/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10855/2//COMMIT_MSG@17
PS2, Line 17: they
> nit: that
Done
http://gerrit.cloudera.org:8080/#/c/10855/2//COMMIT_MSG@19
PS2, Line 19: convertion
> nit: conversion
Done
http://gerrit.cloudera.org:8080/#/c/10855/2//COMMIT_MSG@25
PS2, Line 25:
> Not to be pedantic, but a lot of Thrift structs are being removed as part o
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/exec/hdfs-parquet-table-writer.cc
File be/src/exec/hdfs-parquet-table-writer.cc:
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/exec/hdfs-parquet-table-writer.cc@1183
PS2, Line 1183:
(*parquet_insert_stats_.mutable_per_column_size())[col_name] +=
: col_writer->total_compressed_size();
> nit: Should we have a level of indirection here? It's not very obvious what
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/rpc/thrift-util.h
File be/src/rpc/thrift-util.h:
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/rpc/thrift-util.h@93
PS2, Line 93: Serialize
> Should we explicitly qualify the names of these functions to avoid confusio
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/rpc/thrift-util.h@103
PS2, Line 103: uint8_t* buffer;
: uint32_t len;
: mem_buffer_->getBuffer(, );
: result->assign_copy(buffer, len);
> I think you can avoid the copy here if you do:
I believe faststring manages its own buffer internally so there is no easy way
to share the internal buffer inside mem_buffer_ with the faststring.
If I understand it correctly, mem_buffer_->getBuffer() shouldn't do any data
copying so there should be only one copy into the faststring.
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.h
File be/src/runtime/dml-exec-state.h:
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.h@35
PS2, Line 35: class TInsertResult;
: class TFinalizeParams;
: class TUpdateCatalogRequest;
: class RuntimeProfile;
: class HdfsTableDescriptor;
> nit: Not your change, but the ordering is not alphabetical.
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.h@126
PS2, Line 126: std::map
> I'm wondering if it just makes sense to use google::protobuf::Map here inst
The comment explicitly says "Uses ordered map so that iteration order is
deterministic." so I am sticking to map for now. According to the specification
of protobuf, google::protobuf::Map acts more like an unordered_map
(https://developers.google.com/protocol-buffers/docs/proto#maps-features)
- Wire format ordering and map iteration ordering of map values is undefined,
so you cannot rely on your map items being in a particular order.
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.cc
File be/src/runtime/dml-exec-state.cc:
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.cc@83
PS2, Line 83: for (auto i = parquet_stats.per_column_size().begin();
:i != parquet_stats.per_column_size().end(); ++i) {
> Range based for-loop with const ref.
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.cc@95
PS2, Line 95: auto
> Let's try to stay explicit about data types where we can. Also take it as a
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.cc@95
PS2, Line 95: new_partition_status
> nit: new_per_partition_status_map
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.cc@96
PS2, Line 96: (auto iter = new_partition_status.begin(); iter !=
new_partition_status.end();
:++iter)
> Why not use a range based for loop? It's much more readable. Also, referenc
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.cc@404
PS2, Line 404: for (auto iter = files_to_move_.begin(); iter !=
files_to_move_.end(); ++iter) {
> Range based for loop, and take iterator as const ref.
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.cc@407
PS2, Line 407: for (auto iter = per_partition_status_.begin();
> Same here
Done
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/dml-exec-state.cc@453
PS2, Line 453: KuduDmlStatsPB* kudu_stats
> This can be a reference, no reason to use a pointer here.
kudu_stats is actually being modified here so using a pointer seems to fit the
coding convention.
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:
http://gerrit.cloudera.org:8080/#/c/10855/2/be/src/runtime/query-state.cc@237
[Impala-ASF-CR] IMPALA-6214: Determine and warn about stuck fragment instances.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11021 ) Change subject: IMPALA-6214: Determine and warn about stuck fragment instances. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/43/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I260a1d0a3477e5c6a46094e664500c3e2ed7de62 Gerrit-Change-Number: 11021 Gerrit-PatchSet: 2 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 24 Jul 2018 23:40:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6214: Determine and warn about stuck fragment instances.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/11021 ) Change subject: IMPALA-6214: Determine and warn about stuck fragment instances. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11021/2/be/src/runtime/krpc-data-stream-recvr.cc File be/src/runtime/krpc-data-stream-recvr.cc: http://gerrit.cloudera.org:8080/#/c/11021/2/be/src/runtime/krpc-data-stream-recvr.cc@238 PS2, Line 238: VLOG_QUERY << "wait arrival fragment_instance_id=" VLOG_QUERY is on by default, so this would become very very noisy, once per wait, no? I think we'd only want to log if we have hit a timeout from the below CV wait. Also, I don't know the context of this code quite well enough, but isn't it normal to sometimes wait for minutes on a sender? For example, if the upstream node is a full sort, or a join with a lot of slow parents then the receiver side may block for minutes or even hours before making progress. In that case, I can see surfacing this kind of information in the profile or in some query-scoped log but maybe not in the global impalad log? I think I don't quite understand the end goal of this JIRA well enough to evaluate whether this change is a net help. Why doesn't the existing data_wait_timer already tell us that this node is the blocking culprit, and from there we can just look at the fragment graph to understand what was blocking it? -- To view, visit http://gerrit.cloudera.org:8080/11021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I260a1d0a3477e5c6a46094e664500c3e2ed7de62 Gerrit-Change-Number: 11021 Gerrit-PatchSet: 2 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 24 Jul 2018 23:36:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7333: remove MarkNeedsDeepCopy() in agg and BTS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11007 ) Change subject: IMPALA-7333: remove MarkNeedsDeepCopy() in agg and BTS .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/42/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11007 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91ac53bacc00df4726c015a30ba5a2026aa4b5f5 Gerrit-Change-Number: 11007 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Jul 2018 23:29:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6214: Determine and warn about stuck fragment instances.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11021 ) Change subject: IMPALA-6214: Determine and warn about stuck fragment instances. .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/43/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I260a1d0a3477e5c6a46094e664500c3e2ed7de62 Gerrit-Change-Number: 11021 Gerrit-PatchSet: 2 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 24 Jul 2018 23:13:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6214: Determine and warn about stuck fragment instances.
Hello Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11021 to look at the new patch set (#2). Change subject: IMPALA-6214: Determine and warn about stuck fragment instances. .. IMPALA-6214: Determine and warn about stuck fragment instances. In order to diagnose query hangs, we need to know the fragment execution time on a particular exec node. Inspecting the query run time profile to find the cause of hang does not give much details. This change helps in finding the problematic 'exec node' where the fragment execution is not making progress. The change makes use of timed-wait wait that waits for the 1 min delay if the batch request is not recieved from the sender node. Testing: a) Added a delay on the sender side as a part of manual test case to notice the affect of change. The watchdog prints the detail of fragmentID and nodeID when the watchdog timer expires. b) Ran the core test without failure. Change-Id: I260a1d0a3477e5c6a46094e664500c3e2ed7de62 --- M be/src/runtime/krpc-data-stream-recvr.cc 1 file changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/11021/2 -- To view, visit http://gerrit.cloudera.org:8080/11021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I260a1d0a3477e5c6a46094e664500c3e2ed7de62 Gerrit-Change-Number: 11021 Gerrit-PatchSet: 2 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 ) Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/40/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a Gerrit-Change-Number: 11041 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Jul 2018 23:07:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7333: remove MarkNeedsDeepCopy() in agg and BTS
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11007 ) Change subject: IMPALA-7333: remove MarkNeedsDeepCopy() in agg and BTS .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/11007/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11007/4//COMMIT_MSG@7 PS4, Line 7: TS > Does this patch address joins? Yeah I guess that's a bit confusing, the BufferedTupleStream is used by HashJoin. http://gerrit.cloudera.org:8080/#/c/11007/4/be/src/runtime/buffered-tuple-stream-test.cc File be/src/runtime/buffered-tuple-stream-test.cc: http://gerrit.cloudera.org:8080/#/c/11007/4/be/src/runtime/buffered-tuple-stream-test.cc@678 PS4, Line 678: // on previous GetNext() calls. : ASSERT_EQ(0, stream.BytesPinned(false)); : > Is this comment still accurate? It is 0 now Done -- To view, visit http://gerrit.cloudera.org:8080/11007 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91ac53bacc00df4726c015a30ba5a2026aa4b5f5 Gerrit-Change-Number: 11007 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Jul 2018 23:04:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7333: remove MarkNeedsDeepCopy() in agg and BTS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11007 ) Change subject: IMPALA-7333: remove MarkNeedsDeepCopy() in agg and BTS .. Patch Set 5: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/42/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11007 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91ac53bacc00df4726c015a30ba5a2026aa4b5f5 Gerrit-Change-Number: 11007 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Jul 2018 23:04:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7333: remove MarkNeedsDeepCopy() in agg and BTS
Hello Thomas Marshall, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11007 to look at the new patch set (#5). Change subject: IMPALA-7333: remove MarkNeedsDeepCopy() in agg and BTS .. IMPALA-7333: remove MarkNeedsDeepCopy() in agg and BTS This takes advantage of work (e.g. IMPALA-3200, IMPALA-5844) to remove a couple of uses of the API. Testing: Ran core, ASAN and exhaustive builds. Added a unit test to directly test the attaching behaviour. Change-Id: I91ac53bacc00df4726c015a30ba5a2026aa4b5f5 --- M be/src/exec/grouping-aggregator.cc M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h M be/src/runtime/row-batch.h 5 files changed, 235 insertions(+), 122 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/11007/5 -- To view, visit http://gerrit.cloudera.org:8080/11007 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91ac53bacc00df4726c015a30ba5a2026aa4b5f5 Gerrit-Change-Number: 11007 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: memset NULL
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11042 ) Change subject: IMPALA-5031: Fix undefined behavior: memset NULL .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I804f642f4be3b74c24f871f656c5147ee226d2c8 Gerrit-Change-Number: 11042 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Jul 2018 23:01:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 ) Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/39/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a Gerrit-Change-Number: 11041 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Jul 2018 23:00:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2857/ -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 22:54:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7333: remove MarkNeedsDeepCopy() in aggs and joins
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11007 ) Change subject: IMPALA-7333: remove MarkNeedsDeepCopy() in aggs and joins .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/11007/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11007/4//COMMIT_MSG@7 PS4, Line 7: joins Does this patch address joins? http://gerrit.cloudera.org:8080/#/c/11007/4/be/src/runtime/buffered-tuple-stream-test.cc File be/src/runtime/buffered-tuple-stream-test.cc: http://gerrit.cloudera.org:8080/#/c/11007/4/be/src/runtime/buffered-tuple-stream-test.cc@678 PS4, Line 678: // Note: this should really be 0, but the BufferedTupleStream returns eos before : // attaching the last buffer, rather than after, so the last block isn't deleted : // until the stream is closed. Is this comment still accurate? It is 0 now -- To view, visit http://gerrit.cloudera.org:8080/11007 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91ac53bacc00df4726c015a30ba5a2026aa4b5f5 Gerrit-Change-Number: 11007 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Jul 2018 22:53:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: memset NULL
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11042 Change subject: IMPALA-5031: Fix undefined behavior: memset NULL .. IMPALA-5031: Fix undefined behavior: memset NULL memset has undefined behavior when its first argument is NULL. The instance fixed here was found by Clang's undefined behavior sanitizer. It was found in the end-to-end tests. The interesting part of the stack trace is: be/src/util/bitmap.h:78:12: runtime error: null pointer passed as argument 1, which is declared to never be null /usr/include/string.h:62:79: note: nonnull attribute specified here #0 0x2ccb59b in Bitmap::SetAllBits(bool) be/src/util/bitmap.h:78:5 #1 0x2cb6b9e in NestedLoopJoinNode::ResetMatchingBuildRows(RuntimeState*, long) be/src/exec/nested-loop-join-node.cc:176:27 #2 0x2cb5ad6 in NestedLoopJoinNode::Open(RuntimeState*) be/src/exec/nested-loop-join-node.cc:90:43 Change-Id: I804f642f4be3b74c24f871f656c5147ee226d2c8 --- M be/src/util/bitmap.h 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/11042/1 -- To view, visit http://gerrit.cloudera.org:8080/11042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I804f642f4be3b74c24f871f656c5147ee226d2c8 Gerrit-Change-Number: 11042 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: memset NULL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11042 ) Change subject: IMPALA-5031: Fix undefined behavior: memset NULL .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/41/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I804f642f4be3b74c24f871f656c5147ee226d2c8 Gerrit-Change-Number: 11042 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Jul 2018 22:46:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7242/IMPALA-7243: check for overflow when converting IntVal to DecimalValue
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11020 ) Change subject: IMPALA-7242/IMPALA-7243: check for overflow when converting IntVal to DecimalValue .. Patch Set 1: (2 comments) Couple of minor comments, but looks good. http://gerrit.cloudera.org:8080/#/c/11020/1/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/11020/1/be/src/exprs/expr-test.cc@5382 PS1, Line 5382: // Test when converting num_buckets from a IntVal to Decimal16Val causes an Can you mention the JIRA number here for context? http://gerrit.cloudera.org:8080/#/c/11020/1/be/src/exprs/math-functions-ir.cc File be/src/exprs/math-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/11020/1/be/src/exprs/math-functions-ir.cc@541 PS1, Line 541: stringstream error_msg; Nit: we prefer Substitute() over stringstream where possible (although I guess we're already using it for other cases above). -- To view, visit http://gerrit.cloudera.org:8080/11020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie5808802096bb0e6cfd4ab6464538474e42cab60 Gerrit-Change-Number: 11020 Gerrit-PatchSet: 1 Gerrit-Owner: anujphadke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Jul 2018 22:34:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11041 to look at the new patch set (#2). Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3 .. IMPALA-7170: Update data_generator.py for Hadoop 3 After the move to Hadoop 3, data_generator.py was broken. The issue seems to be that we rely on additional jars not in the classpath. The solution is to pass the location of these jars into the 'hadoop' command using the '-libjars' parameter. This patch also updates tests/comparison/README to add instructions for dealing with Yarn, since during the move to Hadoop 3 we switched to no longer running Yarn as part of the minicluster by default. Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a --- M tests/comparison/README M tests/comparison/data_generator.py 2 files changed, 8 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/11041/2 -- To view, visit http://gerrit.cloudera.org:8080/11041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a Gerrit-Change-Number: 11041 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 ) Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3 .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/40/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a Gerrit-Change-Number: 11041 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Jul 2018 22:28:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11041 ) Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3 .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/39/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a Gerrit-Change-Number: 11041 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Jul 2018 22:24:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7170: Update data generator.py for Hadoop 3
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11041 Change subject: IMPALA-7170: Update data_generator.py for Hadoop 3 .. IMPALA-7170: Update data_generator.py for Hadoop 3 After the move to Hadoop 3, data_generator.py was broken. The issue seems to be that we rely on additional jars not in the classpath. The solution is to pass the location of these jars into the 'hadoop' command using the '-libjars' parameter. This patch also updates tests/comparison/README to add instructions for dealing with Yarn, since during the move to Hadoop 3 we switched to no longer running Yarn as part of the minicluster by default. Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a --- M tests/comparison/README M tests/comparison/cluster.py M tests/comparison/data_generator.py 3 files changed, 9 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/11041/1 -- To view, visit http://gerrit.cloudera.org:8080/11041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I47b7d663174dbd38a5d9c98f1a88f0ebab726d5a Gerrit-Change-Number: 11041 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-7330. After LOAD DATA, only refresh affected partition
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/11014 ) Change subject: IMPALA-7330. After LOAD DATA, only refresh affected partition .. Patch Set 2: Code-Review+2 Forwarding +2 -- To view, visit http://gerrit.cloudera.org:8080/11014 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b29846deac49a89abcd3495e4b757ef536ff331 Gerrit-Change-Number: 11014 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 24 Jul 2018 20:45:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7330. After LOAD DATA, only refresh affected partition
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11014 ) Change subject: IMPALA-7330. After LOAD DATA, only refresh affected partition .. IMPALA-7330. After LOAD DATA, only refresh affected partition This changes LOAD DATA so that, if a specific partition is provided, only the named partition will be refreshed upon completion of the statement. No new tests are added since this code path is covered by existing tests and this just optimizes the metadata reload. I did verify looking at the catalogd logs that only the single specified partition was refreshed when I issued a LOAD statement from the shell. Change-Id: I3b29846deac49a89abcd3495e4b757ef536ff331 Reviewed-on: http://gerrit.cloudera.org:8080/11014 Tested-by: Impala Public Jenkins Reviewed-by: Todd Lipcon --- M be/src/service/client-request-state.cc 1 file changed, 4 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11014 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3b29846deac49a89abcd3495e4b757ef536ff331 Gerrit-Change-Number: 11014 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-6214: Determine and warn about stuck fragment instances.
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/11021 ) Change subject: IMPALA-6214: Determine and warn about stuck fragment instances. .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/11021/1/be/src/runtime/krpc-data-stream-recvr.cc File be/src/runtime/krpc-data-stream-recvr.cc: http://gerrit.cloudera.org:8080/#/c/11021/1/be/src/runtime/krpc-data-stream-recvr.cc@230 PS1, Line 230: stringstream dstid; : : dstid << recvr_->dest_node_id(); : string frgId = PrintId(recvr_->fragment_instance_id()); : string watchdog_str = "fragment_instance_id= " + frgId + " node= " + dstid.str(); : memset(buf, BUFSZ, 0); : sprintf(buf, "%s : %d %s", __FILE__, __LINE__, watchdog_str.c_str()); > constructing this once per batch seems like a big performance red flag. I'll remove it. http://gerrit.cloudera.org:8080/#/c/11021/1/be/src/runtime/krpc-data-stream-recvr.cc@239 PS1, Line 239: kudu::SetStackTraceSignal(SIGRTMIN + 1); > you should call this at init time, not once per batch. I'm not even sure it I'll remove it. http://gerrit.cloudera.org:8080/#/c/11021/1/be/src/runtime/krpc-data-stream-recvr.cc@266 PS1, Line 266: data_arrival_cv_.wait(l); > maybe it would be simpler to just use a timed-wait variant here to periodic Thanks Todd I was thinking along the same lines, it's a good and easy fix for this case. -- To view, visit http://gerrit.cloudera.org:8080/11021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I260a1d0a3477e5c6a46094e664500c3e2ed7de62 Gerrit-Change-Number: 11021 Gerrit-PatchSet: 1 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 24 Jul 2018 20:14:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10813 ) Change subject: IMPALA-7163: Implement a state machine for the QueryState class .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/38/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Gerrit-Change-Number: 10813 Gerrit-PatchSet: 7 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Tue, 24 Jul 2018 20:10:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6214: Determine and warn about stuck fragment instances.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/11021 ) Change subject: IMPALA-6214: Determine and warn about stuck fragment instances. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11021/1/be/src/runtime/krpc-data-stream-recvr.cc File be/src/runtime/krpc-data-stream-recvr.cc: http://gerrit.cloudera.org:8080/#/c/11021/1/be/src/runtime/krpc-data-stream-recvr.cc@266 PS1, Line 266: data_arrival_cv_.wait(l); maybe it would be simpler to just use a timed-wait variant here to periodically log if something gets stuck for a long time? -- To view, visit http://gerrit.cloudera.org:8080/11021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I260a1d0a3477e5c6a46094e664500c3e2ed7de62 Gerrit-Change-Number: 11021 Gerrit-PatchSet: 1 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 24 Jul 2018 20:06:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7257. Support Kudu tables in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10912 ) Change subject: IMPALA-7257. Support Kudu tables in LocalCatalog .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2856/ -- To view, visit http://gerrit.cloudera.org:8080/10912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b6a317ee895e43e00ade953e814867b56b4e6dd Gerrit-Change-Number: 10912 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:59:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10914 ) Change subject: IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2858/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ae47a0b58022ed77abe51d2596c2b1d0111fae3 Gerrit-Change-Number: 10914 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:59:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7307 (part 2). Support TABLESAMPLE in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10972 ) Change subject: IMPALA-7307 (part 2). Support TABLESAMPLE in LocalCatalog .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/33/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10972 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f7baf05f16c6389ed900e0459708005ab44491e Gerrit-Change-Number: 10972 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:57:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7308. Support Avro tables in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10970 ) Change subject: IMPALA-7308. Support Avro tables in LocalCatalog .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/32/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4b86c8203271b773a711ed77558ec3e3070cb69 Gerrit-Change-Number: 10970 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:54:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7330. After LOAD DATA, only refresh affected partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11014 ) Change subject: IMPALA-7330. After LOAD DATA, only refresh affected partition .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11014 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b29846deac49a89abcd3495e4b757ef536ff331 Gerrit-Change-Number: 11014 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 24 Jul 2018 19:51:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2857/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:51:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7340. Only send necessary fields in THdfsPartition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11026 ) Change subject: IMPALA-7340. Only send necessary fields in THdfsPartition .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/36/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97e8402efdfdeea06463bb71a40ebb6abd1f11f0 Gerrit-Change-Number: 11026 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:48:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10914 ) Change subject: IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/31/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ae47a0b58022ed77abe51d2596c2b1d0111fae3 Gerrit-Change-Number: 10914 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:47:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10813 ) Change subject: IMPALA-7163: Implement a state machine for the QueryState class .. Patch Set 7: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/38/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/10813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Gerrit-Change-Number: 10813 Gerrit-PatchSet: 7 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Tue, 24 Jul 2018 19:47:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7140 (part 9): add support for SHOW FILES in LocalFsTable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10973 ) Change subject: IMPALA-7140 (part 9): add support for SHOW FILES in LocalFsTable .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/34/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10973 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I143bea9f72b6a25ae2545663e06f4849b58533ba Gerrit-Change-Number: 10973 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:40:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7320. Avoid calling getFileStatus() for each partition when table is loaded
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11027 ) Change subject: IMPALA-7320. Avoid calling getFileStatus() for each partition when table is loaded .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/37/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11027 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I83e5ebc214d6620d165e13f8cc80f8fdda100734 Gerrit-Change-Number: 11027 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:40:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option Also, clarified cancelled queries vs closed queries Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Reviewed-on: http://gerrit.cloudera.org:8080/11004 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M docs/impala.ditamap M docs/shared/impala_common.xml A docs/topics/impala_idle_session_timeout.xml M docs/topics/impala_timeouts.xml 4 files changed, 177 insertions(+), 33 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Jul 2018 19:35:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. Patch Set 4: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/4/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Jul 2018 19:34:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7311. Allow INSERT on writable partitions even if some other partition is READ ONLY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10974 ) Change subject: IMPALA-7311. Allow INSERT on writable partitions even if some other partition is READ_ONLY .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/35/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dd81100ae73fcabdbfaf679c20cea7dc102cd13 Gerrit-Change-Number: 10974 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:33:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/357/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Jul 2018 19:32:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Jul 2018 19:32:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7307 (part 1). Support stats extrapolation in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10971 ) Change subject: IMPALA-7307 (part 1). Support stats extrapolation in LocalCatalog .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/30/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I479b7f517091dd558768601e1e0704a1902b78a5 Gerrit-Change-Number: 10971 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:25:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. Patch Set 4: Code-Review+2 Move the new doc to the correct place in the ditamap, and corrected spelling errors in the commit message. -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Jul 2018 19:24:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/4/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Jul 2018 19:23:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Hello Zoltan Borok-Nagy, Balazs Jeszenszky, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11004 to look at the new patch set (#4). Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option Also, clarified cancelled queries vs closed queries Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 --- M docs/impala.ditamap M docs/shared/impala_common.xml A docs/topics/impala_idle_session_timeout.xml M docs/topics/impala_timeouts.xml 4 files changed, 177 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/11004/4 -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Sailesh Mukil has posted comments on this change. (
http://gerrit.cloudera.org:8080/10813 )
Change subject: IMPALA-7163: Implement a state machine for the QueryState class
..
Patch Set 7:
(21 comments)
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/fragment-instance-state.cc
File be/src/runtime/fragment-instance-state.cc:
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/fragment-instance-state.cc@a113
PS6, Line 113:
> May make sense to replace this by checking the state. Doing a racy read of
I'm not sure if there's a way to do a racy read on an AtomicEnum, so I've used
current_state_.Load() here.
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h
File be/src/runtime/query-state.h:
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@152
PS6, Line 152: /// Not idempotent, not thread-safe. Must only be called by a
single thread.
> update comment to mention that it blocks till a terminal state has been rea
Done
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@156
PS6, Line 156: mentInstanceState* GetFInstanc
> nit: avoid reference to private methods. how about switching the descriptio
Done
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@198
PS6, Line 198: void DonePreparing() { d
> If you take the suggestion to only keep a single 'query_status_', there doe
Done
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@212
PS6, Line 212: }
> This is only meaningful when there is an error status, right ? Please docum
Done
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@234
PS6, Line 234: on
> fragment instance
Done
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@271
PS6, Line 271:
> Consider calling it "PREPARING"
Done
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@274
PS6, Line 274:
> Consider calling it "EXECUTING"
Done
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@281
PS6, Line 281: / The ove
> when is the cancelled state set?
I've added a change to Cancel() to make the query_status_ Status::CANCELLED.
Anytime UpdateBackendExecState() is called after that, the
BackendExecState::CANCELLED state will be set.
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@289
PS6, Line 289:
> if cancelled, will the status be Status::CANCELLED? if yes that is also cou
You're right. I've fixed that in the next patch set.
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@308
PS6, Line 308:
: /// Number of active fragment instances and coordinators for
this query that may consume
: /// resources for query execution (i.e. threads, memory) on
the Impala daemon.
: /// Query-wide execution resources for this query are released
once this goes to zero.
: AtomicInt32 exec_resource_refcnt_;
:
: /// Temporary files for this query (owned by obj_pool_).
Non-null if spilling is
: /// enabled. Set in Pr
> It seems simpler to keep track of a single status for the first error hit i
I feel that it's not ideal having the fragment instance threads modify the
query wide status. Now the patch looks like this:
The fragment instance threads modify the query wide Status to show that there's
an error, but they don't modify the query BackendExecState. The
BackendExecState is modified by the query state thread. This seems a little
counter intuitive, since the fragment instance thread could itself modify the
BackendExecState if it wants to as it has all the information to do so when it
hits an error. But that would mean that sending final reports, etc. would have
to be driven by the fragment instance thread too, which is not a good idea.
In any case, I'll go ahead with this, as functionally it'll have the same
behavior, and it's not a huge problem, so we can avoid more back and forth
about it.
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.h@320
PS6, Line 320:
> Mind documenting the thread safety about this variable ?
Done
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:
http://gerrit.cloudera.org:8080/#/c/10813/6/be/src/runtime/query-state.cc@233
PS6, Line 233: return query_status_;
> unused variable. maybe just pass a status object to this method if the fins
I was printing this to the logs in the previous patchset, but I removed it
since that was one of the comments. So, in this patch, we aren't using the
'finst_id_' part of the struct, but as part of a following change, we will
require it (IMPALA-4063). This patch is primarily meant as a set up to do that
change, so this state machine isn't really used for anything until that follow
on patch comes in.
In any case,
[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Hello Michael Ho, Joe McDonnell, Bikramjeet Vig, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10813 to look at the new patch set (#7). Change subject: IMPALA-7163: Implement a state machine for the QueryState class .. IMPALA-7163: Implement a state machine for the QueryState class This patch adds a state machine for the QueryState class. The motivation behind this patch is to make the query lifecycle from the point of view of an executor much easier to reason about and this patch is key for a follow on patch for IMPALA-2990 where the status reporting will be per-query rather than per-fragment-instance. Currently, the state machine provides no other purpose, and it will mostly be used for IMPALA-2990. We introduce 5 possible states for the QueryState which include 3 terminal states (FINISHED, CANCELLED and ERROR) and 2 non-terminal states (PREPARING, EXECUTING). The transition from one state to the next is always handled by a single thread which is also the QueryState thread. This thread will additionally bear the purpose of sending periodic updates after IMPALA-4063, which is the primary reason behind having only this thread modify the state of the query. Counting barriers are introduced to keep a count of how many fragment instances have finished Preparing and Executing. These barriers also block until all the fragment instances have finished a respective state. The fragment instances update the query wide query status if an error is hit and unblocks the barrier if it is in the EXECUTING state. The PREPARING state blocks regardless of whether a fragment instance hit an error or not, until all the fragment instances have completed successfully or unsuccessfully, to maintain the invariant that fragment instances cannot be cancelled until the entire QueryState has finished PREPARING. The status reporting protocol has not been changed and remains exactly as it was. Testing: Ran 'core' and 'exhaustive' tests. TODO: Run stress tests. Future related work: 1) IMPALA-2990: Make status reporting per-query. 2) Try to logically align the FIS states with the QueryState states. 3) Consider mirroring the QueryState state machine to CoordinatorBackendState Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe --- M be/src/runtime/coordinator.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h 5 files changed, 231 insertions(+), 63 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/10813/7 -- To view, visit http://gerrit.cloudera.org:8080/10813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Gerrit-Change-Number: 10813 Gerrit-PatchSet: 7 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Jul 2018 18:55:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/356/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Jul 2018 18:55:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE SESSION TIMEOUT query option
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/11004 ) Change subject: IMPALA-5826 IMPALA-7162: [DOCS] Documented the IDLE_SESSION_TIMEOUT query option .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37182a3c5cf19fdcbb5f247ed71d43f963143510 Gerrit-Change-Number: 11004 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Jul 2018 18:55:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7307 (part 2). Support TABLESAMPLE in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10972 ) Change subject: IMPALA-7307 (part 2). Support TABLESAMPLE in LocalCatalog .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/33/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/10972 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f7baf05f16c6389ed900e0459708005ab44491e Gerrit-Change-Number: 10972 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 18:46:23 + Gerrit-HasComments: No
