[Impala-ASF-CR] IMPALA-3282: [DOCS] Adds regexp escape built-in function
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10174 ) Change subject: IMPALA-3282: [DOCS] Adds regexp_escape built-in function .. IMPALA-3282: [DOCS] Adds regexp_escape built-in function Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf Reviewed-on: http://gerrit.cloudera.org:8080/10174 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_string_functions.xml 1 file changed, 48 insertions(+), 0 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf Gerrit-Change-Number: 10174 Gerrit-PatchSet: 3 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-3282: [DOCS] Adds regexp escape built-in function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10174 ) Change subject: IMPALA-3282: [DOCS] Adds regexp_escape built-in function .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf Gerrit-Change-Number: 10174 Gerrit-PatchSet: 2 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Apr 2018 04:44:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3282: [DOCS] Adds regexp escape built-in function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10174 ) Change subject: IMPALA-3282: [DOCS] Adds regexp_escape built-in function .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/268/ -- To view, visit http://gerrit.cloudera.org:8080/10174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf Gerrit-Change-Number: 10174 Gerrit-PatchSet: 2 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 25 Apr 2018 04:41:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3282: [DOCS] Adds regexp escape built-in function
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10174 ) Change subject: IMPALA-3282: [DOCS] Adds regexp_escape built-in function .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf Gerrit-Change-Number: 10174 Gerrit-PatchSet: 2 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Alex Rodoni Gerrit-Comment-Date: Wed, 25 Apr 2018 04:41:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3282: [DOCS] Adds regexp escape built-in function
Alex Rodoni has uploaded a new patch set (#2) to the change originally created by Kim Jin Chul. ( http://gerrit.cloudera.org:8080/10174 ) Change subject: IMPALA-3282: [DOCS] Adds regexp_escape built-in function .. IMPALA-3282: [DOCS] Adds regexp_escape built-in function Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf --- M docs/topics/impala_string_functions.xml 1 file changed, 48 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/10174/2 -- To view, visit http://gerrit.cloudera.org:8080/10174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf Gerrit-Change-Number: 10174 Gerrit-PatchSet: 2 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Alex Rodoni
[Impala-ASF-CR] IMPALA-6916: Implement COMMENT ON DATABASE
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10171 ) Change subject: IMPALA-6916: Implement COMMENT ON DATABASE .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 Gerrit-Change-Number: 10171 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Wed, 25 Apr 2018 04:00:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6916: Implement COMMENT ON DATABASE
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10171 ) Change subject: IMPALA-6916: Implement COMMENT ON DATABASE .. IMPALA-6916: Implement COMMENT ON DATABASE This patch implements updating comment on a database. Syntax: COMMENT ON DATABASE db IS 'comment' Testing: - Added new front-end tests - Ran all front-end tests - Added new end-to-end tests - Ran end-to-end DDL tests Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 --- M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/CommentOnDbStmt.java A fe/src/main/java/org/apache/impala/analysis/CommentOnStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 13 files changed, 216 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10171/5 -- To view, visit http://gerrit.cloudera.org:8080/10171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 Gerrit-Change-Number: 10171 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6916: Implement COMMENT ON DATABASE
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10171 ) Change subject: IMPALA-6916: Implement COMMENT ON DATABASE .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10171/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/10171/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3526 PS4, Line 3526: db.getMetaStoreDb().setDescription(comment); : applyAlterDatabase(db); > nit: make this one line. Done -- To view, visit http://gerrit.cloudera.org:8080/10171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 Gerrit-Change-Number: 10171 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Wed, 25 Apr 2018 04:00:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6916: Implement COMMENT ON DATABASE
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10171 ) Change subject: IMPALA-6916: Implement COMMENT ON DATABASE .. Patch Set 4: (1 comment) last one. http://gerrit.cloudera.org:8080/#/c/10171/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/10171/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3526 PS4, Line 3526: org.apache.hadoop.hive.metastore.api.Database msDb = db.getMetaStoreDb(); : msDb.setDescription(comment); nit: make this one line. -- To view, visit http://gerrit.cloudera.org:8080/10171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 Gerrit-Change-Number: 10171 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Wed, 25 Apr 2018 03:57:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6916: Implement COMMENT ON DATABASE
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10171 ) Change subject: IMPALA-6916: Implement COMMENT ON DATABASE .. Patch Set 4: (4 comments) http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1081 PS2, Line 1081: ; > see comment below, might not be needed. Done http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1578 PS2, Line 1578: ; > see comment below, might not be needed. Done http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2853 PS2, Line 2853: private void applyAlterDatabase(Db db) > See comment below, might not be needed. Done http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3528 PS2, Line 3528: db); > This could just be "db" to avoid the changes to the methods above. Done -- To view, visit http://gerrit.cloudera.org:8080/10171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 Gerrit-Change-Number: 10171 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Wed, 25 Apr 2018 03:53:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6916: Implement COMMENT ON DATABASE
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/10171 ) Change subject: IMPALA-6916: Implement COMMENT ON DATABASE .. IMPALA-6916: Implement COMMENT ON DATABASE This patch implements updating comment on a database. Syntax: COMMENT ON DATABASE db IS 'comment' Testing: - Added new front-end tests - Ran all front-end tests - Added new end-to-end tests - Ran end-to-end DDL tests Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 --- M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/CommentOnDbStmt.java A fe/src/main/java/org/apache/impala/analysis/CommentOnStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 13 files changed, 217 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10171/4 -- To view, visit http://gerrit.cloudera.org:8080/10171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 Gerrit-Change-Number: 10171 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6916: Implement COMMENT ON DATABASE
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10171 ) Change subject: IMPALA-6916: Implement COMMENT ON DATABASE .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1081 PS2, Line 1081: getMetaStoreDb() see comment below, might not be needed. http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1578 PS2, Line 1578: getMetaStoreDb() see comment below, might not be needed. http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2853 PS2, Line 2853: private void applyAlterDatabase(org.apache.hadoop.hive.metastore.api.Database msDb) See comment below, might not be needed. http://gerrit.cloudera.org:8080/#/c/10171/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3528 PS2, Line 3528: msDb This could just be "db" to avoid the changes to the methods above. -- To view, visit http://gerrit.cloudera.org:8080/10171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 Gerrit-Change-Number: 10171 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Wed, 25 Apr 2018 03:53:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6902: Tighten udf test to look for crashes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10180 ) Change subject: IMPALA-6902: Tighten udf test to look for crashes .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2358/ -- To view, visit http://gerrit.cloudera.org:8080/10180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7780ff328fde5cf462cf718a1467b8a032d2c17b Gerrit-Change-Number: 10180 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 25 Apr 2018 03:42:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6740: Fix flaky test cancellation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10177 ) Change subject: IMPALA-6740: Fix flaky test_cancellation .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10177 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 Gerrit-Change-Number: 10177 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 25 Apr 2018 01:25:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6740: Fix flaky test cancellation
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10177 ) Change subject: IMPALA-6740: Fix flaky test_cancellation .. IMPALA-6740: Fix flaky test_cancellation test_shall_commandline:test_cancellation starts an Impala shell process, runs a query, sleeps briefly, and then cancels the query by sending a SIGINT to the process. This has been occasionally failing with either the error 'KeyboardInterrupt' or with the query succeeding instead of being cancelled. The problem occurs if the process hasn't fully started up before the SIGINT is sent - in particular, if ImpalaShell:__init__ hasn't installed the signal handler, which happens sometimes depending on concurrent load on the machine. Depending on the exact timing, this may cause a 'KeyboardInterrupt' that isn't handled, or the signal may be ignored and the query allowed to run to completion. The solution is to increase the time spent sleeping. Testing: - I can reliably repro the problem locally by reducing the sleep time. Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 Reviewed-on: http://gerrit.cloudera.org:8080/10177 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins --- M tests/shell/test_shell_commandline.py 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10177 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 Gerrit-Change-Number: 10177 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR](2.x) IMPALA-6899: Optimize the HDFS commands used in dataload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10167 ) Change subject: IMPALA-6899: Optimize the HDFS commands used in dataload .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I0934353329dc7312394fc4457ab8db2a272c6282 Gerrit-Change-Number: 10167 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 25 Apr 2018 01:22:39 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6899: Optimize the HDFS commands used in dataload
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10167 ) Change subject: IMPALA-6899: Optimize the HDFS commands used in dataload .. IMPALA-6899: Optimize the HDFS commands used in dataload HDFS commandline calls can be expensive due to JVM startup and other costs. Since most HDFS commandline calls can take multiple paths, one way to reduce execution time is to consolidate multiple HDFS commands into a single HDFS call. Since HDFS put commands will follow symbolic links and can copy recursively, this can allow for further consolidation by creating the full directory structure and copying it in a single HDFS call. This does several of these optimizations throughout the dataload codepath. It saves a few seconds here and there: Loading Hive Builtins: 1:10 -> 0:30 Loading custom schemas: 0:35 -> 0:20 Loading Hive UDFs: 0:45 -> 0:25 Conflicts: testdata/bin/copy-udfs-udas.sh - conflict due to "Loosen hive-exec.jar glob pattern..." Change-Id: I0934353329dc7312394fc4457ab8db2a272c6282 Reviewed-on: http://gerrit.cloudera.org:8080/10120 Reviewed-by: Philip Zeyliger Tested-by: Impala Public Jenkins (cherry picked from commit da363a99a4b1afff91600c71650e26932be9350a) Reviewed-on: http://gerrit.cloudera.org:8080/10167 Reviewed-by: Joe McDonnell --- M testdata/bin/copy-udfs-udas.sh M testdata/bin/create-load-data.sh M testdata/bin/load-hive-builtins.sh 3 files changed, 131 insertions(+), 122 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I0934353329dc7312394fc4457ab8db2a272c6282 Gerrit-Change-Number: 10167 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR](asf-site) Update download and signature links for 2.12.0 release.
Sailesh Mukil has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10183 Change subject: Update download and signature links for 2.12.0 release. .. Update download and signature links for 2.12.0 release. Change-Id: I08f652b0b14dc04a72788ae6988014b10e645802 --- M downloads.html 1 file changed, 14 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/10183/1 -- To view, visit http://gerrit.cloudera.org:8080/10183 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newchange Gerrit-Change-Id: I08f652b0b14dc04a72788ae6988014b10e645802 Gerrit-Change-Number: 10183 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil
[Impala-ASF-CR](asf-site) Added changelog for 2.12.0
Sailesh Mukil has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10184 Change subject: Added changelog for 2.12.0 .. Added changelog for 2.12.0 Change-Id: Ie78b1397623a5baac42e5bfd1b23f9234b76958b --- A docs/changelog-2.12.html 1 file changed, 541 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/10184/1 -- To view, visit http://gerrit.cloudera.org:8080/10184 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newchange Gerrit-Change-Id: Ie78b1397623a5baac42e5bfd1b23f9234b76958b Gerrit-Change-Number: 10184 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil
[Impala-ASF-CR] add impala-flake8
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/10182 ) Change subject: add impala-flake8 .. Patch Set 1: Code-Review+2 Thanks for doing this. -- To view, visit http://gerrit.cloudera.org:8080/10182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4ce9eca6f8b55eaec1c96e7db1ff630ac016be0 Gerrit-Change-Number: 10182 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Brown Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 24 Apr 2018 22:40:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6923:Update scripts in benchmark folder to store workload and few minor updates
Nithya Janarthanan has posted comments on this change. ( http://gerrit.cloudera.org:8080/10100 ) Change subject: IMPALA-6923:Update scripts in benchmark folder to store workload and few minor updates .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10100/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10100/5//COMMIT_MSG@7 PS5, Line 7: CDH-65183 Update scripts in benchmark folder to store workload and few minor updates > Please use IMPALA tickets. People who do not work at Cloudera cannot read C Done -- To view, visit http://gerrit.cloudera.org:8080/10100 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ica7c00ad59963d466bae9e607a4692af0138962c Gerrit-Change-Number: 10100 Gerrit-PatchSet: 5 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Nithya Janarthanan Gerrit-Comment-Date: Tue, 24 Apr 2018 22:36:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [Preview]IMPALA-5216: Make admission control queuing async
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/10060 )
Change subject: [Preview]IMPALA-5216: Make admission control queuing async
..
Patch Set 2:
(25 comments)
addressed review comment, working on TODOs
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/runtime/coordinator.h
File be/src/runtime/coordinator.h:
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/runtime/coordinator.h@329
PS2, Line 329: bool execution_started_ = false;
> leftover?
Done
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/runtime/coordinator.cc@160
PS2, Line 160: // Ensure 'this' Coordinator object can be safely
destroyed.
: DCHECK(query_status_.ok());
: query_status_ = prepare_status;
: CancelInternal();
> this is somewhat subtle logic, so it'd be good to combine it with the tail
Done
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.h
File be/src/scheduling/admission-controller.h:
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.h@45
PS2, Line 45: AdmissionStatus
> Maybe it should be AdmissionOutcome? This is really the final outcome of ad
Done
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.h@199
PS2, Line 199: /// returns an OK status, schedule->is_admitted() is true and
admit_status is ADMITTED.
> Is (admit_status.IsSet() && admit_status.Get() == ADMITTED) equivalent to s
Done. It seems like the check for isAdmitted() in ReleaseQuery() was redundant
as it is only ever called if the query was successfully admitted.
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.h@395
PS2, Line 395: Promise* admit_status;
> What owns the memory of admit_status?
Done
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.cc@505
PS2, Line 505: lock_guard lock(admission_ctrl_lock_);
> Do we need to worry about races between admit_status being set to CANCELLED
In case admit_status is set to cancelled before this, the following will happen:
for REJECTED the admission controller thread would simple exit after checking
the status returned by AdmitQuery().
for ADMITTED immediately, the cancellation check after returning from
AdmitQuery will take care of it.
I agree that the admit_status would not be consistent with the admission
decision logged, hence I will change this to return immediately if
admit_status->TrySet fails and let the cancellation check in CRS do the rest.
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.cc@576
PS2, Line 576:
admit_status->TrySet(AdmissionStatus::REJECTED_OR_TIMED_OUT);
> It would be nice to log the outcome of admission in all cases. Maybe we sho
that makes sense, but in this case, for every admission decision, there is more
context that needs to be printed to the log which is done by the callee.
For eg, printing the reason of rejection, if admitted immediately, etc.
If you feel that separately printing the admission outcome when tryset is
invoked will help minimize bugs, then we can add it for sure.
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.cc@595
PS2, Line 595: if (queue->Remove(&queue_node)) {
> One line?
Done
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.cc@885
PS2, Line 885: // TODO: Maybe dont even check cancelled here, just try
admitting it and let the
> +1 to removing a code path if it isn't totally necessary. As long as we cat
I thought about this more and it seems like we should keep it. for the case
where a query completes and calls AdmissionController::ReleaseQuery(), this
will notify the dequeue thread, if cancel is called while dequeue thread is
processing the query and CanAdmitRequest is false, then if we dont check for
cancellation there, the dequeue thread will block until it is awoken again.
This will cause some delay in admitting the query waiting in line after the
cancelled query.
The right behavior should be that if it is cancelled, it should dequeue the
query so that it can process the next one.
http://gerrit.cloudera.org:8080/#/c/10060/2/be/src/scheduling/admission-controller.cc@901
PS2, Line 901: DCHECK
> We should really log the actual value if the DCHECK fails. I wish we have a
Done
http://gerrit.cloudera.org:8080/#/c/10060/1/be/src/scheduling/query-schedule.h
File be/src/scheduling/query-schedule.h:
http://gerrit.cloudera.org:8080/#/c/10060/1/be/src/scheduling/query-schedule.h@274
PS1, Line 274:
> Can the coordinator have a refer
[Impala-ASF-CR] [Preview]IMPALA-5216: Make admission control queuing async
Hello Tim Armstrong, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10060 to look at the new patch set (#3). Change subject: [Preview]IMPALA-5216: Make admission control queuing async .. [Preview]IMPALA-5216: Make admission control queuing async Implement asynchronous admission control queuing. This is achieved by running the admission control code-path in a separate thread. Major changes include: propagating cancellation to the admission control thread and dequeuing thread, and ensuring that the state expected by the impala-server from the ClientRequest objects across its life cycle is the same before and after this patch. Testing: - Added a deterministic end to end test - Ran multiple stress tests successfully with a cancellation probability of 60% and with different values for the following parameters: max_requests, queue_wait_timeout_ms. Ensured that the impalad was in a valid state afterwards (no orphan fragments or wrong metrics). TODO: add stress tests. TODO: add test to verify the codepath where dequeue thread comes across a cancelled query. TODO: change terminology of "in_flight_query" to "submitted_queries" need to identify all refernces of this terminology, eg. in comments, tests, variable names, etc. TODO: Currently the impala debug queries page does not show any details related to the query, this is because the http-server does not display anything if query state is below RUNNING. Solution is to use internal representation of states and show info accordingly. Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf --- M be/src/common/atomic.h M be/src/common/logging.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/promise-test.cc M be/src/util/promise.h M common/thrift/ImpalaService.thrift M tests/custom_cluster/test_admission_controller.py 18 files changed, 448 insertions(+), 179 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/10060/3 -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9930 ) Change subject: IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2357/ -- To view, visit http://gerrit.cloudera.org:8080/9930 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id406f4189e01a909152985fabd5cca7a1527a568 Gerrit-Change-Number: 9930 Gerrit-PatchSet: 7 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Tue, 24 Apr 2018 22:34:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE
Taras Bobrovytsky has posted comments on this change. ( http://gerrit.cloudera.org:8080/9930 ) Change subject: IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE .. Patch Set 7: Code-Review+2 Made a minor fix to widetable.py in patch 6. Rebased. Forwarding the +2 from Alex. -- To view, visit http://gerrit.cloudera.org:8080/9930 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id406f4189e01a909152985fabd5cca7a1527a568 Gerrit-Change-Number: 9930 Gerrit-PatchSet: 7 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Tue, 24 Apr 2018 22:33:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE
Taras Bobrovytsky has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/9930 ) Change subject: IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE .. IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE In this patch we implement strict decimal type checking in the FE in various situations when DECIMAL_V2 is enabled. What is affected: - Union. If we union two decimals and it is not possible to come up with a decimal that will be able to contain all the digits, an error is thrown. For example, the union(decimal(20, 10), decimal(20, 20)) returns decimal(30, 20). However, for union(decimal(38, 0), decimal(38, 38)) the ideal return type would be decimal(76,38), but this is too large, so an error is thrown. - Insert. If we are inserting a decimal value into a column where we are not guaranteed that all digits will fit, an error is thrown. For example, inserting a decimal(38,0) value into a decimal(38,38) column. - Functions such as coalesce(). If we are unable to determine the output type that guarantees that all digits will fit from all the arguments, an error is thrown. For example, coalesce(decimal(38,38), decimal(38,0)) will throw an error. - Hash Join. When joining on two decimals, if a type cannot be determined that both columns can be cast to, we throw an error. For example, join on decimal(38,0) and decimal(38,38) will result in an error. To avoid these errors, you need to use CAST() on some of the decimals. In this patch we also change the output decimal calculation of decimal round, truncate and related functions. If these functions are a no-op, the resulting decimal type is the same as the input type. Testing: - Core build passed. Ran an exhaustive build. The errors discovered by the exhaustive build were fixed. Change-Id: Id406f4189e01a909152985fabd5cca7a1527a568 --- M be/src/exprs/expr-test.cc M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ArithmeticExpr.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/CaseExpr.java M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/InPredicate.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LikePredicate.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/RangePartition.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/TimestampArithmeticExpr.java M fe/src/main/java/org/apache/impala/analysis/TypesUtil.java M fe/src/main/java/org/apache/impala/catalog/Function.java M fe/src/main/java/org/apache/impala/catalog/ScalarType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/analysis/TypesUtilTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/common/widetable.py M testdata/workloads/functional-planner/queries/PlannerTest/complex-types-file-formats.test M testdata/workloads/functional-planner/queries/PlannerTest/insert.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/union.test M testdata/workloads/functional-query/queries/QueryTest/aggregation.test M testdata/workloads/functional-query/queries/QueryTest/avro-writer.test M testdata/workloads/functional-query/queries/QueryTest/decimal-exprs.test M testdata/workloads/functional-query/queries/QueryTest/decimal.test M testdata/workloads/funct
[Impala-ASF-CR] IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE
Taras Bobrovytsky has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/9930 ) Change subject: IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE .. IMPALA-6518,IMPALA-6340: Check that decimal types are compatible in FE In this patch we implement strict decimal type checking in the FE in various situations when DECIMAL_V2 is enabled. What is affected: - Union. If we union two decimals and it is not possible to come up with a decimal that will be able to contain all the digits, an error is thrown. For example, the union(decimal(20, 10), decimal(20, 20)) returns decimal(30, 20). However, for union(decimal(38, 0), decimal(38, 38)) the ideal return type would be decimal(76,38), but this is too large, so an error is thrown. - Insert. If we are inserting a decimal value into a column where we are not guaranteed that all digits will fit, an error is thrown. For example, inserting a decimal(38,0) value into a decimal(38,38) column. - Functions such as coalesce(). If we are unable to determine the output type that guarantees that all digits will fit from all the arguments, an error is thrown. For example, coalesce(decimal(38,38), decimal(38,0)) will throw an error. - Hash Join. When joining on two decimals, if a type cannot be determined that both columns can be cast to, we throw an error. For example, join on decimal(38,0) and decimal(38,38) will result in an error. To avoid these errors, you need to use CAST() on some of the decimals. In this patch we also change the output decimal calculation of decimal round, truncate and related functions. If these functions are a no-op, the resulting decimal type is the same as the input type. Testing: - Core build passed. Ran an exhaustive build. The errors discovered by the exhaustive build were fixed. Change-Id: Id406f4189e01a909152985fabd5cca7a1527a568 --- M be/src/exprs/expr-test.cc M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ArithmeticExpr.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/CaseExpr.java M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/InPredicate.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LikePredicate.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/RangePartition.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/TimestampArithmeticExpr.java M fe/src/main/java/org/apache/impala/analysis/TypesUtil.java M fe/src/main/java/org/apache/impala/catalog/Function.java M fe/src/main/java/org/apache/impala/catalog/ScalarType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/analysis/TypesUtilTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/common/widetable.py M testdata/workloads/functional-planner/queries/PlannerTest/complex-types-file-formats.test M testdata/workloads/functional-planner/queries/PlannerTest/insert.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/union.test M testdata/workloads/functional-query/queries/QueryTest/aggregation.test M testdata/workloads/functional-query/queries/QueryTest/avro-writer.test M testdata/workloads/functional-query/queries/QueryTest/decimal-exprs.test M testdata/workloads/functional-query/queries/QueryTest/decimal.test M testdata/workloads/funct
[Impala-ASF-CR] IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries
Alex Behm has posted comments on this change. (
http://gerrit.cloudera.org:8080/9005 )
Change subject: IMPALA-6314: Add run time scalar subquery check for
uncorrelated subqueries
..
Patch Set 29:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/9005/29/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/9005/29/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@484
PS29, Line 484: if (isRuntimeScalar) {
> I don't think this is right if the subquery already had a order by + limit
Sorry I pasted the wrong query here. This is the correct one:
select count(*) from functional.alltypes
where 7 = (select id from functional.alltypestiny
order by id limit 8 offset 7);
The subquery returns a single "7" so the query should succeed, but currently it
fails.
--
To view, visit http://gerrit.cloudera.org:8080/9005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f52b93a60eeacedd242a2f17fa6b99c4fc38e06
Gerrit-Change-Number: 9005
Gerrit-PatchSet: 29
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Dimitris Tsirogiannis
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 24 Apr 2018 22:30:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] add impala-flake8
Michael Brown has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10182 Change subject: add impala-flake8 .. add impala-flake8 Add flake8 and dependencies to impala-python. The versions are compatible with Python 2.6.6. Add the impala-flake8 entry point, similar to impala-python. Add setup.cfg which defines flake8 special rules and exemptions. They are added to support 2-space indents and a max line length of 90. Contributors writing Python can use impala-flake8 to look for formatting mistakes. The two most common uses would be: impala-flake8 myfile.py or git diff HEAD^ myfile.py | impala-flake8 --diff In the second usage, flake8 will only examine lines changed. This allows a contributor to fix their own code and not be penalized by flake8 violations that may already be present (though they are encouraged to fix them if they can!) Change-Id: Ib4ce9eca6f8b55eaec1c96e7db1ff630ac016be0 --- A bin/impala-flake8 M infra/python/deps/requirements.txt A setup.cfg 3 files changed, 47 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/10182/1 -- To view, visit http://gerrit.cloudera.org:8080/10182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib4ce9eca6f8b55eaec1c96e7db1ff630ac016be0 Gerrit-Change-Number: 10182 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Brown
[Impala-ASF-CR] IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries
Alex Behm has posted comments on this change. (
http://gerrit.cloudera.org:8080/9005 )
Change subject: IMPALA-6314: Add run time scalar subquery check for
uncorrelated subqueries
..
Patch Set 29:
(3 comments)
Getting really close :)
http://gerrit.cloudera.org:8080/#/c/9005/29/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/9005/29/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@484
PS29, Line 484: if (isRuntimeScalar) {
I don't think this is right if the subquery already had a order by + limit +
offset clause. For example, the following query should work but it currently
fails because the plan is wrong (the offset is wiped):
select count(*) from functional.alltypes
where 7 = (select int_col from functional.alltypestiny order by id limit 2
offset 20);
http://gerrit.cloudera.org:8080/#/c/9005/29/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@809
PS29, Line 809: throw new AnalysisException("Subquery must return a
single row: " + stmt.toSql());
Unsupported correlated subquery with runtime scalar check:
http://gerrit.cloudera.org:8080/#/c/9005/29/fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
File fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java:
http://gerrit.cloudera.org:8080/#/c/9005/29/fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java@45
PS29, Line 45: child.setLimit(2);
Is this needed? Should we make this a Preconditions check?
--
To view, visit http://gerrit.cloudera.org:8080/9005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0f52b93a60eeacedd242a2f17fa6b99c4fc38e06
Gerrit-Change-Number: 9005
Gerrit-PatchSet: 29
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Dimitris Tsirogiannis
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 24 Apr 2018 22:27:36 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] CDH-65183 Update scripts in benchmark folder to store workload and few minor updates
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/10100 ) Change subject: CDH-65183 Update scripts in benchmark folder to store workload and few minor updates .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10100/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10100/5//COMMIT_MSG@7 PS5, Line 7: CDH-65183 Update scripts in benchmark folder to store workload and few minor updates Please use IMPALA tickets. People who do not work at Cloudera cannot read CDH tickets. -- To view, visit http://gerrit.cloudera.org:8080/10100 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ica7c00ad59963d466bae9e607a4692af0138962c Gerrit-Change-Number: 10100 Gerrit-PatchSet: 5 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Comment-Date: Tue, 24 Apr 2018 22:20:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6802 (part 1): Clean up authorization tests
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10135 ) Change subject: IMPALA-6802 (part 1): Clean up authorization tests .. Patch Set 5: Code-Review+1 Thanks for the updates. -- To view, visit http://gerrit.cloudera.org:8080/10135 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cd5713607c423f644451af5ebb3494d3a728e3b Gerrit-Change-Number: 10135 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Tue, 24 Apr 2018 22:20:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6802 (part 1): Clean up authorization tests
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10135 ) Change subject: IMPALA-6802 (part 1): Clean up authorization tests .. IMPALA-6802 (part 1): Clean up authorization tests The first patch of this patch is to introduce a new mechanism of testing authorization that tests authorization at every hierarchy. This patch rewrites the authorization tests for select statements. Testing: - Added new authorization tests - Ran all front-end tests Cherry-picks: not for 2.x Change-Id: I9cd5713607c423f644451af5ebb3494d3a728e3b --- A fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java 1 file changed, 729 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/10135/5 -- To view, visit http://gerrit.cloudera.org:8080/10135 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9cd5713607c423f644451af5ebb3494d3a728e3b Gerrit-Change-Number: 10135 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6913: Simple parser for Impala profile logs.
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/9265 )
Change subject: IMPALA-6913: Simple parser for Impala profile logs.
..
Patch Set 4:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/9265/3/bin/parse-thrift-profile.py
File bin/parse-thrift-profile.py:
http://gerrit.cloudera.org:8080/#/c/9265/3/bin/parse-thrift-profile.py@60
PS3, Line 60: base64_encoded = space_separated[0]
> I couldn't find anything in the documentation that was particularly clear.
I see. Ok.
http://gerrit.cloudera.org:8080/#/c/9265/3/bin/parse-thrift-profile.py@65
PS3, Line 65: try:
> Do you have a snippet that would do the trick? I wasn't able to find one qu
We could do something like this,
serialize(tree, protocol_factory=TJSONProtocol.TJSONProtocolFactory())
but TJSONProtocol is in 0.9.1 and later. So don't think we can use that. Nvm.
http://gerrit.cloudera.org:8080/#/c/9265/4/bin/parse-thrift-profile.py
File bin/parse-thrift-profile.py:
http://gerrit.cloudera.org:8080/#/c/9265/4/bin/parse-thrift-profile.py@33
PS4, Line 33: # | head -n 1 | awk '{ print $3 }' | parse-profile.py
Could you add the file example too.
http://gerrit.cloudera.org:8080/#/c/9265/4/bin/parse-thrift-profile.py@45
PS4, Line 45: or sys.argv[1] == "-":
Why this? Whats so specific about "-"?
--
To view, visit http://gerrit.cloudera.org:8080/9265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib1174c65b002f9b71eccb0b56f875449f00eae39
Gerrit-Change-Number: 9265
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 24 Apr 2018 22:10:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6679,IMPALA-6678: reduce scan reservation
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9757 ) Change subject: IMPALA-6679,IMPALA-6678: reduce scan reservation .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/9757/9/be/src/exec/hdfs-scan-node.cc File be/src/exec/hdfs-scan-node.cc: http://gerrit.cloudera.org:8080/#/c/9757/9/be/src/exec/hdfs-scan-node.cc@286 PS9, Line 286: DCHECK(!first_thread); > I actually don't understand how this ever worked - we can bail out here if Filed IMPALA-6920. From what I can tell it can't result in any deadlocks or crashes prior to this DCHECK but it's not clear if it's desirable behaviour. -- To view, visit http://gerrit.cloudera.org:8080/9757 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc80e05118a9eef72cac8e2308418122e3ee0842 Gerrit-Change-Number: 9757 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Apr 2018 22:02:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4025: Part 2: Add percentile cont & median aggregation functions
Tianyi Wang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/9778 ) Change subject: IMPALA-4025: Part 2: Add percentile_cont & median aggregation functions .. IMPALA-4025: Part 2: Add percentile_cont & median aggregation functions percentile_cont is implemented in the similar way as percentile_disc, except for using a BE custom aggregation function for interpolating the final result. median is rewritten into percentile_cont(0.5). Some EE tests are added. Tests not related to error handling are verified against PostgreSQL. Change-Id: I2cc184682bb1bf4a5011b69a89e9ae253f3fd88d --- M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/aggregate-functions.h M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/PercentileAggExpr.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M testdata/workloads/functional-query/queries/QueryTest/aggregation.test 11 files changed, 243 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/9778/4 -- To view, visit http://gerrit.cloudera.org:8080/9778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2cc184682bb1bf4a5011b69a89e9ae253f3fd88d Gerrit-Change-Number: 9778 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-4025: Part 1: Add percentile disc aggregation function
Tianyi Wang has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/9777 ) Change subject: IMPALA-4025: Part 1: Add percentile_disc aggregation function .. IMPALA-4025: Part 1: Add percentile_disc aggregation function This patch adds aggregation function percentile_disc. The implementation is rewriting it into an inline view. The inline view computes the row number on the ordering expr using analytic functions. The parent query then picks the desired row using aggregation. An Example of such rewrite is in StmtRewriter.java. The behavior of this function is mostly the same as in PostgreSQL. The handling of percentile expr not in [0, 1] is different: PostgreSQL throws an error and impala returns NULL. Some FE and EE tests are added. EE tests not related to the above difference are verified against PostgreSQL. Change-Id: Iacef7b3fcd74c4c73d88400ce27307c3baa0121e --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java A fe/src/main/java/org/apache/impala/analysis/PercentileAggExpr.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/catalog/AggregateFunction.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M testdata/workloads/functional-planner/queries/PlannerTest/aggregation.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-query/queries/QueryTest/aggregation.test 23 files changed, 1,263 insertions(+), 127 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/9777/5 -- To view, visit http://gerrit.cloudera.org:8080/9777 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacef7b3fcd74c4c73d88400ce27307c3baa0121e Gerrit-Change-Number: 9777 Gerrit-PatchSet: 5 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6902: Tighten udf test to look for crashes
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/10180 ) Change subject: IMPALA-6902: Tighten udf test to look for crashes .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7780ff328fde5cf462cf718a1467b8a032d2c17b Gerrit-Change-Number: 10180 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 24 Apr 2018 21:50:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6902: Tighten udf test to look for crashes
Vuk Ercegovac has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10180 Change subject: IMPALA-6902: Tighten udf test to look for crashes .. IMPALA-6902: Tighten udf test to look for crashes test_native_functions_race assumes that any exception is grounds for failing the test. However, the flake described in IMPALA-6902 shows an example of an exception that causes a false negative (it should be ignored). This change tightens the test to fail when an impalad crashes, which was the original intent of the test. Tests: - re-ran test_udfs.py Change-Id: I7780ff328fde5cf462cf718a1467b8a032d2c17b --- M tests/query_test/test_udfs.py 1 file changed, 8 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/10180/1 -- To view, visit http://gerrit.cloudera.org:8080/10180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7780ff328fde5cf462cf718a1467b8a032d2c17b Gerrit-Change-Number: 10180 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac
[Impala-ASF-CR] CDH-65183 Update scripts in benchmark folder to store workload and few minor updates
Mostafa Mokhtar has posted comments on this change. (
http://gerrit.cloudera.org:8080/10100 )
Change subject: CDH-65183 Update scripts in benchmark folder to store workload
and few minor updates
..
Patch Set 5:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10100/5/tests/benchmark/perf_result_datastore.py
File tests/benchmark/perf_result_datastore.py:
http://gerrit.cloudera.org:8080/#/c/10100/5/tests/benchmark/perf_result_datastore.py@298
PS5, Line 298: insert into workloadmetrics
Fix indentation
http://gerrit.cloudera.org:8080/#/c/10100/5/tests/benchmark/perf_result_datastore.py@338
PS5, Line 338: profile = profile.replace('\"', '')
Would be good to add a comment explaining why this is happening?
The quotes here " break line below, would single quotes work instead?
http://gerrit.cloudera.org:8080/#/c/10100/5/tests/benchmark/report_benchmark_results.py
File tests/benchmark/report_benchmark_results.py:
http://gerrit.cloudera.org:8080/#/c/10100/5/tests/benchmark/report_benchmark_results.py@735
PS5, Line 735: if not first_exec_summary:
Add comment.
Metadata only queries don't contain a summary, this code is to handle that case.
Metadata query is for something like "alter table foo recover partitions"
http://gerrit.cloudera.org:8080/#/c/10100/5/tests/benchmark/report_benchmark_results.py@1058
PS5, Line 1058: if exec_summaries[0] is None:
Same as comment above, please add a comment line explaining what this does.
--
To view, visit http://gerrit.cloudera.org:8080/10100
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ica7c00ad59963d466bae9e607a4692af0138962c
Gerrit-Change-Number: 10100
Gerrit-PatchSet: 5
Gerrit-Owner: Nithya Janarthanan
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Michael Brown
Gerrit-Reviewer: Mostafa Mokhtar
Gerrit-Comment-Date: Tue, 24 Apr 2018 21:28:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) IMPALA-6899: Optimize the HDFS commands used in dataload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10167 ) Change subject: IMPALA-6899: Optimize the HDFS commands used in dataload .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2356/ -- To view, visit http://gerrit.cloudera.org:8080/10167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I0934353329dc7312394fc4457ab8db2a272c6282 Gerrit-Change-Number: 10167 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 24 Apr 2018 21:28:52 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6899: Optimize the HDFS commands used in dataload
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/10167 ) Change subject: IMPALA-6899: Optimize the HDFS commands used in dataload .. Patch Set 1: The test failure looks like IMPALA-6740. My previous gerrit-verify-dryrun-external run didn't see this issue, retrying. -- To view, visit http://gerrit.cloudera.org:8080/10167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I0934353329dc7312394fc4457ab8db2a272c6282 Gerrit-Change-Number: 10167 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 24 Apr 2018 21:28:03 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6643: Add REFRESH fine-grained privilege
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10179 ) Change subject: IMPALA-6643: Add REFRESH fine-grained privilege .. Patch Set 1: Clean cherry-pick. -- To view, visit http://gerrit.cloudera.org:8080/10179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I4c3c5a51fe493d39fd719c7a388d4d5760049ce4 Gerrit-Change-Number: 10179 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Apr 2018 21:25:51 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6643: Add REFRESH fine-grained privilege
Hello Alex Behm, Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10179 to review the following change. Change subject: IMPALA-6643: Add REFRESH fine-grained privilege .. IMPALA-6643: Add REFRESH fine-grained privilege Before this patch, ALL privilege was required to execute INVALIDATE METADATA and having any privilege allowed executing REFRESH and INVALIDATE METADATA . With this patch, REFRESH privilege is now required to execute INVALIDATE METADATA or REFRESH statement. These are the new GRANT/REVOKE statements introduced at server, database, and table scopes. GRANT REFRESH on SERVER svr TO ROLE testrole; GRANT REFRESH on DATABASE db TO ROLE testrole; GRANT REFRESH on TABLE db.tbl TO ROLE testrole; REVOKE REFRESH on SERVER svr FROM ROLE testrole; REVOKE REFRESH on DATABASE db FROM ROLE testrole; REVOKE REFRESH on TABLE db.tbl FROM ROLE testrole; Testing: - Ran front-end tests Change-Id: I4c3c5a51fe493d39fd719c7a388d4d5760049ce4 Reviewed-on: http://gerrit.cloudera.org:8080/9589 Reviewed-by: Alex Behm Tested-by: Impala Public Jenkins --- M common/thrift/CatalogObjects.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/Privilege.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/resources/authz-policy.ini.template 10 files changed, 204 insertions(+), 51 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/10179/1 -- To view, visit http://gerrit.cloudera.org:8080/10179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I4c3c5a51fe493d39fd719c7a388d4d5760049ce4 Gerrit-Change-Number: 10179 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-6740: Fix flaky test cancellation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10177 ) Change subject: IMPALA-6740: Fix flaky test_cancellation .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2355/ -- To view, visit http://gerrit.cloudera.org:8080/10177 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 Gerrit-Change-Number: 10177 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 24 Apr 2018 21:17:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6740: Fix flaky test cancellation
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/10177 ) Change subject: IMPALA-6740: Fix flaky test_cancellation .. Patch Set 3: Code-Review+2 (1 comment) carrying forward http://gerrit.cloudera.org:8080/#/c/10177/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10177/2//COMMIT_MSG@25 PS2, Line 25: - I can reliably repro the problem locally by reducing the sleep time. > reducing Done -- To view, visit http://gerrit.cloudera.org:8080/10177 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 Gerrit-Change-Number: 10177 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 24 Apr 2018 21:17:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6740: Fix flaky test cancellation
Hello Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10177 to look at the new patch set (#3). Change subject: IMPALA-6740: Fix flaky test_cancellation .. IMPALA-6740: Fix flaky test_cancellation test_shall_commandline:test_cancellation starts an Impala shell process, runs a query, sleeps briefly, and then cancels the query by sending a SIGINT to the process. This has been occasionally failing with either the error 'KeyboardInterrupt' or with the query succeeding instead of being cancelled. The problem occurs if the process hasn't fully started up before the SIGINT is sent - in particular, if ImpalaShell:__init__ hasn't installed the signal handler, which happens sometimes depending on concurrent load on the machine. Depending on the exact timing, this may cause a 'KeyboardInterrupt' that isn't handled, or the signal may be ignored and the query allowed to run to completion. The solution is to increase the time spent sleeping. Testing: - I can reliably repro the problem locally by reducing the sleep time. Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 --- M tests/shell/test_shell_commandline.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/10177/3 -- To view, visit http://gerrit.cloudera.org:8080/10177 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 Gerrit-Change-Number: 10177 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-6821: Push down limits into Kudu
Hello Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10119 to look at the new patch set (#3). Change subject: IMPALA-6821: Push down limits into Kudu .. IMPALA-6821: Push down limits into Kudu This patch takes advantage of a recent change in Kudu (KUDU-16) that exposes the ability to set limits on KuduScanners. Since each KuduScanner corresponds to a scan token, and there will be multiple scan tokens per query, this is just a performance optimization in cases where the limit is smaller than the number of rows per token, and Impala still needs to apply the limit on our side for cases where the limit is greater than the number of rows per token. Testing: - Added e2e tests for various situations where limits are applied at a Kudu scan node. - For the query 'select * from tpch_kudu.lineitem limit 1', a best case perf scenario for this change where the limit is highly effective, the time spent in the Kudu scan node was reduced from 6.107ms to 3.498ms (avg over 3 runs). - For the query 'select count(*) from (select * from tpch_kudu.lineitem limit 100) v', a worst case perf scenario for this change where the limit is ineffective, the time spent in the Kudu scan node was essentially unchanged, 32.815ms previously vs. 29.532ms (avg over 3 runs). Change-Id: Ibe35e70065d8706b575e24fe20902cd405b49941 --- M be/src/exec/kudu-scanner.cc A testdata/workloads/functional-query/queries/QueryTest/kudu_limit.test M tests/query_test/test_kudu.py 3 files changed, 68 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/10119/3 -- To view, visit http://gerrit.cloudera.org:8080/10119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe35e70065d8706b575e24fe20902cd405b49941 Gerrit-Change-Number: 10119 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-6821: Push down limits into Kudu
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/10119 )
Change subject: IMPALA-6821: Push down limits into Kudu
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10119/2/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:
http://gerrit.cloudera.org:8080/#/c/10119/2/tests/query_test/test_queries.py@60
PS2, Line 60: self.run_test_case('QueryTest/limit', vector)
> I'd prefer to revert this move and add a new test specifically for Kudu in
Done
--
To view, visit http://gerrit.cloudera.org:8080/10119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe35e70065d8706b575e24fe20902cd405b49941
Gerrit-Change-Number: 10119
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Tue, 24 Apr 2018 21:14:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6802 (part 1): Clean up authorization tests
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/10135 ) Change subject: IMPALA-6802 (part 1): Clean up authorization tests .. IMPALA-6802 (part 1): Clean up authorization tests The first patch of this patch is to introduce a new mechanism of testing authorization that tests authorization at every hierarchy. This patch rewrites the authorization tests for select statements. Testing: - Added new authorization tests - Ran all front-end tests Cherry-picks: not for 2.x Change-Id: I9cd5713607c423f644451af5ebb3494d3a728e3b --- A fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java 1 file changed, 700 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/10135/4 -- To view, visit http://gerrit.cloudera.org:8080/10135 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9cd5713607c423f644451af5ebb3494d3a728e3b Gerrit-Change-Number: 10135 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] CDH-65183 Update scripts in benchmark folder to store workload and few minor updates
Nithya Janarthanan has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10100 Change subject: CDH-65183 Update scripts in benchmark folder to store workload and few minor updates .. CDH-65183 Update scripts in benchmark folder to store workload and few minor updates Change-Id: Ica7c00ad59963d466bae9e607a4692af0138962c --- M tests/benchmark/create_database.py M tests/benchmark/perf_result_datastore.py M tests/benchmark/report_benchmark_results.py 3 files changed, 180 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/10100/5 -- To view, visit http://gerrit.cloudera.org:8080/10100 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ica7c00ad59963d466bae9e607a4692af0138962c Gerrit-Change-Number: 10100 Gerrit-PatchSet: 5 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Mostafa Mokhtar
[Impala-ASF-CR] IMPALA-5392: Added all stack frames to ThreadInfo summary.
Abhishek Sharma has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/10145 ) Change subject: IMPALA-5392: Added all stack frames to ThreadInfo summary. .. IMPALA-5392: Added all stack frames to ThreadInfo summary. The current implementation uses ThreadInfo.toString. This implementation restricts the number of stack frames to 8. As a part of this fix, only this particular constraint is removed. Now all stack frames are included in the summary. No other detail is added or edited or omitted apart from the constraint stated above. A standard ThreadInfo summary includes - - Thread ID - Thread Name - Thread State - Entire Stack Trace (if any) - Monitor Info (if any) - Lock Info Change-Id: I80ab4aad03e0c1f01fecad6b87779531244c28b7 --- M fe/src/main/java/org/apache/impala/common/JniUtil.java 1 file changed, 76 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/10145/4 -- To view, visit http://gerrit.cloudera.org:8080/10145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I80ab4aad03e0c1f01fecad6b87779531244c28b7 Gerrit-Change-Number: 10145 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Sharma Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Charles Agnello Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] IMPALA-6740: Fix flaky test cancellation
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10177 ) Change subject: IMPALA-6740: Fix flaky test_cancellation .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10177/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10177/2//COMMIT_MSG@25 PS2, Line 25: - I can reliably repro the problem locally by reduing the sleep time. reducing -- To view, visit http://gerrit.cloudera.org:8080/10177 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 Gerrit-Change-Number: 10177 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 24 Apr 2018 20:55:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6740: Fix flaky test cancellation
Philip Zeyliger has posted comments on this change. (
http://gerrit.cloudera.org:8080/10177 )
Change subject: IMPALA-6740: Fix flaky test_cancellation
..
Patch Set 2: Code-Review+2
It's totally academic, but if you want to see if a Linux process is listening
to a signal, you can figure it out based on /proc/self/status:
>>> print bin(int([x for x in file("/proc/self/status").readlines() if "SigCgt"
>>> in x][0].split("\t")[1].strip(), 16))
0b110001010
>>> def f():
...pass
>>> import signal
>>> signal.signal(8, f)
0
>>> print bin(int([x for x in file("/proc/self/status").readlines() if "SigCgt"
>>> in x][0].split("\t")[1].strip(), 16))
0b1100010001010
Note how signal 8 had its bit flipped.
--
To view, visit http://gerrit.cloudera.org:8080/10177
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0
Gerrit-Change-Number: 10177
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Tue, 24 Apr 2018 20:54:54 +
Gerrit-HasComments: No
[native-toolchain-CR] thrift-0.9.3-p4: forward compatibility of TLS protocols
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10129 ) Change subject: thrift-0.9.3-p4: forward compatibility of TLS protocols .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifdca94a9426feff5ab52a80cf4da669a3fbfe812 Gerrit-Change-Number: 10129 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Tue, 24 Apr 2018 20:41:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6679,IMPALA-6678: reduce scan reservation
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9757 ) Change subject: IMPALA-6679,IMPALA-6678: reduce scan reservation .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/9757/9/be/src/exec/hdfs-scan-node.cc File be/src/exec/hdfs-scan-node.cc: http://gerrit.cloudera.org:8080/#/c/9757/9/be/src/exec/hdfs-scan-node.cc@286 PS9, Line 286: DCHECK(!first_thread); > This DCHECK can be hit because in some places the count is decremented befo I actually don't understand how this ever worked - we can bail out here if this is the first thread and I don't see how we're guaranteed to make forward progress in that case. I guess we would depend on whatever else is running invoking the callback at some point, but that doesn't seem to be sufficient to avoid all possible deadlocks. I'm going to investigate further to see if there's a pre-existing bug. -- To view, visit http://gerrit.cloudera.org:8080/9757 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc80e05118a9eef72cac8e2308418122e3ee0842 Gerrit-Change-Number: 9757 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Apr 2018 20:26:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) IMPALA-6899: Optimize the HDFS commands used in dataload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10167 ) Change subject: IMPALA-6899: Optimize the HDFS commands used in dataload .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2354/ -- To view, visit http://gerrit.cloudera.org:8080/10167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I0934353329dc7312394fc4457ab8db2a272c6282 Gerrit-Change-Number: 10167 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 24 Apr 2018 20:22:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5690: Part 2: Upgrade thrift to 0.9.3-p4
Tianyi Wang has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/9300 ) Change subject: IMPALA-5690: Part 2: Upgrade thrift to 0.9.3-p4 .. IMPALA-5690: Part 2: Upgrade thrift to 0.9.3-p4 Dependency changes: - BE and python use thrift 0.9.3-p4 from native-toolchain. - FE uses thrift 0.9.3 from apache maven repo. - Fb303 and http components dependencies are no longer needed in FE and are removed. - The minimum openssl version requirement is increased to 1.0.1. Configuration change: - Thrift codegen option movable_type is enabled. New code no longer needs to use std::swap to avoid copying. Change-Id: I639227721502eaa10398d9490ff6ac63aa71b3a6 --- M CMakeLists.txt M be/src/common/init.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/authentication.cc M be/src/rpc/thrift-server-test.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-thread.h M be/src/rpc/thrift-util.cc M bin/impala-config.sh M buildall.sh M common/thrift/CMakeLists.txt M fe/pom.xml M infra/python/deps/compiled-requirements.txt 15 files changed, 80 insertions(+), 178 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/9300/12 -- To view, visit http://gerrit.cloudera.org:8080/9300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I639227721502eaa10398d9490ff6ac63aa71b3a6 Gerrit-Change-Number: 9300 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: anujphadke
[Impala-ASF-CR] IMPALA-6913: Simple parser for Impala profile logs.
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/9265 ) Change subject: IMPALA-6913: Simple parser for Impala profile logs. .. Patch Set 4: > Patch Set 3: > > (2 comments) > > I added a tiny bit of cmdline parsing to allow specifying input not via > stdin. > I added this as I was looking at the exception handling and decided > to keep it. Just an FYI -- there is something very similar to this in Impala-aux already. I don't recall who wrote it, or if anyone still uses it. Just pointing out its existence. -- To view, visit http://gerrit.cloudera.org:8080/9265 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1174c65b002f9b71eccb0b56f875449f00eae39 Gerrit-Change-Number: 9265 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Apr 2018 19:48:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6889: Fix race around terminating processes
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/10156 ) Change subject: IMPALA-6889: Fix race around terminating processes .. Patch Set 2: Code-Review+2 Thanks again for investigating this. -- To view, visit http://gerrit.cloudera.org:8080/10156 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecf630e0b71d91469650636e81f940a7bec07113 Gerrit-Change-Number: 10156 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Comment-Date: Tue, 24 Apr 2018 19:44:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6740: Fix flaky test cancellation
Thomas Tauber-Marshall has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10177 ) Change subject: IMPALA-6740: Fix flaky test_cancellation .. IMPALA-6740: Fix flaky test_cancellation test_shall_commandline:test_cancellation starts an Impala shell process, runs a query, sleeps briefly, and then cancels the query by sending a SIGINT to the process. This has been occasionally failing with either the error 'KeyboardInterrupt' or with the query succeeding instead of being cancelled. The problem occurs if the process hasn't fully started up before the SIGINT is sent - in particular, if ImpalaShell:__init__ hasn't installed the signal handler, which happens sometimes depending on concurrent load on the machine. Depending on the exact timing, this may cause a 'KeyboardInterrupt' that isn't handled, or the signal may be ignored and the query allowed to run to completion. The solution is to increase the time spent sleeping. Testing: - I can reliably repro the problem locally by reduing the sleep time. Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 --- M tests/shell/test_shell_commandline.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/10177/2 -- To view, visit http://gerrit.cloudera.org:8080/10177 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 Gerrit-Change-Number: 10177 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-6740: Fix flaky test cancellation
Thomas Tauber-Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10177 Change subject: IMPALA-6740: Fix flaky test_cancellation .. IMPALA-6740: Fix flaky test_cancellation test_shall_commandline:test_cancellation starts an Impala shell process, runs a query, sleeps briefly, and then cancels the query by sending a SIGINT to the process. This has been occasionally failing with the error 'KeyboardInterrupt'. The problem occurs if the process hasn't fully started up before the SIGINT is sent - in particular, if ImpalaShell:__init__ hasn't installed the signal handler, which happens sometimes depending on concurrent load on the machine. The solution is to increase the time spent sleeping. Testing: - I can reliably repro the problem locally by reduing the sleep time. Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 --- M tests/shell/test_shell_commandline.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/10177/1 -- To view, visit http://gerrit.cloudera.org:8080/10177 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5d13de6207807e4ba2e2e406a29d670f01d6c3a0 Gerrit-Change-Number: 10177 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-6916: Implement COMMENT ON DATABASE
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/10171 ) Change subject: IMPALA-6916: Implement COMMENT ON DATABASE .. IMPALA-6916: Implement COMMENT ON DATABASE This patch implements updating comment on a database. Syntax: COMMENT ON DATABASE db IS 'comment' Testing: - Added new front-end tests - Ran all front-end tests - Added new end-to-end tests - Ran end-to-end DDL tests Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 --- M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/CommentOnDbStmt.java A fe/src/main/java/org/apache/impala/analysis/CommentOnStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 13 files changed, 221 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10171/3 -- To view, visit http://gerrit.cloudera.org:8080/10171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifcf909c18f97073346f6f603538bf921e69fbb00 Gerrit-Change-Number: 10171 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6802 (part 1): Clean up authorization tests
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10135 ) Change subject: IMPALA-6802 (part 1): Clean up authorization tests .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/10135/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java: http://gerrit.cloudera.org:8080/#/c/10135/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@95 PS2, Line 95: , > Maybe we don't need it for every test, but where is the test to ensure that Done http://gerrit.cloudera.org:8080/#/c/10135/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@509 PS2, Line 509: > The problem is, if I expect an error to be "... not authorized on function Done. Instead of a boolean, there's a custom Matcher that we can use. -- To view, visit http://gerrit.cloudera.org:8080/10135 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cd5713607c423f644451af5ebb3494d3a728e3b Gerrit-Change-Number: 10135 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Tue, 24 Apr 2018 18:54:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6802 (part 1): Clean up authorization tests
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/10135 ) Change subject: IMPALA-6802 (part 1): Clean up authorization tests .. IMPALA-6802 (part 1): Clean up authorization tests The first patch of this patch is to introduce a new mechanism of testing authorization that tests authorization at every hierarchy. This patch rewrites the authorization tests for select statements. Testing: - Added new authorization tests - Ran all front-end tests Cherry-picks: not for 2.x Change-Id: I9cd5713607c423f644451af5ebb3494d3a728e3b --- A fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java 1 file changed, 701 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/10135/3 -- To view, visit http://gerrit.cloudera.org:8080/10135 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9cd5713607c423f644451af5ebb3494d3a728e3b Gerrit-Change-Number: 10135 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6679,IMPALA-6678: reduce scan reservation
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9757 ) Change subject: IMPALA-6679,IMPALA-6678: reduce scan reservation .. Patch Set 9: (2 comments) http://gerrit.cloudera.org:8080/#/c/9757/9/be/src/exec/hdfs-scan-node.cc File be/src/exec/hdfs-scan-node.cc: http://gerrit.cloudera.org:8080/#/c/9757/9/be/src/exec/hdfs-scan-node.cc@286 PS9, Line 286: DCHECK(!first_thread); This DCHECK can be hit because in some places the count is decremented before the thread token is released. There may be a similar bug with reservations. http://gerrit.cloudera.org:8080/#/c/9757/9/be/src/exec/hdfs-scan-node.cc@291 PS9, Line 291: // Deduct the reservation. We haven't dropped the lock since the Stale comment. -- To view, visit http://gerrit.cloudera.org:8080/9757 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc80e05118a9eef72cac8e2308418122e3ee0842 Gerrit-Change-Number: 9757 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Apr 2018 18:47:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3282: [DOCS] Adds regexp escape built-in function
Alex Rodoni has posted comments on this change. (
http://gerrit.cloudera.org:8080/10174 )
Change subject: IMPALA-3282: [DOCS] Adds regexp_escape built-in function
..
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/10174/1/docs/topics/impala_string_functions.xml
File docs/topics/impala_string_functions.xml:
http://gerrit.cloudera.org:8080/#/c/10174/1/docs/topics/impala_string_functions.xml@705
PS1, Line 705: .\+*?[^]$(){}=!<>|:-
Need to put this in CDATA. <> characters confuses XML converter.
http://gerrit.cloudera.org:8080/#/c/10174/1/docs/topics/impala_string_functions.xml@705
PS1, Line 705: RE2's special characters: .\+*?[^]$(){}=!<>|:-
so that the special characters are interpreted literally rather than as special
characters.
http://gerrit.cloudera.org:8080/#/c/10174/1/docs/topics/impala_string_functions.xml@718
PS1, Line 718: [localhost:21000] > select regexp_escape('Hello.world');
: +--+
: | regexp_escape('Hello.world') |
: +--+
: | Hello\.world |
: +--+
: Returned 1 row(s) in 0.12s
:
: This example shows escaping all the special
characters in RE2.
:
: [localhost:21000] > select
regexp_escape('a.b\\c+d*e?f[g]h$i(j)k{l}m=n!oq|r:s-t');
: ++
: | regexp_escape('a.b\\c+d*e?f[g]h$i(j)k{l}m=n!oq|r:s-t') |
: ++
: | a\.b\\c\+d\*e\?f\[g\]h\$i\(j\)k\{l\}m\=n\!o\q\|r\:s\-t |
: ++
: Returned 1 row(s) in 0.12s
Put this in CDATA, as well.
--
To view, visit http://gerrit.cloudera.org:8080/10174
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf
Gerrit-Change-Number: 10174
Gerrit-PatchSet: 1
Gerrit-Owner: Kim Jin Chul
Gerrit-Reviewer: Alex Rodoni
Gerrit-Comment-Date: Tue, 24 Apr 2018 17:44:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6070: Further improvements to test-with-docker.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10123 ) Change subject: IMPALA-6070: Further improvements to test-with-docker. .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/10123/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10123/1//COMMIT_MSG@26 PS1, Line 26: it more in Python, with a class representing a suite with a > Nit: possible typo (it -> bit) Done http://gerrit.cloudera.org:8080/#/c/10123/1/docker/entrypoint.sh File docker/entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/10123/1/docker/entrypoint.sh@379 PS1, Line 379: env > I know this is not likely to run against S3 anytime soon, but would you min Done http://gerrit.cloudera.org:8080/#/c/10123/1/docker/test-with-docker.py File docker/test-with-docker.py: http://gerrit.cloudera.org:8080/#/c/10123/1/docker/test-with-docker.py@283 PS1, Line 283: ne > nit: typo: environmental Done -- To view, visit http://gerrit.cloudera.org:8080/10123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I43fe124f00340afa21ad1eeb6432d6d50151ca7c Gerrit-Change-Number: 10123 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 24 Apr 2018 17:28:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6070: Further improvements to test-with-docker.
Hello Laszlo Gaal, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10123 to look at the new patch set (#3). Change subject: IMPALA-6070: Further improvements to test-with-docker. .. IMPALA-6070: Further improvements to test-with-docker. This commit tackles a few additions and improvements to test-with-docker. In general, I'm adding workloads (e.g., exhaustive, rat-check), tuning memory setting and parallelism, and trying to speed things up. Bug fixes: * Embarassingly, I was still skipping thrift-server-test in the backend tests. This was a mistake in handling feedback from my last review. * I made the timeline a little bit taller to clip less. Adding workloads: * I added the RAT licensing check. * I added exhaustive runs. This led me to model the suites a little bit more in Python, with a class representing a suite with a bunch of data about the suite. It's not perfect and still coupled with the entrypoint.sh shell script, but it feels workable. As part of adding exhaustive tests, I had to re-work the timeout handling, since now different suites meaningfully have different timeouts. Speed ups: * To speed up test runs, I added a mechanism to split py.test suites into multiple shards with a py.test argument. This involved a little bit of work in conftest.py, and exposing $RUN_CUSTOM_CLUSTER_TESTS_ARGS in run-all-tests.sh. Furthermore, I moved a bit more logic about managing the list of suites into Python. * Doing the full build with "-notests" and only building the backend tests in the relevant target that needs them. This speeds up "docker commit" significantly by removing about 20GB from the container. I had to indicates that expr-codegen-test depends on expr-codegen-test-ir, which was missing. * I sped up copying the Kudu data: previously I did both a move and a copy; now I'm doing a move followed by a move. One of the moves is cross-filesystem so is slow, but this does half the amount of copying. Memory usage: * I tweaked the memlimit_gb settings to have a higher default. I've been fighting empirically to have the tests run well on c4.8xlarge and m4.10xlarge. The more memory a minicluster and test suite run uses, the fewer parallel suites we can run. By observing the peak processes at the tail of a run (with a new "memory_usage" function that uses a ps/sort/awk trick) and by observing peak container total_rss, I found that we had several JVMs that didn't have Xmx settings set. I added Xms/Xmx settings in a few places: * The non-first Impalad does very little JVM work, so having an Xmx keeps it small, even in the parallel tests. * Datanodes do work, but they essentially were never garbage collecting, because JVM defaults let them use up to 1/4th the machine memory. (I observed this based on RSS at the end of the run; nothing fancier.) Adding Xms/Xmx settings helped. * Similarly, I piped the settings through to HBase. A few daemons still run without resource limitations, but they don't seem to be a problem. Change-Id: I43fe124f00340afa21ad1eeb6432d6d50151ca7c --- M be/src/exprs/CMakeLists.txt M bin/run-all-tests.sh M docker/entrypoint.sh M docker/monitor.py M docker/test-with-docker.py M docker/timeline.html.template M testdata/bin/run-hbase.sh M testdata/cluster/node_templates/common/etc/init.d/hdfs-common M tests/conftest.py M tests/run-tests.py 10 files changed, 425 insertions(+), 148 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/10123/3 -- To view, visit http://gerrit.cloudera.org:8080/10123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I43fe124f00340afa21ad1eeb6432d6d50151ca7c Gerrit-Change-Number: 10123 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-6913: Simple parser for Impala profile logs.
Philip Zeyliger has posted comments on this change. (
http://gerrit.cloudera.org:8080/9265 )
Change subject: IMPALA-6913: Simple parser for Impala profile logs.
..
Patch Set 3:
(2 comments)
I added a tiny bit of cmdline parsing to allow specifying input not via stdin.
I added this as I was looking at the exception handling and decided to keep it.
http://gerrit.cloudera.org:8080/#/c/9265/3/bin/parse-thrift-profile.py
File bin/parse-thrift-profile.py:
http://gerrit.cloudera.org:8080/#/c/9265/3/bin/parse-thrift-profile.py@60
PS3, Line 60: thrift = possibly_compressed
> nit: Check for specific error code Z_DATA_ERROR, else raise?
I couldn't find anything in the documentation that was particularly clear.
Here's what you can get at :
(Pdb) p e.args
('Error -3 while decompressing data: incorrect header check',)
(Pdb) p e.message
'Error -3 while decompressing data: incorrect header check'
I don't think there's much value in being more clever here; either it'll parse
as Thrift or it won't.
http://gerrit.cloudera.org:8080/#/c/9265/3/bin/parse-thrift-profile.py@65
PS3, Line 65: print tree
> How about printing a JSON instead? That is more readable.
Do you have a snippet that would do the trick? I wasn't able to find one
quickly.
This serves the purpose I have for it, namely looking through the profile logs
from tests to find queries. Future work may extend this.
--
To view, visit http://gerrit.cloudera.org:8080/9265
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib1174c65b002f9b71eccb0b56f875449f00eae39
Gerrit-Change-Number: 9265
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 24 Apr 2018 17:23:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6913: Simple parser for Impala profile logs.
Hello Bharath Vissapragada, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9265 to look at the new patch set (#4). Change subject: IMPALA-6913: Simple parser for Impala profile logs. .. IMPALA-6913: Simple parser for Impala profile logs. It's hard to find an example of how to parse the profiles in Impala test logs, so I've added it to the bin/ directory. The parser's not full-featured, but it may be useful to others. It handles three common cases of Thrift-encoded profiles that we see. Change-Id: Ib1174c65b002f9b71eccb0b56f875449f00eae39 --- A bin/parse-thrift-profile.py 1 file changed, 73 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/9265/4 -- To view, visit http://gerrit.cloudera.org:8080/9265 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib1174c65b002f9b71eccb0b56f875449f00eae39 Gerrit-Change-Number: 9265 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6912: [DOCS] A new query option COMPUTE STATS MIN SAMPLE SIZE
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10172 ) Change subject: IMPALA-6912: [DOCS] A new query option COMPUTE_STATS_MIN_SAMPLE_SIZE .. Patch Set 2: > (1 comment) Yes, I fixed it in a separate patch. -- To view, visit http://gerrit.cloudera.org:8080/10172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibd6f55a4c6c99185baf7974f68935ba4e03579df Gerrit-Change-Number: 10172 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Apr 2018 17:20:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6536: Disable HMS stats collection in createTable().
Alex Behm has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10176 Change subject: IMPALA-6536: Disable HMS stats collection in createTable(). .. IMPALA-6536: Disable HMS stats collection in createTable(). Sets the DO_NOT_UPDATE_STATS table property to true before calling the HMS create_table() RPC to avoid the expensive file listing on the HMS side to compute simple stats. Those simple stats like file count and file sizes are not useful to Impala without also running a full COMPUTE STATS. The expensive file listing can be especially bad for file systems like S3. This patch relies on a recent fix in Hive: HIVE-18743 Testing: - ran test_ddl.py locally - core/hdfs run passed No new tests are added: - Our mini cluster is set up with hive.stats.autogather=false, which already prevents the HMS-side stats computation - Hive has testing as part of HIVE-18743 Change-Id: I2d37f3752106f90e5c0ab2e8c0f5bcca98d6d172 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py 3 files changed, 12 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/10176/1 -- To view, visit http://gerrit.cloudera.org:8080/10176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2d37f3752106f90e5c0ab2e8c0f5bcca98d6d172 Gerrit-Change-Number: 10176 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Behm
[Impala-ASF-CR](2.x) IMPALA-6899: Optimize the HDFS commands used in dataload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10167 ) Change subject: IMPALA-6899: Optimize the HDFS commands used in dataload .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2354/ -- To view, visit http://gerrit.cloudera.org:8080/10167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I0934353329dc7312394fc4457ab8db2a272c6282 Gerrit-Change-Number: 10167 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 24 Apr 2018 16:22:37 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6899: Optimize the HDFS commands used in dataload
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/10167 ) Change subject: IMPALA-6899: Optimize the HDFS commands used in dataload .. Patch Set 1: Code-Review+2 Tests ran without issue, moving forward with backport. -- To view, visit http://gerrit.cloudera.org:8080/10167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I0934353329dc7312394fc4457ab8db2a272c6282 Gerrit-Change-Number: 10167 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 24 Apr 2018 16:22:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6912: [DOCS] A new query option COMPUTE STATS MIN SAMPLE SIZE
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/10172 ) Change subject: IMPALA-6912: [DOCS] A new query option COMPUTE_STATS_MIN_SAMPLE_SIZE .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10172/1/docs/impala_keydefs.ditamap File docs/impala_keydefs.ditamap: http://gerrit.cloudera.org:8080/#/c/10172/1/docs/impala_keydefs.ditamap@10779 PS1, Line 10779: > typo in key, should be "compute" Did you fix this? -- To view, visit http://gerrit.cloudera.org:8080/10172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibd6f55a4c6c99185baf7974f68935ba4e03579df Gerrit-Change-Number: 10172 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Apr 2018 15:50:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6802 (part 1): Clean up authorization tests
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10135 ) Change subject: IMPALA-6802 (part 1): Clean up authorization tests .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/10135/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java: http://gerrit.cloudera.org:8080/#/c/10135/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@95 PS2, Line 95: ); > I don't know if we want to go through every permutation in the error. It ca Maybe we don't need it for every test, but where is the test to ensure that "REFRESH" or other privileges do not unintentionally allow you to do select? Shouldn't that be somewhere with the select tests? http://gerrit.cloudera.org:8080/#/c/10135/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@509 PS2, Line 509: ) > Passing a full error string in expectedErrorString is essentially comparing The problem is, if I expect an error to be "... not authorized on functional", and the error is "... not authorized on functional.alltypes", I have no way to say I got the wrong error, i.e. there's information leakage on the error. -- To view, visit http://gerrit.cloudera.org:8080/10135 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cd5713607c423f644451af5ebb3494d3a728e3b Gerrit-Change-Number: 10135 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Tue, 24 Apr 2018 15:37:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6070: Further improvements to test-with-docker.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/10123 ) Change subject: IMPALA-6070: Further improvements to test-with-docker. .. Patch Set 1: (3 comments) Just a couple of typos and an 'env' filtering request. Looks good; I wonder if an m5 or c5 instance can make it run in 2 hours... http://gerrit.cloudera.org:8080/#/c/10123/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10123/1//COMMIT_MSG@26 PS1, Line 26: it more in Python, with a class representing a suite with a Nit: possible typo (it -> bit) http://gerrit.cloudera.org:8080/#/c/10123/1/docker/entrypoint.sh File docker/entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/10123/1/docker/entrypoint.sh@379 PS1, Line 379: env I know this is not likely to run against S3 anytime soon, but would you mind adding the grep clause to filter out the AWS_ secrets, so that it doesn't bite us later? e.g. env | grep -vE "AWS_(SECRET_)?ACCESS_KEY" http://gerrit.cloudera.org:8080/#/c/10123/1/docker/test-with-docker.py File docker/test-with-docker.py: http://gerrit.cloudera.org:8080/#/c/10123/1/docker/test-with-docker.py@283 PS1, Line 283: ne nit: typo: environmental -- To view, visit http://gerrit.cloudera.org:8080/10123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I43fe124f00340afa21ad1eeb6432d6d50151ca7c Gerrit-Change-Number: 10123 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 24 Apr 2018 13:28:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/9005 ) Change subject: IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries .. Patch Set 29: (1 comment) http://gerrit.cloudera.org:8080/#/c/9005/23/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java: http://gerrit.cloudera.org:8080/#/c/9005/23/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java@1395 PS23, Line 1395: "functional.alltypes)", "Comparison between subqueries is not supported " + > Sorry, I messed up that one critical sentence. Let me try again :) Thanks, it seems to work! :) I added new tests to 'PlannerTest/subquery-rewrite.test' for this case. -- To view, visit http://gerrit.cloudera.org:8080/9005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0f52b93a60eeacedd242a2f17fa6b99c4fc38e06 Gerrit-Change-Number: 9005 Gerrit-PatchSet: 29 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Apr 2018 13:13:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries
Hello Attila Jeges, Dimitris Tsirogiannis, Tim Armstrong, Csaba Ringhofer, Alex Behm, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9005 to look at the new patch set (#29). Change subject: IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries .. IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries If a scalar subquery is used with a binary predicate, or, used in an arithmetic expression, it must return only one row/column to be valid. If this cannot be guaranteed at parse time through a single row aggregate or limit clause, Impala fails the query like such. E.g., currently the following query is not allowed: SELECT bigint_col FROM alltypesagg WHERE id = (SELECT id FROM alltypesagg WHERE id = 1) However, it would be allowed if the query contained a LIMIT 1 clause, or instead of id it was max(id). This commit makes the example valid by introducing a runtime check to test if the subquery returns a single row. If the subquery returns more than one row, it aborts the query with an error. I added a new node type, called CardinalityCheckNode. It is created during planning on top of the subquery when needed, then during execution it checks if its child only returns a single row. I extended the frontend tests and e2e tests as well. Change-Id: I0f52b93a60eeacedd242a2f17fa6b99c4fc38e06 --- M be/src/exec/CMakeLists.txt A be/src/exec/cardinality-check-node.cc A be/src/exec/cardinality-check-node.h M be/src/exec/exec-node.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/ExistsPredicate.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/HdfsCachingOp.java M fe/src/main/java/org/apache/impala/analysis/InPredicate.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/Subquery.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java A fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test M testdata/workloads/functional-query/queries/QueryTest/subquery.test 28 files changed, 1,061 insertions(+), 80 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/9005/29 -- To view, visit http://gerrit.cloudera.org:8080/9005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0f52b93a60eeacedd242a2f17fa6b99c4fc38e06 Gerrit-Change-Number: 9005 Gerrit-PatchSet: 29 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries
Hello Attila Jeges, Dimitris Tsirogiannis, Tim Armstrong, Csaba Ringhofer, Alex Behm, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9005 to look at the new patch set (#28). Change subject: IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries .. IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries If a scalar subquery is used with a binary predicate, or, used in an arithmetic expression, it must return only one row/column to be valid. If this cannot be guaranteed at parse time through a single row aggregate or limit clause, Impala fails the query like such. E.g., currently the following query is not allowed: SELECT bigint_col FROM alltypesagg WHERE id = (SELECT id FROM alltypesagg WHERE id = 1) However, it would be allowed if the query contained a LIMIT 1 clause, or instead of id it was max(id). This commit makes the example valid by introducing a runtime check to test if the subquery returns a single row. If the subquery returns more than one row, it aborts the query with an error. I added a new node type, called CardinalityCheckNode. It is created during planning on top of the subquery when needed, then during execution it checks if its child only returns a single row. I extended the frontend tests and e2e tests as well. Change-Id: I0f52b93a60eeacedd242a2f17fa6b99c4fc38e06 --- M be/src/exec/CMakeLists.txt A be/src/exec/cardinality-check-node.cc A be/src/exec/cardinality-check-node.h M be/src/exec/exec-node.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/ExistsPredicate.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/HdfsCachingOp.java M fe/src/main/java/org/apache/impala/analysis/InPredicate.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/Subquery.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java A fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test M testdata/workloads/functional-query/queries/QueryTest/subquery.test 28 files changed, 1,060 insertions(+), 80 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/9005/28 -- To view, visit http://gerrit.cloudera.org:8080/9005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0f52b93a60eeacedd242a2f17fa6b99c4fc38e06 Gerrit-Change-Number: 9005 Gerrit-PatchSet: 28 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-5842: Write page index in Parquet files
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/9693 ) Change subject: IMPALA-5842: Write page index in Parquet files .. Patch Set 12: (5 comments) Do we have tests for the edge cases? For example a table with columns of 63/64/65 length strings. http://gerrit.cloudera.org:8080/#/c/9693/12/be/src/exec/hdfs-parquet-table-writer.cc File be/src/exec/hdfs-parquet-table-writer.cc: http://gerrit.cloudera.org:8080/#/c/9693/12/be/src/exec/hdfs-parquet-table-writer.cc@87 PS12, Line 87: const int MAX_STAT_VALUE_LENGTH = 64; If different value is used for page and column stats, then the constants should be placed close to each other and their name should reflect their role. http://gerrit.cloudera.org:8080/#/c/9693/12/be/src/exec/hdfs-parquet-table-writer.cc@749 PS12, Line 749: Status s_min = TruncateMinValue(page_stats.min_value, MAX_STAT_VALUE_LENGTH, : &min_val); : Status s_max = TruncateMaxValue(page_stats.max_value, MAX_STAT_VALUE_LENGTH, : &max_val); I can't find TruncateMinValue and TruncateMaxValue in code. http://gerrit.cloudera.org:8080/#/c/9693/12/be/src/exec/hdfs-parquet-table-writer.cc@1221 PS12, Line 1221: column.column_index_.__set_boundary_order(column.row_group_stats_base_->GetBoundaryOrder()); nit: long line http://gerrit.cloudera.org:8080/#/c/9693/12/tests/query_test/test_parquet_page_index.py File tests/query_test/test_parquet_page_index.py: http://gerrit.cloudera.org:8080/#/c/9693/12/tests/query_test/test_parquet_page_index.py@171 PS12, Line 171: len(page_max_value) != len(column_max_value)): The != should be < - what happens if column_max_value (e.g. "bb") is shorter then page_max_value (e.g "")? I would also consider using the exact max length used for page stats, to make the tests as strict as possible. http://gerrit.cloudera.org:8080/#/c/9693/12/tests/query_test/test_parquet_page_index.py@172 PS12, Line 172: assert page_max_value[:-1] <= column_max_value[:len(page_max_value) - 1] This may not be true if the last two characters contain the max char value. -- To view, visit http://gerrit.cloudera.org:8080/9693 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbacf7fe3b7672e3ce719261ecef445b16f8dec9 Gerrit-Change-Number: 9693 Gerrit-PatchSet: 12 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Anonymous Coward #248 Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Apr 2018 10:42:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5706: Parallelise read I/O in sorter
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/9943 ) Change subject: IMPALA-5706: Parallelise read I/O in sorter .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/9943 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Gerrit-Change-Number: 9943 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Apr 2018 10:27:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5706: Parallelise read I/O in sorter
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/9943 )
Change subject: IMPALA-5706: Parallelise read I/O in sorter
..
Patch Set 6:
(2 comments)
Thanks Csaba for taking a look at this review!
Carry +1
http://gerrit.cloudera.org:8080/#/c/9943/3/be/src/runtime/sorter.cc
File be/src/runtime/sorter.cc:
http://gerrit.cloudera.org:8080/#/c/9943/3/be/src/runtime/sorter.cc@1039
PS3, Line 1039:
> You don't have to go into details here, but I still think that the current
Done
http://gerrit.cloudera.org:8080/#/c/9943/5/tests/query_test/test_sort.py
File tests/query_test/test_sort.py:
http://gerrit.cloudera.org:8080/#/c/9943/5/tests/query_test/test_sort.py@137
PS5, Line 137: sort1_profile = self.get_sort_node_profile(1,
result.runtime_profile)
: assert(sort1_profile.find("TotalMergesPerformed: 7") > 0)
:
: # The second sort extends its memory allocation with the
pages the first sort releases
: # when it finishes. As a result it has more memory for in
> Can you add a short explanation about the cause of the difference between t
Good point. Done.
--
To view, visit http://gerrit.cloudera.org:8080/9943
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9
Gerrit-Change-Number: 9943
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 24 Apr 2018 10:27:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5706: Parallelise read I/O in sorter
Hello Tim Armstrong, Csaba Ringhofer, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9943 to look at the new patch set (#6). Change subject: IMPALA-5706: Parallelise read I/O in sorter .. IMPALA-5706: Parallelise read I/O in sorter This patch covers multiple changes with the purpose of optimizing spilling sort mechanism: - Use double-buffering when merging the sorted runs. As a result while a run's page is being processed the next one can be loaded from disk in the background. - Remove the hard-coded maximum limit of buffers that can be used for merging the sorted runs. Instead this number is calculated based on the available memory through buffer pool. - The already sorted runs are distributed evenly between the last intermediate merge and the final merge to avoid that a heavy intermediate merge is followed by a light final merge. - Right before starting the merging phase Sorter tries to allocate additional memory through the buffer pool. - An output run is not allocated anymore for the final merge. Performance measurements were made during manual testing to verify that this is in fact an optimization: - In case doing a sort on top of a join when working with a restricted amount of memory then the Sort node successfully allocates additional memory right before the merging phase. This is feasible because once Join finishes sending new input data and calls InputDone() then it releases memory that can be picked up by the Sorter. This results in shallower merging trees (more runs grabbed for a merge). - Manual tests showed that when a single final merge is performed this change slightly decreases the execution time for sorting. Further testing should be done to cover double-buffering scenarios as my manual testing so far didn't show any performance gain when intermediate merges were performed. Most probably because due to double-buffering the number of runs in a single merge decreases and I have to hit an I/O heavy scenario to overcome this. Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 --- M be/src/runtime/sorter.cc M be/src/runtime/sorter.h M fe/src/main/java/org/apache/impala/planner/SortNode.java M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M testdata/workloads/functional-query/queries/QueryTest/analytic-fns.test M testdata/workloads/functional-query/queries/QueryTest/spilling-naaj-no-deny-reservation.test M testdata/workloads/functional-query/queries/QueryTest/spilling-sorts-exhaustive.test M testdata/workloads/tpch/queries/sort-reservation-usage.test M tests/custom_cluster/test_mem_reservations.py M tests/query_test/test_sort.py 14 files changed, 357 insertions(+), 246 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/9943/6 -- To view, visit http://gerrit.cloudera.org:8080/9943 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Gerrit-Change-Number: 9943 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3282: [DOCS] Adds regexp escape built-in function
Kim Jin Chul has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10174 Change subject: IMPALA-3282: [DOCS] Adds regexp_escape built-in function .. IMPALA-3282: [DOCS] Adds regexp_escape built-in function Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf --- M docs/topics/impala_string_functions.xml 1 file changed, 43 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/10174/1 -- To view, visit http://gerrit.cloudera.org:8080/10174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ied8e757c1b3012dd170b05da190d1598004d12cf Gerrit-Change-Number: 10174 Gerrit-PatchSet: 1 Gerrit-Owner: Kim Jin Chul
