[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349863#comment-16349863 ] Ke Jia commented on HIVE-17139: --- [~mmccline], [~Ferd],[~colinma], I update the patch to fix HIVE-18524 issue and upload to RB . Please help me review. Thanks for your help. > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184358#comment-16184358 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12889222/HIVE-17139.17.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11088 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown] (batchId=232) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=232) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=157) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=171) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=101) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=236) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7025/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7025/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7025/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12889222 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.13.patch, > HIVE-17139.14.patch, HIVE-17139.15.patch, HIVE-17139.16.patch, > HIVE-17139.17.patch, HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184948#comment-16184948 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12889222/HIVE-17139.17.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11088 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown] (batchId=232) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=232) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=157) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=171) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.org.apache.hadoop.hive.cli.TestTezPerfCliDriver (batchId=240) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=203) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7030/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7030/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7030/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12889222 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.13.patch, > HIVE-17139.14.patch, HIVE-17139.15.patch, HIVE-17139.16.patch, > HIVE-17139.17.patch, HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186021#comment-16186021 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12889641/HIVE-17139.18.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11089 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown] (batchId=232) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=232) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=157) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=171) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.org.apache.hadoop.hive.cli.TestTezPerfCliDriver (batchId=240) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=203) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7047/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7047/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7047/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12889641 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.13.patch, > HIVE-17139.14.patch, HIVE-17139.15.patch, HIVE-17139.16.patch, > HIVE-17139.17.patch, HIVE-17139.18.patch, HIVE-17139.1.patch, > HIVE-17139.2.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186989#comment-16186989 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12889834/HIVE-17139.18.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11177 tests executed *Failed tests:* {noformat} TestSSL - did not produce a TEST-*.xml file (likely timed out) (batchId=226) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown] (batchId=232) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=232) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=157) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=171) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=240) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] (batchId=240) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=203) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7064/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7064/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7064/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12889834 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.13.patch, > HIVE-17139.14.patch, HIVE-17139.15.patch, HIVE-17139.16.patch, > HIVE-17139.17.patch, HIVE-17139.18.patch, HIVE-17139.18.patch, > HIVE-17139.1.patch, HIVE-17139.2.patch, HIVE-17139.3.patch, > HIVE-17139.4.patch, HIVE-17139.5.patch, HIVE-17139.6.patch, > HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16189616#comment-16189616 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12890136/HIVE-17139.19.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11193 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown] (batchId=232) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=232) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=157) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=171) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=240) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] (batchId=240) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=203) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7090/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7090/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7090/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12890136 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.13.patch, > HIVE-17139.14.patch, HIVE-17139.15.patch, HIVE-17139.16.patch, > HIVE-17139.17.patch, HIVE-17139.18.patch, HIVE-17139.18.patch, > HIVE-17139.19.patch, HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196418#comment-16196418 ] Ferdinand Xu commented on HIVE-17139: - LGTM +1 > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196484#comment-16196484 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12890998/HIVE-17139.20.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11190 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown] (batchId=231) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=231) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=154) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=171) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] (batchId=239) org.apache.hive.spark.client.rpc.TestRpc.testServerPort (batchId=287) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7188/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7188/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7188/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12890998 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197967#comment-16197967 ] Ferdinand Xu commented on HIVE-17139: - [~Jk_Self], can you please take a look at the failed qtest cases? > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198042#comment-16198042 ] Ke Jia commented on HIVE-17139: --- [~Ferd], thanks for your review. The failed tests seem not patch related. > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198420#comment-16198420 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12890998/HIVE-17139.20.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11149 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[spark_local_queries] (batchId=64) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=239) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] (batchId=239) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.org.apache.hadoop.hive.ql.parse.TestReplicationScenarios (batchId=219) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7208/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7208/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7208/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12890998 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199754#comment-16199754 ] Ke Jia commented on HIVE-17139: --- [~Ferd], about the 5 failed tests, I run locally with the latest hive code. Spark_local_queries,TestReplicationScenarios and optimize_nullscan tests can pass. Query14 and query23 tests failed with or without patch. So I think the failed tests seem not patch related. > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201196#comment-16201196 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12890998/HIVE-17139.20.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 11211 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=239) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] (batchId=239) org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish (batchId=186) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7239/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7239/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7239/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12890998 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095796#comment-16095796 ] Ke Jia commented on HIVE-17139: --- With this patch, I test "select case when a=1 then trim(b) end from test_orc_5000" in my development machine. The data scale is almost 50 million records in table test_orc_5000(a int, b string) stored as ORC. The execution engine is spark. I do three experiments and the average value is as below table. The result shows the execution time of spark from 35.76s to 32.57s, the time cost of VectorSelectOperator from 3.12s to 0.89s and the count of then expression evaluation from 4735 to 5000712. || ||Non-optimization||Optimization||Improvement|| |Hos|35.76s|32.57s|8.9%| |VectorSelectOperator|3.12s|0.89s|7.15%| |count|4735|5000712|8.99%| > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097941#comment-16097941 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12878297/HIVE-17139.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 11092 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case] (batchId=55) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar] (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce_3] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id1] (batchId=149) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case] (batchId=125) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=179) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6120/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6120/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6120/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 14 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12878297 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102899#comment-16102899 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12879091/HIVE-17139.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11012 tests executed *Failed tests:* {noformat} TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed] (batchId=240) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge7] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce_3] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping] (batchId=146) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=99) org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge (batchId=206) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=179) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6144/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6144/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6144/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12879091 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105554#comment-16105554 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12879301/HIVE-17139.4.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11013 tests executed *Failed tests:* {noformat} TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1] (batchId=240) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce_3] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping] (batchId=146) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=99) org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge (batchId=206) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=179) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6173/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6173/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6173/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12879301 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140178#comment-16140178 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12879301/HIVE-17139.4.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10999 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce_3] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping] (batchId=147) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=228) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6519/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6519/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6519/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12879301 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142643#comment-16142643 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12883846/HIVE-17139.5.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11009 tests executed *Failed tests:* {noformat} TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6554/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6554/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6554/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12883846 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151038#comment-16151038 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12884897/HIVE-17139.6.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 11023 tests executed *Failed tests:* {noformat} TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6644/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6644/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6644/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12884897 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, HIVE-17139.6.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156987#comment-16156987 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12885757/HIVE-17139.8.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11028 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6712/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6712/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6712/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12885757 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157994#comment-16157994 ] Ke Jia commented on HIVE-17139: --- Upload the latest patch to fix the failed tests and the remain failed tests seem not patch related. I test the patch with table product_reviews of TPCx-BB using the following sql statement: {code:java} select case when pr_review_rating=4 then upper(pr_review_content) when pr_review_rating=3 then upper(pr_review_content) end from product_reviews; {code} The cluster includes 8 nodes, 230G/per node. CPU is Intel(R) Xeon(R) CPU E5-2699. With 3TB data scale and spark as executor engine, the following is the result: || ||without patch||with patch||improvement(s)||improvement(%)|| |Hos|28.25s|16.14s|12.11s|42.8%| |VectorSelectOperator |2.99s|12.58s|9.59s|76.2%| The result shows the execution time of spark from 28.25s to 16.14s and the time cost of VectorSelectOperator from 12.58s to 2.99s. Here, the total records, "pr_review_rating=4" records and "pr_review_rating=3" records are as following: || ||count|| |total records|9934636| |pr_review_rating=4 records|1897804| |pr_review_rating=3 records|792278| With this patch, only (1897804+792278) records do the upper operation of the above sql statement and without this patch, there are (9934636+9934636) records doing the upper operation. > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157998#comment-16157998 ] Ke Jia commented on HIVE-17139: --- [~Ferd], can you help to review the patch? thanks for your help. > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158008#comment-16158008 ] Ferdinand Xu commented on HIVE-17139: - The patch used in review board is different from your patch 8. No need for file: b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/ConditionalFilter.java > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165784#comment-16165784 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12886850/HIVE-17139.9.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 84 failed/errored test(s), 11040 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[foldts] (batchId=55) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_bitmap_compression] (batchId=76) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] (batchId=56) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_types_non_dictionary_encoding_vectorization] (batchId=81) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_types_vectorization] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_between_columns] (batchId=67) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_binary_join_groupby] (batchId=78) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce] (batchId=10) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_complex_join] (batchId=42) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_custom_udf_configure] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_data_types] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_math_funcs] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_udf2] (batchId=71) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_struct_in] (batchId=44) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_casts] (batchId=79) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_date_funcs] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_distinct_gby] (batchId=71) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_math_funcs] (batchId=19) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet_types] (batchId=64) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_ints_casts] (batchId=47) org.apache.hadoop.hive.cli.TestCompareCliDriver.testCliDriver[vectorized_math_funcs] (batchId=235) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters] (batchId=145) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_types_vectorization] (batchId=141) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_partitioned] (batchId=149) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acidvec_part] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acidvec_table] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_table] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_adaptor_usage_mode] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_auto_smb_mapjoin_14] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_between_columns] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_between_in] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_binary_join_groupby] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_complex_join] (batchId=154) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_data_types] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_math_funcs] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169894#comment-16169894 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887603/HIVE-17139.10.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11041 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_table] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union3] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join_filters] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_number_compare_projection] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf1] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case] (batchId=157) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation (batchId=227) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6863/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6863/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6863/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 16 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12887603 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.1.patch, > HIVE-17139.2.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172144#comment-16172144 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887830/HIVE-17139.11.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11043 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=227) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6886/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6886/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6886/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12887830 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.1.patch, HIVE-17139.2.patch, HIVE-17139.3.patch, > HIVE-17139.4.patch, HIVE-17139.5.patch, HIVE-17139.6.patch, > HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175798#comment-16175798 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12888297/HIVE-17139.12.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11051 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_when_case] (batchId=56) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6932/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6932/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6932/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12888297 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176379#comment-16176379 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12888434/HIVE-17139.13.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6942/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6942/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6942/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-09-22 13:04:03.744 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6942/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-09-22 13:04:03.747 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 266c505 HIVE-17570 : Fix view deletion related test failures (create_view.q etc) (Tao Li via Thejas Nair) + git clean -f -d Removing standalone-metastore/src/gen/org/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 266c505 HIVE-17570 : Fix view deletion related test failures (create_view.q etc) (Tao Li via Thejas Nair) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-09-22 13:04:04.447 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Going to apply patch with: patch -p1 patching file common/src/java/org/apache/hadoop/hive/conf/HiveConf.java Hunk #1 succeeded at 2861 (offset -14 lines). patching file ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/ConditionalFilter.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprColumnNull.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprDoubleColumnDoubleColumn.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprIntervalDayTimeColumnColumn.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprLongColumnLongColumn.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprNullColumn.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprStringGroupColumnStringGroupColumn.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprTimestampColumnColumnBase.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/vector/udf/VectorUDFAdaptor.java + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: protoc version: 250, detected platform: linux/amd64 protoc-jar: executing: [/tmp/protoc8294637482170651910.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] DataNucleus Enhancer (version 4.1.17) for API "JDO" DataNucleus Enhancer : Classpath >> /usr/share/maven/boot/plexus-classworlds-2.x.jar ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFieldSchema ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTy
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177439#comment-16177439 ] Lefty Leverenz commented on HIVE-17139: --- Review of configuration parameter in patch 13: {quote} + HIVE_VECTORIZATION_CASEWHEN_OPTIMIZATION_ENABLED("hive.vectorized.casewhen.optimization.enable", true, + "This flag should be set to true to enable vectorization optimization for casewhen expression.The default value is true."), {quote} 1) Should "casewhen" be "case when" in the description? 2) Please add a space before the description's second sentence. 3) Please use "\n" to split the description into two lines. > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.13.patch, > HIVE-17139.1.patch, HIVE-17139.2.patch, HIVE-17139.3.patch, > HIVE-17139.4.patch, HIVE-17139.5.patch, HIVE-17139.6.patch, > HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177805#comment-16177805 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12888601/HIVE-17139.13.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11059 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge4] (batchId=12) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6959/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6959/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6959/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12888601 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.13.patch, > HIVE-17139.1.patch, HIVE-17139.2.patch, HIVE-17139.3.patch, > HIVE-17139.4.patch, HIVE-17139.5.patch, HIVE-17139.6.patch, > HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178178#comment-16178178 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12888738/HIVE-17139.14.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11059 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6971/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6971/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6971/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12888738 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.13.patch, > HIVE-17139.14.patch, HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178476#comment-16178476 ] Ke Jia commented on HIVE-17139: --- [~leftylev], Thanks for your review. we are not going to use "HIVE_VECTORIZATION_CASEWHEN_OPTIMIZATION_ENABLED" configuration in HiveConf.java. Thanks. > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.11.patch, > HIVE-17139.12.patch, HIVE-17139.13.patch, HIVE-17139.13.patch, > HIVE-17139.14.patch, HIVE-17139.1.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180973#comment-16180973 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12889055/HIVE-17139.16.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 11063 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[optimize_join_ptp] (batchId=70) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_4] (batchId=85) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce] (batchId=10) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_custom_udf_configure] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_varchar_4] (batchId=28) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_casts] (batchId=79) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_math_funcs] (batchId=19) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet_types] (batchId=64) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_ints_casts] (batchId=47) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_types_vectorization] (batchId=141) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_4] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets1] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets4] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets6] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit] (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf1] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_4] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_casts] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_math_funcs] (batchId=149) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_timestamp_ints_casts] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_char_4] (batchId=139) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_varchar_4] (batchId=114) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_math_funcs] (batchId=110) org.apache.hadoop.hive.ql.exec.vector.udf.TestVectorUDFAdaptor.testLongUDF (batchId=273) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6991/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6991/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6991/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 33 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12889055 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-1
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181290#comment-16181290 ] Hive QA commented on HIVE-17139: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12889055/HIVE-17139.16.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 11062 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_4] (batchId=85) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce] (batchId=10) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_custom_udf_configure] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_varchar_4] (batchId=28) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_casts] (batchId=79) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_math_funcs] (batchId=19) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet_types] (batchId=64) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_ints_casts] (batchId=47) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_types_vectorization] (batchId=141) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_4] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets1] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets4] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets6] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit] (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf1] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_4] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_casts] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_math_funcs] (batchId=149) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_timestamp_ints_casts] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_char_4] (batchId=139) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_varchar_4] (batchId=114) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_math_funcs] (batchId=110) org.apache.hadoop.hive.ql.exec.vector.udf.TestVectorUDFAdaptor.testLongUDF (batchId=273) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6993/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6993/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6993/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 33 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12889055 - PreCommit-HIVE-Build > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: