[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501228#comment-16501228
]
Hive QA commented on HIVE-19358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926107/HIVE-19358.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14461 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11514/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11514/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11514/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926107 - PreCommit-HIVE-Build
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.05.patch,
> HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501203#comment-16501203
]
Hive QA commented on HIVE-19358:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
29s{color} | {color:blue} ql in master has 2277 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 1 fixed
= 17 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11514/dev-support/hive-personality.sh
|
| git revision | master / 91cdd4f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11514/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11514/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.05.patch,
> HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[ https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499314#comment-16499314 ] Ashutosh Chauhan commented on HIVE-19358: - +1 > CBO decorrelation logic should generate Hive operators > -- > > Key: HIVE-19358 > URL: https://issues.apache.org/jira/browse/HIVE-19358 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch, > HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.05.patch, > HIVE-19358.patch, fix.patch > > > Decorrelation logic may generate logical instances of the operators in the > plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while > costing the tree in the Volcano planner (used in MV rewriting), since logical > operators do not have a cost associated to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498128#comment-16498128
]
Jesus Camacho Rodriguez commented on HIVE-19358:
{{TestLlapTaskCommunicator.testFinishableStateUpdateFailure}} seems flaky.
{{TestJdbcWithMiniLlapArrow.testDataTypes}} was broken by HIVE-19723 (reverted
now).
I have uploaded patch to get a new run. [~vgarg], [~ashutoshc], please take a
look whenever you can. This is critical for MV rewriting in complex queries.
Thanks
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.05.patch,
> HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498083#comment-16498083
]
Hive QA commented on HIVE-19358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925642/HIVE-19358.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 1 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
(batchId=261)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testDataTypes (batchId=242)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11413/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11413/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11413/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925642 - PreCommit-HIVE-Build
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498006#comment-16498006
]
Hive QA commented on HIVE-19358:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 2278 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 1 fixed
= 17 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11413/dev-support/hive-personality.sh
|
| git revision | master / 28779d2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11413/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11413/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[ https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494282#comment-16494282 ] Jesus Camacho Rodriguez commented on HIVE-19358: Fixed issues (it should come back clean from QA) and uploaded patch to https://reviews.apache.org/r/67364/ . [~vgarg], [~ashutoshc], could you take a look? New methods in HiveRelOptUtil are brought from Calcite and extended to make them work with custom operator instances (probably those methods should be generalized in Calcite so we do not have two copies in codebase). Thanks > CBO decorrelation logic should generate Hive operators > -- > > Key: HIVE-19358 > URL: https://issues.apache.org/jira/browse/HIVE-19358 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch, > HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.patch, fix.patch > > > Decorrelation logic may generate logical instances of the operators in the > plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while > costing the tree in the Volcano planner (used in MV rewriting), since logical > operators do not have a cost associated to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493825#comment-16493825
]
Hive QA commented on HIVE-19358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925445/HIVE-19358.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 14412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=184)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in]
(batchId=137)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=140)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=114)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1]
(batchId=256)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query30]
(batchId=256)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query32]
(batchId=256)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query6]
(batchId=256)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query81]
(batchId=256)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query92]
(batchId=256)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query32]
(batchId=254)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query92]
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11343/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11343/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11343/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925445 - PreCommit-HIVE-Build
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.03.patch, HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493757#comment-16493757
]
Hive QA commented on HIVE-19358:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 2331 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 1 fixed
= 17 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11343/dev-support/hive-personality.sh
|
| git revision | master / 99ed2bc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11343/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11343/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.03.patch, HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463116#comment-16463116
]
Hive QA commented on HIVE-19358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921694/fix.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 14316 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=253)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_1]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_2]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative3]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_1_23]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_skew_1_23]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_13]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sort_merge_join_desc_7]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insertsel_fail]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_data_parquet_empty]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_1]
(batchId=137)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_2]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin_negative3]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_sort_1_23]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_sort_skew_1_23]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_13]
(batchId=122)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10661/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10661/console
Test logs: http
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463030#comment-16463030
]
Hive QA commented on HIVE-19358:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10661/dev-support/hive-personality.sh
|
| git revision | master / db26f34 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10661/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[ https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463005#comment-16463005 ] Jesus Camacho Rodriguez commented on HIVE-19358: [~vgarg], awesome, thanks. That seems to fix the issue that I had encountered. I have uploaded a new patch that includes your changes to trigger a new ptest run. > CBO decorrelation logic should generate Hive operators > -- > > Key: HIVE-19358 > URL: https://issues.apache.org/jira/browse/HIVE-19358 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch, > HIVE-19358.patch, fix.patch > > > Decorrelation logic may generate logical instances of the operators in the > plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while > costing the tree in the Volcano planner (used in MV rewriting), since logical > operators do not have a cost associated to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461958#comment-16461958
]
Vineet Garg commented on HIVE-19358:
[~jcamachorodriguez] I have attached {{fix.patch}}. This should fix this issue.
This patch can be applied on your branch. Let me know if this doesn't work out
for you.
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459422#comment-16459422
]
Hive QA commented on HIVE-19358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921306/HIVE-19358.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 55 failed/errored test(s), 14270 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqualcolumnrefs]
(batchId=18)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_in]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insertsel_fail]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_subq_in]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1]
(batchId=255)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query30]
(batchId=255)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query32]
(batchId=255)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query6]
(batchId=255)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query81]
(batchId=255)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query92]
(batchId=255)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query32]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query92]
(batchId=253)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=309)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNo
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459394#comment-16459394
]
Hive QA commented on HIVE-19358:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 1 fixed
= 17 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10592/dev-support/hive-personality.sh
|
| git revision | master / ee6a532 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10592/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10592/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459075#comment-16459075
]
Jesus Camacho Rodriguez commented on HIVE-19358:
[~vgarg], would you mind taking a look at this issue?
Latest patch has some updated q files for which the plan improves, including
TPC-DS queries. The reason is that {{ReduceExpressions}} rule matches on
{{HiveProject}}, hence we were not matching {{LogicalProject}} operator and we
were missing optimization opportunities, including predicate propagation.
In turn, there are some plans that change quite a bit (e.g. new semijoin
operators appear), results are still the same but probably we need to double
check whether all is correct.
Finally, there are tests that are now failing. For instance,
{{subquery_unqualcolumnrefs.q}} is failing. The stacktrace is below.
{code:sql}
explain
select *
from src b
where b.key in
(select distinct key
from src
where b.value = value and key > '9'
);
{code}
{code}
2018-04-30T13:27:41,704 DEBUG [1c1ab7bd-d3ae-4957-b15c-6e960c853230 main]
calcite.sql2rel: Plan after trimming unused fields
HiveFilter(condition=[IN($0, {
HiveProject(key=[$0])
HiveAggregate(group=[{0}])
HiveProject($f0=[$0])
HiveFilter(condition=[AND(=($cor0.value, $1), >($0, _UTF-16LE'9'))])
HiveTableScan(table=[[default.src]], table:alias=[src])
})])
HiveProject(key=[$0], value=[$1])
HiveTableScan(table=[[default.src]], table:alias=[b])
2018-04-30T13:27:41,704 DEBUG [1c1ab7bd-d3ae-4957-b15c-6e960c853230 main]
parse.CalcitePlanner: Plan before removing subquery:
HiveProject(key=[$0], value=[$1])
HiveFilter(condition=[IN($0, {
HiveProject(key=[$0])
HiveAggregate(group=[{0}])
HiveProject($f0=[$0])
HiveFilter(condition=[AND(=($cor0.value, $1), >($0, _UTF-16LE'9'))])
HiveTableScan(table=[[default.src]], table:alias=[src])
})])
HiveTableScan(table=[[default.src]], table:alias=[b])
2018-04-30T13:27:41,716 DEBUG [1c1ab7bd-d3ae-4957-b15c-6e960c853230 main]
parse.CalcitePlanner: Plan just after removing subquery:
HiveProject(key=[$0], value=[$1])
HiveFilter(condition=[=($0, $5)])
LogicalCorrelate(correlation=[$cor0], joinType=[semi],
requiredColumns=[{1}])
HiveTableScan(table=[[default.src]], table:alias=[b])
HiveProject(key=[$0])
HiveAggregate(group=[{0}])
HiveProject($f0=[$0])
HiveFilter(condition=[AND(=($cor0.value, $1), >($0, _UTF-16LE'9'))])
HiveTableScan(table=[[default.src]], table:alias=[src])
2018-04-30T13:27:41,718 ERROR [1c1ab7bd-d3ae-4957-b15c-6e960c853230 main]
parse.CalcitePlanner: CBO failed, skipping CBO.
java.lang.IllegalStateException: Unable to convert SEMI to JoinRelType
at org.apache.calcite.sql.SemiJoinType.toJoinType(SemiJoinType.java:83)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator$RemoveCorrelationForScalarAggregateRule.onMatch(HiveRelDecorrelator.java:2470)
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:317)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:415)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:252)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:211)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:198)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.removeCorrelationViaRule(HiveRelDecorrelator.java:327)
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateQuery(HiveRelDecorrelator.java:220)
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1709)
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1613)
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
~[calcite-core-1.16.0.jar:1.16.0]
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052)
~[calcite-core-1.16.0.jar:1.16.0]
at org.apache.
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458252#comment-16458252
]
Hive QA commented on HIVE-19358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921193/HIVE-19358.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 75 failed/errored test(s), 14286 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=254)
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notexists]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notin_having]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqualcolumnrefs]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[uniquejoin] (batchId=72)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_in]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_groupingset_bug]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in_having]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_views]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_subq_in]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1]
(batchId=255)
org
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458234#comment-16458234
]
Hive QA commented on HIVE-19358:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 1 fixed
= 17 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10578/dev-support/hive-personality.sh
|
| git revision | master / a095c3d |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10578/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10578/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
