[jira] [Commented] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[ https://issues.apache.org/jira/browse/HIVE-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168611#comment-17168611 ] Stamatis Zampetakis commented on HIVE-23963: I will provide soon a patch (basically a new QTest driver) where the problem can be reproduced easily. > UnsupportedOperationException in queries 74 and 84 while applying > HiveCardinalityPreservingJoinRule > --- > > Key: HIVE-23963 > URL: https://issues.apache.org/jira/browse/HIVE-23963 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Stamatis Zampetakis >Priority: Major > Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt > > > The following TPC-DS queries: > * cbo_query74.q > * cbo_query84.q > * query74.q > * query84.q > fail on the metastore with the partitioned TPC-DS 30TB dataset. > The stacktraces for cbo_query74 and cbo_query84 show that the problem > originates while applying HiveCardinalityPreservingJoinRule. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[ https://issues.apache.org/jira/browse/HIVE-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169820#comment-17169820 ] Stamatis Zampetakis commented on HIVE-23963: The problem can be reproduced with the patch in [PR#1347|https://github.com/apache/hive/pull/1347]. > UnsupportedOperationException in queries 74 and 84 while applying > HiveCardinalityPreservingJoinRule > --- > > Key: HIVE-23963 > URL: https://issues.apache.org/jira/browse/HIVE-23963 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Stamatis Zampetakis >Assignee: Krisztian Kasa >Priority: Major > Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt > > > The following TPC-DS queries: > * cbo_query74.q > * cbo_query84.q > * query74.q > * query84.q > fail on the metastore with the partitioned TPC-DS 30TB dataset. > The stacktraces for cbo_query74 and cbo_query84 show that the problem > originates while applying HiveCardinalityPreservingJoinRule. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[ https://issues.apache.org/jira/browse/HIVE-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169985#comment-17169985 ] Krisztian Kasa commented on HIVE-23963: --- When cost is calculated inĀ HiveOnTezCostModel https://github.com/apache/hive/blob/28b6384e9ba287188015418b4c38c85dfdde8133/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveOnTezCostModel.java#L403 We need the RelDistribution of a Project node. The RelDistribution is calculated by: 1. get the Project input's RelDistribution. In this case this is a TableScan on the customer table partitioned by the c_customer_sk column -> HiveRelDistribution(hash[0]) - the number *0* indicates that the c_customer_sk column is the 0th in the customer table but in general it can be anywhere. 2. The c_customer_sk is the 0th expression in the project but in general it can be anywhere so we need a mapping to map the key indexes in the RelDistribution. This mapping is an INVERSE_FUNCTION https://github.com/apache/calcite/blob/2088488ac8327b19512a76a122cae2961fc551c3/core/src/main/java/org/apache/calcite/rel/core/Project.java#L375 which does not support *getTargetOpt* [~jcamachorodriguez] Can the mapping type changed to something else which support getTargetOpt? > UnsupportedOperationException in queries 74 and 84 while applying > HiveCardinalityPreservingJoinRule > --- > > Key: HIVE-23963 > URL: https://issues.apache.org/jira/browse/HIVE-23963 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Stamatis Zampetakis >Assignee: Krisztian Kasa >Priority: Major > Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt > > > The following TPC-DS queries: > * cbo_query74.q > * cbo_query84.q > * query74.q > * query84.q > fail on the metastore with the partitioned TPC-DS 30TB dataset. > The stacktraces for cbo_query74 and cbo_query84 show that the problem > originates while applying HiveCardinalityPreservingJoinRule. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[ https://issues.apache.org/jira/browse/HIVE-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170071#comment-17170071 ] Jesus Camacho Rodriguez commented on HIVE-23963: [~kkasa], I think that should be explored on Calcite side indeed. I was checking the code and may need to wait for a release though? I saw we rely on that method in {{HiveRelDistribution}}. Is it possible to create the map using {{mapping.iterator()}} and rely on that map to get the value for each key? I think that may provide a valid workaround. > UnsupportedOperationException in queries 74 and 84 while applying > HiveCardinalityPreservingJoinRule > --- > > Key: HIVE-23963 > URL: https://issues.apache.org/jira/browse/HIVE-23963 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Stamatis Zampetakis >Assignee: Krisztian Kasa >Priority: Major > Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt > > > The following TPC-DS queries: > * cbo_query74.q > * cbo_query84.q > * query74.q > * query84.q > fail on the metastore with the partitioned TPC-DS 30TB dataset. > The stacktraces for cbo_query74 and cbo_query84 show that the problem > originates while applying HiveCardinalityPreservingJoinRule. -- This message was sent by Atlassian Jira (v8.3.4#803005)