[
https://issues.apache.org/jira/browse/HIVE-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14228661#comment-14228661
]
Pengcheng Xiong commented on HIVE-8997:
---------------------------------------
according to [~jpullokkaran]'s comments
> Groupby index will fail if an indexed group by operator is followed by a
> non-indexed group by operator
> ------------------------------------------------------------------------------------------------------
>
> Key: HIVE-8997
> URL: https://issues.apache.org/jira/browse/HIVE-8997
> Project: Hive
> Issue Type: Bug
> Reporter: Pengcheng Xiong
> Assignee: Pengcheng Xiong
>
> following ql_rewrite_gbtoidx.q, if we run
> explain
> select ckeysum, count(ckeysum)
> from
> (select l_shipdate, count(l_shipdate) as ckeysum
> from lineitem_ix
> group by l_shipdate) tabA
> group by ckeysum
> We will get an error:
> junit.framework.AssertionFailedError: Client Execution failed with error code
> = 40000 running
> The trace is
> MismatchedTokenException(-1!=12)
> at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.charSetStringLiteral(HiveParser_IdentifiersParser.java:6099)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.constant(HiveParser_IdentifiersParser.java:5891)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6478)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6641)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:7026)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:7086)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7270)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7430)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7590)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7750)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7909)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8439)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9452)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9571)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9730)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6363)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.groupByExpression(HiveParser_IdentifiersParser.java:1386)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.groupByClause(HiveParser_IdentifiersParser.java:774)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.groupByClause(HiveParser.java:44007)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.singleSelectStatement(HiveParser.java:41504)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41135)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41072)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40125)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40001)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1519)
> at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1057)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at
> org.apache.hadoop.hive.ql.optimizer.index.RewriteParseContextGenerator.generateOperatorTree(RewriteParseContextGenerator.java:67)
> at
> org.apache.hadoop.hive.ql.optimizer.index.RewriteQueryUsingAggregateIndex$NewQueryGroupbySchemaProc.process(RewriteQueryUsingAggregateIndex.java:255)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.index.RewriteQueryUsingAggregateIndexCtx.invokeRewriteQueryProc(RewriteQueryUsingAggregateIndexCtx.java:151)
> at
> org.apache.hadoop.hive.ql.optimizer.index.RewriteGBUsingIndex.rewriteOriginalQuery(RewriteGBUsingIndex.java:382)
> at
> org.apache.hadoop.hive.ql.optimizer.index.RewriteGBUsingIndex.transform(RewriteGBUsingIndex.java:135)
> at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:177)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10177)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1107)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1169)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1034)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:199)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:362)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:297)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:849)
> at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:136)
> at
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx_notwork(TestCliDriver.java:120)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Exception: Client Execution failed with error code = 40000 running
> The main reason is that
> After Optimizer,org.apache.hadoop.hive.ql.optimizer.ColumnPruner, we should
> get the operator tree as
> TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-SEL[5]-SEL[6]-GBY[7]-RS[8]-GBY[9]-SEL[10]-FS[11]
> Here, TS[0], SEL[1], GBY[2], GBY[4] should be rewritten. However, because the
> default graph walker is used in the code, GBY[9] and GBY[7] are going to be
> rewritten. This causes the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
