[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[ https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581911#comment-14581911 ] Xuefu Zhang commented on HIVE-10971: Similar to TestCliDriver, TestSparkCliDriver is generated as part of .q test. > count(*) with count(distinct) gives wrong results when > hive.groupby.skewindata=true > --- > > Key: HIVE-10971 > URL: https://issues.apache.org/jira/browse/HIVE-10971 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Fix For: 1.2.1 > > Attachments: HIVE-10971.01.patch, HIVE-10971.1.patch > > > When hive.groupby.skewindata=true, the following query based on TPC-H gives > wrong results: > {code} > set hive.groupby.skewindata=true; > select l_returnflag, count(*), count(distinct l_linestatus) > from lineitem > group by l_returnflag > limit 10; > {code} > The query plan shows that it generates only one MapReduce job instead of two > theoretically, which is dictated by hive.groupby.skewindata=true. > The problem arises only when {noformat}count(*){noformat} and > {noformat}count(distinct){noformat} exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[ https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581350#comment-14581350 ] wangmeng commented on HIVE-10971: - Thanks [~ashutoshc] for fixing the tests! [~xuefuz], I checked the latest code and could not find TestSparkCliDriver. Am I missing something here? > count(*) with count(distinct) gives wrong results when > hive.groupby.skewindata=true > --- > > Key: HIVE-10971 > URL: https://issues.apache.org/jira/browse/HIVE-10971 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.2.0 >Reporter: wangmeng >Assignee: wangmeng > Fix For: 1.2.1 > > Attachments: HIVE-10971.01.patch, HIVE-10971.1.patch > > > When hive.groupby.skewindata=true, the following query based on TPC-H gives > wrong results: > {code} > set hive.groupby.skewindata=true; > select l_returnflag, count(*), count(distinct l_linestatus) > from lineitem > group by l_returnflag > limit 10; > {code} > The query plan shows that it generates only one MapReduce job instead of two > theoretically, which is dictated by hive.groupby.skewindata=true. > The problem arises only when {noformat}count(*){noformat} and > {noformat}count(distinct){noformat} exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[ https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14580469#comment-14580469 ] Xuefu Zhang commented on HIVE-10971: Hi [~ashutoshc], it seems that the patch brought one additional test failure as shown above. Don't we expect that the failure is analyzed/fixed before we commit the patch? Thanks. > count(*) with count(distinct) gives wrong results when > hive.groupby.skewindata=true > --- > > Key: HIVE-10971 > URL: https://issues.apache.org/jira/browse/HIVE-10971 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.2.0 >Reporter: wangmeng >Assignee: wangmeng > Fix For: 1.2.1 > > Attachments: HIVE-10971.01.patch, HIVE-10971.1.patch > > > When hive.groupby.skewindata=true, the following query based on TPC-H gives > wrong results: > {code} > set hive.groupby.skewindata=true; > select l_returnflag, count(*), count(distinct l_linestatus) > from lineitem > group by l_returnflag > limit 10; > {code} > The query plan shows that it generates only one MapReduce job instead of two > theoretically, which is dictated by hive.groupby.skewindata=true. > The problem arises only when {noformat}count(*){noformat} and > {noformat}count(distinct){noformat} exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[ https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14580047#comment-14580047 ] Hive QA commented on HIVE-10971: {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12738683/HIVE-10971.1.patch {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9006 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2 org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4234/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4234/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4234/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12738683 - PreCommit-HIVE-TRUNK-Build > count(*) with count(distinct) gives wrong results when > hive.groupby.skewindata=true > --- > > Key: HIVE-10971 > URL: https://issues.apache.org/jira/browse/HIVE-10971 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.2.0 >Reporter: wangmeng >Assignee: wangmeng > Attachments: HIVE-10971.01.patch, HIVE-10971.1.patch > > > When hive.groupby.skewindata=true, the following query based on TPC-H gives > wrong results: > {code} > set hive.groupby.skewindata=true; > select l_returnflag, count(*), count(distinct l_linestatus) > from lineitem > group by l_returnflag > limit 10; > {code} > The query plan shows that it generates only one MapReduce job instead of two > theoretically, which is dictated by hive.groupby.skewindata=true. > The problem arises only when {noformat}count(*){noformat} and > {noformat}count(distinct){noformat} exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[ https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579482#comment-14579482 ] Ashutosh Chauhan commented on HIVE-10971: - LGTM +1 need to update test files though. > count(*) with count(distinct) gives wrong results when > hive.groupby.skewindata=true > --- > > Key: HIVE-10971 > URL: https://issues.apache.org/jira/browse/HIVE-10971 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.2.0 >Reporter: wangmeng >Assignee: wangmeng > Attachments: HIVE-10971.01.patch > > > When hive.groupby.skewindata=true, the following query based on TPC-H gives > wrong results: > {code} > set hive.groupby.skewindata=true; > select l_returnflag, count(*), count(distinct l_linestatus) > from lineitem > group by l_returnflag > limit 10; > {code} > The query plan shows that it generates only one MapReduce job instead of two > theoretically, which is dictated by hive.groupby.skewindata=true. > The problem arises only when {noformat}count(*){noformat} and > {noformat}count(distinct){noformat} exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[ https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578802#comment-14578802 ] Hive QA commented on HIVE-10971: {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12738553/HIVE-10971.01.patch {color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 9004 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8_map_skew org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_cube1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_rollup1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby2 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_groupby2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby10 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby11 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8_map_skew org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_cube1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_rollup1 org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4223/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4223/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4223/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 20 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12738553 - PreCommit-HIVE-TRUNK-Build > count(*) with count(distinct) gives wrong results when > hive.groupby.skewindata=true > --- > > Key: HIVE-10971 > URL: https://issues.apache.org/jira/browse/HIVE-10971 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.2.0 >Reporter: wangmeng >Assignee: wangmeng > Attachments: HIVE-10971.01.patch > > > When hive.groupby.skewindata=true, the following query based on TPC-H gives > wrong results: > {code} > set hive.groupby.skewindata=true; > select l_returnflag, count(*), count(distinct l_linestatus) > from lineitem > group by l_returnflag > limit 10; > {code} > The query plan shows that it generates only one MapReduce job instead of two > theoretically, which is dictated by hive.groupby.skewindata=true. > The problem arises only when {noformat}count(*){noformat} and > {noformat}count(distinct){noformat} exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[ https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578602#comment-14578602 ] wangmeng commented on HIVE-10971: - {code} hive> set hive.groupby.skewindata=true; hive> explain select l_returnflag,count(*),count(distinct l_linestatus) from lineitem group by l_returnflag limit 10; OK STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: lineitem Statistics: Num rows: 1008537518 Data size: 201707503616 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: l_returnflag (type: string), l_linestatus (type: string) outputColumnNames: l_returnflag, l_linestatus Statistics: Num rows: 1008537518 Data size: 201707503616 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(), count(DISTINCT l_linestatus) keys: l_returnflag (type: string), l_linestatus (type: string) mode: hash outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 1008537518 Data size: 201707503616 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) sort order: ++ Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 1008537518 Data size: 201707503616 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: bigint) Reduce Operator Tree: Group By Operator aggregations: count(VALUE._col0), count(DISTINCT KEY._col1:0._col0) keys: KEY._col0 (type: string) mode: complete outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 504268759 Data size: 100853751808 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: bigint), _col2 (type: bigint) outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 504268759 Data size: 100853751808 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: true Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: 10 {code} When hive.groupby.skewindata=false, the Group By operator has mode "mergepartial", which gives the correct results. > count(*) with count(distinct) gives wrong results when > hive.groupby.skewindata=true > --- > > Key: HIVE-10971 > URL: https://issues.apache.org/jira/browse/HIVE-10971 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.0 >Reporter: wangmeng >Assignee: wangmeng > > When hive.groupby.skewindata=true, the following query based on TPC-H gives > wrong results: > {code} > set hive.groupby.skewindata=true; > select l_returnflag, count(*), count(distinct l_linestatus) > from lineitem > group by l_returnflag > limit 10; > {code} > The query plan shows that it generates only one MapReduce job instead of two > theoretically, which is dictated by hive.groupby.skewindata=true. > The problem arises only when {noformat}count(*){noformat} and > {noformat}count(distinct){noformat} exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)