[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581911#comment-14581911
]
Xuefu Zhang commented on HIVE-10971:
Similar to TestCliDriver, TestSparkCliDriver is generated as part of .q test.
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 1.2.0
Reporter: WangMeng
Assignee: WangMeng
Fix For: 1.2.1
Attachments: HIVE-10971.01.patch, HIVE-10971.1.patch
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578602#comment-14578602
]
wangmeng commented on HIVE-10971:
-
{code}
hive set hive.groupby.skewindata=true;
hive explain select l_returnflag,count(*),count(distinct l_linestatus) from
lineitem group by l_returnflag limit 10;
OK
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: lineitem
Statistics: Num rows: 1008537518 Data size: 201707503616 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: l_returnflag (type: string), l_linestatus (type:
string)
outputColumnNames: l_returnflag, l_linestatus
Statistics: Num rows: 1008537518 Data size: 201707503616 Basic
stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(), count(DISTINCT l_linestatus)
keys: l_returnflag (type: string), l_linestatus (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 1008537518 Data size: 201707503616 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1008537518 Data size: 201707503616
Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: bigint)
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), count(DISTINCT KEY._col1:0._col0)
keys: KEY._col0 (type: string)
mode: complete
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 504268759 Data size: 100853751808 Basic stats:
COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: bigint), _col2
(type: bigint)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 504268759 Data size: 100853751808 Basic
stats: COMPLETE Column stats: NONE
Limit
Number of rows: 10
Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE
Column stats: NONE
File Output Operator
compressed: true
Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE
Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: 10
{code}
When hive.groupby.skewindata=false, the Group By operator has mode
mergepartial, which gives the correct results.
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579482#comment-14579482
]
Ashutosh Chauhan commented on HIVE-10971:
-
LGTM +1 need to update test files though.
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
Attachments: HIVE-10971.01.patch
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578802#comment-14578802
]
Hive QA commented on HIVE-10971:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12738553/HIVE-10971.01.patch
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 9004 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_rollup1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_groupby2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8_map_skew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_rollup1
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4223/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4223/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4223/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12738553 - PreCommit-HIVE-TRUNK-Build
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
Attachments: HIVE-10971.01.patch
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14580047#comment-14580047
]
Hive QA commented on HIVE-10971:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12738683/HIVE-10971.1.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9006 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4234/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4234/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4234/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12738683 - PreCommit-HIVE-TRUNK-Build
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
Attachments: HIVE-10971.01.patch, HIVE-10971.1.patch
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
