[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13745735#comment-13745735
]
Vikram Dixit K commented on HIVE-5129:
--
[~navis] This issue is related to HIVE-4692. I have attached a patch which
retains the same behavior as was before that patch without affecting the test
added as part of it. However, I am not sure what else is affected by this down
stream. I have started to run the unit tests to make sure I haven't broken
other things. It would be great if you could take a look and provide feedback
on this. Please let me know if you have a better approach in mind.
Thanks!
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13745739#comment-13745739
]
Vikram Dixit K commented on HIVE-5129:
--
Review board request:
https://reviews.apache.org/r/13697/
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13745873#comment-13745873
]
Navis commented on HIVE-5129:
-
mGBY-1RS optimization is really confusing with distinct functions. IMHO, it
should not be allowed to mix distinct and non-distinct cases into one group.
For example,
{noformat}
from src tablesample (10 ROWS)
insert overwrite table src_a select key, count(distinct key) + count(distinct
value) group by key;
{noformat}
makes 20 rows for src_a, and,
{noformat}
from src tablesample (10 ROWS)
insert overwrite table src_b select key, count(value) group by key, value;
{noformat}
makes 10 rows for src_b, but,
{noformat}
from src tablesample (10 ROWS)
insert overwrite table src_a select key, count(distinct key) + count(distinct
value) group by key
insert overwrite table src_b select key, count(value) group by key, value;
{noformat}
makes 20 rows for src_a and src_b, and lastly,
{noformat}
from src tablesample (10 ROWS)
insert overwrite table src_b select key, count(value) group by key, value
insert overwrite table src_a select key, count(distinct key) + count(distinct
value) group by key;
{noformat}
is not working as described in this issue. After applying your patch, it
succeeded with 10 rows for both.
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747173#comment-13747173
]
Navis commented on HIVE-5129:
-
It seemed current distinct implementation has flaw.
{noformat}
select key, count(distinct key) + count(distinct value) from src tablesample
(10 ROWS) group by key;
100 1
val_100 1
165 1
val_165 1
238 1
val_238 1
255 1
val_255 1
27 1
val_27 1
278 1
val_278 1
311 1
val_311 1
409 1
val_409 1
86 1
val_86 1
98 1
val_98 1
{noformat}
this does not make sense.
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt,
> HIVE-5129.2.WIP.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748951#comment-13748951
]
Harish Butani commented on HIVE-5129:
-
Looks good. Can you add a test that has different distincts, which will cause
multiple Jobs to be genned.
We should look at combining such clauses in the future.
[~navis] we think your e.g. is a separate issue. What we observe is that the
Key OI setup in your e.g. has only 1 field which is the Union Filed for key and
value columns.
For some reason the key column from the group by is removed from the Key OI. So
what happens is each spray instance of the original row is being output.
So can we look at this in a separate jira.
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt,
> HIVE-5129.2.WIP.patch.txt, HIVE-5129.3.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755324#comment-13755324
]
Harish Butani commented on HIVE-5129:
-
+1
will commit if tests pass
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt,
> HIVE-5129.2.WIP.patch.txt, HIVE-5129.3.patch.txt, HIVE-5129.4.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755407#comment-13755407
]
Hive QA commented on HIVE-5129:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12600910/HIVE-5129.4.patch
{color:green}SUCCESS:{color} +1 2903 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/581/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/581/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt,
> HIVE-5129.2.WIP.patch.txt, HIVE-5129.3.patch.txt, HIVE-5129.4.patch,
> HIVE-5129.4.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13756916#comment-13756916
]
Hudson commented on HIVE-5129:
--
FAILURE: Integrated in Hive-trunk-hadoop2 #401 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/401/])
HIVE-5129 Multiple table insert fails on count distinct (Vikram Dixit via
Harish Butani) (rhbutani:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1519764)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
* /hive/trunk/ql/src/test/queries/clientpositive/multi_insert_gby3.q
* /hive/trunk/ql/src/test/results/clientpositive/multi_insert_gby3.q.out
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.12.0
>
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt,
> HIVE-5129.2.WIP.patch.txt, HIVE-5129.3.patch.txt, HIVE-5129.4.patch,
> HIVE-5129.4.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13756812#comment-13756812
]
Harish Butani commented on HIVE-5129:
-
Committed to trunk. Thanks, Vikram!
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt,
> HIVE-5129.2.WIP.patch.txt, HIVE-5129.3.patch.txt, HIVE-5129.4.patch,
> HIVE-5129.4.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13757851#comment-13757851
]
Hudson commented on HIVE-5129:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #83 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/83/])
HIVE-5129 Multiple table insert fails on count distinct (Vikram Dixit via
Harish Butani) (rhbutani:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1519764)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
* /hive/trunk/ql/src/test/queries/clientpositive/multi_insert_gby3.q
* /hive/trunk/ql/src/test/results/clientpositive/multi_insert_gby3.q.out
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.12.0
>
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt,
> HIVE-5129.2.WIP.patch.txt, HIVE-5129.3.patch.txt, HIVE-5129.4.patch,
> HIVE-5129.4.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13757954#comment-13757954
]
Hudson commented on HIVE-5129:
--
FAILURE: Integrated in Hive-trunk-hadoop1-ptest #150 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/150/])
HIVE-5129 Multiple table insert fails on count distinct (Vikram Dixit via
Harish Butani) (rhbutani:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1519764)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
* /hive/trunk/ql/src/test/queries/clientpositive/multi_insert_gby3.q
* /hive/trunk/ql/src/test/results/clientpositive/multi_insert_gby3.q.out
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.12.0
>
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt,
> HIVE-5129.2.WIP.patch.txt, HIVE-5129.3.patch.txt, HIVE-5129.4.patch,
> HIVE-5129.4.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5129) Multiple table insert fails on count(distinct)
[
https://issues.apache.org/jira/browse/HIVE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758024#comment-13758024
]
Hudson commented on HIVE-5129:
--
SUCCESS: Integrated in Hive-trunk-h0.21 #2309 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/2309/])
HIVE-5129 Multiple table insert fails on count distinct (Vikram Dixit via
Harish Butani) (rhbutani:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1519764)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
* /hive/trunk/ql/src/test/queries/clientpositive/multi_insert_gby3.q
* /hive/trunk/ql/src/test/results/clientpositive/multi_insert_gby3.q.out
> Multiple table insert fails on count(distinct)
> --
>
> Key: HIVE-5129
> URL: https://issues.apache.org/jira/browse/HIVE-5129
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.12.0
>
> Attachments: aggrTestMultiInsertData1.txt,
> aggrTestMultiInsertData.txt, aggrTestMultiInsert.q, HIVE-5129.1.patch.txt,
> HIVE-5129.2.WIP.patch.txt, HIVE-5129.3.patch.txt, HIVE-5129.4.patch,
> HIVE-5129.4.patch.txt
>
>
> Hive fails with a class cast exception on queries of the form:
> {noformat}
> from studenttab10k
> insert overwrite table multi_insert_2_1
> select name, avg(age) as avgage
> group by name
> insert overwrite table multi_insert_2_2
> select name, age, sum(gpa) as sumgpa
> group by name, age
> insert overwrite table multi_insert_2_3
> select name, count(distinct age) as distage
> group by name;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
