[jira] [Commented] (HIVE-10755) Rework on HIVE-5193 to enhance the column oriented table acess
[
https://issues.apache.org/jira/browse/HIVE-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940647#comment-14940647
]
Daniel Dai commented on HIVE-10755:
---
The approach should be fine. We also need to avoid appending ids multiple times
to conf to avoid unnecessary warning from
ColumnProjectionUtils.getReadColumnIDs. We shall also add a test case which
cause HIVE-10752. A simple join + foreach should reproduce HIVE-10752:
{code}
A = load '" + COMPLEX_TABLE + "' using
org.apache.hive.hcatalog.pig.HCatLoader();
B = load '" + COMPLEX_TABLE + "' using
org.apache.hive.hcatalog.pig.HCatLoader();
C = join A by name, B by name;
D = foreach C generate B::studentid;
{code}
> Rework on HIVE-5193 to enhance the column oriented table acess
> --
>
> Key: HIVE-10755
> URL: https://issues.apache.org/jira/browse/HIVE-10755
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.0.0
>
> Attachments: HIVE-10755.patch
>
>
> Add the support of column pruning for column oriented table access which was
> done in HIVE-5193 but was reverted due to the join issue in HIVE-10720.
> In 1.3.0, the patch posted by Viray didn't work, probably due to some jar
> reference. That seems to get fixed and that patch works in 2.0.0 now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10755) Rework on HIVE-5193 to enhance the column oriented table acess
[ https://issues.apache.org/jira/browse/HIVE-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941620#comment-14941620 ] Aihua Xu commented on HIVE-10755: - Thanks for reviewing Daniel. I will take a look and try to add the test case. > Rework on HIVE-5193 to enhance the column oriented table acess > -- > > Key: HIVE-10755 > URL: https://issues.apache.org/jira/browse/HIVE-10755 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 1.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Fix For: 2.0.0 > > Attachments: HIVE-10755.patch > > > Add the support of column pruning for column oriented table access which was > done in HIVE-5193 but was reverted due to the join issue in HIVE-10720. > In 1.3.0, the patch posted by Viray didn't work, probably due to some jar > reference. That seems to get fixed and that patch works in 2.0.0 now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10755) Rework on HIVE-5193 to enhance the column oriented table acess
[
https://issues.apache.org/jira/browse/HIVE-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942264#comment-14942264
]
Hive QA commented on HIVE-10755:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12743955/HIVE-10755.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9649 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5510/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5510/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5510/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12743955 - PreCommit-HIVE-TRUNK-Build
> Rework on HIVE-5193 to enhance the column oriented table acess
> --
>
> Key: HIVE-10755
> URL: https://issues.apache.org/jira/browse/HIVE-10755
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.0.0
>
> Attachments: HIVE-10755.patch
>
>
> Add the support of column pruning for column oriented table access which was
> done in HIVE-5193 but was reverted due to the join issue in HIVE-10720.
> In 1.3.0, the patch posted by Viray didn't work, probably due to some jar
> reference. That seems to get fixed and that patch works in 2.0.0 now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10755) Rework on HIVE-5193 to enhance the column oriented table acess
[
https://issues.apache.org/jira/browse/HIVE-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944083#comment-14944083
]
Aihua Xu commented on HIVE-10755:
-
[~daijy] I'm working on adding the unit tests you mentioned to
TestHCatLoader.java. Actually I have problem to load complex_table with the
following code.
{noformat}
PigServer server = new PigServer(ExecType.LOCAL);
server.registerQuery("A = load '" + COMPLEX_TABLE + "' using
org.apache.hive.hcatalog.pig.HCatLoader();");
server.registerQuery("B = load '" + COMPLEX_TABLE + "' using
org.apache.hive.hcatalog.pig.HCatLoader();");
server.registerQuery("C = join A by name, B by name;");
server.registerQuery("D = foreach C generate B::studentid;");
{noformat}
{{Iterator iter = server.openIterator("A");}} won't be able to access
the data, while if I replace with basic_table it's fine.
Don't know too much pig/hcatalog. I'm wondering if I did anything wrong?
> Rework on HIVE-5193 to enhance the column oriented table acess
> --
>
> Key: HIVE-10755
> URL: https://issues.apache.org/jira/browse/HIVE-10755
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.0.0
>
> Attachments: HIVE-10755.patch
>
>
> Add the support of column pruning for column oriented table access which was
> done in HIVE-5193 but was reverted due to the join issue in HIVE-10720.
> In 1.3.0, the patch posted by Viray didn't work, probably due to some jar
> reference. That seems to get fixed and that patch works in 2.0.0 now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10755) Rework on HIVE-5193 to enhance the column oriented table acess
[ https://issues.apache.org/jira/browse/HIVE-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616724#comment-14616724 ] Aihua Xu commented on HIVE-10755: - [~viraj] and [~mithun] I spent time trying to work on HIVE-5193, but later found out your patch for HIVE-10720 does fix the issue in 2.0.0 now. I'm wondering if we had some older versions of the jars which causes the incorrect test result at that time. But now with the test against hadoop2.6.0, the join works with correct result. Sorry that I didn't have enough time to investigate further at that time and had to revert HIVE-5193. [~viraj] Please assign this to you since basically it's your patch. > Rework on HIVE-5193 to enhance the column oriented table acess > -- > > Key: HIVE-10755 > URL: https://issues.apache.org/jira/browse/HIVE-10755 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 1.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Fix For: 2.0.0 > > Attachments: HIVE-10755.patch > > > Add the support of column pruning for column oriented table access which was > done in HIVE-5193 but was reverted due to the join issue in HIVE-10720. > In 1.3.0, the patch posted by Viray didn't work, probably due to some jar > reference. That seems to get fixed and that patch works in 2.0.0 now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10755) Rework on HIVE-5193 to enhance the column oriented table acess
[ https://issues.apache.org/jira/browse/HIVE-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617185#comment-14617185 ] Viraj Bhat commented on HIVE-10755: --- Thanks for letting me know about the current status. Really appreciate. Mithun is currently on vacation, when he is back I will work with him to get the fix to HIVE-10720 and also HIVE-5193. We at Yahoo! did not find the incorrect results when we internally patched it with HIVE-10720. Viraj > Rework on HIVE-5193 to enhance the column oriented table acess > -- > > Key: HIVE-10755 > URL: https://issues.apache.org/jira/browse/HIVE-10755 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 1.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Fix For: 2.0.0 > > Attachments: HIVE-10755.patch > > > Add the support of column pruning for column oriented table access which was > done in HIVE-5193 but was reverted due to the join issue in HIVE-10720. > In 1.3.0, the patch posted by Viray didn't work, probably due to some jar > reference. That seems to get fixed and that patch works in 2.0.0 now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10755) Rework on HIVE-5193 to enhance the column oriented table acess
[ https://issues.apache.org/jira/browse/HIVE-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14650060#comment-14650060 ] Aihua Xu commented on HIVE-10755: - [~viraj] and [~mithun] How is everything? The attached patch should fix both HIVE-5193 and HIVE-10720. Can we get it submitted? > Rework on HIVE-5193 to enhance the column oriented table acess > -- > > Key: HIVE-10755 > URL: https://issues.apache.org/jira/browse/HIVE-10755 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 1.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Fix For: 2.0.0 > > Attachments: HIVE-10755.patch > > > Add the support of column pruning for column oriented table access which was > done in HIVE-5193 but was reverted due to the join issue in HIVE-10720. > In 1.3.0, the patch posted by Viray didn't work, probably due to some jar > reference. That seems to get fixed and that patch works in 2.0.0 now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
