[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432147#comment-15432147 ] Daniel Dai commented on PIG-2125: - If something went wrong, I believe any MiniCluster or local unit test will fail. But for this change, we definitely need to run through all unit tests and system tests cuz the consequence is not obvious. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.9.1, 0.10.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.9.2, 0.10.0 > > Attachments: ContextFactory.java, PIG-2125-1.patch, > PIG-2125-10.patch, PIG-2125-10_0.9.patch, PIG-2125-2.patch, PIG-2125-3.patch, > PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, PIG-2125-7.patch, > PIG-2125-7_0.9.patch, PIG-2125-8.patch, PIG-2125-9.patch, > PIG-2125-9_0.9.patch, PIG-2125-buildxml-0.9.patch, PIG-2125-commitJob.patch, > PIG-2125-ivy-0.9-3.patch, PIG-2125-zebra.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432037#comment-15432037 ] Nandor Kollar commented on PIG-2125: How can I make sure that if I remove the ClientProtocolProvider, it won't break the Hadoop integration? Is there a specific unit test for that, or I should execute the entire test suite (which is takes a lot of time)? > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.9.1, 0.10.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.9.2, 0.10.0 > > Attachments: ContextFactory.java, PIG-2125-1.patch, > PIG-2125-10.patch, PIG-2125-10_0.9.patch, PIG-2125-2.patch, PIG-2125-3.patch, > PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, PIG-2125-7.patch, > PIG-2125-7_0.9.patch, PIG-2125-8.patch, PIG-2125-9.patch, > PIG-2125-9_0.9.patch, PIG-2125-buildxml-0.9.patch, PIG-2125-commitJob.patch, > PIG-2125-ivy-0.9-3.patch, PIG-2125-zebra.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431871#comment-15431871 ] Daniel Dai commented on PIG-2125: - org.apache.pig.ExecType is required, Pig use this to enumerate all exectype supported. org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider is used to work around an issue in Hadoop23, I think probably this is fixed in newer Hadoop and we can remove this if tested working. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.9.1, 0.10.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.9.2, 0.10.0 > > Attachments: ContextFactory.java, PIG-2125-1.patch, > PIG-2125-10.patch, PIG-2125-10_0.9.patch, PIG-2125-2.patch, PIG-2125-3.patch, > PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, PIG-2125-7.patch, > PIG-2125-7_0.9.patch, PIG-2125-8.patch, PIG-2125-9.patch, > PIG-2125-9_0.9.patch, PIG-2125-buildxml-0.9.patch, PIG-2125-commitJob.patch, > PIG-2125-ivy-0.9-3.patch, PIG-2125-zebra.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427388#comment-15427388 ] Nandor Kollar commented on PIG-2125: It looks like in this change two services are registered in META-INF/services, which are implemented in Hadoop client dependency. [~daijy] do you recall why these are registered in Pig an not just in Hadoop mapreduce client? It looks like due to this service registration and MAPREDUCE-6473 Pig related unit test break in Hive. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.9.1, 0.10.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.9.2, 0.10.0 > > Attachments: ContextFactory.java, PIG-2125-1.patch, > PIG-2125-10.patch, PIG-2125-10_0.9.patch, PIG-2125-2.patch, PIG-2125-3.patch, > PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, PIG-2125-7.patch, > PIG-2125-7_0.9.patch, PIG-2125-8.patch, PIG-2125-9.patch, > PIG-2125-9_0.9.patch, PIG-2125-buildxml-0.9.patch, PIG-2125-commitJob.patch, > PIG-2125-ivy-0.9-3.patch, PIG-2125-zebra.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142483#comment-13142483 ] Daniel Dai commented on PIG-2125: - After PIG-2125-10.patch, we can compile and run end-to-end tests on 23. There are still ~20 unit tests fail on 23 and we are working on that. Since all patches attached to the Jira is committed, and this thread is becoming big and confusing, I would suggest to close this ticket and create a separate ticket to track unit test fixes for hadoop 23, agree? > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: ContextFactory.java, PIG-2125-1.patch, > PIG-2125-10.patch, PIG-2125-10_0.9.patch, PIG-2125-2.patch, PIG-2125-3.patch, > PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, PIG-2125-7.patch, > PIG-2125-7_0.9.patch, PIG-2125-8.patch, PIG-2125-9.patch, > PIG-2125-9_0.9.patch, PIG-2125-buildxml-0.9.patch, PIG-2125-commitJob.patch, > PIG-2125-ivy-0.9-3.patch, PIG-2125-zebra.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141787#comment-13141787 ] Thomas Weise commented on PIG-2125: --- +1 ant compile-test is good on branch-0.9 now > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: ContextFactory.java, PIG-2125-1.patch, PIG-2125-2.patch, > PIG-2125-3.patch, PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, > PIG-2125-7.patch, PIG-2125-7_0.9.patch, PIG-2125-8.patch, PIG-2125-9.patch, > PIG-2125-9_0.9.patch, PIG-2125-buildxml-0.9.patch, PIG-2125-commitJob.patch, > PIG-2125-ivy-0.9-3.patch, PIG-2125-zebra.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126955#comment-13126955 ] Thejas M Nair commented on PIG-2125: +1 for PIG-2125-commitJob.patch > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: ContextFactory.java, PIG-2125-1.patch, PIG-2125-2.patch, > PIG-2125-3.patch, PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, > PIG-2125-7.patch, PIG-2125-7_0.9.patch, PIG-2125-buildxml-0.9.patch, > PIG-2125-commitJob.patch, PIG-2125-ivy-0.9.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126945#comment-13126945 ] Olga Natkovich commented on PIG-2125: - I think that is good enough. As long as we do not regress on the current Hadoop I think we are good. Just need a review then. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: ContextFactory.java, PIG-2125-1.patch, PIG-2125-2.patch, > PIG-2125-3.patch, PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, > PIG-2125-7.patch, PIG-2125-7_0.9.patch, PIG-2125-buildxml-0.9.patch, > PIG-2125-commitJob.patch, PIG-2125-ivy-0.9.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126928#comment-13126928 ] Daniel Dai commented on PIG-2125: - All tests done on 20.X. I only did sanity tests on .Next, since I don't have the facility to run the whole thing currently. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: ContextFactory.java, PIG-2125-1.patch, PIG-2125-2.patch, > PIG-2125-3.patch, PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, > PIG-2125-7.patch, PIG-2125-7_0.9.patch, PIG-2125-buildxml-0.9.patch, > PIG-2125-commitJob.patch, PIG-2125-ivy-0.9.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126785#comment-13126785 ] Olga Natkovich commented on PIG-2125: - Did the tests pass for both .Next and 20.x version? > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: ContextFactory.java, PIG-2125-1.patch, PIG-2125-2.patch, > PIG-2125-3.patch, PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, > PIG-2125-7.patch, PIG-2125-7_0.9.patch, PIG-2125-buildxml-0.9.patch, > PIG-2125-commitJob.patch, PIG-2125-ivy-0.9.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126756#comment-13126756 ] Daniel Dai commented on PIG-2125: - Tests pass for PIG-2125-commitJob.patch. No test included since it solve the problem only in hadoop 23, which is not released. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: ContextFactory.java, PIG-2125-1.patch, PIG-2125-2.patch, > PIG-2125-3.patch, PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, > PIG-2125-7.patch, PIG-2125-7_0.9.patch, PIG-2125-buildxml-0.9.patch, > PIG-2125-commitJob.patch, PIG-2125-ivy-0.9.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119812#comment-13119812 ] Arun C Murthy commented on PIG-2125: I've committed MAPREDUCE-3138 i.e. the ContextFactory patch to Hadoop MapReduce. Thanks. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: ContextFactory.java, PIG-2125-1.patch, PIG-2125-2.patch, > PIG-2125-3.patch, PIG-2125-4.patch, PIG-2125-5.patch, PIG-2125-6.patch, > PIG-2125-7.patch, PIG-2125-7_0.9.patch, e2e-hadoop23.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078313#comment-13078313 ] Daniel Dai commented on PIG-2125: - Put ContextFactory.java into $PIG_HOME/src/org/apache/hadoop/mapreduce, then compile. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: ContextFactory.java, PIG-2125-1.patch, PIG-2125-2.patch, > PIG-2125-3.patch, PIG-2125-4.patch, PIG-2125-5.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069124#comment-13069124 ] Daniel Dai commented on PIG-2125: - Steps to build pig.jar for hadoop 23: 1. change "build.xml", modify "hadoopversion" to "23" 2. change "ivy.xml", remove the lines: 3. Build hadoop 23, copy hadoop 23 libraries into lib, here is a list: aopalliance-1.0.jar commons-cli-1.2.jar commons-codec-1.4.jar commons-collections-3.2.1.jar commons-configuration-1.6.jar commons-httpclient-3.1.jar guice-2.0.jar guice-servlet-2.0.jar hadoop-common-0.22.0-SNAPSHOT.jar hadoop-common-test-0.22.0-SNAPSHOT.jar hadoop-hdfs-0.22.0-SNAPSHOT.jar hadoop-hdfs-test-0.22.0-SNAPSHOT.jar hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar hadoop-mapreduce-client-common-1.0-SNAPSHOT.jar hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar hadoop-mapreduce-client-hs-1.0-SNAPSHOT.jar hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT-tests.jar hadoop-mapreduce-client-shuffle-1.0-SNAPSHOT.jar hadoop-metrics2-0.22.0-SNAPSHOT.jar jetty-6.1.26.jar log4j-1.2.16.jar protobuf-java-2.4.0a.jar servlet-api-2.5-20081211.jar slf4j-api-1.6.1.jar slf4j-log4j12-1.6.1.jar yarn-api-1.0-SNAPSHOT.jar yarn-common-1.0-SNAPSHOT.jar yarn-server-common-1.0-SNAPSHOT.jar yarn-server-nodemanager-1.0-SNAPSHOT.jar yarn-server-resourcemanager-1.0-SNAPSHOT.jar yarn-server-tests-1.0-SNAPSHOT-tests.jar 4. Then, do an "ant clean jar" 5. You may also need to recompile your udf with the new hadoop libraries. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: PIG-2125-1.patch, PIG-2125-2.patch, PIG-2125-3.patch, > PIG-2125-4.patch, PIG-2125-5.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068744#comment-13068744 ] Daniel Dai commented on PIG-2125: - PIG-2125-5.patch committed to both trunk and 0.9 branch. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: PIG-2125-1.patch, PIG-2125-2.patch, PIG-2125-3.patch, > PIG-2125-4.patch, PIG-2125-5.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068739#comment-13068739 ] Richard Ding commented on PIG-2125: --- +1 > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: PIG-2125-1.patch, PIG-2125-2.patch, PIG-2125-3.patch, > PIG-2125-4.patch, PIG-2125-5.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067428#comment-13067428 ] Daniel Dai commented on PIG-2125: - PIG-2125-4.patch committed to both trunk and 0.9 branch. This is the first phase work (mapreduce and local mode end-to-end), I will continue work on the next phase (unit test). > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: PIG-2125-1.patch, PIG-2125-2.patch, PIG-2125-3.patch, > PIG-2125-4.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064268#comment-13064268 ] Daniel Dai commented on PIG-2125: - bq. 1. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA. They need to copy all hadoop 23 jars and dependencies into lib. This step should go away once hadoop 23 publish maven artifacts bq. 2. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed. Sure. I will add bq. 3. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better. How about org.apache.pig.backend.hadoop.shims? > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: PIG-2125-1.patch, PIG-2125-2.patch, PIG-2125-3.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064053#comment-13064053 ] Alan Gates commented on PIG-2125: - In general the patch looks good. I have a few questions/comments. # How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA. # Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed. # I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: PIG-2125-1.patch, PIG-2125-2.patch, PIG-2125-3.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049970#comment-13049970 ] Olga Natkovich commented on PIG-2125: - We don't have any current plans to use 22 at Yahoo or for Yahoo engineers to do the port. But of course a patch is welcome from anybody interested in taking this project on. > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: PIG-2125-1.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2125) Make Pig work with hadoop .NEXT
[ https://issues.apache.org/jira/browse/PIG-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049942#comment-13049942 ] Dmitriy V. Ryaboy commented on PIG-2125: are you guys planning on skipping 22 altogether? > Make Pig work with hadoop .NEXT > --- > > Key: PIG-2125 > URL: https://issues.apache.org/jira/browse/PIG-2125 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.10 > > Attachments: PIG-2125-1.patch > > > We need to make Pig work with hadoop .NEXT, the svn branch currently is: > https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
