Cheolsoo, just to make sure I do not misinterpret your words: - there were a change in YARN packaging between 2.0.0 and 2.0.3 that move some APIs into separate artifact hadoop-yarn-client - adding hadoop-yarn-client is only relevant for 2.0.3 and later and won't work on 2.0.0-2.0.2 versions
Does that represent your point of view correctly? If so, increamenting the version does't look like a big deal anyway, 'cause we are talking about _minor_ Hadoop version increment, normally reflecting bug fixes, etc. If the above mistates the situation that I don't see why the version needs to be bumped up at all. Please advise. With regards, Cos On Mon, May 13, 2013 at 09:35AM, Cheolsoo Park wrote: > Hi Cos, > > I am happy to backport it to branch-011. But I am not entirely sure whether > we should bump the Hadoop version in a minor release. As of now, > branch-0.11 unit tests run fine with 2.0.0-alpha. > > Please let me know if anyone has an opinion on this. > > Thanks, > Cheolsoo > > > On Sun, May 12, 2013 at 11:45 PM, Konstantin Boudnik <c...@apache.org> wrote: > > > If unit tests can't be run - which seems to be the case - it sound like a > > release blocker to me. Any plans to back port this fix into 0.11.2? > > 0.11.1 doesn't have it, right? > > > > Cos > > > > On Sun, May 12, 2013 at 09:12PM, Cheolsoo Park wrote: > > > Hi Andrew, > > > > > > >> I'm trying HEAD of branch-0.11. > > > > > > You need PIG-3248: https://issues.apache.org/jira/browse/PIG-3248. It's > > not > > > in branch-0.11 but in trunk. > > > > > > In particular, you need to add the following lines to ivy.xml: > > > > > > + <dependency org="org.apache.hadoop" name="hadoop-yarn-client" > > > + rev="${hadoop-mapreduce.version}" conf="hadoop23->master"/> > > > > > > Thanks, > > > Cheolsoo > > > > > > > > > > > > On Sun, May 12, 2013 at 7:13 PM, Andrew Purtell <apurt...@apache.org> > > wrote: > > > > > > > It happens consistently on multiple systems and different operating > > systems > > > > and JDK versions, so while is a build (or more specifically a unit test > > > > environment) problem, it is unlikely an issue singular to me. > > > > > > > > On Saturday, May 11, 2013, Konstantin Boudnik wrote: > > > > > > > > > As you know we've been testing Pig 0.11 vs 2.0.4-alpha as a part of > > > > > Bigtop's > > > > > validation for the latest hadoop release and it worked ok. Bigtop > > doesn't > > > > > run > > > > > unit tests though, so it seems like a build issue to me. > > > > > > > > > > Cos > > > > > > > > > > On Sat, May 11, 2013 at 09:19AM, Andrew Purtell wrote: > > > > > > I've tried that, thanks. I did a bit more investigation and it > > seems > > > > the > > > > > > issue is recent Hadoop 2 releases. Has anyone tried running Pig > > unit > > > > > tests > > > > > > using a more recent Hadoop release than 2.0.0-alpha? Maybe my > > trouble > > > > is > > > > > a > > > > > > simple thing that someone with more experience with Pig internals > > would > > > > > see > > > > > > right away? Cluster testing seems ok. It's just unit tests that > > fail. > > > > But > > > > > > that is concerning. > > > > > > > > > > > > I'm trying HEAD of branch-0.11. > > > > > > > > > > > > My Java is version "1.6.0_43" Java(TM) SE Runtime Environment > > (build > > > > > > 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, > > mixed > > > > > > mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64). > > > > > > > > > > > > With defaults and only -Dhadoopversion=23 on the Ant command line, > > it > > > > > seems > > > > > > ok. > > > > > > > > > > > > With build.properties of: > > > > > > > > > > > > hadoopversion=23 > > > > > > hadoop-common.version=2.0.4-alpha > > > > > > hadoop-hdfs.version=2.0.4-alpha > > > > > > hadoop-mapreduce.version=2.0.4-alpha > > > > > > > > > > > > > > > > > > or defined on the Ant command line, I'll see unit test failures > > like: > > > > > > > > > > > > Testcase: testAccumWithDistinct took 0.868 sec > > > > > > Caused an ERROR > > > > > > org/apache/hadoop/mapred/ResourceMgrDelegate > > > > > > java.lang.NoClassDefFoundError: > > > > > org/apache/hadoop/mapred/ResourceMgrDelegate > > > > > > at > > > > > org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112) > > > > > > at > > > > > > > > > > > > > > > > > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > > > > > > at > > > > > org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94) > > > > > > at > > org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81) > > > > > > at > > org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74) > > > > > > at > > org.apache.hadoop.mapred.JobClient.init(JobClient.java:482) > > > > > > at > > > > org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461) > > > > > > at > > > > > > > > > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152) > > > > > > at org.apache.pig.PigServer.launchPlan(PigServer.java:1264) > > > > > > at > > > > > > > > > > > > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249) > > > > > > at org.apache.pig.PigServer.storeEx(PigServer.java:931) > > > > > > at org.apache.pig.PigServer.store(PigServer.java:898) > > > > > > at > > org.apache.pig.PigServer.openIterator(PigServer.java:811) > > > > > > at > > > > > > > > > > > > > > > > > org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424) > > > > > > > > > > > > That suggests a cause but I've not started spelunking code with the > > > > hope > > > > > > this is something simple that someone has already encountered. > > > > > > > > > > > > > > > > > > On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang < > > xiao...@cloudera.com> > > > > > wrote: > > > > > > > > > > > > > Hi, Andrew: > > > > > > > Does something like "-Dhadoopversion=23" help ? eg. ant clean > > test > > > > > > > -Dhadoopversion=23 -Dtest.junit.output.format=xml > > > > > > > > > > > > > > Johnny > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2013 at 3:39 AM, Andrew Purtell < > > apurt...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Please pardon the basic question. I'm building Pig > > 0.11.2-SNAPSHOT > > > > > > > against > > > > > > > > Hadoop 2.0.4. 'ant package' and full cluster tests work fine, > > but > > > > > I'm not > > > > > > > > having much luck with running the unit tests, 'ant test-core' > > or > > > > 'ant > > > > > > > > test'. The problem looks to be a MR app classpath issue. > > > > > > > > > > > > > > > > Sometimes: java.lang.NoClassDefFoundError: > > > > > > > > org/apache/hadoop/yarn/client/YarnClientImpl > > > > > > > > > > > > > > > > Sometimes: java.lang.NoClassDefFoundError: > > > > > > > > org/apache/hadoop/mapred/ResourceMgrDelegate > > > > > > > > > > > > > > > > A few Google searches have turned up no useful pointers. Maybe > > > > there > > > > > is > > > > > > > > something simple I am mis > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > > > > > - Andy > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein > > > > (via Tom White) > > > > > >
signature.asc
Description: Digital signature