In case you wanna know the actual cause of the issue, please look at BIGTOP-906. It seems that published Pig artifacts are missing pieces of what's available in the fat-pig-jar aka pig.jar. And seemingly the only way to deal with it is to avoid using Maven artifacts and start using the fat one.
I wonder if the artifacts can be fixed somehow...
Cos
On Fri, Mar 29, 2013 at 03:45PM, Konstantin Boudnik wrote:
> Hey Johnny.
>
> Yup, I've been using pretty much this without -hs artifact. However, adding
> hadoop-mapreduce-client-hs didn't help ;(
>
> Cos
>
> On Fri, Mar 29, 2013 at 03:25PM, Johnny Zhang wrote:
> > Hi, cos:
> > I met the same problem before. Can you try below in your Bigtop pig
> > maven execution .pom ? It will actually resolve other problem as well (for
> > example, you may see Pig cannot get kerberos principle in secure YARN
> > cluster)
> >
> >
> > <dependencies>
> >
> > <dependency>
> >
> > <groupId>org.apache.pig</groupId>
> >
> > <artifactId>pigsmoke</artifactId>
> >
> > <version>${pig-smoke.version}</version>
> >
> > <exclusions>
> >
> > <exclusion>
> >
> > <groupId>org.apache.pig</groupId>
> >
> > <artifactId>pig</artifactId>
> >
> > </exclusion>
> >
> > </exclusions>
> >
> > </dependency>
> >
> > <dependency>
> >
> > <groupId>org.apache.hadoop</groupId>
> >
> > <artifactId>hadoop-common</artifactId>
> >
> > <version>${hadoop.version}</version>
> >
> > </dependency>
> >
> > <dependency>
> >
> > <groupId>org.apache.hadoop</groupId>
> >
> > <artifactId>hadoop-hdfs</artifactId>
> >
> > <version>${hadoop.version}</version>
> >
> > </dependency>
> >
> > <dependency>
> >
> > <groupId>org.apache.hadoop</groupId>
> >
> > <artifactId>hadoop-mapreduce-client-core</artifactId>
> >
> > <version>${hadoop.version}</version>
> >
> > </dependency>
> >
> > <dependency>
> >
> > <groupId>org.apache.hadoop</groupId>
> >
> > <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
> >
> > <version>${hadoop.version}</version>
> >
> > </dependency>
> >
> > <dependency>
> >
> > <groupId>org.apache.hadoop</groupId>
> >
> > <artifactId>hadoop-mapreduce-client-hs</artifactId>
> >
> > <version>${hadoop.version}</version>
> >
> > </dependency>
> >
> >
> > Johnny
> >
> >
> >
> > On Fri, Mar 29, 2013 at 2:54 PM, Konstantin Boudnik <c...@apache.org> wrote:
> >
> > > I certainly agree with you. I have managed to narrow it down to the
> > > difference
> > > between pig-0.10.0-withouthadoop.jar and maven pig-0.10.0.jar.
> > >
> > > Basically, I can run:
> > >
> > > java -cp \
> > > /etc/hadoop/conf:\
> > > /etc/pig/conf:\
> > > /root/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:\
> > >
> > > /root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
> > >
> > > /root/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
> > > /root/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:\
> > > /root/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
> > > /usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:\
> > > /root/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-common/2.0.3.1-SNAPSHOT/hadoop-common-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-auth/2.0.3.1-SNAPSHOT/hadoop-auth-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.3.1-SNAPSHOT/hadoop-hdfs-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.0.3.1-SNAPSHOT/hadoop-yarn-api-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.0.3.1-SNAPSHOT/hadoop-yarn-client-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.0.3.1-SNAPSHOT/hadoop-yarn-common-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-core-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-common-2.0.3.1-SNAPSHOT.jar:\
> > >
> > > /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-jobclient-2.0.3.1-SNAPSHOT.jar:\
> > > /usr/lib/pig/pig-0.10.0-withouthadoop.jar:\
> > > /root/.m2/repository/com/google/guava/guava/11.0/guava-11.0.jar:\
> > > /root/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:\
> > > /root/.m2/repository/org/antlr/antlr-runtime/3.4/antlr-runtime-3.4.jar:\
> > >
> > > /root/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:\
> > >
> > > /root/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:\
> > >
> > > /root/.m2/repository/org/antlr/stringtemplate/3.2.1/stringtemplate-3.2.1.jar:\
> > > /usr/lib/pig/lib/automaton.jar:\
> > > build/ivy/lib/Pig/junit-4.5.jar:\
> > > build/pig-0.10.0-SNAPSHOT-smoketests.jar:\
> > >
> > > /root/.m2/repository/org/apache/pig/pigunit/0.10.0-SNAPSHOT/pigunit-0.10.0-SNAPSHOT.jar
> > > \
> > > -Dpigunit.exectype.cluster=true \
> > > org.junit.runner.JUnitCore org.apache.pig.test.pigunit.TestPigTest
> > >
> > > But once
> > > /usr/lib/pig/pig-0.10.0-withouthadoop.jar
> > > is replaced with
> > > /root/.m2/repository/org/apache/pig/pig/0.10.0/pig-0.10.0.jar
> > > all hell breaks loose and submission doesn't happen anymore ;( And as
> > > before -
> > > no error messages nor exceptions appear anywhere on the surface.
> > >
> > > Cos
> > >
> > > On Fri, Mar 29, 2013 at 11:52AM, Dmitriy Ryaboy wrote:
> > > > I missed the part where you are running it from JUnit. Still suspect a
> > > > classpath issue :). Can you dump your classpath as seen by JUnit, and
> > > what
> > > > happens when you run "-secretDebugCmd" and see if something is missing?
> > > >
> > > > -D
> > > >
> > > >
> > > > On Fri, Mar 29, 2013 at 10:38 AM, Konstantin Boudnik <c...@apache.org>
> > > wrote:
> > > >
> > > > > There's no problem with Pig itself. If I use Pig CLI I can run some
> > > > > PigLatin
> > > > > scripts and they go all the way to the completion.
> > > > >
> > > > > The problem arises while running TestPigTest _programmatically_ e.g.
> > > using
> > > > > JUnit runner. Is there any secret debug options in this case? I've
> > > looked
> > > > > through the code but didn't spot anything immediately.
> > > > >
> > > > > I will give "-secretDebugCmd" a try anyway - it might give me some
> > > ideas...
> > > > >
> > > > > Appreciate the help!
> > > > > Cos
> > > > >
> > > > > On Fri, Mar 29, 2013 at 09:08AM, Dmitriy Ryaboy wrote:
> > > > > > Can you run `pig -secretDebugCmd` and look at your paths? I suspect
> > > > > there's
> > > > > > something going on there, like not having the right configs or
> > > > > > hadoop
> > > > > jars.
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 28, 2013 at 11:32 PM, Konstantin Boudnik
> > > > > > <c...@apache.org
> > > >
> > > > > wrote:
> > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > I am trying to run TestPigTest against a fanctional Hadoop 2.0
> > > > > > > YARN
> > > > > > > cluster -
> > > > > > > I can run MR jobs using Hive or plain MR examples. However, it is
> > > going
> > > > > > > somewhat unsuccessful. While using pig "far" jar file - e.g. with
> > > > > > > everything
> > > > > > > repacked inside of it - I seeminly can run TestPigTest against a
> > > real
> > > > > > > cluster.
> > > > > > >
> > > > > > > However, while using pig-withouthadoop.jar and adding all needed
> > > > > > > dependencies
> > > > > > > to the classpath, Pig is failing to communicate with YARN nor
> > > submit
> > > > > any MR
> > > > > > > jobs. There's no exceptions, not error messages anywhere in the
> > > logs.
> > > > > Have
> > > > > > > anyone seen anything similar?
> > > > > > >
> > > > > > > Any input is highly appreciated!
> > > > > > >
> > > > > > > --
> > > > > > > With regards,
> > > > > > > Cos
> > > > > > >
> > > > > > >
> > > > >
> > >
signature.asc
Description: Digital signature
