Hi managed to overcome the HDFS HA settings but I am facing a strange error
right now:
2014-10-16 10:38:06,146 [main] WARN PigJobControl.<clinit> - falling back
to default JobControl (not using hadoop 0.20 ?)
java.lang.NoSuchFieldException: runnerState
at java.lang.Class.getDeclaredField(Class.java:1948)
at
org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
at
org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:98)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:287)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:190)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.storeEx(PigServer.java:978)
at org.apache.pig.PigServer.store(PigServer.java:942)
at org.apache.pig.PigServer.openIterator(PigServer.java:855)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.PigServer.registerScript(PigServer.java:649)
at org.apache.pig.PigServer.registerScript(PigServer.java:726)
at org.apache.pig.PigServer.registerScript(PigServer.java:699)
at com.ncr.bigdata.mr.pig.PigFromJava$1.run(PigFromJava.java:32)
at com.ncr.bigdata.mr.pig.PigFromJava$1.run(PigFromJava.java:29)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at com.ncr.bigdata.mr.pig.PigFromJava.runPig(PigFromJava.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at
com.intellij.junit4.JUnit4TestRunnerUtil$IgnoreIgnoredTestJUnit4ClassRunner.runChild(JUnit4TestRunnerUtil.java:310)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
2014-10-16 10:38:06,155 [main] INFO ScriptState.addSettingsToConf - Pig
script settings are added to the job
It seems that I am using incompatible libs or so. My maven dependency looks
following:
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pigunit</artifactId>
<version>${version.pig}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>${version.pig}</version>
<scope>test</scope>
</dependency>
where
<version.hadoop>2.2.0</version.hadoop>
<version.pig>0.12.0</version.pig>
Any hint what might go wrong?
Thanks a lot
jakub
On 16 October 2014 09:46, Jakub Stransky <[email protected]> wrote:
> I am using Hortonworks HDP 2.0. I know that this setting is in I guess in
> Yarn-site.xml. Just not sure how to pass this configuration to PigServer.
>
> On 15 October 2014 16:58, Serega Sheypak <[email protected]> wrote:
>
>> Hi, you need to pass several properties because of NN HA.
>> What distro do you use?
>> see here:
>>
>> http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html#Configuration_overview
>> Configuration details
>>
>> <property>
>> <name>dfs.ha.namenodes.mycluster</name>
>> <value>nn1,nn2</value>
>> </property>
>>
>> <property>
>> <name>dfs.namenode.rpc-address.mycluster.nn1</name>
>> <value>machine1.example.com:8020</value>
>> </property>
>> <property>
>> <name>dfs.namenode.rpc-address.mycluster.nn2</name>
>> <value>machine2.example.com:8020</value>
>> </property>
>>
>>
>> You sohuld pass these three properties to your "client" (pig). Then
>> you can refer your nameservice as hdfs://mycluster/path/to/input
>>
>>
>> 2014-10-15 18:16 GMT+04:00 Jakub Stransky <[email protected]>:
>>
>> > Thanks! Just giving a try but I am not able to figure out the hostname
>> of
>> > dhfs because we run cluster in HA mode. Is there a way how to pass
>> hadoop
>> > config? Because othervise I don't see any other option how that would be
>> > resolved.
>> >
>> > On 15 October 2014 14:50, Serega Sheypak <[email protected]>
>> wrote:
>> >
>> > > Have a look at
>> > > http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html
>> > > We use it in our integration test utility since default pig-unit
>> doesn't
>> > > fit our needs.
>> > > You can register script and run it.
>> > >
>> > > 2014-10-15 16:45 GMT+04:00 Jakub Stransky <[email protected]>:
>> > >
>> > > > That's probably not the option as I need to wrap it in oozie java
>> > action
>> > > > and not sure that pig will be there. May be I am completely mistaken
>> > > >
>> > > > On 15 October 2014 14:01, praveenesh kumar <[email protected]>
>> > wrote:
>> > > >
>> > > > > You can write a wrapper (like a shell/ruby/python script) which
>> can
>> > do
>> > > > the
>> > > > > parameter generation and then pass those parameters using -p
>> > parameter
>> > > in
>> > > > > your pig calling statement. Wouldn't that work ?
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Oct 15, 2014 at 12:16 PM, Jakub Stransky <
>> > > [email protected]>
>> > > > > wrote:
>> > > > >
>> > > > > > Hello experienced users,
>> > > > > >
>> > > > > > I have a pig script where I need dynamically change input
>> > parameters
>> > > > e.g.
>> > > > > > input paths and for every single combination I need to submit a
>> pig
>> > > > > > script. Esentially I am walking around missing features of
>> loops
>> > or
>> > > > for
>> > > > > > cycles. The only solution which came to my mind is put the whole
>> > > > dynamic
>> > > > > > logic in java activity just need to fine a way how to submit a
>> pig
>> > > > script
>> > > > > > from java.
>> > > > > >
>> > > > > > Could someone please point me to right direction?
>> > > > > >
>> > > > > > Many thanks
>> > > > > > Jakub
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Jakub Stransky
>> > > > cz.linkedin.com/in/jakubstransky
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Jakub Stransky
>> > cz.linkedin.com/in/jakubstransky
>> >
>>
>
>
>
> --
> Jakub Stransky
> cz.linkedin.com/in/jakubstransky
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
