Hi, you need to pass several properties because of NN HA. What distro do you use? see here: http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html#Configuration_overview Configuration details
<property> <name>dfs.ha.namenodes.mycluster</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.mycluster.nn1</name> <value>machine1.example.com:8020</value> </property> <property> <name>dfs.namenode.rpc-address.mycluster.nn2</name> <value>machine2.example.com:8020</value> </property> You sohuld pass these three properties to your "client" (pig). Then you can refer your nameservice as hdfs://mycluster/path/to/input 2014-10-15 18:16 GMT+04:00 Jakub Stransky <stransky...@gmail.com>: > Thanks! Just giving a try but I am not able to figure out the hostname of > dhfs because we run cluster in HA mode. Is there a way how to pass hadoop > config? Because othervise I don't see any other option how that would be > resolved. > > On 15 October 2014 14:50, Serega Sheypak <serega.shey...@gmail.com> wrote: > > > Have a look at > > http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html > > We use it in our integration test utility since default pig-unit doesn't > > fit our needs. > > You can register script and run it. > > > > 2014-10-15 16:45 GMT+04:00 Jakub Stransky <stransky...@gmail.com>: > > > > > That's probably not the option as I need to wrap it in oozie java > action > > > and not sure that pig will be there. May be I am completely mistaken > > > > > > On 15 October 2014 14:01, praveenesh kumar <praveen...@gmail.com> > wrote: > > > > > > > You can write a wrapper (like a shell/ruby/python script) which can > do > > > the > > > > parameter generation and then pass those parameters using -p > parameter > > in > > > > your pig calling statement. Wouldn't that work ? > > > > > > > > > > > > > > > > On Wed, Oct 15, 2014 at 12:16 PM, Jakub Stransky < > > stransky...@gmail.com> > > > > wrote: > > > > > > > > > Hello experienced users, > > > > > > > > > > I have a pig script where I need dynamically change input > parameters > > > e.g. > > > > > input paths and for every single combination I need to submit a pig > > > > > script. Esentially I am walking around missing features of loops > or > > > for > > > > > cycles. The only solution which came to my mind is put the whole > > > dynamic > > > > > logic in java activity just need to fine a way how to submit a pig > > > script > > > > > from java. > > > > > > > > > > Could someone please point me to right direction? > > > > > > > > > > Many thanks > > > > > Jakub > > > > > > > > > > > > > > > > > > > > > -- > > > Jakub Stransky > > > cz.linkedin.com/in/jakubstransky > > > > > > > > > -- > Jakub Stransky > cz.linkedin.com/in/jakubstransky >
