If your hdfs path is visible to master, and since oozie is running in your master, if you add the path to "oozie.libpath" wont it work?
Thanks & Regards, Harish.T.K On Tue, Jan 29, 2013 at 12:15 PM, Jinye Luo <j...@merkleinc.com> wrote: > Robert, I guess I wasn't 100% clear on what I was trying to do in my > previous message. > > I am looking for something in Oozie to perform equivalent to "hadoop fs > -copyFromLocal /home/cloudera/input.log /user/cloudera/input". > > As you said, "//user/cloudera/input/input.log" is visible to all nodes if > "hadoop fs..." is successful, but /home/cloudera/input.log is not, and that > is my problem. > > -----Original Message----- > From: Robert Kanter [mailto:rkan...@cloudera.com] > Sent: Tuesday, January 29, 2013 2:06 PM > To: user@oozie.apache.org > Cc: Mohammad Islam > Subject: Re: Is it possible to use Oozie Java/Shell action to put a file > into hdfs? > > Hi Jinye, > > When you run that hadoop fs command, you're copying input.log from your > local FS to HDFS, so it should be visible on all of your nodes. > > Are you sure that you're pointing the Oozie action to the right place? > When you run hadoop fs -copyFromLocal /home/cloudera/input.log > /user/cloudera/input I believe you are actually uploading the file to > /user/cloudera/input/input.log because you didn't specify a file extension > in the destination path; did you mean to upload it to > /user/cloudera/input.log? If so, try running hadoop fs -copyFromLocal > /home/cloudera/input.log /user/cloudera/input.log > > - Robert > > > On Tue, Jan 29, 2013 at 11:23 AM, Jinye Luo <j...@merkleinc.com> wrote: > > > Let's say I have a four node cluster, one master and three slaves. > > > > The master is running: > > > > hadoop-yarn-resourcemanager > > hadoop-hdfs-namenode > > hadoop--mapreduce-historyserver > > zookeeper-server > > hbase-master > > oozie > > > > > > The slaves are running: > > hadoop-yarn-nodemanager > > hadoop-hdfs-datanode > > hbase-regionserver > > > > From the master, I run the following to put a file into hdfs, then > > kick off a M/R job: > > hadoop fs -copyFromLocal /home/cloudera/input.log > > /user/cloudera/input > > > > hadoop jar log-processing.jar ... > > > > Obviously, input.log is not visible to slaves, and Oozie action is > > always executed in slaves, hence the Oozie action will always fail. > > > > -----Original Message----- > > From: Mohammad Islam [mailto:misla...@yahoo.com] > > Sent: Tuesday, January 29, 2013 1:02 PM > > To: user@oozie.apache.org > > Subject: Re: Is it possible to use Oozie Java/Shell action to put a > > file into hdfs? > > > > Hi Jinye, > > Where is your source file (hdfs/local)? Would you please copy paste > > the exact/equivalent command that you want to execute? > > > > Regards, > > Mohammad > > > > > > ________________________________ > > From: Jinye Luo <j...@merkleinc.com> > > To: Mona Chitnis <chit...@yahoo-inc.com>; "user@oozie.apache.org" < > > user@oozie.apache.org>; Mohammad Islam <misla...@yahoo.com> > > Sent: Monday, January 28, 2013 9:31 PM > > Subject: RE: Is it possible to use Oozie Java/Shell action to put a > > file into hdfs? > > > > Mohammad/Mona, Thx for your reply. > > > > What I did is pretty much the same as what in the Cookbook, without > > having security enabled. > > > > I think/hope I found what went wrong. The Cookbook reads "Assume a > > local file $filename can be accessed by all cluster nodes." It is > > certainly not the case for me. In my clustered environment, oozie is > > running in an "edge node", which is the only one with access to > > (local) input files, and all other nodes, where M/R are running, are not > able to see this file. > > > > So I am out of luck with oozie, or there is a way to get around it? > > > > -----Original Message----- > > From: Mona Chitnis [mailto:chit...@yahoo-inc.com] > > Sent: Monday, January 28, 2013 12:37 PM > > To: user@oozie.apache.org; Mohammad Islam > > Subject: Re: Is it possible to use Oozie Java/Shell action to put a > > file into hdfs? > > > > Hello Jinye, > > > > In our Java Cookbook doc - > > https://cwiki.apache.org/confluence/display/OOZIE/Java+Cookbook > > > > Refer to section "Examples and Use Cases" #1. That should help. > > > > -- > > Mona > > > > On 1/26/13 12:06 AM, "Mohammad Islam" <misla...@yahoo.com> wrote: > > > > >Hi Jinye, > > >Are you using secure Hadoop with Kerberoes? What version? > > > > > >I think both ways (in OOzie 3.2+), you should be able to create a > > >file in HDFS. > > > > > >How did you try? and why they failed with what error ? > > > > > >Regards, > > >Mohammad > > > > > > > > >________________________________ > > > From: Jinye Luo <j...@merkleinc.com> > > >To: "user@oozie.apache.org" <user@oozie.apache.org> > > >Sent: Friday, January 25, 2013 2:10 PM > > >Subject: Is it possible to use Oozie Java/Shell action to put a file > > >into hdfs? > > > > > >I have tried a hundred different ways without success, any one has > > >better luck? If so, share a sample please. > > > > > >
