Jinye Luo, I'm afraid it is not possible to do what you want to do in the way you want to do it.
Another approach would be that you have a java action that starts a connection to your data source and writes it to HDFS. For example, if your datasource is exposed with an HTTP server this would be quite easy (read from HTTP stream, write to HDFS stream). Thx On Tue, Jan 29, 2013 at 1:50 PM, Harish Krishnan < harish.t.krish...@gmail.com> wrote: > If your hdfs path is visible to master, and since oozie is running in your > master, if you add the path to "oozie.libpath" wont it work? > > Thanks & Regards, > Harish.T.K > > > On Tue, Jan 29, 2013 at 12:15 PM, Jinye Luo <j...@merkleinc.com> wrote: > > > Robert, I guess I wasn't 100% clear on what I was trying to do in my > > previous message. > > > > I am looking for something in Oozie to perform equivalent to "hadoop fs > > -copyFromLocal /home/cloudera/input.log /user/cloudera/input". > > > > As you said, "//user/cloudera/input/input.log" is visible to all nodes if > > "hadoop fs..." is successful, but /home/cloudera/input.log is not, and > that > > is my problem. > > > > -----Original Message----- > > From: Robert Kanter [mailto:rkan...@cloudera.com] > > Sent: Tuesday, January 29, 2013 2:06 PM > > To: user@oozie.apache.org > > Cc: Mohammad Islam > > Subject: Re: Is it possible to use Oozie Java/Shell action to put a file > > into hdfs? > > > > Hi Jinye, > > > > When you run that hadoop fs command, you're copying input.log from your > > local FS to HDFS, so it should be visible on all of your nodes. > > > > Are you sure that you're pointing the Oozie action to the right place? > > When you run hadoop fs -copyFromLocal /home/cloudera/input.log > > /user/cloudera/input I believe you are actually uploading the file to > > /user/cloudera/input/input.log because you didn't specify a file > extension > > in the destination path; did you mean to upload it to > > /user/cloudera/input.log? If so, try running hadoop fs -copyFromLocal > > /home/cloudera/input.log /user/cloudera/input.log > > > > - Robert > > > > > > On Tue, Jan 29, 2013 at 11:23 AM, Jinye Luo <j...@merkleinc.com> wrote: > > > > > Let's say I have a four node cluster, one master and three slaves. > > > > > > The master is running: > > > > > > hadoop-yarn-resourcemanager > > > hadoop-hdfs-namenode > > > hadoop--mapreduce-historyserver > > > zookeeper-server > > > hbase-master > > > oozie > > > > > > > > > The slaves are running: > > > hadoop-yarn-nodemanager > > > hadoop-hdfs-datanode > > > hbase-regionserver > > > > > > From the master, I run the following to put a file into hdfs, then > > > kick off a M/R job: > > > hadoop fs -copyFromLocal /home/cloudera/input.log > > > /user/cloudera/input > > > > > > hadoop jar log-processing.jar ... > > > > > > Obviously, input.log is not visible to slaves, and Oozie action is > > > always executed in slaves, hence the Oozie action will always fail. > > > > > > -----Original Message----- > > > From: Mohammad Islam [mailto:misla...@yahoo.com] > > > Sent: Tuesday, January 29, 2013 1:02 PM > > > To: user@oozie.apache.org > > > Subject: Re: Is it possible to use Oozie Java/Shell action to put a > > > file into hdfs? > > > > > > Hi Jinye, > > > Where is your source file (hdfs/local)? Would you please copy paste > > > the exact/equivalent command that you want to execute? > > > > > > Regards, > > > Mohammad > > > > > > > > > ________________________________ > > > From: Jinye Luo <j...@merkleinc.com> > > > To: Mona Chitnis <chit...@yahoo-inc.com>; "user@oozie.apache.org" < > > > user@oozie.apache.org>; Mohammad Islam <misla...@yahoo.com> > > > Sent: Monday, January 28, 2013 9:31 PM > > > Subject: RE: Is it possible to use Oozie Java/Shell action to put a > > > file into hdfs? > > > > > > Mohammad/Mona, Thx for your reply. > > > > > > What I did is pretty much the same as what in the Cookbook, without > > > having security enabled. > > > > > > I think/hope I found what went wrong. The Cookbook reads "Assume a > > > local file $filename can be accessed by all cluster nodes." It is > > > certainly not the case for me. In my clustered environment, oozie is > > > running in an "edge node", which is the only one with access to > > > (local) input files, and all other nodes, where M/R are running, are > not > > able to see this file. > > > > > > So I am out of luck with oozie, or there is a way to get around it? > > > > > > -----Original Message----- > > > From: Mona Chitnis [mailto:chit...@yahoo-inc.com] > > > Sent: Monday, January 28, 2013 12:37 PM > > > To: user@oozie.apache.org; Mohammad Islam > > > Subject: Re: Is it possible to use Oozie Java/Shell action to put a > > > file into hdfs? > > > > > > Hello Jinye, > > > > > > In our Java Cookbook doc - > > > https://cwiki.apache.org/confluence/display/OOZIE/Java+Cookbook > > > > > > Refer to section "Examples and Use Cases" #1. That should help. > > > > > > -- > > > Mona > > > > > > On 1/26/13 12:06 AM, "Mohammad Islam" <misla...@yahoo.com> wrote: > > > > > > >Hi Jinye, > > > >Are you using secure Hadoop with Kerberoes? What version? > > > > > > > >I think both ways (in OOzie 3.2+), you should be able to create a > > > >file in HDFS. > > > > > > > >How did you try? and why they failed with what error ? > > > > > > > >Regards, > > > >Mohammad > > > > > > > > > > > >________________________________ > > > > From: Jinye Luo <j...@merkleinc.com> > > > >To: "user@oozie.apache.org" <user@oozie.apache.org> > > > >Sent: Friday, January 25, 2013 2:10 PM > > > >Subject: Is it possible to use Oozie Java/Shell action to put a file > > > >into hdfs? > > > > > > > >I have tried a hundred different ways without success, any one has > > > >better luck? If so, share a sample please. > > > > > > > > > > > -- Alejandro
