Hi Hdfs shell command standard I/O is supported. I think if you using it for avoiding temporary save to local file system.
For example: wget https://web-server/file-path -O - | hadoop fs -put - hdfs://nn.example.com/hadoop/hadoopfile Refer to this URL http://hadoop.apache.org/common/docs/current/hdfs_shell.html#put Hope that helps zhuweimin > -----Original Message----- > From: zenMonkey [mailto:numan.sal...@gmail.com] > Sent: Sunday, March 07, 2010 4:23 AM > To: hadoop-u...@lucene.apache.org > Subject: Copying files between two remote hadoop clusters > > > I want to write a script that pulls data (flat files) from a remote machine > and pushes that into its hadoop cluster. > > At the moment, it is done in two steps: > > 1 - Secure copy the remote files > 2 - Put the files into HDFS > > I was wondering if it was possible to optimize this by avoiding copying > to > local fs before pushing to hdfs; and instead write directly to hdfs. I am > not sure if this is something that hadoop tools already provide. > > Thanks for any help. > > -- > View this message in context: > http://old.nabble.com/Copying-files-between-two-remote-hadoop-clusters > -tp27799963p27799963.html > Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
