Jayaseelan, Am not sure if your NFS can take that much load, but do try with a smaller dataset to see if it works. You'll need the destination path available on all the MR nodes for distcp to succeed -- cause its essentially going to try and write from each of a map task (distcp runs a job).
And, copyToLocal will work for directories too - its not just for files. On Wed, Dec 21, 2011 at 3:31 PM, Jayaseelan E <jayaseela...@ericsson.com> wrote: > Hi Harsh > basically i have some 40 folders each folder size is approximately 3GB .To > move tis folder from HDFS to local location.I used the distcp.. > Iam not copying files, but iam copying the whole folders which contains > folders and files. > So i cannot use the copyToLocal command. > May be i will try this distcp with NFS location > > thanks indeed > jayaseelan > > ________________________________ > From: Harsh J [mailto:ha...@cloudera.com] > Sent: Wednesday, December 21, 2011 3:26 PM > To: hdfs-user@hadoop.apache.org > Subject: Re: REG:error details(temporary directory does not exist) > > Hey Jayaseelan, > > You can't distcp to a file:// destination URI, unless that location is a NFS > point commonly available across all your nodes. > > What you probably need for 3 GB worth of data is a simple "hadoop fs > -copyToLocal src dest" command. > > On 21-Dec-2011, at 12:20 PM, Jayaseelan E wrote: > > > Hi Folks! > I am have some 40 folders whose size of each folders are aproximately 3GB.I > used distcp command to take a copy from HDFS to local machine. > But it fails with the following error. > /opt/hadoop/bin/hadoop distcp hdfs://bl460x1890:9000/<<Source > location>> file:///<<targetpath>> > > java.io.IOException: The temporary job-output > directory file:/archive/_distcp_logs_ndyr8m/_temporarydoesn't exist! > at > org.apache.hadoop.mapred.FileOutputCommitter.getWorkPath(FileOutputCommitter.java:204) > at > org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:234) > > My hadoop version is 2.0.1 > If any one can please help me resolve this > > Thanks indeed > jayaseelan > > > -- Harsh J
