Did you run the copy command from machine A? I believe that if you do the copy 
from an hdfs client that is on the same machine as a data node, then for each 
block the primary copy always goes to that data node, and only the replicas get 
distributed among other data nodes. I ran into this issue once -- I had to have 
the client doing the copy either on the master or on an "off-cluster" node.
-TCK



--- On Thu, 2/12/09, Saptarshi Guha <[email protected]> wrote:
From: Saptarshi Guha <[email protected]>
Subject: Very large file copied to cluster, and the copy fails. All blocks bad
To: "[email protected]" <[email protected]>
Date: Thursday, February 12, 2009, 9:50 PM

hello,
I have a 42 GB file on the local fs(call the machine A)  which i need
to copy to a HDFS (replicattion 1), according the HDFS webtracker it
has 208GB across 7 machines.
Note, the machine A has about 80 GB total, so there is no place to
store copies of the file.
Using the command bin/hadoop dfs -put /local/x /remote/tmp/ fails,
with all blocks being bad. This is not surprising since the file is
copied entirely to the HDFS region that resides on A. Had the file
been copied across all machines, this would not have failed.

I have more experience with mapreduce and not much with the hdfs side
of things.
Is there a configuration option i'm missing that forces the file to be
split across the machines(when it is being copied)?
-- 
Saptarshi Guha - [email protected]



      

Reply via email to