...@netapp.com]
Sent: Monday, August 19, 2013 10:37 AM
To: hdfs-dev@hadoop.apache.org
Subject: RE: mapred replication
Thanks Chris and others for the detailed explanation.
I was aware of the basic rationale behind having a higher replication factor
for mapred job files - thanks taking the time to elaborate
for the NameNode to
keep a replica's metadata in the BlocksMap after it has already decided to
invalidate said replica?
Eric
-Original Message-
From: Robert Evans [mailto:ev...@yahoo-inc.com]
Sent: Monday, August 19, 2013 10:11 AM
To: hdfs-dev@hadoop.apache.org
Subject: Re: mapre
Without the stack trace of the exceptions it is hard to tell. The pruning
is asynchronous, but so is a node crashing with a replica on it. The
client is supposed to detect this situation and find a new replica that
works. I am not that familiar with the code, but I believe in some if not
all of
Why should this lead to an IOException? Is it because the pruning of
replicas is asynchronous and the datanodes try to access nonexistent
files? If so that seems like a pretty major bug
On Fri, Aug 16, 2013 at 5:21 PM, Chris Nauroth wrote:
> Hi Eric,
>
> Yes, this is intentional. The job.xml
Hi Eric,
Yes, this is intentional. The job.xml file and the job jar file get read
from every node running a map or reduce task. Because of this, using a
higher than normal replication factor on these files improves locality.
More than 3 task slots will have access to local replicas. These file
In debugging some replication issues in our HDFS environment, I noticed that
the MapReduce framework uses the following algorithm for setting the
replication on submitted job files:
1. Create the file with *default* DFS replication factor (i.e.
'dfs.replication')
2. Subsequently alter
