1000 concurrent read on a single file failing the task/client
---------------------------------------------------------------
Key: HADOOP-3859
URL: https://issues.apache.org/jira/browse/HADOOP-3859
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.17.1
Environment: 0.17.2
(0.17.1-H3002-H3633-H3681-H3685-H3370-H3707-H3760-H3758)
Reporter: Koji Noguchi
After fixing Hadoop-3633, some users started seeing their tasks fail with
{noformat}
08/07/29 05:13:07 INFO mapred.JobClient: Task Id :
task_200807290511_0001_m_000846_0, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7893038518783920880
file=/tmp/files111
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1430)
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1281)
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1385)
at java.io.DataInputStream.read(DataInputStream.java:83)
at
org.apache.hadoop.mapred.LineRecordReader$LineReader.backfill(LineRecordReader.java:88)
at
org.apache.hadoop.mapred.LineRecordReader$LineReader.readLine(LineRecordReader.java:114)
at
org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:179)
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:211)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2122)
{noformat}
This happened when hundreds of mappers pulled the same file concurrently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
