[
https://issues.apache.org/jira/browse/HADOOP-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190006#comment-13190006
]
Koji Noguchi commented on HADOOP-7973:
--------------------------------------
bq. Add tests.
Looking at HADOOP-7973-4.patch, does it reproduce the issue we are seeing?
In my environment, this test succeeds without your change. I guess you would
need minidfs cluster instead of running miniMR on top of local filesystem?
Also, for the actual change, is this sufficient to avoid the close failure? I
thought that FileSystem throwing the actual error is not from the MapTask
directly but the one opened inside TextInputFormat->LineRecordReader
{noformat}
2012-01-13 00:08:09,268 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:241)
at org.apache.hadoop.hdfs.DFSClient.access$800(DFSClient.java:74)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:1959)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at
org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:168)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:254)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:440)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1082)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
{noformat}
bq. If the MR tasks opens the input/output streams ...
Does this apply to output stream as well? If it really does, then something
needs to be changed on the reducer side as well? (I couldn't confirm this.)
bq. I'm attempting to only restore 204 behavior with this jira.
Just to make sure everyone is one the same page,
the close issue itself has been there even before 0.20.205/1.0.
With input/output paths without any 'hdfs://...' set, "Filesystem closed"
exception above happens on any hadoop 0.20 versions.
||version|| fs.default.name ||
|before 0.20.205| hdfs://hostname.domain (without port) |
|on 0.20.205 | hdfs://hostname.domain:8020 (with port) |
> DistributedFileSystem close has severe consequences
> ---------------------------------------------------
>
> Key: HADOOP-7973
> URL: https://issues.apache.org/jira/browse/HADOOP-7973
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 1.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Blocker
> Attachments: HADOOP-7973-2.patch, HADOOP-7973-3.patch,
> HADOOP-7973-4.patch, HADOOP-7973.patch
>
>
> The way {{FileSystem#close}} works is very problematic. Since the
> {{FileSystems}} are cached, any {{close}} by any caller will cause problems
> for every other reference to it. Will add more detail in the comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
