[ https://issues.apache.org/jira/browse/HADOOP-8045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204468#comment-13204468 ]
Harsh J commented on HADOOP-8045: --------------------------------- You should get them in the DN logs (count exceeded form of messages). Are they not present? > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs does not handle many > files well > -------------------------------------------------------------------------------------- > > Key: HADOOP-8045 > URL: https://issues.apache.org/jira/browse/HADOOP-8045 > Project: Hadoop Common > Issue Type: Bug > Components: io > Affects Versions: 0.21.0, 1.0.0 > Environment: Cloudera CH3 release. > Reporter: Tarjei Huse > Labels: patch > Attachments: hadoop-multiple-outputs.patch > > > We were tryong to use MultipleOutputs to write one file per key. This > produced the error: > exception: > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File > /user/me/part6/_temporary/ > _attempt_201202071305_0017_r_000000_2/2011-11-18-22- > attempt_201202071305_0017_r_000000_2-r-00000 > could only be replicated to 0 nodes, instead of 1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java: > 1520) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java: > 665) > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: > 25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java: > 1157) > When the nr. of files processed increased over 20 on a single developer > system. > The solution proved to be to close each RecordWriter when the reducer was > finished with a key, something that required that we extended the multiple > outputs to fetch the recordwriter - not a good solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira