[ http://issues.apache.org/jira/browse/HADOOP-141?page=comments#action_12375167 ]
Doug Cutting commented on HADOOP-141: ------------------------------------- It would be good to find which of these changes actually made the difference for you. Each TaskTracker's map output server only accepts up to mapred.tasktracker.tasks.maximum connections at once. Since this is typically around 2, I would be surprised if a small buffer size results in lots of seeks, since the OS should perform readaheads in its buffer cache. What OS are you using? What do you have mapred.tasktracker.tasks.maximum set to? If on linux, what does 'iostat -x 1 10' show when things are slow? How about 'sar -n DEV 1 10'? > Disk thrashing / task timeouts during map output copy phase > ----------------------------------------------------------- > > Key: HADOOP-141 > URL: http://issues.apache.org/jira/browse/HADOOP-141 > Project: Hadoop > Type: Bug > Components: mapred > Environment: linux > Reporter: paul sutter > > MapOutputProtocol connections cause timeouts because of system thrashing and > transferring the same file over and over again, ultimately leading to making > no forward progress(medium sized job, 500GB input file, map output about as > large as the input, 10 node cluster). > There are several bugs behind this, but the following two changes improved > matters considerably. > (1) > The buffersize in MapOutputFile is currently hardcoded to 8192 bytes (for > both reads and writes). By changing this buffer size to 256KB, the number of > disk seeks are reduced and the problem went away. > Ideally there would be a buffer size parameter for this that is separate from > the DFS io buffer size. > (2) > I also added the following code to the socket configuration in both > Server.java and Client.java. No linger is a minor good idea in an enivronment > with some packet loss (and you will have that when all the nodes get busy at > once), but 256KB buffers is probably excessive, especially on a LAN, but it > takes me two hours to test changes so I havent experimented. > socket.setSendBufferSize(256*1024); > socket.setReceiveBufferSize(256*1024); > socket.setSoLinger(false, 0); > socket.setKeepAlive(true); -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
