On Fri, May 9, 2008 at 12:00 PM, Hairong Kuang <[EMAIL PROTECTED]> wrote: >> I'm using the machine running the namenode to run maps as well. > Please do not run maps on the machine that is running the namenode. This > would cause CPU contention and slow down namenode. Thus more easily to see > SocketTimeoutException. > > Hairong
I've turned off running tasks on the master, and I'm not seeing those errors. The behavior was interesting. On one job, I saw a total of 11 timeout failures (where the map was reported as a failure), but all of them happened in the first few minutes. After that it worked well and completed correctly. I'm wondering if it's worth it, though. If the number of maps/reduces that the master machine can run is substantially greater than the number of failures due to timeouts, isn't it worth having the master run tasks? It seems like there's probably a point where the number of machines in the cluster makes having a separate master a requirement, but at 20 8-core machines, it's not clear that dedicating a box to being the master is a win. (And having a smaller machine dedicated to being the master is cheaper, but annoying. I'd rather have N identical boxes running the same AMI, etc.) To anyone using amazon - definitely upgrade to the new kernels. I now have have very few instances of the 'Exception in createBlockOutputStream' error that started this thread in my logs. (These are different than the 11 timeouts I mentioned above, FYI). The ones that are there all happened in one burst at 03:59:22 this afternoon: [EMAIL PROTECTED]:~/dev/hadoop$ bin/slaves.sh grep -r 'Exception in createBlockOutputStream' ~/dev/hadoop/logs/ domU-12-31-38-00-04-51.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000024_0/syslog:2008-05-09 03:59:22,713 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException domU-12-31-38-00-D6-21.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000048_0/syslog:2008-05-09 03:59:22,989 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.252.22.111:50010 domU-12-31-38-00-D6-21.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000061_0/syslog:2008-05-09 03:59:22,398 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException domU-12-31-38-00-60-D1.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000017_0/syslog:2008-05-09 03:59:22,880 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.252.217.203:50010 domU-12-31-38-00-CD-41.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000051_0/syslog:2008-05-09 03:59:23,012 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.252.34.31:50010 domU-12-31-38-00-D5-E1.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000026_0/syslog:2008-05-09 03:59:24,551 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.252.15.47:50010 domU-12-31-38-00-1D-D1.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000056_0/syslog:2008-05-09 03:59:23,504 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.252.11.159:50010 domU-12-31-38-00-1D-D1.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000050_0/syslog:2008-05-09 03:59:22,454 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException domU-12-31-38-00-1D-D1.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000009_0/syslog:2008-05-09 03:59:22,944 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException domU-12-31-38-00-D8-81.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000002_0/syslog:2008-05-09 03:59:22,420 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException domU-12-31-38-00-D8-81.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000072_0/syslog:2008-05-09 03:59:22,318 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException domU-12-31-38-00-08-C1.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000021_0/syslog:2008-05-09 03:59:24,150 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.252.22.111:50010 domU-12-31-38-00-C9-51.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000045_0/syslog:2008-05-09 03:59:24,470 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.252.22.111:50010 domU-12-31-38-00-C9-51.compute-1.internal: /home/james/dev/hadoop/logs/userlogs/task_200805082159_0043_r_000055_0/syslog:2008-05-09 03:59:21,588 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException -- James Moore | [EMAIL PROTECTED] blog.restphone.com
