Hey, difficult to tell, can you grep through the tasklogs on each groom and look after the exception? I hope we get our new web interface up soon, so you don't have to search through the logs there.
Thanks! 2012/9/15 Yuesheng Hu <[email protected]> > Hi, > > I am testing k-means' performance on large data. I generated 5 bilion > vectors via mapreduce(about 140GB), and launched 45 tasks to handle the 45 > input files in the input directory. Every taks need to process about 3GB > data. But the job failed when the iteration ends. The following is log > message: > > attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO > zookeeper.ZooKeeper: Client environment:os.version=3.3.4-5.fc17.x86_64 > attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO > zookeeper.ZooKeeper: Client environment:user.name=hadoop > attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO > zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop > attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO > zookeeper.ZooKeeper: Client > > environment:user.dir=/tmp/hadoop-hadoop/bsp/local/groomServer/attempt_201209141233_0001_000001_0/work > attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO > zookeeper.ZooKeeper: Initiating client connection, connectString= > 192.168.1.200:21810 sessionTimeout=1200000 > watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl@76dfa2ae > attempt_201209141233_0001_000001_0: 12/09/14 12:35:41 INFO > zookeeper.ClientCnxn: Opening socket connection to server / > 192.168.1.200:21810 > attempt_201209141233_0001_000001_0: 12/09/14 12:35:41 INFO > zookeeper.ClientCnxn: Socket connection established to namenode/ > 192.168.1.200:21810, initiating session > attempt_201209141233_0001_000001_0: 12/09/14 12:35:41 INFO > sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At datanode01/ > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:35:41 INFO > zookeeper.ClientCnxn: Session establishment complete on server namenode/ > 192.168.1.200:21810, sessionid = 0x139c30dba450011, negotiated timeout = > 1200000 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x602cc296, /192.168.1.211:56957 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7ca7700a, /192.168.1.218:39111 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x5a716968, /192.168.1.217:46340 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7d083fe3, /192.168.1.218:39112 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x30a945f8, /192.168.1.216:56518 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x3930b5bd, /192.168.1.216:56519 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x2ceba8a0, /192.168.1.212:56842 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x238bb76a, /192.168.1.212:56843 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x602cc296, /192.168.1.211:56957 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x602cc296, /192.168.1.211:56957 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.211:56957 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x30a945f8, /192.168.1.216:56518 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x30a945f8, /192.168.1.216:56518 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.216:56518 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7ca7700a, /192.168.1.218:39111 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7ca7700a, /192.168.1.218:39111 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.218:39111 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x3930b5bd, /192.168.1.216:56519 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x3930b5bd, /192.168.1.216:56519 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.216:56519 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x5a716968, /192.168.1.217:46340 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x5a716968, /192.168.1.217:46340 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.217:46340 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x2ceba8a0, /192.168.1.212:56842 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x2ceba8a0, /192.168.1.212:56842 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.212:56842 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x626bb08a, /192.168.1.219:50114 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x0c136681, /192.168.1.219:50115 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x626bb08a, /192.168.1.219:50114 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x626bb08a, /192.168.1.219:50114 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.219:50114 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x0c136681, /192.168.1.219:50115 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x0c136681, /192.168.1.219:50115 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.219:50115 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x74a77044, /192.168.1.215:35165 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1cbbbc31, /192.168.1.215:35166 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x694043b3, /192.168.1.217:46341 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7a1cd2c0, /192.168.1.214:39405 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x694043b3, /192.168.1.217:46341 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x694043b3, /192.168.1.217:46341 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.217:46341 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7a1cd2c0, /192.168.1.214:39405 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7a1cd2c0, /192.168.1.214:39405 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.214:39405 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x638c7ceb, /192.168.1.216:56520 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x04a7b933, /192.168.1.214:39406 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1f6fc5a8, /192.168.1.212:56844 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x6319a8b7, /192.168.1.216:56521 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x6319a8b7, /192.168.1.216:56521 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x6319a8b7, /192.168.1.216:56521 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.216:56521 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7d083fe3, /192.168.1.218:39112 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7d083fe3, /192.168.1.218:39112 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.218:39112 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x238bb76a, /192.168.1.212:56843 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x238bb76a, /192.168.1.212:56843 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.212:56843 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1cbbbc31, /192.168.1.215:35166 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1cbbbc31, /192.168.1.215:35166 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.215:35166 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x04a7b933, /192.168.1.214:39406 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x04a7b933, /192.168.1.214:39406 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.214:39406 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x21645be6, /192.168.1.213:50616 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x0c7b7e61, /192.168.1.218:39113 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x0c7b7e61, /192.168.1.218:39113 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x0c7b7e61, /192.168.1.218:39113 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.218:39113 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7fa618ad, /192.168.1.217:46342 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x4e5cd46b, /192.168.1.214:39407 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x65371c10, /192.168.1.211:56958 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7f7d0638, /192.168.1.218:39114 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1bb08b3a, /192.168.1.213:50617 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x63849b09, /192.168.1.213:50618 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x16de9eea, /192.168.1.214:39408 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x73b034c8, /192.168.1.218:39115 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x44d5ce53, /192.168.1.217:46343 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x30dcce73, /192.168.1.219:50116 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x6dbf1737, /192.168.1.219:50117 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1c8d542b, /192.168.1.212:56845 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x0290e1b2, /192.168.1.216:56522 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x75d16e6e, /192.168.1.219:50118 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x669b0d36, /192.168.1.215:35167 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x2b5cf2a8, /192.168.1.217:46344 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x633f09d5, /192.168.1.212:56846 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO > ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO > ipc.NettyTransceiver: [id: 0x32e796ac] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x02551050, /192.168.1.215:35168 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x75304935, /192.168.1.215:35169 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x351672ae, /192.168.1.211:56959 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1ca2d8f6, /192.168.1.213:50619 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x6bee8942, /192.168.1.211:56960 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x3f801e2c, /192.168.1.213:50620 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x743cfb45, /192.168.1.211:56961 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x69b24aa6, /192.168.1.214:39409 => /192.168.1.211:61002] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO > ipc.NettyTransceiver: [id: 0x32e796ac, /192.168.1.211:56960 => datanode01/ > 192.168.1.211:61002] BOUND: /192.168.1.211:56960 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO > ipc.NettyTransceiver: [id: 0x32e796ac, /192.168.1.211:56960 => datanode01/ > 192.168.1.211:61002] CONNECTED: datanode01/192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x74a77044, /192.168.1.215:35165 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x74a77044, /192.168.1.215:35165 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.215:35165 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x638c7ceb, /192.168.1.216:56520 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x638c7ceb, /192.168.1.216:56520 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.216:56520 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7f7d0638, /192.168.1.218:39114 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7f7d0638, /192.168.1.218:39114 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.218:39114 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x73b034c8, /192.168.1.218:39115 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x73b034c8, /192.168.1.218:39115 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.218:39115 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1c8d542b, /192.168.1.212:56845 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1c8d542b, /192.168.1.212:56845 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.212:56845 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x2b5cf2a8, /192.168.1.217:46344 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x2b5cf2a8, /192.168.1.217:46344 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.217:46344 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x351672ae, /192.168.1.211:56959 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x351672ae, /192.168.1.211:56959 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.211:56959 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x743cfb45, /192.168.1.211:56961 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x743cfb45, /192.168.1.211:56961 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.211:56961 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1f6fc5a8, /192.168.1.212:56844 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x4e5cd46b, /192.168.1.214:39407 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1f6fc5a8, /192.168.1.212:56844 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.212:56844 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7fa618ad, /192.168.1.217:46342 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x4e5cd46b, /192.168.1.214:39407 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.214:39407 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x7fa618ad, /192.168.1.217:46342 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.217:46342 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x63849b09, /192.168.1.213:50618 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1bb08b3a, /192.168.1.213:50617 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1bb08b3a, /192.168.1.213:50617 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.213:50617 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x44d5ce53, /192.168.1.217:46343 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x63849b09, /192.168.1.213:50618 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.213:50618 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x44d5ce53, /192.168.1.217:46343 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.217:46343 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x30dcce73, /192.168.1.219:50116 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x0290e1b2, /192.168.1.216:56522 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x0290e1b2, /192.168.1.216:56522 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.216:56522 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x633f09d5, /192.168.1.212:56846 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x30dcce73, /192.168.1.219:50116 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.219:50116 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x75d16e6e, /192.168.1.219:50118 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x75d16e6e, /192.168.1.219:50118 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.219:50118 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x02551050, /192.168.1.215:35168 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x02551050, /192.168.1.215:35168 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.215:35168 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x6bee8942, /192.168.1.211:56960 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x6bee8942, /192.168.1.211:56960 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.211:56960 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x633f09d5, /192.168.1.212:56846 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.212:56846 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1ca2d8f6, /192.168.1.213:50619 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x1ca2d8f6, /192.168.1.213:50619 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.213:50619 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x69b24aa6, /192.168.1.214:39409 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x69b24aa6, /192.168.1.214:39409 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.214:39409 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x21645be6, /192.168.1.213:50616 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x21645be6, /192.168.1.213:50616 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.213:50616 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x65371c10, /192.168.1.211:56958 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x65371c10, /192.168.1.211:56958 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.211:56958 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x16de9eea, /192.168.1.214:39408 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x16de9eea, /192.168.1.214:39408 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.214:39408 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x6dbf1737, /192.168.1.219:50117 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x6dbf1737, /192.168.1.219:50117 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.219:50117 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x669b0d36, /192.168.1.215:35167 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x669b0d36, /192.168.1.215:35167 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.215:35167 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x75304935, /192.168.1.215:35169 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x75304935, /192.168.1.215:35169 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.215:35169 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x3f801e2c, /192.168.1.213:50620 => /192.168.1.211:61002] BOUND: / > 192.168.1.211:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer: > [id: 0x3f801e2c, /192.168.1.213:50620 => /192.168.1.211:61002] CONNECTED: > / > 192.168.1.213:50620 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO > ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO > ipc.NettyTransceiver: [id: 0x693e6031] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO > ipc.NettyTransceiver: [id: 0x693e6031, /192.168.1.211:38151 => datanode02/ > 192.168.1.212:61001] BOUND: /192.168.1.211:38151 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO > ipc.NettyTransceiver: [id: 0x693e6031, /192.168.1.211:38151 => datanode02/ > 192.168.1.212:61001] CONNECTED: datanode02/192.168.1.212:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x4d506e73] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x4d506e73, /192.168.1.211:53194 => datanode01/ > 192.168.1.211:61001] BOUND: /192.168.1.211:53194 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x4d506e73, /192.168.1.211:53194 => datanode01/ > 192.168.1.211:61001] CONNECTED: datanode01/192.168.1.211:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x122e8a3c] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x122e8a3c, /192.168.1.211:40394 => datanode02/ > 192.168.1.212:61003] BOUND: /192.168.1.211:40394 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x122e8a3c, /192.168.1.211:40394 => datanode02/ > 192.168.1.212:61003] CONNECTED: datanode02/192.168.1.212:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x6eb54678] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x6eb54678, /192.168.1.211:56565 => datanode03/ > 192.168.1.213:61002] BOUND: /192.168.1.211:56565 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x6eb54678, /192.168.1.211:56565 => datanode03/ > 192.168.1.213:61002] CONNECTED: datanode03/192.168.1.213:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x48b565d2] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x48b565d2, /192.168.1.211:53664 => datanode01/ > 192.168.1.211:61004] BOUND: /192.168.1.211:53664 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x48b565d2, /192.168.1.211:53664 => datanode01/ > 192.168.1.211:61004] CONNECTED: datanode01/192.168.1.211:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x1b7d8bfa] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x1b7d8bfa, /192.168.1.211:36872 => datanode04/ > 192.168.1.214:61001] BOUND: /192.168.1.211:36872 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x1b7d8bfa, /192.168.1.211:36872 => datanode04/ > 192.168.1.214:61001] CONNECTED: datanode04/192.168.1.214:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x41424f86] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x41424f86, /192.168.1.211:39778 => datanode02/ > 192.168.1.212:61002] BOUND: /192.168.1.211:39778 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x41424f86, /192.168.1.211:39778 => datanode02/ > 192.168.1.212:61002] CONNECTED: datanode02/192.168.1.212:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x53b252a4] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x53b252a4, /192.168.1.211:45705 => datanode03/ > 192.168.1.213:61001] BOUND: /192.168.1.211:45705 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x53b252a4, /192.168.1.211:45705 => datanode03/ > 192.168.1.213:61001] CONNECTED: datanode03/192.168.1.213:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x0be83fed] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x0be83fed, /192.168.1.211:38537 => datanode01/ > 192.168.1.211:61003] BOUND: /192.168.1.211:38537 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x0be83fed, /192.168.1.211:38537 => datanode01/ > 192.168.1.211:61003] CONNECTED: datanode01/192.168.1.211:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x45697ab0] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x45697ab0, /192.168.1.211:39655 => datanode07/ > 192.168.1.217:61001] BOUND: /192.168.1.211:39655 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x45697ab0, /192.168.1.211:39655 => datanode07/ > 192.168.1.217:61001] CONNECTED: datanode07/192.168.1.217:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x2f555d7c] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x2f555d7c, /192.168.1.211:59371 => datanode06/ > 192.168.1.216:61002] BOUND: /192.168.1.211:59371 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x2f555d7c, /192.168.1.211:59371 => datanode06/ > 192.168.1.216:61002] CONNECTED: datanode06/192.168.1.216:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x78634aff] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x78634aff, /192.168.1.211:53018 => datanode03/ > 192.168.1.213:61005] BOUND: /192.168.1.211:53018 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x78634aff, /192.168.1.211:53018 => datanode03/ > 192.168.1.213:61005] CONNECTED: datanode03/192.168.1.213:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x79b35d3a] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x79b35d3a, /192.168.1.211:58515 => datanode04/ > 192.168.1.214:61004] BOUND: /192.168.1.211:58515 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x79b35d3a, /192.168.1.211:58515 => datanode04/ > 192.168.1.214:61004] CONNECTED: datanode04/192.168.1.214:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x3c90cbce] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x3c90cbce, /192.168.1.211:37935 => datanode05/ > 192.168.1.215:61003] BOUND: /192.168.1.211:37935 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x3c90cbce, /192.168.1.211:37935 => datanode05/ > 192.168.1.215:61003] CONNECTED: datanode05/192.168.1.215:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x2c1768fe] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x2c1768fe, /192.168.1.211:60504 => datanode08/ > 192.168.1.218:61001] BOUND: /192.168.1.211:60504 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x2c1768fe, /192.168.1.211:60504 => datanode08/ > 192.168.1.218:61001] CONNECTED: datanode08/192.168.1.218:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x240e7423] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x240e7423, /192.168.1.211:39924 => datanode07/ > 192.168.1.217:61002] BOUND: /192.168.1.211:39924 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x240e7423, /192.168.1.211:39924 => datanode07/ > 192.168.1.217:61002] CONNECTED: datanode07/192.168.1.217:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x75302bb6] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x75302bb6, /192.168.1.211:47363 => datanode06/ > 192.168.1.216:61003] BOUND: /192.168.1.211:47363 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x75302bb6, /192.168.1.211:47363 => datanode06/ > 192.168.1.216:61003] CONNECTED: datanode06/192.168.1.216:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x6392c99c] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x6392c99c, /192.168.1.211:50035 => datanode04/ > 192.168.1.214:61005] BOUND: /192.168.1.211:50035 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x6392c99c, /192.168.1.211:50035 => datanode04/ > 192.168.1.214:61005] CONNECTED: datanode04/192.168.1.214:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x798623a3] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x798623a3, /192.168.1.211:53262 => datanode05/ > 192.168.1.215:61004] BOUND: /192.168.1.211:53262 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x798623a3, /192.168.1.211:53262 => datanode05/ > 192.168.1.215:61004] CONNECTED: datanode05/192.168.1.215:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x13b71ee8] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x13b71ee8, /192.168.1.211:57211 => datanode03/ > 192.168.1.213:61003] BOUND: /192.168.1.211:57211 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x13b71ee8, /192.168.1.211:57211 => datanode03/ > 192.168.1.213:61003] CONNECTED: datanode03/192.168.1.213:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO > ipc.NettyTransceiver: [id: 0x4f08370b] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x4f08370b, /192.168.1.211:41614 => datanode01/ > 192.168.1.211:61005] BOUND: /192.168.1.211:41614 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x4f08370b, /192.168.1.211:41614 => datanode01/ > 192.168.1.211:61005] CONNECTED: datanode01/192.168.1.211:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x57e13166] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x57e13166, /192.168.1.211:48980 => datanode02/ > 192.168.1.212:61004] BOUND: /192.168.1.211:48980 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x57e13166, /192.168.1.211:48980 => datanode02/ > 192.168.1.212:61004] CONNECTED: datanode02/192.168.1.212:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x1aa0c35b] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x1aa0c35b, /192.168.1.211:56959 => datanode04/ > 192.168.1.214:61002] BOUND: /192.168.1.211:56959 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x1aa0c35b, /192.168.1.211:56959 => datanode04/ > 192.168.1.214:61002] CONNECTED: datanode04/192.168.1.214:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x484eff79] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x484eff79, /192.168.1.211:50664 => datanode05/ > 192.168.1.215:61001] BOUND: /192.168.1.211:50664 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x484eff79, /192.168.1.211:50664 => datanode05/ > 192.168.1.215:61001] CONNECTED: datanode05/192.168.1.215:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x10bd314a] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x10bd314a, /192.168.1.211:40716 => datanode06/ > 192.168.1.216:61001] BOUND: /192.168.1.211:40716 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x10bd314a, /192.168.1.211:40716 => datanode06/ > 192.168.1.216:61001] CONNECTED: datanode06/192.168.1.216:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x5b12bba0] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x5b12bba0, /192.168.1.211:50285 => datanode02/ > 192.168.1.212:61005] BOUND: /192.168.1.211:50285 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x5b12bba0, /192.168.1.211:50285 => datanode02/ > 192.168.1.212:61005] CONNECTED: datanode02/192.168.1.212:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x7659657a] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x7659657a, /192.168.1.211:44243 => datanode03/ > 192.168.1.213:61004] BOUND: /192.168.1.211:44243 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x7659657a, /192.168.1.211:44243 => datanode03/ > 192.168.1.213:61004] CONNECTED: datanode03/192.168.1.213:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x64fed7fa] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x64fed7fa, /192.168.1.211:37357 => datanode04/ > 192.168.1.214:61003] BOUND: /192.168.1.211:37357 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x64fed7fa, /192.168.1.211:37357 => datanode04/ > 192.168.1.214:61003] CONNECTED: datanode04/192.168.1.214:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x500dffbf] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x500dffbf, /192.168.1.211:36378 => datanode05/ > 192.168.1.215:61002] BOUND: /192.168.1.211:36378 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x500dffbf, /192.168.1.211:36378 => datanode05/ > 192.168.1.215:61002] CONNECTED: datanode05/192.168.1.215:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x4861c19c] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x4861c19c, /192.168.1.211:47948 => datanode09/ > 192.168.1.219:61003] BOUND: /192.168.1.211:47948 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x4861c19c, /192.168.1.211:47948 => datanode09/ > 192.168.1.219:61003] CONNECTED: datanode09/192.168.1.219:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x3a67ba87] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x3a67ba87, /192.168.1.211:52403 => datanode08/ > 192.168.1.218:61004] BOUND: /192.168.1.211:52403 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x3a67ba87, /192.168.1.211:52403 => datanode08/ > 192.168.1.218:61004] CONNECTED: datanode08/192.168.1.218:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x389f2c6e] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x389f2c6e, /192.168.1.211:43475 => datanode07/ > 192.168.1.217:61005] BOUND: /192.168.1.211:43475 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x389f2c6e, /192.168.1.211:43475 => datanode07/ > 192.168.1.217:61005] CONNECTED: datanode07/192.168.1.217:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x6786091b] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x6786091b, /192.168.1.211:53587 => datanode09/ > 192.168.1.219:61004] BOUND: /192.168.1.211:53587 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x6786091b, /192.168.1.211:53587 => datanode09/ > 192.168.1.219:61004] CONNECTED: datanode09/192.168.1.219:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x02cb7647] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x02cb7647, /192.168.1.211:47875 => datanode08/ > 192.168.1.218:61005] BOUND: /192.168.1.211:47875 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x02cb7647, /192.168.1.211:47875 => datanode08/ > 192.168.1.218:61005] CONNECTED: datanode08/192.168.1.218:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x4158fa62] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x4158fa62, /192.168.1.211:38157 => datanode09/ > 192.168.1.219:61001] BOUND: /192.168.1.211:38157 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x4158fa62, /192.168.1.211:38157 => datanode09/ > 192.168.1.219:61001] CONNECTED: datanode09/192.168.1.219:61001 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x3ed62895] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x3ed62895, /192.168.1.211:60073 => datanode08/ > 192.168.1.218:61002] BOUND: /192.168.1.211:60073 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x3ed62895, /192.168.1.211:60073 => datanode08/ > 192.168.1.218:61002] CONNECTED: datanode08/192.168.1.218:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x12977ada] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x12977ada, /192.168.1.211:48821 => datanode07/ > 192.168.1.217:61003] BOUND: /192.168.1.211:48821 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x12977ada, /192.168.1.211:48821 => datanode07/ > 192.168.1.217:61003] CONNECTED: datanode07/192.168.1.217:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x42e95cbf] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x42e95cbf, /192.168.1.211:36170 => datanode06/ > 192.168.1.216:61004] BOUND: /192.168.1.211:36170 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x42e95cbf, /192.168.1.211:36170 => datanode06/ > 192.168.1.216:61004] CONNECTED: datanode06/192.168.1.216:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x37e66f47] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x37e66f47, /192.168.1.211:43133 => datanode05/ > 192.168.1.215:61005] BOUND: /192.168.1.211:43133 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x37e66f47, /192.168.1.211:43133 => datanode05/ > 192.168.1.215:61005] CONNECTED: datanode05/192.168.1.215:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x69bf5fb1] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x69bf5fb1, /192.168.1.211:39261 => datanode09/ > 192.168.1.219:61002] BOUND: /192.168.1.211:39261 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x69bf5fb1, /192.168.1.211:39261 => datanode09/ > 192.168.1.219:61002] CONNECTED: datanode09/192.168.1.219:61002 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x36d0c62b] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x36d0c62b, /192.168.1.211:55267 => datanode08/ > 192.168.1.218:61003] BOUND: /192.168.1.211:55267 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x36d0c62b, /192.168.1.211:55267 => datanode08/ > 192.168.1.218:61003] CONNECTED: datanode08/192.168.1.218:61003 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x1bac8b63] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x1bac8b63, /192.168.1.211:48745 => datanode07/ > 192.168.1.217:61004] BOUND: /192.168.1.211:48745 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x1bac8b63, /192.168.1.211:48745 => datanode07/ > 192.168.1.217:61004] CONNECTED: datanode07/192.168.1.217:61004 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x1e59a7ba] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x1e59a7ba, /192.168.1.211:39669 => datanode06/ > 192.168.1.216:61005] BOUND: /192.168.1.211:39669 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO > ipc.NettyTransceiver: [id: 0x1e59a7ba, /192.168.1.211:39669 => datanode06/ > 192.168.1.216:61005] CONNECTED: datanode06/192.168.1.216:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:04 INFO > ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61005 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:04 INFO > ipc.NettyTransceiver: [id: 0x6bbccfbc] OPEN > attempt_201209141233_0001_000001_0: 12/09/14 12:49:04 INFO > ipc.NettyTransceiver: [id: 0x6bbccfbc, /192.168.1.211:45969 => datanode09/ > 192.168.1.219:61005] BOUND: /192.168.1.211:45969 > attempt_201209141233_0001_000001_0: 12/09/14 12:49:04 INFO > ipc.NettyTransceiver: [id: 0x6bbccfbc, /192.168.1.211:45969 => datanode09/ > 192.168.1.219:61005] CONNECTED: datanode09/192.168.1.219:61005 > attempt_201209141233_0001_000001_0: 12/09/14 15:08:29 INFO > clustering.KMeansBSPV02: Finished! Writing the assignments... > 12/09/14 15:09:44 INFO bsp.BSPJobClient: Job failed. > > All about zookeeper. I don't what happend? Any suggestions? > > btw, our cluster has 9 nodes (RAM: 4GB/node). I configured 5 tasks per > node, 512MB memory per task. > > Regards, > Hu >
