Looking my task logs there is a big gap in time I do not understand. The task connects to zookeeper creates the entries and from: 2012-10-24 12:25:24 to 2012-10-24 13:08:03 logs nothing. Doing map reduce I guess.
2012-10-24 12:25:23,323 INFO org.apache.zookeeper.ClientCnxn: Sessionestablishment complete on server 2012-10-24 12:25:24,266 INFO org.apache.hadoop.hbase.mapreduce.TableOutputFormat: Created table instance for conf2_events 2012-10-24 12:25:24,361 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2012-10-24 12:25:24,461 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin org.apache.hadoop.util.LinuxResourceCalculatorPlugin@13394344 2012-10-24 12:25:24,615 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy Snappy native library not loaded 2012-10-24 13:08:03,738 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x13a91f1e41000c0 2012-10-24 13:08:03,751 INFO org.apache.zookeeper.ZooKeeper: Session:0x13a91f1e41000c0 closed 2012-10-24 13:08:03,751 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-10-24 13:08:03,751 INFO org.apache.hadoop.mapred.Task: Task:attempt_201210241044_0005_m_000000_0 is done. And is in the process of commiting Map reduce side the job is being run 2012-10-24 12:25:19,212 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_.. given task: attempt_201210241044_0005_m_000002_0 2012-10-24 12:25:19,308 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_.. given task: attempt_201210241044_0005_m_000012_0 2012-10-24 12:25:19,347 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_.. given task: attempt_201210241044_0005_m_000003_0 2012-10-24 12:25:19,510 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_.. given task: attempt_201210241044_0005_m_000010_0 2012-10-24 12:25:19,525 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_899418193 given task: attempt_201210241044_0005_m_000007_0 2012-10-24 12:25:19,526 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_-1509383641 given task: attempt_201210241044_0005_m_000001_0 2012-10-24 12:25:19,708 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_-19778997 given task: attempt_201210241044_0005_m_000004_0 2012-10-24 12:25:19,822 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_-4189743 given task: attempt_201210241044_0005_m_000009_0 2012-10-24 12:25:19,980 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_-661677671 given task: attempt_201210241044_0005_m_000005_0 2012-10-24 12:25:20,044 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_1898916331 given task: attempt_201210241044_0005_m_000000_0 2012-10-24 12:25:20,167 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_1123667416 given task: attempt_201210241044_0005_m_000008_0 2012-10-24 12:25:20,392 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_1621934208 given task: attempt_201210241044_0005_m_000006_0 2012-10-24 12:25:20,500 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_-538140840 given task: attempt_201210241044_0005_m_000013_0 2012-10-24 12:25:20,602 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201210241044_0005_m_-1673565310 given task: attempt_201210241044_0005_m_000011_0 2012-10-24 12:25:27,566 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201210241044_0005_m_000012_0 0.005804179% 2012-10-24 12:25:27,719 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201210241044_0005_m_000002_0 0.005184336% 2012-10-24 12:25:27,745 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201210241044_0005_m_000003_0 0.008510194% Datanode logs: 2012-10-24 12:26:31,373 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_8158410273681796837_11398 at file /home/runner/app/hadoop/dfs/data/current/subdir41/blk_8158410273681796837 2012-10-24 12:26:32,576 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /37.59.44.188:36421, dest: /91.121.69.14:50010, bytes: 6543254, op: HDFS_WRITE, cliID: DFSClient_hb_rs_slave2,60020,1351068281823, offset: 0, srvID: DS-747375281-91.121.69.14-50010-1350487134487, blockid: blk_-6002773137274160991_11407, duration: 1749006835 2012-10-24 12:26:32,576 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_-6002773137274160991_11407 terminating 2012-10-24 12:26:33,807 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /94.23.225.32:54226, dest: /91.121.69.14:50010, bytes: 25497785, op: HDFS_WRITE, cliID: DFSClient_hb_rs_slave2,60020,1351068281823, offset: 0, srvID: DS-747375281-91.121.69.14-50010-1350487134487, blockid: blk_-869989770332149129_11406, duration: 3505041135 2012-10-24 12:26:33,807 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-869989770332149129_11406 terminating 2012-10-24 12:26:34,165 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_3644089242601024939_11408 src: /37.59.44.188:36433 dest: /91.121.69.14:50010 2012-10-24 12:26:34,347 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-5745640874810358842_11401 file /home/runner/app/hadoop/dfs/data/current/subdir41/blk_-5745640874810358842 for deletion 2012-10-24 12:26:34,347 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1293587588119122949_11402 file /home/runner/app/hadoop/dfs/data/current/subdir41/blk_1293587588119122949 for deletion 2012-10-24 12:26:34,347 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2164482121237588223_11404 file /home/runner/app/hadoop/dfs/data/current/subdir41/blk_2164482121237588223 for deletion 2012-10-24 12:26:34,347 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4223515637631646553_11405 file /home/runner/app/hadoop/dfs/data/current/subdir41/blk_4223515637631646553 for deletion 2012-10-24 12:26:34,349 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-5745640874810358842_11401 at file /home/runner/app/hadoop/dfs/data/current/subdir41/blk_-5745640874810358842 2012-10-24 12:26:34,351 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_1293587588119122949_11402 at file /home/runner/app/hadoop/dfs/data/current/subdir41/blk_1293587588119122949 2012-10-24 12:26:34,353 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_2164482121237588223_11404 at file /home/runner/app/hadoop/dfs/data/current/subdir41/blk_2164482121237588223 2012-10-24 12:26:34,355 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_4223515637631646553_11405 at file /home/runner/app/hadoop/dfs/data/current/subdir41/blk_4223515637631646553 2012-10-24 12:26:34,991 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /37.59.44.188:36433, dest: /91.121.69.14:50010, bytes: 6245768, op: HDFS_WRITE, cliID: DFSClient_hb_rs_slave2,60020,1351068281823, offset: 0, srvID: DS-747375281-91.121.69.14-50010-1350487134487, blockid: blk_3644089242601024939_11408, duration: 821738497 2012-10-24 12:26:34,991 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_3644089242601024939_11408 terminating 2012-10-24 12:26:35,134 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-1806359844563012739_11409 src: /94.23.225.32:54236 dest: /91.121.69.14:50010 2012-10-24 12:26:36,874 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-1348441749762311112_11410 src: /37.59.44.188:36444 dest: /91.121.69.14:5001
