Me again ;) I did some more investigation.
And I found that: http://pastebin.com/Bedm6Ldy Seems that no region is serving my logs. That's strange because all my servers are up and fsck is telling me that FS is clean. Can I just delete those files? What's the impact of such delete? I don't really worrie about loosing some data. It's a test environment. But I really need it to start again. Thanks, JM 2012/8/3, Jean-Marc Spaggiari <jean-m...@spaggiari.org>: > Here us the complete log. And seems it's every 30 seconds and not > every 20 seconds... > > http://pastebin.com/gMiURnnj > > 2012/8/3, Jean-Marc Spaggiari <jean-m...@spaggiari.org>: >> Hi, >> >> I'm using HBase 0.94.0. >> >> I stopped the cluster for some maintenance, and I'm have some troubles >> to restart it. >> >> I'm getting one line every about >> >> Start Time Description State Status >> Fri Aug 03 08:59:54 EDT 2012 Doing distributed log split in >> [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting, >> hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting, >> hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting, >> hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting, >> hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting, >> hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting, >> hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting, >> hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting, >> hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting, >> hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting, >> hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting, >> hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting, >> hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting, >> hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting, >> hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting, >> hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting, >> hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting, >> hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting, >> hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting] >> RUNNING (since 3sec ago) Waiting for distributed tasks to finish. >> scheduled=1 done=0 error=0 (since 0sec ago) >> >> If I let it run, it will run like that for hours. Adding lines and >> lines and lines until I stop it. >> >> >> On the master logs, I can see that: >> 2012-08-03 09:02:49,788 INFO >> org.apache.hadoop.hbase.master.SplitLogManager: task >> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297 >> entered state err node4,60020,1343998592129 >> 2012-08-03 09:02:49,788 WARN >> org.apache.hadoop.hbase.master.SplitLogManager: Error splitting >> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297 >> 2012-08-03 09:02:49,788 WARN >> org.apache.hadoop.hbase.master.SplitLogManager: error while splitting >> logs in >> [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting, >> hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting, >> hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting, >> hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting, >> hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting, >> hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting, >> hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting, >> hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting, >> hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting, >> hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting, >> hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting, >> hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting, >> hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting, >> hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting, >> hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting, >> hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting, >> hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting, >> hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting, >> hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting] >> installed = 1 but only 0 done >> 2012-08-03 09:02:49,788 WARN >> org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting of >> [latitude,60020,1343908057839, latitude,60020,1343998595290, >> node1,60020,1343908057567, node1,60020,1343939284240, >> node1,60020,1343998593757, node2,60020,1343908059614, >> node2,60020,1343939286369, node2,60020,1343998595830, >> node3,60020,1343908054414, node3,60020,1343939282294, >> node3,60020,1343998590612, node4,60020,1343908056186, >> node4,60020,1343939282889, node4,60020,1343998592129, >> node5,60020,1343908059158, node5,60020,1343998594856, >> phenom,60020,1343908053256, phenom,60020,1343939281065, >> phenom,60020,1343998580375] >> java.io.IOException: error or interrupt while splitting logs in >> [hdfs://node3:9000/hbase/.logs/latitude,60020,1343908057839-splitting, >> hdfs://node3:9000/hbase/.logs/latitude,60020,1343998595290-splitting, >> hdfs://node3:9000/hbase/.logs/node1,60020,1343908057567-splitting, >> hdfs://node3:9000/hbase/.logs/node1,60020,1343939284240-splitting, >> hdfs://node3:9000/hbase/.logs/node1,60020,1343998593757-splitting, >> hdfs://node3:9000/hbase/.logs/node2,60020,1343908059614-splitting, >> hdfs://node3:9000/hbase/.logs/node2,60020,1343939286369-splitting, >> hdfs://node3:9000/hbase/.logs/node2,60020,1343998595830-splitting, >> hdfs://node3:9000/hbase/.logs/node3,60020,1343908054414-splitting, >> hdfs://node3:9000/hbase/.logs/node3,60020,1343939282294-splitting, >> hdfs://node3:9000/hbase/.logs/node3,60020,1343998590612-splitting, >> hdfs://node3:9000/hbase/.logs/node4,60020,1343908056186-splitting, >> hdfs://node3:9000/hbase/.logs/node4,60020,1343939282889-splitting, >> hdfs://node3:9000/hbase/.logs/node4,60020,1343998592129-splitting, >> hdfs://node3:9000/hbase/.logs/node5,60020,1343908059158-splitting, >> hdfs://node3:9000/hbase/.logs/node5,60020,1343998594856-splitting, >> hdfs://node3:9000/hbase/.logs/phenom,60020,1343908053256-splitting, >> hdfs://node3:9000/hbase/.logs/phenom,60020,1343939281065-splitting, >> hdfs://node3:9000/hbase/.logs/phenom,60020,1343998580375-splitting] >> Task = installed = 1 done = 0 error = 1 >> at >> org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:269) >> at >> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:277) >> at >> org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:219) >> at >> org.apache.hadoop.hbase.master.HMaster.splitLogAfterStartup(HMaster.java:577) >> at >> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:522) >> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:343) >> at java.lang.Thread.run(Thread.java:722) >> 2012-08-03 09:02:49,891 DEBUG >> org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: >> deleted >> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1343908057567-splitting%2Fnode1%252C60020%252C1343908057567.1343914548297 >> >> I would like to try with 0.94.1 but I don't know where to find the >> files. Does any one have any idea where this is coming from and where >> I can found 0.94.1RC1? >> >> Thanks, >> >> JM >> >
