Never ending distributed log split

2012-08-03 Thread Jean-Marc Spaggiari
Hi, I'm using HBase 0.94.0. I stopped the cluster for some maintenance, and I'm have some troubles to restart it. I'm getting one line every about Start Time Description State Status Fri Aug 03 08:59:54 EDT 2012Doing distributed log split in [hdfs://node3:9000/hbase/.logs/latitud

Never ending distributed log split

2013-06-02 Thread Jean-Marc Spaggiari
My HBase was in a bad state recently. HBCK did a slow but good job and everything is now almost stable. However, I still have one log split which is not working. Every minute, the SplitLogManager try to split the log, fails, and retry. It's always the same file. It's assigned to different nodes, bu

Re: Never ending distributed log split

2012-08-03 Thread Jean-Marc Spaggiari
Here us the complete log. And seems it's every 30 seconds and not every 20 seconds... http://pastebin.com/gMiURnnj 2012/8/3, Jean-Marc Spaggiari : > Hi, > > I'm using HBase 0.94.0. > > I stopped the cluster for some maintenance, and I'm have some troubles > to restart it. > > I'm getting one line

Re: Never ending distributed log split

2012-08-03 Thread Jean-Marc Spaggiari
Me again ;) I did some more investigation. And I found that: http://pastebin.com/Bedm6Ldy Seems that no region is serving my logs. That's strange because all my servers are up and fsck is telling me that FS is clean. Can I just delete those files? What's the impact of such delete? I don't real

Re: Never ending distributed log split

2012-08-03 Thread Jean-Daniel Cryans
On Fri, Aug 3, 2012 at 8:15 AM, Jean-Marc Spaggiari wrote: > Me again ;) > > I did some more investigation. It would really help to see the region server log although the fsck output might be enough. BTW you'll find 0.94.1 RC1 here: http://people.apache.org/~larsh/hbase-0.94.1-rc1/ > > And I fo

Re: Never ending distributed log split

2012-08-03 Thread Jean-Marc Spaggiari
2012/8/3, Jean-Daniel Cryans : > On Fri, Aug 3, 2012 at 8:15 AM, Jean-Marc Spaggiari > wrote: >> Me again ;) >> >> I did some more investigation. > > It would really help to see the region server log although the fsck > output might be enough. I looked under evey directory and only one is contain

Re: Never ending distributed log split

2013-06-02 Thread Ted Yu
Can you search for 1d44b0630ed7785106a87a2bd4993551/recovered.edits to see when it was created ? Namenode log would be a good place to start with. bq. we can also rename it so if really required we can replay it later? The above is a better way of handling the situation. What version of HBase ar

Re: Never ending distributed log split

2013-06-02 Thread Jean-Marc Spaggiari
I'm using 0.94.7 since I did not get the chance to deploye the last RC... I will wait for some more feedback regarding the option (delete or rename) and most probably will open a JIRA. Regardeing recovered.editsI don't have this file anymore, but I just found another one which is blocking some ot

Re: Never ending distributed log split

2013-06-02 Thread Stack
On Sun, Jun 2, 2013 at 8:09 AM, Jean-Marc Spaggiari wrote: > So, 2 things again here. > > 1) Should the region server send more information of the failure to > the master the the master can display the failure cause on the logs? > Yes. You shouldn't have to work so hard to figure root cause (sm