If the region servers are dying then there logs are more likely to be helpful then the master.

Billy




"Yabo-Arber Xu" <[email protected]> wrote in message news:[email protected]...
Hi J-D,

Thanks for your reply. We have a 10-node cluster installed with HBase/Hadoop 0.19.1. We wrote about 1.8M records into HBase, and it appeared fine. We can use 'count' on shell to get the number. But after a while ( not sure exactly when ) all nodes crashed right away. I checked the log and but did not find anything particular at that moment. When I restart it again, all the rows in
the tables are lost but the tables are still there.

I have difficulty to connect IRC channel now ( will try it later ). For now,
I shared the master's log
here<http://cid-c31c56409ed94abd.skydrive.live.com/self.aspx/Public/Summba/hbase-cloudadmin-master-A6.log.2009-06-07>,
and the crash happens before the line "18:46:54 CST Starting master on A6".
You would notice that the period from "2009-06-07 17:00:57" to "2009-06-07
18:46:54" is empty ( due to the crash).

Thanks for your help!

Best,
Arber


On Sun, Jun 7, 2009 at 11:56 PM, Jean-Daniel Cryans <[email protected]>wrote:

Arber,

From your email I have a hard time understanding exactly what happened
on your cluster. But, before asking any question, I have to say that I
never saw data just "disappearing" like this on the production cluster
I've been managing for the last year.

So, what happened? Did you lose the region server holding .META or
-ROOT-? Was it after an importing job and all nodes crashed right
away? If any crash, do you know why it happened?

Giving us those details (and more) will help us solve your problem.
You can also drop by the hbase IRC channel, there's always people
there happy to take a look at your logs and debugging goes much
faster.

Thx,

J-D

On Sun, Jun 7, 2009 at 9:21 AM, Yabo-Arber Xu<[email protected]>
wrote:
> Hi there,
>
> I had a hbase data lost couple of days ago due to the crash of HBase. > At > that time i asked on the list and was told that it may be due to the > lost
of
> META data on the master ( not flushed into disk during crash ). Just
several
> days after, this happened to me again, this time all the data is gone,
while
> all the tables are still there. This time is on a 10-node hbase > cluster.
>
> I checked the log but did not find anything strange. Could anybody shed > a > light on this? Given such stability, i really worried whether we can > use
it
> in production phase.
>
> Best,
> Arber
>
> On Mon, May 25, 2009 at 10:25 AM, Yabo-Arber Xu <
[email protected]>wrote:
>
>> Hi there,
>>
>> I had a single-node cluster up and running. And yesterday the node
crashed
>> for unknown reason, and when I restart it, everything appears to work
except
>> that all the tables are LOST( UI says that there is no user tables)!! >> I >> checked the log file and didn't find any clue; while I found the >> tables
>> files are still there on HDFS.
>>
>> Anybody has any clue?
>>
>> It's quite urgent. Any help will be really appreciated.
>>
>> Best,
>> Arber
>>
>>
>




Reply via email to