Re: HMaster restart with error

2015-05-18 Thread Ted Yu
The exception originated from Web UI corresponding to HBaseAdmin.listTables(). At that moment, master was unable to process the request - it needed some more time. Cheers On Sun, May 17, 2015 at 11:03 PM, Louis Hust louis.h...@gmail.com wrote: Yes, ted, can you tell me what the following

Re: HMaster restart with error

2015-05-18 Thread Louis Hust
But from the log: 2015-05-15 12:16:40,522 INFO [MASTER_SERVER_OPERATIONS-l-namenode1:6-0] handler.ServerShutdownHandler: Finished processing of shutdown of l-hbase31.data.cn8.qunar.com,60020,1427789773001 2015-05-15 12:17:11,301 WARN [686544788@qtp-660252776-212]

Re: HMaster restart with error

2015-05-18 Thread Louis Hust
Yes, ted, can you tell me what the following excpetion means in l-namenode1.log? 2015-05-15 12:16:40,522 INFO [MASTER_SERVER_OPERATIONS-l-namenode1:6-0] handler.ServerShutdownHandler: Finished processing of shutdown of l-hbase31.data.cn8.qunar.com,60020,1427789773001 2015-05-15 12:17:11,301

Re: HMaster restart with error

2015-05-17 Thread Louis Hust
Hi, ted, Thanks for your reply!! I found the log in l-namenode2.dba.cn8 during the restarting progress: 2015-05-15 12:11:36,540 INFO [master:l-namenode2:6] master.ServerManager: Finished waiting for region servers count to settle; checked in 5, slept for 4511 ms, expecting minimum of 1,

Re: HMaster restart with error

2015-05-17 Thread Ted Yu
bq. the backup master take over at 2015-05-15 12:16:40,024 ? The switch of active master should be earlier than 12:16:40,024 - shortly after 12:15:58 l-namenode1 would do some initialization (such as waiting for region servers count to settle) after it became active master. I tried to download

Re: HMaster restart with error

2015-05-17 Thread Ted Yu
After l-namenode1 became active master , it assigned regions: 2015-05-15 12:16:40,432 INFO [master:l-namenode1:6] master.RegionStates: Transitioned {6f806bb62b347c992cd243fc909276ff state=OFFLINE, ts=1431663400432, server=null} to {6f806bb62b347c992cd243fc909276ff state=OPEN,

Re: HMaster restart with error

2015-05-16 Thread Louis Hust
hi, TED, Any idea? When the HMaster restart, how can i know when it is really can handle request from application? is there any mark in logs? 2015-05-16 14:05 GMT+08:00 Louis Hust louis.h...@gmail.com: @Ted, plz see the log from 12:11:29 to 12:15:28, this timerange the HMaster is in

Re: HMaster restart with error

2015-05-16 Thread Ted Yu
In the period you identified, master was assigning regions. e.g. 2015-05-15 12:13:09,683 INFO [l-namenode2.dba.cn8.qunar.com,6,1431663090427-GeneralBulkAssigner-0] master.RegionStates: Transitioned {c634280ce287b2d6cebd88b61accf685 state=OFFLINE, ts=1431663189621, server=null} to

Re: HMaster restart with error

2015-05-16 Thread Louis Hust
OK, you can get the log from http://pan.baidu.com/s/1pqS6E 2015-05-16 13:26 GMT+08:00 Ted Yu yuzhih...@gmail.com: Can you check server log on 192.168.39.22 http://l-namenode2.dba.cn8.qunar.com/192.168.39.22:6 ? That should give you some clue. Cheers On Fri, May 15, 2015 at 8:22 PM,

Re: HMaster restart with error

2015-05-16 Thread Louis Hust
@Ted, plz see the log from 12:11:29 to 12:15:28, this timerange the HMaster is in restarting stage, but can not handle request from client? Is the HMaster recovering or do something else? 2015-05-16 13:59 GMT+08:00 Louis Hust louis.h...@gmail.com: OK, you can get the log from

Re: HMaster restart with error

2015-05-16 Thread Louis Hust
hi,ted, Thanks very much! Namenode process was not running on l-namenode2.dba.cn8(192.168.39.22), just HMaster run on it。 So you means that at 2015-05-15 12:15:04, the HMaster is handling two region server down, and not ready to handle client request? And how can i tell when the HMaster is

Re: HMaster restart with error

2015-05-16 Thread Ted Yu
bq. the HMaster is handling two region server down, and not ready to handle client request? I didn't mean that - for a functioning master, handling region server shutdown is part of the master's job. You should see something similar to the following in (functioning) master log: 2015-05-13

HMaster restart with error

2015-05-15 Thread Louis Hust
Hi all, I use hbase0.96.0 with hadoop 2.2.0, and the custom said they can not write into hbase cluster, So i stop the HMaster and start it soon, But it seems that the HMaster not response to request, following is the HMaster log: {logs} 2015-05-15 12:13:33,136 INFO [AM.ZK.Worker-pool2-t16]

Re: HMaster restart with error

2015-05-15 Thread Ted Yu
Can you check server log on 192.168.39.22 http://l-namenode2.dba.cn8.qunar.com/192.168.39.22:6 ? That should give you some clue. Cheers On Fri, May 15, 2015 at 8:22 PM, Louis Hust louis.h...@gmail.com wrote: Hi all, I use hbase0.96.0 with hadoop 2.2.0, and the custom said they can not