Ok. So. You HBase system is inconsistent. And I can't see you table " t111" anywhere. On the shell, if you do a "list", which tables are you getting? I guess t111 will not be there. First thing to do is to fix that.
Can you provide more logs from the hbase check? It might tell you what the issues are. You can past it on pastebin.com JM 2013/1/17, hua beatls <bea...@gmail.com>: > On Thu, Jan 17, 2013 at 8:21 PM, Jean-Marc Spaggiari < > jean-m...@spaggiari.org> wrote: > >> Hi, >> > *below is snapshot of webui,* tabel 'cdrtestsnap' and > 'cdrtestsnappy' is on the list , as well as their regions. > > cdrtestGZ <http://192.168.250.106:60010/table.jsp?name=cdrtestGZ> {NAME => > 'cdrtestGZ', FAMILIES => [{NAME => 'cf', COMPRESSION => 'GZ', MIN_VERSIONS > => '0'}]} > cdrtestsnap<http://192.168.250.106:60010/table.jsp?name=cdrtestsnap> > {NAME > => 'cdrtestsnap', FAMILIES => [{NAME => 'cf', COMPRESSION => 'SNAPPY', > MIN_VERSIONS => '0'}]} > cdrtestsnappy<http://192.168.250.106:60010/table.jsp?name=cdrtestsnappy> > {NAME > => 'cdrtestsnappy', FAMILIES => [{NAME => 'cf', COMPRESSION => 'SNAPPY', > MIN_VERSIONS => '0'}]} > > region information of table 'cdrtestsnap' show below: > Table Attributes Attribute Name Value Description Enabled true Is > the table enabled Compaction > > ------------------------------ > > Actions: > > Region Key (optional): This action will force a compaction of all > regions of the table, or, if a key is supplied, only the region containing > the given key. Region Key (optional): This action will force a split of > all eligible regions of the table, or, if a key is supplied, only the > region containing the given key. An eligible region is one that does not > contain any references to other regions. Split requests for noneligible > regions will be ignored. > > > *below is excerption hbase hbck:* > Summary: > -ROOT- is okay. > Number of regions: 1 > Deployed on: hadoop2,60020,1358418770574 > .META. is okay. > Number of regions: 1 > Deployed on: hadoop2,60020,1358418770574 > cdrtest is okay. > Number of regions: 122 > Deployed on: hadoop3,60020,1358418770520 > cdrtest1 is okay. > Number of regions: 1 > Deployed on: hadoop3,60020,1358418770520 > cdrtestGZ is okay. > Number of regions: 1 > Deployed on: hadoop2,60020,1358418770574 > filelog is okay. > Number of regions: 1 > Deployed on: hadoop3,60020,1358418770520 > merge_bigtable is okay. > Number of regions: 1 > Deployed on: hadoop1,60020,1358418784993 > zlytest is okay. > Number of regions: 1 > Deployed on: hadoop1,60020,1358418784993 > zlytest1 is okay. > Number of regions: 1 > Deployed on: hadoop3,60020,1358418770520 > 2 inconsistencies detected. > Status: INCONSISTENT > [hadoop@hadoop2 ~]$ d > > * these two table are in the transition state.* > Regions in Transition Region State > 89d3b8af7181ad8b857ef7f55c3abe46 > cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > state=PENDING_OPEN, ts=Thu Jan 17 20:19:13 CST 2013 (945s ago), > server=hadoop3,60020,1358418770520 c56109758acea3619fb9c656b8b45aa3 > cdrtestsnap,,1358321463618.c56109758acea3619fb9c656b8b45aa3. > state=PENDING_OPEN, ts=Thu Jan 17 20:15:59 CST 2013 (1138s ago), > server=null > > masterlog: > 2013-01-17 20:17:13,630 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_FAILED_OPEN, server=hadoop3,60020,1358418770520, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,630 DEBUG > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED > event for 89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,630 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; > was=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > state=CLOSED, ts=1358425018648, server=hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,630 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > master:60000-0x33c48075af00003 Creating (or updating) unassigned node for > 89d3b8af7181ad8b857ef7f55c3abe46 with OFFLINE state > 2013-01-17 20:17:13,639 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan > for cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > destination server is hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,639 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=M_ZK_REGION_OFFLINE, server=hadoop1,60000,1358418784101, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,639 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan > for region cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46.; > plan=hri=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46., > src=, dest=hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,639 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region > cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. to > hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,639 DEBUG org.apache.hadoop.hbase.master.ServerManager: > New connection to hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,648 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_OPENING, server=hadoop3,60020,1358418770520, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,656 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_FAILED_OPEN, server=hadoop3,60020,1358418770520, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,656 DEBUG > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED > event for 89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,656 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; > was=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > state=CLOSED, ts=1358425018687, server=hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,656 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > master:60000-0x33c48075af00003 Creating (or updating) unassigned node for > 89d3b8af7181ad8b857ef7f55c3abe46 with OFFLINE state > 2013-01-17 20:17:13,664 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=M_ZK_REGION_OFFLINE, server=hadoop1,60000,1358418784101, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,664 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan > for cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > destination server is hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,664 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan > for region cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46.; > plan=hri=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46., > src=, dest=hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,664 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region > cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. to > hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,664 DEBUG org.apache.hadoop.hbase.master.ServerManager: > New connection to hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,675 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_OPENING, server=hadoop3,60020,1358418770520, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,683 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_FAILED_OPEN, server=hadoop3,60020,1358418770520, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,684 DEBUG > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED > event for 89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,684 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; > was=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > state=CLOSED, ts=1358425018714, server=hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,684 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > master:60000-0x33c48075af00003 Creating (or updating) unassigned node for > 89d3b8af7181ad8b857ef7f55c3abe46 with OFFLINE state > 2013-01-17 20:17:13,692 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan > for cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > destination server is hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,692 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan > for region cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46.; > plan=hri=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46., > src=, dest=hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,692 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region > cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. to > hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,692 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=M_ZK_REGION_OFFLINE, server=hadoop1,60000,1358418784101, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,692 DEBUG org.apache.hadoop.hbase.master.ServerManager: > New connection to hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,700 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_OPENING, server=hadoop3,60020,1358418770520, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,708 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_FAILED_OPEN, server=hadoop3,60020,1358418770520, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,709 DEBUG > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED > event for 89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,709 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; > was=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > state=CLOSED, ts=1358425018739, server=hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,709 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > master:60000-0x33c48075af00003 Creating (or updating) unassigned node for > 89d3b8af7181ad8b857ef7f55c3abe46 with OFFLINE state > 2013-01-17 20:17:13,717 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan > for cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > destination server is hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,717 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan > for region cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46.; > plan=hri=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46., > src=, dest=hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,717 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region > cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. to > hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,717 DEBUG org.apache.hadoop.hbase.master.ServerManager: > New connection to hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,717 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=M_ZK_REGION_OFFLINE, server=hadoop1,60000,1358418784101, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,725 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_OPENING, server=hadoop3,60020,1358418770520, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,733 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_FAILED_OPEN, server=hadoop3,60020,1358418770520, > region=89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,733 DEBUG > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED > event for 89d3b8af7181ad8b857ef7f55c3abe46 > 2013-01-17 20:17:13,734 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; > was=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > state=CLOSED, ts=1358425018764, server=hadoop3,60020,1358418770520 > 2013-01-17 20:17:13,734 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > master:60000-0x33c48075af00003 Creating (or updating) unassigned node for > 89d3b8af7181ad8b857ef7f55c3abe46 with OFFLINE state > 2013-01-17 20:17:13,742 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=M_ZK_REGION_OFFLINE, server=hadoop1,60000,1358418784101, > region=89d3b8af718 > [root@hadoop1 logs]# tail -f hbase-hadoop-master-hadoop1.log > 2013-01-17 20:38:25,958 DEBUG > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker > org.apache.hadoop.hbase.catalog.CatalogTracker@7decd1e2 > 2013-01-17 20:38:25,962 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x23c4806bfd50012 closed > 2013-01-17 20:38:25,962 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2013-01-17 20:39:35,138 DEBUG org.apache.hadoop.hbase.master.HMaster: Not > running balancer because 2 region(s) in transition: > {89d3b8af7181ad8b857ef7f55c3abe46=cdrtestsnappy,,1358316641849.89d3b8af7181ad8b857ef7f55c3abe46. > state=PENDING_OPEN, ts=1358425153007, server=hadoop3,60020,1358418770520, > c56109758acea3619fb9c656b8b45aa3=cdrtestsnap,,1358321463618.c56109758acea3619fb9c6... > 2013-01-17 20:39:35,139 DEBUG > org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner over > .META. starting at key '' > 2013-01-17 20:39:35,139 DEBUG > org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal > scanner to startKey at '' > 2013-01-17 20:39:35,212 DEBUG > org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with scanning > at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => > 1028785192,} > 2013-01-17 20:39:35,212 DEBUG > org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 130 catalog row(s) > and gc'd 0 unreferenced parent region(s) > 2013-01-17 20:40:11,469 INFO org.apache.hadoop.hbase.master.HMaster: > Balance=false > 2013-01-17 20:40:13,028 INFO org.apache.hadoop.hbase.master.HMaster: > Balance=true > > > use hbase zkcli to delete the /hbase/unassigned, still not > works. > > Thanks! > > beatls > > > > > > >> What do you mean by "the file system is ok"? Have you run both the >> HBase and the Hadoop checks and both of them aren't returning any >> error? >> >> Also, what do you have on the webui? Can yo usee the table? The regions? >> >> When you are trying to delete the table, what do you havae on you >> master logs? Can you share this log? >> >> Thanks, >> >> JM >> >> 2013/1/17, hua beatls <bea...@gmail.com>: >> > Hi, >> > the file system is ok. >> > >> > any other idea? >> > >> > beatls >> > >> > On Thu, Jan 17, 2013 at 7:38 PM, Jean-Marc Spaggiari < >> > jean-m...@spaggiari.org> wrote: >> > >> >> Hi Beat, >> >> >> >> Have you tried to check the filesystem using fsck to see if there is >> >> no inconsistencies? >> >> >> >> JM >> >> >> >> 2013/1/17, hua beatls <bea...@gmail.com>: >> >> > hi, >> >> > A table mistakenly created cannot be dropped? >> >> > first we create it with 'hbase shell', below: >> >> > [hadoop@hadoop3 ~]$ >> >> > [hadoop@hadoop3 ~]$ hbase shell >> >> > 13/01/17 15:49:35 WARN conf.Configuration: hadoop.native.lib is >> >> deprecated. >> >> > Instead, use io.native.lib.available >> >> > HBase Shell; enter 'help<RETURN>' for list of supported commands. >> >> > Type "exit<RETURN>" to leave the HBase Shell >> >> > Version 0.92.1-cdh4.1.2, r, Thu Nov 1 17:54:35 PDT 2012 >> >> > hbase(main):001:0> create >> >> > 't111',{NAME=>'CF111',COMPRESSION=>'snappy'} >> >> > SLF4J: Class path contains multiple SLF4J bindings. >> >> > SLF4J: Found binding in >> >> > >> >> >> [jar:file:/app/hbase/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> >> > SLF4J: Found binding in >> >> > >> >> >> [jar:file:/app/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >> >> > explanation. >> >> > ^C[hadoop@hadoop3 ~]$ >> >> > (this command hang after i issue "ctrl-c". (please be noted that >> >> 'snappy' >> >> > is not installed/config in this hbase.) >> >> > second, we 'describe' this table, below, >> >> > hbase(main):003:0> describe 't111' >> >> > DESCRIPTION >> >> > ENABLED >> >> > {NAME => 't111', FAMILIES => [{NAME => 'CF111', BLOOMFILTER => >> 'NONE', >> >> > REPLICATION_SCOPE => '0', VERSIONS >> >> > true >> >> > => '3', COMPRESSION => 'SNAPPY', MIN_VERSIONS => '0', TTL => >> >> '2147483647', >> >> > BLOCKSIZE => '65536', >> >> > IN_MEMORY >> >> > => 'false', BLOCKCACHE => >> >> > 'true'}]} >> >> > >> >> > 1 row(s) in 0.0470 seconds >> >> > 3rd, but this table is in the 'transition' state, and remains >> >> > in >> >> this >> >> > state. >> >> > 4th, i want to 'drop' this table, but 'drop' hang and failed. >> >> > hbase(main):002:0> disable 't111' >> >> > 0 row(s) in 1.1560 seconds >> >> > hbase(main):005:0> drop 't111' >> >> > >> >> > 'drop' is hang until i 'ctrl-c'. >> >> > >> >> > is there any other way to drop this table forcefully? >> >> > >> >> > >> >> > Beat R. >> >> > >> >> > beatls >> >> > >> >> >> > >> >
