This also reproduces on HBase-0.18.1 and HBase-0.18.2. Apparently its org.apache.hadoop.hbase.DroppedSnapshotException that is causing the cache data loss.
If between step 4 and step 5 the cache is flushed (e.g. Hbase cluster restart) no data is loss. However it means that if you stop a region server with dirty cache you will loose some data. Step 5 is hbase/bin/hbase-daemon.sh stop regionserver I raised https://issues.apache.org/jira/browse/HBASE-1052 On 12/9/08 4:53 PM, "Cosmin Lehene" <[EMAIL PROTECTED]> wrote: Hi, It seems https://issues.apache.org/jira/browse/HBASE-1010 has some ugly effects: data from entire table is missing after stopping the HRegionServer Hadoop 0.18.2 Hbase 0.18.0 Reproducing it is simple: 1. Start a Hbase cluster 2. Create a table t1: create 't1', {NAME => 'f1'} 3. Put a cell in the table: put 't1', 'r1', 'f1:', 'value' 4. Scan it, see it's fine 5. Stop the HRegionSever hosting the t1 region. Watch the region being reassigned from the original HRegionServer 6. Scan the t1 table again. It's empty now. The problem we initially found was that a few hours after restarting a HREgionServer we couldn't instantiate any scanners on the affected table. The problem could have been solved with a HRegionServer restart. However when trying to reproduce it the entire table content disappeared . Thanks, Cosmin Original HRegionServer: java version "1.6.0_06" Java(TM) SE Runtime Environment (build 1.6.0_06-b02) Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode) ulimit -n 32768 2008-12-09 06:35:26,761 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=HRegionServer, port=60020 2008-12-09 06:35:26,862 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at h2:60000 that we are up 2008-12-09 06:35:27,559 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.regionserver.address=10.131.237.51 2008-12-09 06:35:27,559 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: fs.default.name=hdfs://h1:54310/hbase 2008-12-09 06:35:27,559 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.rootdir=hdfs://h1:54310/hbase 2008-12-09 06:35:27,731 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir hdfs://h1:54310/hbase/log_10.131.237.51_1228833326838_60020 2008-12-09 06:35:27,835 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /hbase/log_10.131.237.51_1228833326838_60020/hlog.dat.1228833327777 2008-12-09 06:35:27,908 INFO org.mortbay.util.Credential: Checking Resource aliases 2008-12-09 06:35:27,956 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4 2008-12-09 06:35:27,957 INFO org.mortbay.util.Container: Started HttpContext[/static,/static] 2008-12-09 06:35:27,957 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs] 2008-12-09 06:35:28,510 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED] 2008-12-09 06:35:28,565 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/] 2008-12-09 06:35:28,568 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60030 2008-12-09 06:35:28,568 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED] 2008-12-09 06:35:28,568 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2008-12-09 06:35:28,570 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60020: starting 2008-12-09 06:35:28,570 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020: starting 2008-12-09 06:35:28,577 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020: starting 2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020: starting 2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020: starting 2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020: starting 2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020: starting 2008-12-09 06:35:28,579 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020: starting 2008-12-09 06:35:28,579 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020: starting 2008-12-09 06:35:28,579 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020: starting 2008-12-09 06:35:28,579 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started at: 10.131.237.51:60020 2008-12-09 06:35:28,580 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020: starting 2008-12-09 06:36:04,729 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: t1,,1228833363456 2008-12-09 06:36:04,730 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: t1,,1228833363456 2008-12-09 06:36:04,815 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}} 2008-12-09 06:36:04,840 DEBUG org.apache.hadoop.hbase.RegionHistorian: Onlined 2008-12-09 06:36:04,851 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region t1,,1228833363456/2133923429 2008-12-09 06:36:04,881 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region t1,,1228833363456 is 0 2008-12-09 06:36:04,883 INFO org.apache.hadoop.hbase.regionserver.HRegion: region t1,,1228833363456/2133923429 available 2008-12-09 06:36:04,884 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region: t1,,1228833363456 2008-12-09 06:36:04,884 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region t1,,1228833363456 2008-12-09 06:36:04,886 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region t1,,1228833363456 in 0sec Tue Dec 9 16:37:46 EET 2008 Stopping regionserver Tue Dec 9 16:37:46 EET 2008 Killing regionserver 2008-12-09 06:37:46,119 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown thread. 2008-12-09 06:37:46,871 DEBUG org.apache.hadoop.hbase.RegionHistorian: Offlined 2008-12-09 06:37:46,871 INFO org.apache.hadoop.ipc.Server: Stopping server on 60020 2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60020 2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020: exiting 2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020: exiting 2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020: exiting 2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020: exiting 2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020: exiting 2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020: exiting 2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020: exiting 2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020: exiting 2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020: exiting 2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020: exiting 2008-12-09 06:37:46,874 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer 2008-12-09 06:37:46,874 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2008-12-09 06:37:46,874 INFO org.mortbay.util.ThreadedServer: Stopping Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030] 2008-12-09 06:37:46,886 INFO org.mortbay.http.SocketListener: Stopped SocketListener on 0.0.0.0:60030 2008-12-09 06:37:46,948 INFO org.mortbay.util.Container: Stopped HttpContext[/static,/static] 2008-12-09 06:37:47,007 INFO org.mortbay.util.Container: Stopped HttpContext[/logs,/logs] 2008-12-09 06:37:47,007 INFO org.mortbay.util.Container: Stopped [EMAIL PROTECTED] 2008-12-09 06:37:47,094 INFO org.mortbay.util.Container: Stopped WebApplicationContext[/,/] 2008-12-09 06:37:47,094 INFO org.mortbay.util.Container: Stopped [EMAIL PROTECTED] 2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: closing region t1,,1228833363456 2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region t1,,1228833363456 2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region t1,,1228833363456 2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region t1,,1228833363456 2008-12-09 06:37:47,095 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region t1,,1228833363456 2008-12-09 06:37:47,095 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region t1,,1228833363456 2008-12-09 06:37:47,095 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region t1,,1228833363456. Current region memcache size 18.0 2008-12-09 06:37:47,095 INFO org.apache.hadoop.hbase.regionserver.Flusher: regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting 2008-12-09 06:37:47,096 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting. 2008-12-09 06:37:47,096 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting 2008-12-09 06:37:47,099 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: error closing region t1,,1228833363456 org.apache.hadoop.hbase.DroppedSnapshotException: region: t1,,1228833363456 at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1071) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:619) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeAllRegions(HRegionServer.java:951) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:459) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: Filesystem closed at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:196) at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:564) at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:390) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:667) at org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:152) at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:599) at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:577) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1058) ... 4 more 2008-12-09 06:37:47,100 DEBUG org.apache.hadoop.hbase.regionserver.HLog: closing log writer in hdfs://h1:54310/hbase/log_10.131.237.51_1228833326838_60020 2008-12-09 06:37:47,101 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Close and delete failed java.io.IOException: Filesystem closed at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:196) at org.apache.hadoop.dfs.DFSClient.access$600(DFSClient.java:59) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2689) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2655) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79) at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:962) at org.apache.hadoop.hbase.regionserver.HLog.close(HLog.java:349) at org.apache.hadoop.hbase.regionserver.HLog.closeAndDelete(HLog.java:333) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:461) at java.lang.Thread.run(Thread.java:619) 2008-12-09 06:37:47,102 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: telling master that region server is shutting down at: 10.131.237.51:60020 2008-12-09 06:37:47,104 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server at: 10.131.237.51:60020 2008-12-09 06:37:47,882 INFO org.apache.hadoop.hbase.Leases: regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closing leases 2008-12-09 06:37:47,882 INFO org.apache.hadoop.hbase.Leases: regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closed leases 2008-12-09 06:37:54,919 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting 2008-12-09 06:37:54,920 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/0:0:0:0:0:0:0:0:60020 exiting 2008-12-09 06:37:54,920 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete Second HRegion Server: 2008-12-09 06:35:28,710 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started at: 10.131.237.52:60020 2008-12-09 06:35:28,711 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020: starting 2008-12-09 06:37:49,965 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: t1,,1228833363456 2008-12-09 06:37:49,966 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: t1,,1228833363456 2008-12-09 06:37:50,055 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}} 2008-12-09 06:37:50,069 DEBUG org.apache.hadoop.hbase.RegionHistorian: Onlined 2008-12-09 06:37:50,080 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region t1,,1228833363456/2133923429 2008-12-09 06:37:50,110 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region t1,,1228833363456 is 0 2008-12-09 06:37:50,112 INFO org.apache.hadoop.hbase.regionserver.HRegion: region t1,,1228833363456/2133923429 available 2008-12-09 06:37:50,112 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region: t1,,1228833363456 2008-12-09 06:37:50,114 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region t1,,1228833363456 2008-12-09 06:37:50,118 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region t1,,1228833363456 in 0sec HMaster Server: java version "1.6.0_06" Java(TM) SE Runtime Environment (build 1.6.0_06-b02) Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode) ulimit -n 32768 2008-12-09 06:35:24,944 INFO org.apache.hadoop.hbase.master.HMaster: Root region dir: hdfs://h1:54310/hbase/-ROOT-/70236052 2008-12-09 06:35:25,109 INFO org.apache.hadoop.hbase.master.HMaster: BOOTSTRAP: creating ROOT and first META regions 2008-12-09 06:35:25,156 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /hbase/-ROOT-/70236052/log/hlog.dat.1228833325121 2008-12-09 06:35:25,160 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region -ROOT-,,0/70236052 2008-12-09 06:35:25,197 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region -ROOT-,,0 is 0 2008-12-09 06:35:25,200 INFO org.apache.hadoop.hbase.regionserver.HRegion: region -ROOT-,,0/70236052 available 2008-12-09 06:35:25,219 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /hbase/.META./1028785192/log/hlog.dat.1228833325214 2008-12-09 06:35:25,220 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region .META.,,1/1028785192 2008-12-09 06:35:25,252 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region .META.,,1 is 0 2008-12-09 06:35:25,256 INFO org.apache.hadoop.hbase.regionserver.HRegion: region .META.,,1/1028785192 available 2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region -ROOT-,,0 2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region -ROOT-,,0 2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region -ROOT-,,0 2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region -ROOT-,,0 2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region -ROOT-,,0 2008-12-09 06:35:25,261 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region -ROOT-,,0. Current region memcache size 445.0 2008-12-09 06:35:25,297 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2008-12-09 06:35:25,299 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 2008-12-09 06:35:25,301 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor 2008-12-09 06:35:25,555 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2008-12-09 06:35:25,556 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2008-12-09 06:35:25,556 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2008-12-09 06:35:25,556 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/-ROOT-/70236052/info/mapfiles/2955877428110101962 with 1 entries, sequence id 1, data size 445.0, file size 569.0 2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region -ROOT-,,0 in 310ms, sequence id=1, compaction requested=false 2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 70236052/info 2008-12-09 06:35:25,570 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed -ROOT-,,0 2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HLog: closing log writer in hdfs://h1:54310/hbase/-ROOT-/70236052/log 2008-12-09 06:35:25,590 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region .META.,,1 2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region .META.,,1 2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region .META.,,1 2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region .META.,,1 2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region .META.,,1 2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1028785192/historian 2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1028785192/info 2008-12-09 06:35:25,591 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed .META.,,1 2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HLog: closing log writer in hdfs://h1:54310/hbase/.META./1028785192/log 2008-12-09 06:35:25,917 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=HMaster, port=60000 2008-12-09 06:35:26,088 INFO org.apache.hadoop.hbase.master.HMaster: HMaster initialized on 10.131.237.52:60000 2008-12-09 06:35:26,152 INFO org.mortbay.util.Credential: Checking Resource aliases 2008-12-09 06:35:26,208 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4 2008-12-09 06:35:26,209 INFO org.mortbay.util.Container: Started HttpContext[/static,/static] 2008-12-09 06:35:26,209 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs] 2008-12-09 06:35:27,051 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED] 2008-12-09 06:35:27,133 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/] 2008-12-09 06:35:27,493 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED] 2008-12-09 06:35:27,495 INFO org.mortbay.util.Container: Started WebApplicationContext[/api,rest] 2008-12-09 06:35:27,498 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60010 2008-12-09 06:35:27,498 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED] 2008-12-09 06:35:27,498 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2008-12-09 06:35:27,499 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60000: starting 2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60000: starting 2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60000: starting 2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60000: starting 2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000: starting 2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60000: starting 2008-12-09 06:35:27,502 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60000: starting 2008-12-09 06:35:27,505 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60000: starting 2008-12-09 06:35:27,507 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60000: starting 2008-12-09 06:35:27,522 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60000: starting 2008-12-09 06:35:27,522 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60000: starting 2008-12-09 06:35:27,523 DEBUG org.apache.hadoop.hbase.master.HMaster: Started service threads 2008-12-09 06:35:27,529 INFO org.apache.hadoop.hbase.master.ServerManager: Received start message from: 10.131.237.54:60020 2008-12-09 06:35:27,535 INFO org.apache.hadoop.hbase.master.ServerManager: Received start message from: 10.131.237.51:60020 2008-12-09 06:35:27,538 INFO org.apache.hadoop.hbase.master.ServerManager: Received start message from: 10.131.237.53:60020 2008-12-09 06:35:27,705 INFO org.apache.hadoop.hbase.master.ServerManager: Received start message from: 10.131.237.52:60020 2008-12-09 06:35:28,648 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region -ROOT-,,0 to server 10.131.237.54:60020 2008-12-09 06:35:28,651 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 0, Num Servers: 4, Avg Load: 0.0 2008-12-09 06:35:31,673 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: -ROOT-,,0 from 10.131.237.54:60020 2008-12-09 06:35:31,675 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} 2008-12-09 06:35:31,769 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '', STARTCODE => 0 2008-12-09 06:35:31,769 DEBUG org.apache.hadoop.hbase.master.BaseScanner: Current assignment of .META.,,1 is not valid: serverInfo: null, passed startCode: 0, storedInfo.startCode: -1, unassignedRegions: false, pendingRegions: false 2008-12-09 06:35:31,772 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} complete 2008-12-09 06:35:34,661 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region .META.,,1 to server 10.131.237.53:60020 2008-12-09 06:35:36,721 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}} 2008-12-09 06:35:36,730 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1>. Retrying after sleep of 2000 2008-12-09 06:35:36,734 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}} 2008-12-09 06:35:37,685 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: .META.,,1 from 10.131.237.53:60020 2008-12-09 06:35:37,685 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: .META.,,1 from 10.131.237.53:60020 2008-12-09 06:35:37,689 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from 10.131.237.53:60020 2008-12-09 06:35:37,691 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: .META.,,1 open on 10.131.237.53:60020 2008-12-09 06:35:37,691 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row .META.,,1 in region -ROOT-,,0 with startcode 1228833326870 and server 10.131.237.53:60020 2008-12-09 06:35:38,744 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}} 2008-12-09 06:35:38,754 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020, location region name .META.,,1 2008-12-09 06:35:38,754 DEBUG org.apache.hadoop.hbase.RegionHistorian: Onlined 2008-12-09 06:35:38,755 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020, location region name .META.,,1 2008-12-09 06:35:38,755 DEBUG org.apache.hadoop.hbase.master.ProcessRegionOpen$1: Adding to onlineMetaRegions: {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} 2008-12-09 06:35:38,758 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} 2008-12-09 06:35:38,831 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} complete 2008-12-09 06:35:38,831 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned 2008-12-09 06:35:38,931 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020, location region name .META.,,1 2008-12-09 06:35:39,697 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020, location region name .META.,,1 2008-12-09 06:35:40,435 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020, location region name .META.,,1 2008-12-09 06:35:41,223 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020, location region name .META.,,1 2008-12-09 06:35:43,673 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 2, Num Servers: 4, Avg Load: 1.0 2008-12-09 06:35:58,697 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 2, Num Servers: 4, Avg Load: 1.0 2008-12-09 06:36:03,477 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /hbase/t1/2133923429/log/hlog.dat.1228833363473 2008-12-09 06:36:03,478 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region t1,,1228833363456/2133923429 2008-12-09 06:36:03,489 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region t1,,1228833363456 is 0 2008-12-09 06:36:03,491 INFO org.apache.hadoop.hbase.regionserver.HRegion: region t1,,1228833363456/2133923429 available 2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region t1,,1228833363456 2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region t1,,1228833363456 2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region t1,,1228833363456 2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region t1,,1228833363456 2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region t1,,1228833363456 2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2133923429/f1 2008-12-09 06:36:03,495 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed t1,,1228833363456 2008-12-09 06:36:03,495 DEBUG org.apache.hadoop.hbase.regionserver.HLog: closing log writer in hdfs://h1:54310/hbase/t1/2133923429/log 2008-12-09 06:36:03,521 INFO org.apache.hadoop.hbase.master.HMaster: created table t1 2008-12-09 06:36:04,705 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region t1,,1228833363456 to server 10.131.237.51:60020 2008-12-09 06:36:07,726 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: t1,,1228833363456 from 10.131.237.51:60020 2008-12-09 06:36:07,727 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from 10.131.237.51:60020 2008-12-09 06:36:07,727 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: t1,,1228833363456 open on 10.131.237.51:60020 2008-12-09 06:36:07,727 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1 2008-12-09 06:36:07,727 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row t1,,1228833363456 in region .META.,,1 with startcode 1228833326838 and server 10.131.237.51:60020 2008-12-09 06:36:11,335 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020, location region name .META.,,1 2008-12-09 06:36:13,735 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 4, Avg Load: 1.0 2008-12-09 06:36:24,695 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020, location region name .META.,,1 2008-12-09 06:36:28,756 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 4, Avg Load: 1.0 2008-12-09 06:36:31,782 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} 2008-12-09 06:36:31,815 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870 2008-12-09 06:36:31,817 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} complete 2008-12-09 06:36:38,766 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} 2008-12-09 06:36:38,773 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.51:60020', STARTCODE => 1228833326838 2008-12-09 06:36:38,775 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} complete 2008-12-09 06:36:38,775 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned 2008-12-09 06:36:43,777 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 4, Avg Load: 1.0 2008-12-09 06:36:56,635 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:36:58,801 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 4, Avg Load: 1.0 2008-12-09 06:37:13,820 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 4, Avg Load: 1.0 2008-12-09 06:37:26,769 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:37:28,841 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 4, Avg Load: 1.0 2008-12-09 06:37:31,792 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} 2008-12-09 06:37:31,835 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870 2008-12-09 06:37:31,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} complete 2008-12-09 06:37:38,776 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} 2008-12-09 06:37:38,786 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.51:60020', STARTCODE => 1228833326838 2008-12-09 06:37:38,789 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} complete 2008-12-09 06:37:38,789 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned 2008-12-09 06:37:43,860 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 4, Avg Load: 1.0 2008-12-09 06:37:47,095 INFO org.apache.hadoop.hbase.master.ServerManager: Cancelling lease for 10.131.237.51:60020 2008-12-09 06:37:47,095 INFO org.apache.hadoop.hbase.master.ServerManager: Region server 10.131.237.51:60020: MSG_REPORT_EXITING -- lease cancelled 2008-12-09 06:37:47,096 INFO org.apache.hadoop.hbase.master.ServerManager: Processing MSG_REPORT_CLOSE: t1,,1228833363456 from 10.131.237.51:60020 2008-12-09 06:37:49,949 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region t1,,1228833363456 to server 10.131.237.52:60020 2008-12-09 06:37:52,974 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: t1,,1228833363456 from 10.131.237.52:60020 2008-12-09 06:37:52,974 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: t1,,1228833363456 from 10.131.237.52:60020 2008-12-09 06:37:52,975 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from 10.131.237.52:60020 2008-12-09 06:37:52,975 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: t1,,1228833363456 open on 10.131.237.52:60020 2008-12-09 06:37:52,975 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1 2008-12-09 06:37:52,975 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row t1,,1228833363456 in region .META.,,1 with startcode 1228833327601 and server 10.131.237.52:60020 2008-12-09 06:37:56,912 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:37:58,877 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:38:13,894 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:38:27,035 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:38:28,914 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:38:31,802 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} 2008-12-09 06:38:31,833 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870 2008-12-09 06:38:31,835 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} complete 2008-12-09 06:38:38,787 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} 2008-12-09 06:38:38,792 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.52:60020', STARTCODE => 1228833327601 2008-12-09 06:38:38,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} complete 2008-12-09 06:38:38,794 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned 2008-12-09 06:38:43,930 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:38:57,168 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:38:58,950 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:39:13,970 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:39:27,310 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:39:28,996 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:39:31,813 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} 2008-12-09 06:39:31,844 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870 2008-12-09 06:39:31,846 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} complete 2008-12-09 06:39:38,797 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} 2008-12-09 06:39:38,806 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.52:60020', STARTCODE => 1228833327601 2008-12-09 06:39:38,808 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} complete 2008-12-09 06:39:38,808 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned 2008-12-09 06:39:44,016 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:39:57,462 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:39:58,440 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:39:59,036 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:40:14,056 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:40:28,547 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:40:29,075 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:40:31,823 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} 2008-12-09 06:40:31,849 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870 2008-12-09 06:40:31,851 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020} complete 2008-12-09 06:40:38,808 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} 2008-12-09 06:40:38,812 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '10.131.237.52:60020', STARTCODE => 1228833327601 2008-12-09 06:40:38,814 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020} complete 2008-12-09 06:40:38,814 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned 2008-12-09 06:40:44,096 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0 2008-12-09 06:40:58,661 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020, location region name t1,,1228833363456 2008-12-09 06:40:59,115 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3, Num Servers: 3, Avg Load: 1.0
