Hi,
It seems https://issues.apache.org/jira/browse/HBASE-1010 has some ugly
effects: data from entire table is missing after stopping the HRegionServer
Hadoop 0.18.2
Hbase 0.18.0
Reproducing it is simple:
1. Start a Hbase cluster
2. Create a table t1: create 't1', {NAME => 'f1'}
3. Put a cell in the table: put 't1', 'r1', 'f1:', 'value'
4. Scan it, see it's fine
5. Stop the HRegionSever hosting the t1 region. Watch the region being
reassigned from the original HRegionServer
6. Scan the t1 table again. It's empty now.
The problem we initially found was that a few hours after restarting a
HREgionServer we couldn't instantiate any scanners on the affected table. The
problem could have been solved with a HRegionServer restart. However when
trying to reproduce it the entire table content disappeared .
Thanks,
Cosmin
Original HRegionServer:
java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode)
ulimit -n 32768
2008-12-09 06:35:26,761 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=HRegionServer, port=60020
2008-12-09 06:35:26,862 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at h2:60000
that we are up
2008-12-09 06:35:27,559 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master:
hbase.regionserver.address=10.131.237.51
2008-12-09 06:35:27,559 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master:
fs.default.name=hdfs://h1:54310/hbase
2008-12-09 06:35:27,559 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master:
hbase.rootdir=hdfs://h1:54310/hbase
2008-12-09 06:35:27,731 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir
hdfs://h1:54310/hbase/log_10.131.237.51_1228833326838_60020
2008-12-09 06:35:27,835 INFO org.apache.hadoop.hbase.regionserver.HLog: New log
writer created at
/hbase/log_10.131.237.51_1228833326838_60020/hlog.dat.1228833327777
2008-12-09 06:35:27,908 INFO org.mortbay.util.Credential: Checking Resource
aliases
2008-12-09 06:35:27,956 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2008-12-09 06:35:27,957 INFO org.mortbay.util.Container: Started
HttpContext[/static,/static]
2008-12-09 06:35:27,957 INFO org.mortbay.util.Container: Started
HttpContext[/logs,/logs]
2008-12-09 06:35:28,510 INFO org.mortbay.util.Container: Started [EMAIL
PROTECTED]
2008-12-09 06:35:28,565 INFO org.mortbay.util.Container: Started
WebApplicationContext[/,/]
2008-12-09 06:35:28,568 INFO org.mortbay.http.SocketListener: Started
SocketListener on 0.0.0.0:60030
2008-12-09 06:35:28,568 INFO org.mortbay.util.Container: Started [EMAIL
PROTECTED]
2008-12-09 06:35:28,568 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2008-12-09 06:35:28,570 INFO org.apache.hadoop.ipc.Server: IPC Server listener
on 60020: starting
2008-12-09 06:35:28,570 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 60020: starting
2008-12-09 06:35:28,577 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1
on 60020: starting
2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2
on 60020: starting
2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 60020: starting
2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4
on 60020: starting
2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5
on 60020: starting
2008-12-09 06:35:28,579 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6
on 60020: starting
2008-12-09 06:35:28,579 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7
on 60020: starting
2008-12-09 06:35:28,579 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8
on 60020: starting
2008-12-09 06:35:28,579 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started at:
10.131.237.51:60020
2008-12-09 06:35:28,580 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9
on 60020: starting
2008-12-09 06:36:04,729 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
t1,,1228833363456
2008-12-09 06:36:04,730 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
t1,,1228833363456
2008-12-09 06:36:04,815 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE',
VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
BLOCKCACHE => 'false'}]}}
2008-12-09 06:36:04,840 DEBUG org.apache.hadoop.hbase.RegionHistorian: Onlined
2008-12-09 06:36:04,851 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region t1,,1228833363456/2133923429
2008-12-09 06:36:04,881 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Next sequence id for region t1,,1228833363456 is 0
2008-12-09 06:36:04,883 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region t1,,1228833363456/2133923429 available
2008-12-09 06:36:04,884 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
for region: t1,,1228833363456
2008-12-09 06:36:04,884 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region t1,,1228833363456
2008-12-09 06:36:04,886 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region t1,,1228833363456 in 0sec
Tue Dec 9 16:37:46 EET 2008 Stopping regionserver
Tue Dec 9 16:37:46 EET 2008 Killing regionserver
2008-12-09 06:37:46,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown thread.
2008-12-09 06:37:46,871 DEBUG org.apache.hadoop.hbase.RegionHistorian: Offlined
2008-12-09 06:37:46,871 INFO org.apache.hadoop.ipc.Server: Stopping server on
60020
2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
listener on 60020
2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1
on 60020: exiting
2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6
on 60020: exiting
2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2
on 60020: exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 60020: exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4
on 60020: exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5
on 60020: exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7
on 60020: exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8
on 60020: exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9
on 60020: exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 60020: exiting
2008-12-09 06:37:46,874 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2008-12-09 06:37:46,874 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
Responder
2008-12-09 06:37:46,874 INFO org.mortbay.util.ThreadedServer: Stopping Acceptor
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
2008-12-09 06:37:46,886 INFO org.mortbay.http.SocketListener: Stopped
SocketListener on 0.0.0.0:60030
2008-12-09 06:37:46,948 INFO org.mortbay.util.Container: Stopped
HttpContext[/static,/static]
2008-12-09 06:37:47,007 INFO org.mortbay.util.Container: Stopped
HttpContext[/logs,/logs]
2008-12-09 06:37:47,007 INFO org.mortbay.util.Container: Stopped [EMAIL
PROTECTED]
2008-12-09 06:37:47,094 INFO org.mortbay.util.Container: Stopped
WebApplicationContext[/,/]
2008-12-09 06:37:47,094 INFO org.mortbay.util.Container: Stopped [EMAIL
PROTECTED]
2008-12-09 06:37:47,094 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
t1,,1228833363456
2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Compactions and cache flushes disabled for region t1,,1228833363456
2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Scanners disabled for region t1,,1228833363456
2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more active scanners for region t1,,1228833363456
2008-12-09 06:37:47,095 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region t1,,1228833363456
2008-12-09 06:37:47,095 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more row locks outstanding on region t1,,1228833363456
2008-12-09 06:37:47,095 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memcache flush for region t1,,1228833363456. Current region memcache
size 18.0
2008-12-09 06:37:47,095 INFO org.apache.hadoop.hbase.regionserver.Flusher:
regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting
2008-12-09 06:37:47,096 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2008-12-09 06:37:47,096 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting
2008-12-09 06:37:47,099 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: error closing region
t1,,1228833363456
org.apache.hadoop.hbase.DroppedSnapshotException: region: t1,,1228833363456
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1071)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:619)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.closeAllRegions(HRegionServer.java:951)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:459)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:196)
at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:564)
at
org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:390)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:667)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:152)
at
org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:599)
at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:577)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1058)
... 4 more
2008-12-09 06:37:47,100 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
closing log writer in
hdfs://h1:54310/hbase/log_10.131.237.51_1228833326838_60020
2008-12-09 06:37:47,101 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Close and delete failed
java.io.IOException: Filesystem closed
at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:196)
at org.apache.hadoop.dfs.DFSClient.access$600(DFSClient.java:59)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2689)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2655)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:962)
at org.apache.hadoop.hbase.regionserver.HLog.close(HLog.java:349)
at org.apache.hadoop.hbase.regionserver.HLog.closeAndDelete(HLog.java:333)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:461)
at java.lang.Thread.run(Thread.java:619)
2008-12-09 06:37:47,102 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: telling master that region
server is shutting down at: 10.131.237.51:60020
2008-12-09 06:37:47,104 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server at:
10.131.237.51:60020
2008-12-09 06:37:47,882 INFO org.apache.hadoop.hbase.Leases:
regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closing leases
2008-12-09 06:37:47,882 INFO org.apache.hadoop.hbase.Leases:
regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closed leases
2008-12-09 06:37:54,919 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2008-12-09 06:37:54,920 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
regionserver/0:0:0:0:0:0:0:0:60020 exiting
2008-12-09 06:37:54,920 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
Second HRegion Server:
2008-12-09 06:35:28,710 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started at:
10.131.237.52:60020
2008-12-09 06:35:28,711 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9
on 60020: starting
2008-12-09 06:37:49,965 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
t1,,1228833363456
2008-12-09 06:37:49,966 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
t1,,1228833363456
2008-12-09 06:37:50,055 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE',
VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
BLOCKCACHE => 'false'}]}}
2008-12-09 06:37:50,069 DEBUG org.apache.hadoop.hbase.RegionHistorian: Onlined
2008-12-09 06:37:50,080 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region t1,,1228833363456/2133923429
2008-12-09 06:37:50,110 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Next sequence id for region t1,,1228833363456 is 0
2008-12-09 06:37:50,112 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region t1,,1228833363456/2133923429 available
2008-12-09 06:37:50,112 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
for region: t1,,1228833363456
2008-12-09 06:37:50,114 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region t1,,1228833363456
2008-12-09 06:37:50,118 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region t1,,1228833363456 in 0sec
HMaster Server:
java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode)
ulimit -n 32768
2008-12-09 06:35:24,944 INFO org.apache.hadoop.hbase.master.HMaster: Root
region dir: hdfs://h1:54310/hbase/-ROOT-/70236052
2008-12-09 06:35:25,109 INFO org.apache.hadoop.hbase.master.HMaster: BOOTSTRAP:
creating ROOT and first META regions
2008-12-09 06:35:25,156 INFO org.apache.hadoop.hbase.regionserver.HLog: New log
writer created at /hbase/-ROOT-/70236052/log/hlog.dat.1228833325121
2008-12-09 06:35:25,160 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region -ROOT-,,0/70236052
2008-12-09 06:35:25,197 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Next sequence id for region -ROOT-,,0 is 0
2008-12-09 06:35:25,200 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region -ROOT-,,0/70236052 available
2008-12-09 06:35:25,219 INFO org.apache.hadoop.hbase.regionserver.HLog: New log
writer created at /hbase/.META./1028785192/log/hlog.dat.1228833325214
2008-12-09 06:35:25,220 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region .META.,,1/1028785192
2008-12-09 06:35:25,252 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Next sequence id for region .META.,,1 is 0
2008-12-09 06:35:25,256 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region .META.,,1/1028785192 available
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Compactions and cache flushes disabled for region -ROOT-,,0
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Scanners disabled for region -ROOT-,,0
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more active scanners for region -ROOT-,,0
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region -ROOT-,,0
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more row locks outstanding on region -ROOT-,,0
2008-12-09 06:35:25,261 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memcache flush for region -ROOT-,,0. Current region memcache size 445.0
2008-12-09 06:35:25,297 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded
the native-hadoop library
2008-12-09 06:35:25,299 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory:
Successfully loaded & initialized native-zlib library
2008-12-09 06:35:25,301 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new compressor
2008-12-09 06:35:25,555 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new decompressor
2008-12-09 06:35:25,556 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new decompressor
2008-12-09 06:35:25,556 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new decompressor
2008-12-09 06:35:25,556 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new decompressor
2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Added /hbase/-ROOT-/70236052/info/mapfiles/2955877428110101962 with 1 entries,
sequence id 1, data size 445.0, file size 569.0
2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region -ROOT-,,0 in 310ms, sequence id=1,
compaction requested=false
2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 70236052/info
2008-12-09 06:35:25,570 INFO org.apache.hadoop.hbase.regionserver.HRegion:
closed -ROOT-,,0
2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
closing log writer in hdfs://h1:54310/hbase/-ROOT-/70236052/log
2008-12-09 06:35:25,590 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Compactions and cache flushes disabled for region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Scanners disabled for region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more active scanners for region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more row locks outstanding on region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1028785192/historian
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1028785192/info
2008-12-09 06:35:25,591 INFO org.apache.hadoop.hbase.regionserver.HRegion:
closed .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
closing log writer in hdfs://h1:54310/hbase/.META./1028785192/log
2008-12-09 06:35:25,917 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=HMaster, port=60000
2008-12-09 06:35:26,088 INFO org.apache.hadoop.hbase.master.HMaster: HMaster
initialized on 10.131.237.52:60000
2008-12-09 06:35:26,152 INFO org.mortbay.util.Credential: Checking Resource
aliases
2008-12-09 06:35:26,208 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2008-12-09 06:35:26,209 INFO org.mortbay.util.Container: Started
HttpContext[/static,/static]
2008-12-09 06:35:26,209 INFO org.mortbay.util.Container: Started
HttpContext[/logs,/logs]
2008-12-09 06:35:27,051 INFO org.mortbay.util.Container: Started [EMAIL
PROTECTED]
2008-12-09 06:35:27,133 INFO org.mortbay.util.Container: Started
WebApplicationContext[/,/]
2008-12-09 06:35:27,493 INFO org.mortbay.util.Container: Started [EMAIL
PROTECTED]
2008-12-09 06:35:27,495 INFO org.mortbay.util.Container: Started
WebApplicationContext[/api,rest]
2008-12-09 06:35:27,498 INFO org.mortbay.http.SocketListener: Started
SocketListener on 0.0.0.0:60010
2008-12-09 06:35:27,498 INFO org.mortbay.util.Container: Started [EMAIL
PROTECTED]
2008-12-09 06:35:27,498 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2008-12-09 06:35:27,499 INFO org.apache.hadoop.ipc.Server: IPC Server listener
on 60000: starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 60000: starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1
on 60000: starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2
on 60000: starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 60000: starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4
on 60000: starting
2008-12-09 06:35:27,502 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5
on 60000: starting
2008-12-09 06:35:27,505 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6
on 60000: starting
2008-12-09 06:35:27,507 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7
on 60000: starting
2008-12-09 06:35:27,522 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9
on 60000: starting
2008-12-09 06:35:27,522 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8
on 60000: starting
2008-12-09 06:35:27,523 DEBUG org.apache.hadoop.hbase.master.HMaster: Started
service threads
2008-12-09 06:35:27,529 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: 10.131.237.54:60020
2008-12-09 06:35:27,535 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: 10.131.237.51:60020
2008-12-09 06:35:27,538 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: 10.131.237.53:60020
2008-12-09 06:35:27,705 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: 10.131.237.52:60020
2008-12-09 06:35:28,648 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region -ROOT-,,0 to server 10.131.237.54:60020
2008-12-09 06:35:28,651 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 0, Num Servers: 4, Avg Load: 0.0
2008-12-09 06:35:31,673 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: -ROOT-,,0 from 10.131.237.54:60020
2008-12-09 06:35:31,675 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:35:31,769 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian',
BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH
=> '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}},
SERVER => '', STARTCODE => 0
2008-12-09 06:35:31,769 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
Current assignment of .META.,,1 is not valid: serverInfo: null, passed
startCode: 0, storedInfo.startCode: -1, unassignedRegions: false,
pendingRegions: false
2008-12-09 06:35:31,772 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
<>, server: 10.131.237.54:60020} complete
2008-12-09 06:35:34,661 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region .META.,,1 to server 10.131.237.53:60020
2008-12-09 06:35:36,721 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE',
VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
BLOCKCACHE => 'false'}]}}
2008-12-09 06:35:36,730 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10
failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: No
server address listed in -ROOT- for region .META.,,1>. Retrying after sleep of
2000
2008-12-09 06:35:36,734 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE',
VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
BLOCKCACHE => 'false'}]}}
2008-12-09 06:35:37,685 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_PROCESS_OPEN: .META.,,1 from 10.131.237.53:60020
2008-12-09 06:35:37,685 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: .META.,,1 from 10.131.237.53:60020
2008-12-09 06:35:37,689 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: PendingOpenOperation from 10.131.237.53:60020
2008-12-09 06:35:37,691 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: .META.,,1 open on
10.131.237.53:60020
2008-12-09 06:35:37,691 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row .META.,,1 in
region -ROOT-,,0 with startcode 1228833326870 and server 10.131.237.53:60020
2008-12-09 06:35:38,744 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE',
VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
BLOCKCACHE => 'false'}]}}
2008-12-09 06:35:38,754 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName .META.: location server
10.131.237.53:60020, location region name .META.,,1
2008-12-09 06:35:38,754 DEBUG org.apache.hadoop.hbase.RegionHistorian: Onlined
2008-12-09 06:35:38,755 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName .META.: location server
10.131.237.53:60020, location region name .META.,,1
2008-12-09 06:35:38,755 DEBUG
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: Adding to
onlineMetaRegions: {regionname: .META.,,1, startKey: <>, server:
10.131.237.53:60020}
2008-12-09 06:35:38,758 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:35:38,831 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey:
<>, server: 10.131.237.53:60020} complete
2008-12-09 06:35:38,831 INFO org.apache.hadoop.hbase.master.BaseScanner: all
meta regions scanned
2008-12-09 06:35:38,931 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName .META.: location server
10.131.237.53:60020, location region name .META.,,1
2008-12-09 06:35:39,697 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName .META.: location server
10.131.237.53:60020, location region name .META.,,1
2008-12-09 06:35:40,435 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName .META.: location server
10.131.237.53:60020, location region name .META.,,1
2008-12-09 06:35:41,223 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName .META.: location server
10.131.237.53:60020, location region name .META.,,1
2008-12-09 06:35:43,673 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 2, Num Servers: 4, Avg Load: 1.0
2008-12-09 06:35:58,697 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 2, Num Servers: 4, Avg Load: 1.0
2008-12-09 06:36:03,477 INFO org.apache.hadoop.hbase.regionserver.HLog: New log
writer created at /hbase/t1/2133923429/log/hlog.dat.1228833363473
2008-12-09 06:36:03,478 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region t1,,1228833363456/2133923429
2008-12-09 06:36:03,489 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Next sequence id for region t1,,1228833363456 is 0
2008-12-09 06:36:03,491 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region t1,,1228833363456/2133923429 available
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Compactions and cache flushes disabled for region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Scanners disabled for region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more active scanners for region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more row locks outstanding on region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 2133923429/f1
2008-12-09 06:36:03,495 INFO org.apache.hadoop.hbase.regionserver.HRegion:
closed t1,,1228833363456
2008-12-09 06:36:03,495 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
closing log writer in hdfs://h1:54310/hbase/t1/2133923429/log
2008-12-09 06:36:03,521 INFO org.apache.hadoop.hbase.master.HMaster: created
table t1
2008-12-09 06:36:04,705 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region t1,,1228833363456 to server 10.131.237.51:60020
2008-12-09 06:36:07,726 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: t1,,1228833363456 from 10.131.237.51:60020
2008-12-09 06:36:07,727 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: PendingOpenOperation from 10.131.237.51:60020
2008-12-09 06:36:07,727 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: t1,,1228833363456 open on
10.131.237.51:60020
2008-12-09 06:36:07,727 DEBUG
org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1,
onlineMetaRegions.size(): 1
2008-12-09 06:36:07,727 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
t1,,1228833363456 in region .META.,,1 with startcode 1228833326838 and server
10.131.237.51:60020
2008-12-09 06:36:11,335 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName .META.: location server
10.131.237.53:60020, location region name .META.,,1
2008-12-09 06:36:13,735 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 4, Avg Load: 1.0
2008-12-09 06:36:24,695 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName .META.: location server
10.131.237.53:60020, location region name .META.,,1
2008-12-09 06:36:28,756 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 4, Avg Load: 1.0
2008-12-09 06:36:31,782 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:36:31,815 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian',
BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH
=> '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}},
SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:36:31,817 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
<>, server: 10.131.237.54:60020} complete
2008-12-09 06:36:38,766 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:36:38,773 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY =>
'', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT =>
'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER =>
'10.131.237.51:60020', STARTCODE => 1228833326838
2008-12-09 06:36:38,775 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey:
<>, server: 10.131.237.53:60020} complete
2008-12-09 06:36:38,775 INFO org.apache.hadoop.hbase.master.BaseScanner: all
meta regions scanned
2008-12-09 06:36:43,777 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 4, Avg Load: 1.0
2008-12-09 06:36:56,635 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:36:58,801 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 4, Avg Load: 1.0
2008-12-09 06:37:13,820 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 4, Avg Load: 1.0
2008-12-09 06:37:26,769 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:37:28,841 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 4, Avg Load: 1.0
2008-12-09 06:37:31,792 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:37:31,835 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian',
BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH
=> '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}},
SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:37:31,837 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
<>, server: 10.131.237.54:60020} complete
2008-12-09 06:37:38,776 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:37:38,786 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY =>
'', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT =>
'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER =>
'10.131.237.51:60020', STARTCODE => 1228833326838
2008-12-09 06:37:38,789 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey:
<>, server: 10.131.237.53:60020} complete
2008-12-09 06:37:38,789 INFO org.apache.hadoop.hbase.master.BaseScanner: all
meta regions scanned
2008-12-09 06:37:43,860 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 4, Avg Load: 1.0
2008-12-09 06:37:47,095 INFO org.apache.hadoop.hbase.master.ServerManager:
Cancelling lease for 10.131.237.51:60020
2008-12-09 06:37:47,095 INFO org.apache.hadoop.hbase.master.ServerManager:
Region server 10.131.237.51:60020: MSG_REPORT_EXITING -- lease cancelled
2008-12-09 06:37:47,096 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE: t1,,1228833363456 from 10.131.237.51:60020
2008-12-09 06:37:49,949 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region t1,,1228833363456 to server 10.131.237.52:60020
2008-12-09 06:37:52,974 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_PROCESS_OPEN: t1,,1228833363456 from 10.131.237.52:60020
2008-12-09 06:37:52,974 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: t1,,1228833363456 from 10.131.237.52:60020
2008-12-09 06:37:52,975 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: PendingOpenOperation from 10.131.237.52:60020
2008-12-09 06:37:52,975 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: t1,,1228833363456 open on
10.131.237.52:60020
2008-12-09 06:37:52,975 DEBUG
org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1,
onlineMetaRegions.size(): 1
2008-12-09 06:37:52,975 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
t1,,1228833363456 in region .META.,,1 with startcode 1228833327601 and server
10.131.237.52:60020
2008-12-09 06:37:56,912 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:37:58,877 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:38:13,894 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:38:27,035 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:38:28,914 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:38:31,802 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:38:31,833 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian',
BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH
=> '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}},
SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:38:31,835 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
<>, server: 10.131.237.54:60020} complete
2008-12-09 06:38:38,787 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:38:38,792 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY =>
'', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT =>
'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER =>
'10.131.237.52:60020', STARTCODE => 1228833327601
2008-12-09 06:38:38,794 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey:
<>, server: 10.131.237.53:60020} complete
2008-12-09 06:38:38,794 INFO org.apache.hadoop.hbase.master.BaseScanner: all
meta regions scanned
2008-12-09 06:38:43,930 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:38:57,168 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:38:58,950 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:39:13,970 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:39:27,310 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:39:28,996 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:39:31,813 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:39:31,844 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian',
BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH
=> '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}},
SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:39:31,846 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
<>, server: 10.131.237.54:60020} complete
2008-12-09 06:39:38,797 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:39:38,806 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY =>
'', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT =>
'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER =>
'10.131.237.52:60020', STARTCODE => 1228833327601
2008-12-09 06:39:38,808 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey:
<>, server: 10.131.237.53:60020} complete
2008-12-09 06:39:38,808 INFO org.apache.hadoop.hbase.master.BaseScanner: all
meta regions scanned
2008-12-09 06:39:44,016 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:39:57,462 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:39:58,440 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:39:59,036 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:40:14,056 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:40:28,547 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:40:29,075 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:40:31,823 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:40:31,849 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian',
BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH
=> '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}},
SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:40:31,851 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
<>, server: 10.131.237.54:60020} complete
2008-12-09 06:40:38,808 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:40:38,812 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME => 't1,,1228833363456', STARTKEY =>
'', ENDKEY => '', ENCODED => 2133923429, TABLE => {{NAME => 't1', IS_ROOT =>
'false', IS_META => 'false', FAMILIES => [{NAME => 'f1', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL =>
'-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER =>
'10.131.237.52:60020', STARTCODE => 1228833327601
2008-12-09 06:40:38,814 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey:
<>, server: 10.131.237.53:60020} complete
2008-12-09 06:40:38,814 INFO org.apache.hadoop.hbase.master.BaseScanner: all
meta regions scanned
2008-12-09 06:40:44,096 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
2008-12-09 06:40:58,661 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName t1: location server
10.131.237.51:60020, location region name t1,,1228833363456
2008-12-09 06:40:59,115 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 3, Num Servers: 3, Avg Load: 1.0
