Using 0.20.0 version of HBase, I frequently face the exception
"NotServingRegionException" caused by the .META. table.
After searching for the log, I found out the following log, which looks very
suspicious.The log is attached to this mail within a text file named
"piece_of_log.txt".
Why would the .META. region be closed?
Any idea or suggestion, please?
FYI, I`m running HBase 0.20.0 on 4 machines which has Intel Quad Core CPU
and each has 5GB of Physical Memory.
HBase is running with 1536MB size of heap. The network bandwidth between the
node is 1Gb.
My HBase cluster has almost 1000 regions, so assigned 250 to each
regionserver, using lzo compression.
Sincerely,
Woosuk Suh.
P.S :: Spam score error makes me unable to attach log to the mail directly
:(
============== Log from RegionServer ==================
2009-10-08 12:40:15,294 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/hadoop02.in.cloudinsight.net,60020,1254972065689/hlog.dat.1254972878222,
entries=2036, calcsize=63866852, fil
esize=63669928. New hlog
/hbase/.logs/hadoop02.in.cloudinsight.net,60020,1254972065689/hlog.dat.1254973215262
2009-10-08 12:45:46,182 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/hadoop02.in.cloudinsight.net,60020,1254972065689/hlog.dat.1254973215262,
entries=2394, calcsize=63765842, fil
esize=63534513. New hlog
/hbase/.logs/hadoop02.in.cloudinsight.net,60020,1254972065689/hlog.dat.1254973546158
2009-10-08 12:53:59,947 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP
2009-10-08 12:53:59,991 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader:
Loaded native gpl library
2009-10-08 12:54:00,019 INFO com.hadoop.compression.lzo.LzoCodec: Successfully
loaded & initialized native-lzo library
2009-10-08 12:54:00,019 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new compressor
(As you may understand, this line is not from log!) All of a sudden, the region
starts to be closed.
2009-10-08 12:54:00,035 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed pages,http:\x2F\x2Fdestin4disaster.deviantart.com\x2F,1253266574365
....(Many lines of Logs with Closed pages...).....
2009-10-08 12:54:02,363 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed
pages,http:\x2F\x2Fbeautiful-leipzig-pre-teens.teen-model-24.com\x2F,1254861622977
2009-10-08 12:54:02,387 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed .META.,,1
2009-10-08 12:54:02,439 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed pages,http:\x2F\x2Falien789.deviantart.com\x2F,1253980691740
....(Many lines of Logs with Closed pages...).....
2009-10-08 12:54:02,479 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1752)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-08 12:54:02,495 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed pages,http:\x2F\x2Fhellasproducts.net\x2F,1253267001655
2009-10-08 12:54:02,523 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 24 on 60020, call getClosestRowBefore([...@5f524524, [...@12396f60,
[...@78bf10f8) from 10.10.2.201:59470: error: org.apac
he.hadoop.hbase.NotServingRegionException: .META.,,1
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1752)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-08 12:54:02,547 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed pages,http:\x2F\x2Fclub-young.privateamateurcam.com\x2F,1253119982057
....(Many lines of Logs with Closed pages...).....
2009-10-08 12:54:03,187 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed pages,http:\x2F\x2Fbild-kontakte-marl.fuck-videoz.com\x2F,1253256117743
2009-10-08 12:54:03,199 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1877)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
========================== Log of Master ===================================
2009-10-08 12:54:01,244 INFO org.apache.hadoop.hbase.master.ServerManager: 4
region servers, 0 dead, average load 209.2
2009-10-08 12:54:03,172 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 10.
10.2.204:60020, regionname: -ROOT-,,0, startKey: <>}
2009-10-08 12:54:03,176 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {
server: 10.10.2.204:60020, regionname: -ROOT-,,0, startKey: <>} complete
2009-10-08 12:54:03,184 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {server: 10.
10.2.202:60020, regionname: .META.,,1, startKey: <>}
2009-10-08 12:54:03,184 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan
one META region: {server: 10.10.2.202:60020, regionnam
e: .META.,,1, startKey: <>}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1877)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:186)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2009-10-08 12:54:03,188 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1
.META. region(s) scanned
2009-10-08 12:54:08,384 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: hadoop02.in.cloudinsight.net
,60020,1254974048391
2009-10-08 12:54:13,752 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: hadoop03.local,60020,1254974
053761
2009-10-08 12:54:14,628 INFO org.apache.hadoop.hbase.master.RegionManager:
Skipped 0 region(s) that are in transition states
2009-10-08 12:54:17,688 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE: pages,http:\x2F\x2Fwww.hrok.
hr\x2F,1254861504185 from hadoop04.in.cloudinsight.net,60020,1254972065466; 1
of 10
2009-10-08 12:54:17,708 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE: pages,http:\x2F\x2Fthaigsdcl
ub.org\x2F,1253269091063 from hadoop04.in.cloudinsight.net,60020,1254972065466;
2 of 10
2009-10-08 12:54:17,708 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE: pages,http:\x2F\x2Fleftofcol
legestation.com\x2F,1253107991710 from
hadoop04.in.cloudinsight.net,60020,1254972065466; 3 of 10
2009-10-08 12:54:17,708 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: region set as unassigned:
pages,http:\x2F\x2Fwww.
hrok.hr\x2F,1254861504185
2009-10-08 12:54:17,708 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE: pages,http:\x2F\x2Faesthetic
economy.com\x2F,1253189834309 from
hadoop04.in.cloudinsight.net,60020,1254972065466; 4 of 10
2009-10-08 12:54:17,708 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: region set as unassigned:
pages,http:\x2F\x2Fthai
gsdclub.org\x2F,1253269091063
2009-10-08 12:54:17,708 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE: pages,http:\x2F\x2Fwww.bbns.
org\x2F,1253878122792 from hadoop04.in.cloudinsight.net,60020,1254972065466; 5
of 10
2009-10-08 12:54:17,708 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: region set as unassigned:
pages,http:\x2F\x2Fleft
ofcollegestation.com\x2F,1253107991710
2009-10-08 12:54:17,708 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE: pages,http:\x2F\x2Frateitall
.com\x2F,1254861414711 from hadoop04.in.cloudinsight.net,60020,1254972065466; 6
of 10
2009-10-08 12:54:17,708 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: region set as unassigned:
pages,http:\x2F\x2Faest
heticeconomy.com\x2F,1253189834309
2009-10-08 12:54:17,708 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_CLOSE: pages,http:\x2F\x2Fmegabrues
te.asianperls.com\x2F,1253173694617 from
hadoop04.in.cloudinsight.net,60020,1254972065466; 7 of 10
2009-10-08 12:54:17,708 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: region set as unassigned:
pages,http:\x2F\x2Fwww.
bbns.org\x2F,1253878122792
