[
https://issues.apache.org/jira/browse/HADOOP-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HADOOP-2251:
--------------------------
Component/s: contrib/hbase
Looks like just saw another incidence of this internally but hard to tell
because DEBUG log level not enabled.... master wouldn't go down... was
complaining that it couldn't talk to the server w/ ROOT:
{code}
2007-11-22 05:02:43,504 INFO org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner scanning meta region regionname: -ROOT-,,0, startKey: <>,
server: 208.76.45.216:60020}
2007-11-22 05:02:47,346 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scanning meta region regionname: .META.,,1, startKey: <>,
server: 208.76.45.131:60020}
2007-11-22 05:02:47,532 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scan of meta region regionname: .META.,,1, startKey: <>,
server: 208.76.45.131:60020} complete
2007-11-22 05:02:47,532 INFO org.apache.hadoop.hbase.HMaster: all meta regions
scanned
2007-11-22 05:03:43,508 WARN org.apache.hadoop.hbase.HMaster: Scan ROOT region
java.net.SocketTimeoutException: timed out waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:484)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
at $Proxy2.openScanner(Unknown Source)
at
org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(HMaster.java:216)
at
org.apache.hadoop.hbase.HMaster$RootScanner.scanRoot(HMaster.java:505)
at
org.apache.hadoop.hbase.HMaster$RootScanner.maintenanceScan(HMaster.java:548)
at org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java:197)
at org.apache.hadoop.hbase.Chore.run(Chore.java:58)
2007-11-22 05:03:47,347 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scanning meta region regionname: .META.,,1, startKey: <>,
server: 208.76.45.131:60020}
2007-11-22 05:03:47,530 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scan of meta region regionname: .META.,,1, startKey: <>,
server: 208.76.45.131:60020} complete
2007-11-22 05:03:47,530 INFO org.apache.hadoop.hbase.HMaster: all meta regions
scanned
2007-11-22 05:03:53,511 INFO org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner scanning meta region regionname: -ROOT-,,0, startKey: <>,
server: 208.76.45.216:60020}
2007-11-22 05:04:47,349 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scanning meta region regionname: .META.,,1, startKey: <>,
server: 208.76.45.131:60020}
2007-11-22 05:04:47,534 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scan of meta region regionname: .META.,,1, startKey: <>,
server: 208.76.45.131:60020} complete
2007-11-22 05:04:47,534 INFO org.apache.hadoop.hbase.HMaster: all meta regions
scanned
2007-11-22 05:04:53,512 ERROR org.apache.hadoop.hbase.HMaster: Scan ROOT region
java.net.SocketTimeoutException: timed out waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:484)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
at $Proxy2.openScanner(Unknown Source)
at
org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(HMaster.java:216)
at
org.apache.hadoop.hbase.HMaster$RootScanner.scanRoot(HMaster.java:505)
at
org.apache.hadoop.hbase.HMaster$RootScanner.maintenanceScan(HMaster.java:548)
at org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java:197)
at org.apache.hadoop.hbase.Chore.run(Chore.java:58)
2007-11-22 05:05:03,514 INFO org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner scanning meta region regionname: -ROOT-,,0, startKey: <>,
server: 208.76.45.216:60020}
2007-11-22 05:05:47,351 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scanning meta region regionname: .META.,,1, startKey: <>,
server: 208.76.45.131:60020}
2007-11-22 05:05:47,531 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scan of meta region regionname: .META.,,1, startKey: <>,
server: 208.76.45.131:60020} complete
2007-11-22 05:05:47,531 INFO org.apache.hadoop.hbase.HMaster: all meta regions
scanned
2007-11-22 05:06:03,515 ERROR org.apache.hadoop.hbase.HMaster: Scan ROOT region
java.net.SocketTimeoutException: timed out waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:484)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
at $Proxy2.openScanner(Unknown Source)
at
org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(HMaster.java:216)
at
org.apache.hadoop.hbase.HMaster$RootScanner.scanRoot(HMaster.java:505)
at
org.apache.hadoop.hbase.HMaster$RootScanner.maintenanceScan(HMaster.java:548)
at org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java:197)
at org.apache.hadoop.hbase.Chore.run(Chore.java:58)
....
{code}
..and so on for ever. Why ain't we noticing the ROOT has gone bad?
> [hbase] master won't go down if root region was not found
> ---------------------------------------------------------
>
> Key: HADOOP-2251
> URL: https://issues.apache.org/jira/browse/HADOOP-2251
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: stack
> Priority: Minor
>
> psaab couldn't shut down his master... was getting reams of the below:
> {code}
> 2007-11-21 13:45:30,421 INFO hbase.HMaster - process shutdown of server
> 38.99.76.15:60020: logSplit: true, rootChecked: true, rootRescanned: false,
> numberOfMetaRegions: 0, onlineMetaRegions.size(): 0
> 2007-11-21 13:45:30,421 DEBUG hbase.HMaster - process server shutdown
> scanning root region cancelled because rootRegionLocation is null
> 2007-11-21 13:45:30,421 DEBUG hbase.HMaster - Put PendingServerShutdown of
> 38.99.76.15:60020 back on queue
> 2007-11-21 13:45:30,421 DEBUG hbase.HMaster - Main processing loop:
> PendingServerShutdown of 38.99.76.21:60020
> 2007-11-21 13:45:30,422 INFO hbase.HMaster - process shutdown of server
> 38.99.76.21:60020: logSplit: true, rootChecked: true, rootRescanned: false,
> numberOfMetaRegions: 0, onlineMetaRegions.size(): 0
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - process server shutdown
> scanning root region cancelled because rootRegionLocation is null
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - Put PendingServerShutdown of
> 38.99.76.21:60020 back on queue
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - Main processing loop:
> PendingServerShutdown of 38.99.76.31:60020
> 2007-11-21 13:45:30,422 INFO hbase.HMaster - process shutdown of server
> 38.99.76.31:60020: logSplit: true, rootChecked: true, rootRescanned: false,
> numberOfMetaRegions: 0, onlineMetaRegions.size(): 0
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - process server shutdown
> scanning root region cancelled because rootRegionLocation is null
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - Put PendingServerShutdown of
> 38.99.76.31:60020 back on queue
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - Main processing loop:
> PendingServerShutdown of 38.99.76.17:60020
> ..
> {code}
> Looks like a shutdown soon after startup so should be reproducible.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
