[
https://issues.apache.org/jira/browse/HADOOP-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560620#action_12560620
]
Billy Pearson commented on HADOOP-2662:
---------------------------------------
The server with the above WARN was 10.0.0.4
{code}
2008-01-18 17:39:37,239 INFO org.apache.hadoop.hbase.HMaster: Cluster shutdown
requested. Starting to quiesce servers
2008-01-18 17:39:54,249 INFO org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>,
server: 10.0.0.4:60020}
2008-01-18 17:39:54,264 DEBUG org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner regioninfo: {regionname: .META.,,1, startKey: <>, endKey:
<>, encodedName(1028785192) tableDesc: {name: .META., families: {info:={name:
info, max versions: 1, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}}}}, server: 10.0.0.3:60020, startCode:
1200691304598
2008-01-18 17:39:54,266 INFO org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>,
server: 10.0.0.4:60020} complete
2008-01-18 17:39:59,102 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>,
server: 10.0.0.3:60020}
2008-01-18 17:39:59,132 DEBUG org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner regioninfo: {regionname: webdata,,1200625220917, startKey:
<>, endKey: <au.org.acl.www/national/browse.stw?article_id=18842:http>,
encodedName(1954097628) tableDesc: {name: webdata, families: {anchor:={name:
anchor, max versions: 1, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, first_seen:={name: first_seen, max versions:
1, compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, in_rank:={name: in_rank, max versions: 1, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}, last_seen:={name:
last_seen, max versions: 1, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, rank_total:={name: rank_total, max versions:
1, compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, rank_visual:={name: rank_visual, max versions: 1, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, size:={name: size,
max versions: 1, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, source2:={name: source2, max versions: 1, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none},
source:={name: source, max versions: 1, compression: NONE, in memory: false,
max length: 2147483647, bloom filter: none}, stime:={name: stime, max versions:
1, compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}}}}, server: 10.0.0.4:60020, startCode: 1200691304484
..
remove lot of regioninfo: from meta scan
..
2008-01-18 17:39:59,278 DEBUG org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner regioninfo: {regionname:
webdata,us.ashlee/component/option,com_weblinks/Itemid,4/:http,1200695516443,
startKey: <us.ashlee/component/option,com_weblinks/Itemid,4/:http>, endKey: <>,
encodedName(63818931) tableDesc: {name: webdata, families: {anchor:={name:
anchor, max versions: 1, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, first_seen:={name: first_seen, max versions:
1, compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, in_rank:={name: in_rank, max versions: 1, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}, last_seen:={name:
last_seen, max versions: 1, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, rank_total:={name: rank_total, max versions:
1, compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, rank_visual:={name: rank_visual, max versions: 1, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, size:={name: size,
max versions: 1, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, source2:={name: source2, max versions: 1, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none},
source:={name: source, max versions: 1, compression: NONE, in memory: false,
max length: 2147483647, bloom filter: none}, stime:={name: stime, max versions:
1, compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}}}}, server: 10.0.0.2:60020, startCode: 1200691304075
2008-01-18 17:39:59,284 DEBUG org.apache.hadoop.hbase.HMaster: Checked
webdata,com.espion-on-line.www/dossiers.htm:http,1200697196799 for references:
splitA: true, splitB: true
2008-01-18 17:39:59,289 DEBUG org.apache.hadoop.hbase.HMaster: Checked
webdata,com.diagonale-chute-libre.www/html/voyages.htm:http,1200695228115 for
references: splitA: true, splitB: true
2008-01-18 17:39:59,294 DEBUG org.apache.hadoop.hbase.HMaster: Checked
webdata,ca.supersites.compatibleintroductions/questionnaire/:http,1200618005435
for references: splitA: true, splitB: true
2008-01-18 17:39:59,313 DEBUG org.apache.hadoop.hbase.HMaster:
webdata,com.eddiebonfigli.www/index2a.html:http,1200697196798 no longer has
references to webdata,com.eddiebonfigli.www/index2a.html:http,1200617086404
2008-01-18 17:39:59,313 INFO org.apache.hadoop.hbase.HMaster: Deleting region
webdata,com.eddiebonfigli.www/index2a.html:http,1200617086404 because daughter
splits no longer hold references
2008-01-18 17:39:59,314 DEBUG org.apache.hadoop.hbase.HRegion: DELETING region
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/2101627985
2008-01-18 17:39:59,925 DEBUG org.apache.hadoop.hbase.HMaster: Checked
webdata,de.disc-fix/parking.php4?task=search&domain=disc-fix.de&s=5429b889900dd49c8824&conf_id=0&a_id=2&language=en&pgt=1VDyC9X12y4KEwirgdSI1OKQAhUJQWgKHboh_ggYACAAMIrwoAM4DQ&ags=2leZCbd6jukKEwirgdSI1OKQAhUJQWgKHboh_ggYAyAAMIrwoAM4DQ&pxy_t=254&start=1&add_c=5b1303ed4714dfb6e9a1bf98bc801db7&keyword=Internet&pos=6&sub=1:http,1200697045811
for references: splitA: true, splitB: true
2008-01-18 17:39:59,931 DEBUG org.apache.hadoop.hbase.HMaster: Checked
webdata,com.americas-fair.www/newsletter.php:http,1200697093033 for references:
splitA: true, splitB: true
2008-01-18 17:39:59,935 DEBUG org.apache.hadoop.hbase.HMaster: Checked
webdata,com.flickr.www/photos/tigerfur:http,1200617086405 for references:
splitA: true, splitB: true
2008-01-18 17:39:59,946 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>,
server: 10.0.0.3:60020} complete
2008-01-18 17:39:59,946 INFO org.apache.hadoop.hbase.HMaster: all meta regions
scanned
2008-01-18 17:40:19,243 INFO org.apache.hadoop.hbase.Leases:
HMaster.leaseChecker lease expired 1481598689/1481598689
2008-01-18 17:40:19,243 INFO org.apache.hadoop.hbase.HMaster: 10.0.0.2:60020
lease expired
2008-01-18 17:40:21,467 INFO org.apache.hadoop.hbase.Leases:
HMaster.leaseChecker lease expired -1038361245/-1038361245
2008-01-18 17:40:21,493 INFO org.apache.hadoop.hbase.HMaster: 10.0.0.4:60020
lease expired
2008-01-18 17:40:21,734 INFO org.apache.hadoop.hbase.Leases:
HMaster.leaseChecker lease expired -1925864926/-1925864926
2008-01-18 17:40:21,734 INFO org.apache.hadoop.hbase.HMaster: 10.0.0.3:60020
lease expired
2008-01-18 17:40:21,735 INFO org.apache.hadoop.hbase.Leases:
HMaster.leaseChecker lease expired 594095008/594095008
2008-01-18 17:40:21,735 INFO org.apache.hadoop.hbase.HMaster: 10.0.0.1:60020
lease expired
2008-01-18 17:40:22,278 DEBUG org.apache.hadoop.hbase.HMaster: telling root
scanner to stop
2008-01-18 17:40:22,278 DEBUG org.apache.hadoop.hbase.HMaster: telling meta
scanner to stop
2008-01-18 17:40:22,278 DEBUG org.apache.hadoop.hbase.HMaster: meta and root
scanners notified
2008-01-18 17:40:22,279 DEBUG org.apache.hadoop.hbase.HMaster: telling root
scanner to stop
2008-01-18 17:40:22,279 INFO org.apache.hadoop.hbase.HMaster$RootScanner:
HMaster.rootScanner exiting
2008-01-18 17:40:22,279 INFO org.apache.hadoop.hbase.HMaster$MetaScanner:
HMaster.metaScanner exiting
2008-01-18 17:40:22,279 DEBUG org.apache.hadoop.hbase.HMaster: telling meta
scanner to stop
2008-01-18 17:40:22,280 DEBUG org.apache.hadoop.hbase.HMaster: meta and root
scanners notified
2008-01-18 17:40:22,280 INFO org.apache.hadoop.hbase.HMaster: Stopping
infoServer
2008-01-18 17:40:22,620 INFO org.mortbay.util.ThreadedServer: Stopping Acceptor
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60010]
2008-01-18 17:40:22,874 INFO org.mortbay.http.SocketListener: Stopped
SocketListener on 0.0.0.0:60010
{code}
> master stopping before regions finish stopping
> ----------------------------------------------
>
> Key: HADOOP-2662
> URL: https://issues.apache.org/jira/browse/HADOOP-2662
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: Billy Pearson
> Priority: Minor
> Fix For: 0.16.1
>
>
> I get this on my region servers logs sometimes when I shutdown the cluster it
> repeats several times sometimes trying to find the master
> I thank we need to look at the master and make sure we do not stop the master
> on exit before all region servers report down.
> I am not sure if there could be data loss or not but we should not leave
> region servers looking for the master unless it has failed on it own.
> {code}
> 2008-01-18 17:40:42,009 WARN org.apache.hadoop.hbase.HRegionServer: Failed to
> send exiting message to master:
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:159)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:575)
> at org.apache.hadoop.ipc.Client.call(Client.java:498)
> at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:210)
> at $Proxy0.regionServerReport(Unknown Source)
> at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:898)
> at java.lang.Thread.run(Thread.java:595)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
