[
https://issues.apache.org/jira/browse/HBASE-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763270#action_12763270
]
stack commented on HBASE-1894:
------------------------------
Good stuff elsif. I wonder if this is one of the double assignments addressed
in 0.20.1?
Grep 1209236691 in your master. Its the encoded region name. Once you have
its actual region name. Grep on that. Look for assignments after its already
been assigned.
> Region server references deleted compaction log
> -----------------------------------------------
>
> Key: HBASE-1894
> URL: https://issues.apache.org/jira/browse/HBASE-1894
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Environment: HBase Version 0.20.0, r805538
> HBase Compiled Wed Aug 19 14:38:36 PDT 2009, root
> Hadoop Version 0.20.0-plus4681, r767961
> Hadoop Compiled Mon Jun 8 18:18:06 UTC 2009, stack
> Reporter: elsif
>
> The region server appears to reference blocks from deleted log files after
> compaction. Stopping the affected region server causes the region to be
> reassigned and appears to function properly again.
> We have seen two instances this week where this caused the region server to
> get stuck with the following error:
> hbase-root-regionserver-hfs-030015.log:2009-10-07 12:24:41,853 INFO
> org.apache.hadoop.hdfs.DFSClient: Could not obtain block
> blk_6092375271544310747_250240 from any node: java.io.IOException: No live
> nodes contain current block
> .
> .
> .
> /opt/hbase/logs/hbase-root-regionserver-hfs-030015.log:java.io.IOException:
> Cannot open filename /hbase/fc_test/1209236691/json/9131415140626575165
> /opt/hbase/logs/hbase-root-regionserver-hfs-030015.log:2009-10-07
> 13:09:47,343 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block
> blk_6092375271544310747_250240 from any node: java.io.IOException: No live
> nodes contain current block
> From the hdfs log we can see that the block in question was part of a
> compaction log:
> hadoop-root-namenode-hfs-030010.log:2009-10-07 08:48:31,271 INFO
> org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock:
> /hbase/fc_test/compaction.dir/1209236691/2099727166320402414.
> blk_6092375271544310747_250240
> hadoop-root-namenode-hfs-030010.log:2009-10-07 08:48:31,337 INFO
> org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock:
> blockMap updated: 10.4.30.39:50010 is added to blk_6092375271544310747_250240
> size 360813
> hadoop-root-namenode-hfs-030010.log:2009-10-07 08:48:31,338 INFO
> org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock:
> blockMap updated: 10.4.30.23:50010 is added to blk_6092375271544310747_250240
> size 360813
> hadoop-root-namenode-hfs-030010.log:2009-10-07 08:48:31,339 INFO
> org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock:
> blockMap updated: 10.4.30.16:50010 is added to blk_6092375271544310747_250240
> size 360813
> From the same log we see the block deleted:
> hadoop-root-namenode-hfs-030010.log:2009-10-07 12:13:20,738 INFO
> org.apache.hadoop.hdfs.StateChange: BLOCK* ask 10.4.30.39:50010 to delete
> blk_-8366798629992987785_250239 blk_-278212026264018289_224708
> blk_4131756856877489141_230749 blk_6092375271544310747_250240
> blk_-1303356519220271320_144130
> hadoop-root-namenode-hfs-030010.log:2009-10-07 12:13:26,739 INFO
> org.apache.hadoop.hdfs.StateChange: BLOCK* ask 10.4.30.23:50010 to delete
> blk_3405064566022477311_135928 blk_-7806500863137897827_139699
> blk_6092375271544310747_250240
> hadoop-root-namenode-hfs-030010.log:2009-10-07 12:13:29,739 INFO
> org.apache.hadoop.hdfs.StateChange: BLOCK* ask 10.4.30.16:50010 to delete
> blk_-3938002954120963098_139149 blk_2284074818158875873_144134
> blk_519099260905493191_222718 blk_7684159545746656462_144137
> blk_7857071994919747100_144127 blk_6311161030310597085_135919
> blk_-8366798629992987785_250239 blk_5638304438869005376_142347
> blk_-1019479064035180992_143997 blk_4131756856877489141_230749
> blk_6092375271544310747_250240 blk_-1520990752128999182_192595
> blk_1714900485284856973_230761 blk_-1303356519220271320_144130
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
