[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510798#comment-13510798 ]
Sergey Shelukhin commented on HBASE-7268: ----------------------------------------- ok, I got repro... will attach patch after cleanup of debug logging/etc. I'd prefer to have TS in meta but this is a simpler fix for now. The logging with patch looks like this: {code} 2012-12-05 12:06:08,285 DEBUG [Thread-521] util.ChaosMonkey$Action(203): Removing 13 regions from 10.10.11.17,53406,1354737903944 ... 2012-12-05 12:06:08,765 INFO [am-zkevent-worker-pool-2-thread-2] master.RegionStates(249): Region {NAME => 'IntegrationTestRebalanceAndKillServersTargeted,7ffffff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.', STARTKEY => '7ffffff8', ENDKEY => '8cccccc4', ENCODED => 89483778064d05b1f2e1c0d20bcabc16,} transitioned from {IntegrationTestRebalanceAndKillServersTargeted,7ffffff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16. state=PENDING_OPEN, ts=1354737968742, server=10.10.11.17,53407,1354737903960} to {IntegrationTestRebalanceAndKillServersTargeted,7ffffff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16. state=OPENING, ts=1354737968765, server=10.10.11.17,53407,1354737903960} ... 2012-12-05 12:06:10,549 INFO [Thread-521] util.ChaosMonkey$Action(179): Killing region server:10.10.11.17,53407,1354737903960 ... 2012-12-05 12:06:39,233 INFO [am-zkevent-worker-pool-2-thread-2] master.RegionStates(249): Region {NAME => 'IntegrationTestRebalanceAndKillServersTargeted,7ffffff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.', STARTKEY => '7ffffff8', ENDKEY => '8cccccc4', ENCODED => 89483778064d05b1f2e1c0d20bcabc16,} transitioned from {IntegrationTestRebalanceAndKillServersTargeted,7ffffff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16. state=OPENING, ts=1354737999228, server=10.10.11.17,53404,1354737903902} to {IntegrationTestRebalanceAndKillServersTargeted,7ffffff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16. state=OPEN, ts=1354737999232, server=10.10.11.17,53404,1354737903902} ... 2012-12-05 12:06:40,276 INFO [HBaseWriterThread_4] client.HConnectionManager$HConnectionImplementation(1776): Received an error from 10.10.11.17:53407 for region IntegrationTestRebalanceAndKillServersTargeted,7ffffff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.; not removing 10.10.11.17:53404 from cache. ... 2012-12-05 12:06:40,381 INFO [HBaseWriterThread_15] client.HConnectionManager$HConnectionImplementation(1809): Region IntegrationTestRebalanceAndKillServersTargeted,7ffffff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16. moved to 10.10.11.17:53407 according to 10.10.11.17:53406 2012-12-05 12:06:40,381 DEBUG [HBaseWriterThread_15] client.HConnectionManager$HConnectionImplementation(1342): Ignoring stale location update for IntegrationTestRebalanceAndKillServersTargeted,7ffffff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.: 10.10.11.17:53407 at 1354737968725; local 10.10.11.17:53404 at 1354738000265 {code} > correct local region location cache information can be overwritten w/stale > information from an old server > --------------------------------------------------------------------------------------------------------- > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug > Affects Versions: 0.96.0 > Reporter: Sergey Shelukhin > Assignee: Sergey Shelukhin > Priority: Minor > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira