[ https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598118#comment-14598118 ]
Hudson commented on HBASE-13938: -------------------------------- FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #988 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/988/]) HBASE-13938 Deletes done during the region merge transaction may get eclipsed (ddas, enis, ndimiduk) (ndimiduk: rev 605fd4f33f6284ee818359911ddbea77111efb27) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeRequest.java * hbase-server/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-protocol/src/main/protobuf/Admin.proto * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/VisibilityLabelsProtos.java * hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java > Deletes done during the region merge transaction may get eclipsed > ----------------------------------------------------------------- > > Key: HBASE-13938 > URL: https://issues.apache.org/jira/browse/HBASE-13938 > Project: HBase > Issue Type: Bug > Components: master, regionserver > Reporter: Devaraj Das > Assignee: Enis Soztutar > Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 > > Attachments: 13938-branch-1.1.txt, hbase-13938_master.patch, > hbase-13938_v2-branch-1.1.patch, hbase-13938_v3-0.98.patch, > hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch, > hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch, > hbase-13938_v3-branch-1.patch > > > Was looking at an issue from our internal testing. It seems the Deletes of > the region rows from the meta done during the merge transaction could be > eclipsed by the Put of a region row that might have happened moments before. > The master logs this for the merge: > {noformat} > 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12] > master.AssignmentManager: Handled MERGED event; > merged=IntegrationTestIngest,a666665c,1434633226681.0927319db6bf5e128e3bec2a420819aa., > > region_a=IntegrationTestIngest,a666665c,1434631353820.8b911862d7705ac808b8d132d0154c16., > > region_b=IntegrationTestIngest,acccccc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec., > on ddas-2-5.openstacklocal,16020,1434632778438 > {noformat} > One of the regions that got merged got Opened a few seconds back: > {noformat} > 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1] > regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next > sequenceid=182988 > {noformat} > The above would have done a Put in the meta. > Looking at the raw scan of the meta, for the new merged region, the creation > timestamp is 1434633226101: > {noformat} > > IntegrationTestIngest,a666665c,1434633226681.0927319db6bf5e128e3bec2a420819aa. > column=info:regioninfo, timestamp=1434633226101, value={ENCODED => > 0927319db6bf5e128e3bec2a420819aa, NAME => > 'IntegrationTestIngest,a666665c,1434633226681.0927319db6bf5e128e3bec2a420819aa.', > STARTKEY => 'a666665c', ENDKEY => 'b3333328'} > {noformat} > Looking at the raw scan of the meta, the timestamp for the region open of the > already merged region is 1434633226600. This is a little after the merge > transaction's timestamp. > {noformat} > IntegrationTestIngest,acccccc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec. > column=info:seqnumDuringOpen, timestamp=1434633226600, > value=\x00\x00\x00\x00\x00\x02\xCA\xCC > > IntegrationTestIngest,acccccc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec. > column=info:server, timestamp=1434633226600, > value=ddas-2-5.openstacklocal:16020 > > IntegrationTestIngest,acccccc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec. > column=info:serverstartcode, timestamp=1434633226600, value=1434632778438 > {noformat} > We need to fix it so that the merge region transaction also takes the > master's timestamp. Similar to HBASE-13875. > When this happens, clients start to see a row in the meta with an empty > HRegionInfo (this is because the Put done during the region open only updates > the location information but not the HRI, and the HRI deleted during the > merge transaction "remains deleted"). -- This message was sent by Atlassian JIRA (v6.3.4#6332)