[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093444#comment-14093444 ] Enis Soztutar commented on HBASE-11567: --- I would also say that logging the request for bulk load is not as important as logging the bulk load commit. I do not think of any use for the bulk load request wal marker. Do you mind putting an RB for easier review. I think we would like to use BULK_LOAD for being consistent: {code} + static final byte [] BULK = Bytes.toBytes("HBASE::BULK"); {code} About jmock it should be ok, but we may want to the escalate the discussion to dev@ so that other devs are aware. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093425#comment-14093425 ] Enis Soztutar commented on HBASE-11567: --- bq. One question I have is, currently I am logging the request by the user for HBase HFile bulk loading. That being said, does it make sense to have a different WAL entry when it completes successfully? Indeed. The whole point of this issue is to add a WAL marker for the COMMIT of files for bulk load. Since we cannot do an atomic commit of bulk load files in the FS and also in WAL, we can do the commit in FS first, then write the WAL entry next, failing the RS if the WAL sync fails. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088347#comment-14088347 ] Alex Newman commented on HBASE-11567: - Thanks Ryan. To double up, JMock makes creating our own testing DSLs really easy. We learned from C5, once you get enough of this infrastructure in place, testing becomes a great pleasure. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088336#comment-14088336 ] ryan rawson commented on HBASE-11567: - JMock has a few features that helps us keep our tests clean. First it only mocks interfaces, encouraging good interface-driven design. Secondly JMock has a nice little mini DSL which reads nicely. Lastly, JMock is a very strict mocker, it doesnt requires a separate validation step and requires complete specification of dependent calls. Mockito and JMock are slightly different tools, and have their purposes each. I'd prefer new tests be done with JMock, and using mockito to leverage older code into a newer style over time. Since the dependency is test-only, there is no run-time impact. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088321#comment-14088321 ] Alex Newman commented on HBASE-11567: - Good news message FamilyPath { required bytes family = 1; required string path = 2; } includes the path for the file. It's a bit annoying but I wanted to reuse the FamilyPath proto Also I am meeting with horton next week in person. So yea I'll double check my work with them. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088317#comment-14088317 ] stack commented on HBASE-11567: --- Oh, also, make case for the new dependency here and up in release notes. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088314#comment-14088314 ] stack commented on HBASE-11567: --- [~posix4e] Yes. Can you include files that were bulk loaded in the list? Thats what the read replicas fellas would like if I were to guess so they know what to open on the slave on completion. Maybe wait on [~enis] or one of the other lads for confirmation before doing work. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088177#comment-14088177 ] Alex Newman commented on HBASE-11567: - Daww thanks stack. One question I have is, currently I am logging the request by the user for HBase HFile bulk loading. That being said, does it make sense to have a different WAL entry when it completes successfully? > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088170#comment-14088170 ] stack commented on HBASE-11567: --- Took a quick look. Looks good to me. We could commit this but agree probably better if read-side goes in around same time. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088082#comment-14088082 ] Alex Newman commented on HBASE-11567: - I think we should wait to commit this until the reader side is done. I think figuring out those tests are important for the acceptability of this patch. That being said, I hope this is a useful first step for whoever (most likely me) has to do it. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081875#comment-14081875 ] Hadoop QA commented on HBASE-11567: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12659013/HBASE-11567-v2.patch against trunk revision . ATTACHMENT ID: 12659013 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 12 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + new java.lang.String[] { "TableName", "EncodedRegionName", "FamilyPath", "AssignSeqNum", }); + * @param assignSeqId Force a flush, get it's sequenceId to preserve the guarantee that all the + * edits lower than the highest sequential ID from all the HFiles are flushed {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestReplicasClient org.apache.hadoop.hbase.master.TestRestartCluster org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction org.apache.hadoop.hbase.regionserver.TestRegionReplicas org.apache.hadoop.hbase.TestIOFencing Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10252//console This message is automatically generated. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081539#comment-14081539 ] Alex Newman commented on HBASE-11567: - OK I addressed everything I could. Next step jiras are probably - SOLID-IFY HRegion, verify testing behavior (We will actually have to have a user story here I think) - With the above user story, we need more integration tests. I added some integration with an existing test to make sure the marker is being made however. - Change the HFile /HFile load API so we can test it without writing out files - Improve HLog pluggability (This is important for a bunch of reason) There's another thing. I am convinced that this should be 5 patches, but I am not sure how we want to do it. The patch comments are a little wonkly but here it is nonetheless https://github.com/posix4e/hbase/pull/1. I will attach a combined one to this jira as well. [~ryanobjc] for the sake of doing things in an impartial way perhaps we should get a non wandisco employee to the next round of reviews. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080422#comment-14080422 ] ryan rawson commented on HBASE-11567: - also what's up with the failing tests, how could this patch conceivably affect org.apache.hadoop.hbase.migration.TestNamespaceUpgrade Just a general expression of annoyance at test brittleness > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080421#comment-14080421 ] ryan rawson commented on HBASE-11567: - Good generally speaking. The bulk load tests should really verify more behavior. In the 'successful' cases, what kind of behavior has changed about the HRegion that we could check? (if anything - it might not be feasible since HRegion isn't SOLID) I also could imagine a series of integration tests that verify that the data in the bulk loaded file is readable. There is also some stress tests that involve loading during concurrent operations and what should happen in those cases. That is a separate integration test. commented on 154f10b hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java:L2748: I want to see a space between ){ commented on 154f10b hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java:L2750: what's the line length max? Are we on Java7, can we use diamonds now? commented on 154f10b hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java:L2751: spacing here, between the : and the ){ commented on 154f10b hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java:L322: im a little vaugely unhappy how creating all these structures is spread about, but that is a greater issue, and we cant fix it now. commented on 154f10b hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:L3667: spacing here: if (log != null) { commented on 154f10b hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:L3672 : so the log doesnt really know how to write business logicy entries, so we have a utility class that does it. So who is really responsible for doing these things? The obvious design solution is a wrapper around HLog which allows pluggable log under it, we should file a new HBase JIRA. one thing that bothers me about this line is we are hiding a pretty heavy duty IO event, and its not really apparent, maybe change the name to HLogUtil.writeBulkLoadMarkerAndSync() so its apparent commented on 154f10b hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L46 : writes to files = not unit test commented on 154f10b hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L64: are these constants? if so, final them, upcaps them commented on 154f10b hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L76: Isn't there already a constant that has this in it somewhere else? the answer is yes, a package protected one in WALEdit - lets not copy magic constants about, DRY that up commented on 154f10b hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L121: hide this (and others like this) behind a static function commented on 154f10b hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L144 : space commented on 154f10b hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L168: maybe call this bulkLogWalEditType -- it might read better when it's being used. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080408#comment-14080408 ] Hadoop QA commented on HBASE-11567: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658800/HBASE-11567-v1.patch against trunk revision . ATTACHMENT ID: 12658800 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + * Generates a marker for the WAL so that we propagate the notion of a bulk region load throughout the WAL. + * @param familyPaths A list of pairs which maps the name of the column family to the location on disk where it is + * @param assignSeqId Whether or not to assign a new sequence ID (by way of calling flush) or to use the one provided + List> familyPaths, + new java.lang.String[] { "TableName", "EncodedRegionName", "FamilyPath", "AssignSeqNum", }); + WALProtos.BulkLoadDescriptor loadDescriptor = ProtobufUtil.toBulkLoadDescriptor(this.getRegionInfo().getTable(), + * @param sequenceId The current sequenceId in the log at the time when we were to write the bulk load marker. + public static WALEdit createBulkLoadEvent(HRegionInfo hri, WALProtos.BulkLoadDescriptor bulkLoadDescriptor) { +testRegionWithFamilies(family1, family2).bulkLoadHFiles(withFamilyPathsFor(family1, family2), false); + public void bulkHLogShouldThrowErrorWhenFamilySpecifiedAndHFileExistsButNotInTableDescriptor() throws IOException { {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.TestIOFencing org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction org.apache.hadoop.hbase.regionserver.TestRegionReplicas org.apache.hadoop.hbase.migration.TestNamespaceUpgrade org.apache.hadoop.hbase.client.TestReplicasClient org.apache.hadoop.hbase.master.TestRestartCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10237//console This message is automatically generated. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080314#comment-14080314 ] Alex Newman commented on HBASE-11567: - OK here is my first pass at it. I talked to stack offline about the jmock changes. I actually think we should break this up into three commits 1. JMock changes to pom 2. Protobuf changes (and generated files) 3. The actual changes But I'll do it as one patch first. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alex Newman > Attachments: HBASE-11567-v1.patch > > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14070805#comment-14070805 ] Enis Soztutar commented on HBASE-11567: --- Sure thing. You can look at HBASE-11511, and HBASE-11512 as a template for code. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14070793#comment-14070793 ] Alex Newman commented on HBASE-11567: - I would love to take this up. > Write bulk load events to WAL > - > > Key: HBASE-11567 > URL: https://issues.apache.org/jira/browse/HBASE-11567 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar > > Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and > region open/close (HBASE-11512) , we should persist bulk load events to WAL. > This is especially important for secondary region replicas, since we can use > this information to pick up primary regions' files from secondary replicas. > A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)