[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093425#comment-14093425 ] Enis Soztutar commented on HBASE-11567: --- bq. One question I have is, currently I am logging the request by the user for HBase HFile bulk loading. That being said, does it make sense to have a different WAL entry when it completes successfully? Indeed. The whole point of this issue is to add a WAL marker for the COMMIT of files for bulk load. Since we cannot do an atomic commit of bulk load files in the FS and also in WAL, we can do the commit in FS first, then write the WAL entry next, failing the RS if the WAL sync fails. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093444#comment-14093444
]
Enis Soztutar commented on HBASE-11567:
---
I would also say that logging the request for bulk load is not as important as
logging the bulk load commit. I do not think of any use for the bulk load
request wal marker.
Do you mind putting an RB for easier review.
I think we would like to use BULK_LOAD for being consistent:
{code}
+ static final byte [] BULK = Bytes.toBytes(HBASE::BULK);
{code}
About jmock it should be ok, but we may want to the escalate the discussion to
dev@ so that other devs are aware.
Write bulk load events to WAL
-
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088082#comment-14088082 ] Alex Newman commented on HBASE-11567: - I think we should wait to commit this until the reader side is done. I think figuring out those tests are important for the acceptability of this patch. That being said, I hope this is a useful first step for whoever (most likely me) has to do it. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088170#comment-14088170 ] stack commented on HBASE-11567: --- Took a quick look. Looks good to me. We could commit this but agree probably better if read-side goes in around same time. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088177#comment-14088177 ] Alex Newman commented on HBASE-11567: - Daww thanks stack. One question I have is, currently I am logging the request by the user for HBase HFile bulk loading. That being said, does it make sense to have a different WAL entry when it completes successfully? Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088314#comment-14088314 ] stack commented on HBASE-11567: --- [~posix4e] Yes. Can you include files that were bulk loaded in the list? Thats what the read replicas fellas would like if I were to guess so they know what to open on the slave on completion. Maybe wait on [~enis] or one of the other lads for confirmation before doing work. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088317#comment-14088317 ] stack commented on HBASE-11567: --- Oh, also, make case for the new dependency here and up in release notes. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088321#comment-14088321
]
Alex Newman commented on HBASE-11567:
-
Good news
message FamilyPath {
required bytes family = 1;
required string path = 2;
}
includes the path for the file. It's a bit annoying but I wanted to reuse the
FamilyPath proto
Also I am meeting with horton next week in person. So yea I'll double check my
work with them.
Write bulk load events to WAL
-
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088336#comment-14088336 ] ryan rawson commented on HBASE-11567: - JMock has a few features that helps us keep our tests clean. First it only mocks interfaces, encouraging good interface-driven design. Secondly JMock has a nice little mini DSL which reads nicely. Lastly, JMock is a very strict mocker, it doesnt requires a separate validation step and requires complete specification of dependent calls. Mockito and JMock are slightly different tools, and have their purposes each. I'd prefer new tests be done with JMock, and using mockito to leverage older code into a newer style over time. Since the dependency is test-only, there is no run-time impact. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088347#comment-14088347 ] Alex Newman commented on HBASE-11567: - Thanks Ryan. To double up, JMock makes creating our own testing DSLs really easy. We learned from C5, once you get enough of this infrastructure in place, testing becomes a great pleasure. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081539#comment-14081539 ] Alex Newman commented on HBASE-11567: - OK I addressed everything I could. Next step jiras are probably - SOLID-IFY HRegion, verify testing behavior (We will actually have to have a user story here I think) - With the above user story, we need more integration tests. I added some integration with an existing test to make sure the marker is being made however. - Change the HFile /HFile load API so we can test it without writing out files - Improve HLog pluggability (This is important for a bunch of reason) There's another thing. I am convinced that this should be 5 patches, but I am not sure how we want to do it. The patch comments are a little wonkly but here it is nonetheless https://github.com/posix4e/hbase/pull/1. I will attach a combined one to this jira as well. [~ryanobjc] for the sake of doing things in an impartial way perhaps we should get a non wandisco employee to the next round of reviews. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081875#comment-14081875
]
Hadoop QA commented on HBASE-11567:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659013/HBASE-11567-v2.patch
against trunk revision .
ATTACHMENT ID: 12659013
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ new java.lang.String[] { TableName, EncodedRegionName,
FamilyPath, AssignSeqNum, });
+ * @param assignSeqId Force a flush, get it's sequenceId to preserve
the guarantee that all the
+ * edits lower than the highest sequential ID from
all the HFiles are flushed
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestReplicasClient
org.apache.hadoop.hbase.master.TestRestartCluster
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.regionserver.TestRegionReplicas
org.apache.hadoop.hbase.TestIOFencing
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10252//console
This message is automatically generated.
Write bulk load events to WAL
-
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080314#comment-14080314 ] Alex Newman commented on HBASE-11567: - OK here is my first pass at it. I talked to stack offline about the jmock changes. I actually think we should break this up into three commits 1. JMock changes to pom 2. Protobuf changes (and generated files) 3. The actual changes But I'll do it as one patch first. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080408#comment-14080408
]
Hadoop QA commented on HBASE-11567:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658800/HBASE-11567-v1.patch
against trunk revision .
ATTACHMENT ID: 12658800
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the trunk's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ * Generates a marker for the WAL so that we propagate the notion of a
bulk region load throughout the WAL.
+ * @param familyPaths A list of pairs which maps the name of the column
family to the location on disk where it is
+ * @param assignSeqId Whether or not to assign a new sequence ID (by way of
calling flush) or to use the one provided
+
ListPairbyte[], String familyPaths,
+ new java.lang.String[] { TableName, EncodedRegionName,
FamilyPath, AssignSeqNum, });
+ WALProtos.BulkLoadDescriptor loadDescriptor =
ProtobufUtil.toBulkLoadDescriptor(this.getRegionInfo().getTable(),
+ * @param sequenceId The current sequenceId in the log at the time when we
were to write the bulk load marker.
+ public static WALEdit createBulkLoadEvent(HRegionInfo hri,
WALProtos.BulkLoadDescriptor bulkLoadDescriptor) {
+testRegionWithFamilies(family1,
family2).bulkLoadHFiles(withFamilyPathsFor(family1, family2), false);
+ public void
bulkHLogShouldThrowErrorWhenFamilySpecifiedAndHFileExistsButNotInTableDescriptor()
throws IOException {
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestIOFencing
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.regionserver.TestRegionReplicas
org.apache.hadoop.hbase.migration.TestNamespaceUpgrade
org.apache.hadoop.hbase.client.TestReplicasClient
org.apache.hadoop.hbase.master.TestRestartCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10237//console
This message is automatically generated.
Write bulk load events to WAL
-
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080421#comment-14080421
]
ryan rawson commented on HBASE-11567:
-
Good generally speaking.
The bulk load tests should really verify more behavior. In the 'successful'
cases, what kind of behavior has changed about the HRegion that we could check?
(if anything - it might not be feasible since HRegion isn't SOLID)
I also could imagine a series of integration tests that verify that the data in
the bulk loaded file is readable. There is also some stress tests that involve
loading during concurrent operations and what should happen in those cases.
That is a separate integration test.
commented on 154f10b
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java:L2748:
I want to see a space between ){
commented on 154f10b
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java:L2750:
what's the line length max? Are we on Java7, can we use diamonds now?
commented on 154f10b
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java:L2751:
spacing here, between the : and the ){
commented on 154f10b
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java:L322:
im a little vaugely unhappy how creating all these structures is spread about,
but that is a greater issue, and we cant fix it now.
commented on 154f10b
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:L3667:
spacing here:
if (log != null) {
commented on 154f10b
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:L3672
:
so the log doesnt really know how to write business logicy entries, so we have
a utility class that does it. So who is really responsible for doing these
things? The obvious design solution is a wrapper around HLog which allows
pluggable log under it, we should file a new HBase JIRA.
one thing that bothers me about this line is we are hiding a pretty heavy duty
IO event, and its not really apparent, maybe change the name to
HLogUtil.writeBulkLoadMarkerAndSync() so its apparent
commented on 154f10b
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L46
:
writes to files = not unit test
commented on 154f10b
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L64:
are these constants? if so, final them, upcaps them
commented on 154f10b
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L76:
Isn't there already a constant that has this in it somewhere else?
the answer is yes, a package protected one in WALEdit - lets not copy magic
constants about, DRY that up
commented on 154f10b
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L121:
hide this (and others like this) behind a static function
commented on 154f10b
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L144
:
space
commented on 154f10b
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java:L168:
maybe call this
bulkLogWalEditType -- it might read better when it's being used.
Write bulk load events to WAL
-
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080422#comment-14080422 ] ryan rawson commented on HBASE-11567: - also what's up with the failing tests, how could this patch conceivably affect org.apache.hadoop.hbase.migration.TestNamespaceUpgrade Just a general expression of annoyance at test brittleness Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14070793#comment-14070793 ] Alex Newman commented on HBASE-11567: - I would love to take this up. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14070805#comment-14070805 ] Enis Soztutar commented on HBASE-11567: --- Sure thing. You can look at HBASE-11511, and HBASE-11512 as a template for code. Write bulk load events to WAL - Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
