[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310302#comment-14310302
]
Enis Soztutar commented on HBASE-11567:
---
Can you change this following:
{code}
+WALKey key = new WALKey(info.getEncodedNameAsBytes(), tn);
{code}
to
{code}
// we use HLogKey here instead of WALKey directly to support legacy
coprocessors.
WALKey key = new HLogKey(info.getEncodedNameAsBytes(), tn);
{code}
The other similar methods in WALUtil (writeCompactionMarker) does that. Other
than that +1.
I have checked the PB refactor I was talking about earlier. It seems it is
compatible in wire format, and I also tried manually serializing and
deserializing with this change.
Write bulk load COMMIT events to WAL
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch,
HBASE-11567-v4-rebase.patch, hbase-11567-v3.patch, hbase-11567-v4.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310527#comment-14310527 ] Hudson commented on HBASE-11567: FAILURE: Integrated in HBase-1.0 #720 (See [https://builds.apache.org/job/HBase-1.0/720/]) HBASE-11567 Write bulk load COMMIT events to WAL (Only partial patch containing PB changes) (enis: rev 2395d69c23dccba029659a5f28d72631494fee8f) * hbase-protocol/src/main/protobuf/WAL.proto * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/VisibilityLabelsProtos.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Fix For: 2.0.0, 1.1.0 Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, HBASE-11567-v4-rebase.patch, hbase-11567-branch-1.0-partial.patch, hbase-11567-v3.patch, hbase-11567-v4.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310551#comment-14310551 ] Hudson commented on HBASE-11567: FAILURE: Integrated in HBase-1.1 #154 (See [https://builds.apache.org/job/HBase-1.1/154/]) HBASE-11567 Write bulk load COMMIT events to WAL (jeffreyz: rev b0b0a74fef6382643c6ff8d07167ad90ff0d7c43) * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/FilterProtos.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * pom.xml * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java * hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/VisibilityLabelsProtos.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java * hbase-protocol/src/main/protobuf/WAL.proto * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Fix For: 2.0.0, 1.1.0 Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, HBASE-11567-v4-rebase.patch, hbase-11567-branch-1.0-partial.patch, hbase-11567-v3.patch, hbase-11567-v4.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310523#comment-14310523
]
Hadoop QA commented on HBASE-11567:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697205/hbase-11567-branch-1.0-partial.patch
against branch-1.0 branch at commit 7f4146bf4d4df84041b284a76d917d602b5531da.
ATTACHMENT ID: 12697205
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 9
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12730//console
This message is automatically generated.
Write bulk load COMMIT events to WAL
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch,
HBASE-11567-v4-rebase.patch, hbase-11567-branch-1.0-partial.patch,
hbase-11567-v3.patch, hbase-11567-v4.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310550#comment-14310550 ] Hudson commented on HBASE-11567: FAILURE: Integrated in HBase-TRUNK #6103 (See [https://builds.apache.org/job/HBase-TRUNK/6103/]) HBASE-11567 Write bulk load COMMIT events to WAL (Alex Newman, Jeffrey Zhong) (jeffreyz: rev 3f4427739d9ff698d39f2687f11f65967c67340d) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoad.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/FilterProtos.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java * pom.xml * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/VisibilityLabelsProtos.java * hbase-protocol/src/main/protobuf/WAL.proto * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java * hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Fix For: 2.0.0, 1.1.0 Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, HBASE-11567-v4-rebase.patch, hbase-11567-branch-1.0-partial.patch, hbase-11567-v3.patch, hbase-11567-v4.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306818#comment-14306818
]
Hadoop QA commented on HBASE-11567:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696680/HBASE-11567-v4-rebase.patch
against master branch at commit 8aeb3acaf959e2905191fd6c92fa56300f7d3597.
ATTACHMENT ID: 12696680
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12705//console
This message is automatically generated.
Write bulk load COMMIT events to WAL
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch,
HBASE-11567-v4-rebase.patch, hbase-11567-v3.patch, hbase-11567-v4.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307865#comment-14307865
]
Enis Soztutar commented on HBASE-11567:
---
One unfortunate thing is that the refactor of {{message StoreDescriptor }} may
not be compatible. It seems that branch-1.0 contains the RegionEventDescriptor.
If 1.0.0 does not sink, we may not be able to do that change. Let me check
that.
Write bulk load COMMIT events to WAL
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch,
HBASE-11567-v4-rebase.patch, hbase-11567-v3.patch, hbase-11567-v4.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306512#comment-14306512
]
Hadoop QA commented on HBASE-11567:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696622/HBASE-11567-v4-rebase.patch
against master branch at commit 8aeb3acaf959e2905191fd6c92fa56300f7d3597.
ATTACHMENT ID: 12696622
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1946 checkstyle errors (more than the master's current 1941 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ new java.lang.String[] { TableName, EncodedRegionName,
Stores, BulkloadSeqNum, });
+ * @param assignSeqId Force a flush, get it's sequenceId to preserve
the guarantee that all the
+ * edits lower than the highest sequential ID from
all the HFiles are flushed
+ list.append(\n).append(Bytes.toString(p.getFirst())).append( :
).append(p.getSecond());
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.camel.component.jetty.jettyproducer.HttpJettyProducerRecipientListCustomThreadPoolTest.testRecipientList(HttpJettyProducerRecipientListCustomThreadPoolTest.java:40)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12700//console
This message is automatically generated.
Write bulk load COMMIT events to WAL
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch,
HBASE-11567-v4-rebase.patch, hbase-11567-v3.patch, hbase-11567-v4.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176101#comment-14176101 ] Alex Newman commented on HBASE-11567: - Can this be closed? Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, hbase-11567-v3.patch, hbase-11567-v4.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176138#comment-14176138 ] stack commented on HBASE-11567: --- bq. Can this be closed? Is it committed? It doesn't seem to be. Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, hbase-11567-v3.patch, hbase-11567-v4.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106539#comment-14106539 ] Alex Newman commented on HBASE-11567: - +1 overall (although my vote doesn't count). Sorry for not getting this done sooner, I have been distracted trying to make the build cleaner. On your comments 1) agreed 2) That makes sense, I did to avoid making fields public. I am down either way - Overall it looks good however i have some concerns. - It seems as though the formatting is wrong in some places. Please double check that you autoindent everything - I am curious if we should add a unit test (as opposed to regression or acceptance test) so that the wal entry is only written if everything succeeds. It would verify that your change is successful. Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, hbase-11567-v3.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106568#comment-14106568
]
Hadoop QA commented on HBASE-11567:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663565/hbase-11567-v3.patch
against trunk revision .
ATTACHMENT ID: 12663565
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ new java.lang.String[] { TableName, EncodedRegionName,
Stores, BulkloadSeqNum, });
+ * @param assignSeqId Force a flush, get it's sequenceId to preserve
the guarantee that all the
+ * edits lower than the highest sequential ID from
all the HFiles are flushed
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//console
This message is automatically generated.
Write bulk load COMMIT events to WAL
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch,
hbase-11567-v3.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107648#comment-14107648 ] Alex Newman commented on HBASE-11567: - Sounds good, although I can't really +1 since I am not a committer. But upon brief glance, it seems as though you have. Good on you! Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, hbase-11567-v3.patch, hbase-11567-v4.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1410#comment-1410
]
Hadoop QA commented on HBASE-11567:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663781/hbase-11567-v4.patch
against trunk revision .
ATTACHMENT ID: 12663781
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ new java.lang.String[] { TableName, EncodedRegionName,
Stores, BulkloadSeqNum, });
+ * @param assignSeqId Force a flush, get it's sequenceId to preserve
the guarantee that all the
+ * edits lower than the highest sequential ID from
all the HFiles are flushed
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestRegionRebalancing
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations.testClusterIdMismatchAtStartupWithHA(TestDataNodeMultipleRegistrations.java:249)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10544//console
This message is automatically generated.
Write bulk load COMMIT events to WAL
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch,
hbase-11567-v3.patch, hbase-11567-v4.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093474#comment-14093474 ] Alex Newman commented on HBASE-11567: - Sounds good What do you mean failing the RS if the Wal sync fails? Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093520#comment-14093520 ] Enis Soztutar commented on HBASE-11567: --- After the bulk load files are committed to the region, we will call writeBulkLoadMarkerAndSync(). But if that fails with IOException or smt, then we should abort the region server similar to what we do for exception handling in flush event markers in HBASE-11511. Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
