[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274275#comment-15274275
]
Hudson commented on HBASE-15669:
FAILURE: Integrated in HBase-Trunk_matrix #898 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/898/])
HBASE-15669 HFile size is not considered correctly in a replication
(ashishsinghi: rev 34e9a6ff301f40aa3f6ce33ac1b86f9e50fa6694)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* hbase-protocol/src/main/protobuf/WAL.proto
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.branch-1.patch,
> HBASE-15669.branch-1.v1.patch, HBASE-15669.patch, HBASE-15669.v1.patch,
> HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274168#comment-15274168
]
Hudson commented on HBASE-15669:
FAILURE: Integrated in HBase-1.3-IT #651 (See
[https://builds.apache.org/job/HBase-1.3-IT/651/])
HBASE-15669 HFile size is not considered correctly in a replication
(ashishsinghi: rev 5ef9d475281b498a3c97b3842aa15699965109a7)
* hbase-protocol/src/main/protobuf/WAL.proto
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.branch-1.patch,
> HBASE-15669.branch-1.v1.patch, HBASE-15669.patch, HBASE-15669.v1.patch,
> HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274163#comment-15274163
]
Hudson commented on HBASE-15669:
FAILURE: Integrated in HBase-1.3 #688 (See
[https://builds.apache.org/job/HBase-1.3/688/])
HBASE-15669 HFile size is not considered correctly in a replication
(ashishsinghi: rev 5ef9d475281b498a3c97b3842aa15699965109a7)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
*
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* hbase-protocol/src/main/protobuf/WAL.proto
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.branch-1.patch,
> HBASE-15669.branch-1.v1.patch, HBASE-15669.patch, HBASE-15669.v1.patch,
> HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274104#comment-15274104
]
Hudson commented on HBASE-15669:
SUCCESS: Integrated in HBase-1.4 #140 (See
[https://builds.apache.org/job/HBase-1.4/140/])
HBASE-15669 HFile size is not considered correctly in a replication
(ashishsinghi: rev 0964884b925f251725bcd101f23f77a5d3d829e1)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* hbase-protocol/src/main/protobuf/WAL.proto
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
*
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.branch-1.patch,
> HBASE-15669.branch-1.v1.patch, HBASE-15669.patch, HBASE-15669.v1.patch,
> HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268390#comment-15268390
]
Ashish Singhi commented on HBASE-15669:
---
I will commit this later in the day if no objections.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.branch-1.patch,
> HBASE-15669.branch-1.v1.patch, HBASE-15669.patch, HBASE-15669.v1.patch,
> HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266724#comment-15266724
]
Hadoop QA commented on HBASE-15669:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 3m
51s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 32s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 4s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266533#comment-15266533
]
Ashish Singhi commented on HBASE-15669:
---
Made a silly mistake, fixed it. So the test case is fixed.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.branch-1.patch,
> HBASE-15669.branch-1.v1.patch, HBASE-15669.patch, HBASE-15669.v1.patch,
> HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266486#comment-15266486
]
Hadoop QA commented on HBASE-15669:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 3m
48s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 32s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 56s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266467#comment-15266467
]
Ashish Singhi commented on HBASE-15669:
---
branch-1 patch will apply to branch-1.3 also.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.branch-1.patch, HBASE-15669.patch,
> HBASE-15669.v1.patch, HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266317#comment-15266317
]
Ashish Singhi commented on HBASE-15669:
---
Attached patch for branch-1. Checking for branch-1.3.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.branch-1.patch, HBASE-15669.patch,
> HBASE-15669.v1.patch, HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253110#comment-15253110
]
Ashish Singhi commented on HBASE-15669:
---
I'm travelling out of city, will be away from my computer. I will come back by
next week end and attach the patch.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch, HBASE-15669.v1.patch,
> HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252187#comment-15252187
]
Ted Yu commented on HBASE-15669:
Please attach patch for branch-1.3
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch, HBASE-15669.v1.patch,
> HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251922#comment-15251922
]
Ted Yu commented on HBASE-15669:
+1
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch, HBASE-15669.v1.patch,
> HBASE-15669.v2.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251566#comment-15251566
]
Hadoop QA commented on HBASE-15669:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 13m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 13m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m 24s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250288#comment-15250288
]
Ted Yu commented on HBASE-15669:
Can you fix the FindBugs warning w.r.t. autoboxing ?
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch, HBASE-15669.v1.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250043#comment-15250043
]
Hadoop QA commented on HBASE-15669:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 23s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 17m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 13m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 46s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 59s
{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 19s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 156m 31s
{color} | {color:red} hbase-server in the patch failed. {color} |
|
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249583#comment-15249583
]
Ashish Singhi commented on HBASE-15669:
---
Sure. Will try to do by tomorrow (IST) if not after 10 days I will be on
personal leaves and will be away from the computer.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249581#comment-15249581
]
Anoop Sam John commented on HBASE-15669:
bq.Shall I do that as part of another jira as I know there are many places
where we can do such kind off optimization for bulk loaded data replication
Ya that will be better and handle it in all places.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249569#comment-15249569
]
Ashish Singhi commented on HBASE-15669:
---
{quote}
How that will be possible really? I dont think so.. It will be of great help
while handling the normal WALEdits (for normal writes). That may contain many
cells and the Q check can be limited to 1 per edit. Now it will be so many
compares to handle the bulk load replication. We even had a boolean to enable
the bulk load replication right? Even that check also not done here? We should
not be adding so many unwanted compare ops.
{quote}
OK. Shall I do that as part of another jira as I know there are many places
where we can do such kind off optimization for bulk loaded data replication.
bq. I was just asking to add like a safe guard
Will address in the next version of my patch.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249568#comment-15249568
]
Anoop Sam John commented on HBASE-15669:
bq.By default its 0.
It is a PB object right? If not set, the getter may throw exception. Yes we are
setting it.. I was just asking to add like a safe guard.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249567#comment-15249567
]
Anoop Sam John commented on HBASE-15669:
bq.The same thing came up when we were working on the main jira (HBASE-13153),
but we are not sure if in future an edit can contain a mix of mutation and bulk
load marker cells. If that happens then it will break the replication. So to
avoid that we are handling it in that way
How that will be possible really? I dont think so.. It will be of great help
while handling the normal WALEdits (for normal writes). That may contain many
cells and the Q check can be limited to 1 per edit. Now it will be so many
compares to handle the bulk load replication. We even had a boolean to enable
the bulk load replication right? Even that check also not done here?We
should not be adding so many unwanted compare ops.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249558#comment-15249558
]
Ashish Singhi commented on HBASE-15669:
---
[~tedyu],
{quote} What if there is no size for this file ?
I see LOG.warn() below. Is that enough ?{quote}
That's not possible, even if get a exception we will set it 0 so that should be
enough.
[~anoop.hbase],
bq. Do we need a check like hasStoreFileSize()? getStoreFileSize(): 0?
By default its 0.
bq. totalEdits?
totalCells :)
{quote}
In loop condition part u can have i < cells.size()? Other places also similar
way. Will it add more burden on other normal edits size calc? Like we have
qualifier check on each and every cell.
There can be one WALEdit with a mix of bulk load cells + normal cells? I dont
think so. So we can early out when 1st cell in WALEdit is not a bulk load cell?
May be this optimization can come in some other places also?
{quote}
The same thing came up when we were working on the main jira (HBASE-13153), but
we are not sure if in future an edit can contain a mix of mutation and bulk
load marker cells. If that happens then it will break the replication. So to
avoid that we are handling it in that way.
Thanks for the reviews.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249291#comment-15249291
]
Anoop Sam John commented on HBASE-15669:
bq.stores.get(j).getStoreFileSize();
Do we need a check like hasStoreFileSize()? getStoreFileSize(): 0?
int totalEdits = edit.size();
752 for (int i = 0; i < totalEdits; i++) {
totalEdits? This is cells in this edit right? In loop condition part u can
have i < cells.size()? Other places also similar way. Will it add more burden
on other normal edits size calc? Like we have qualifier check on each and
every cell.
There can be one WALEdit with a mix of bulk load cells + normal cells? I dont
think so. So we can early out when 1st cell in WALEdit is not a bulk load
cell? May be this optimization can come in some other places also?
Can we mark the WALEdit itself as bulk load related? Am not sure.. May be see
that later.
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248755#comment-15248755
]
Hadoop QA commented on HBASE-15669:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 12m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 12m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 15s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 10s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 147m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense
[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request
[
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248289#comment-15248289
]
Ted Yu commented on HBASE-15669:
{code}
3063 * @param storeFilesSize Map of store files and its length
{code}
'its length' -> 'their lengths'
{code}
3083builder.setStoreFileSize(storeFilesSize.get(name));
{code}
What if there is no size for this file ?
I see LOG.warn() below. Is that enough ?
{code}
5314Map storeFilesSize = new HashMap();
{code}
Rename the variable storeFilesSizes
{code}
762 LOG.error("Failed to deserialize bulk load entry from wal
edit. "
763 + "This its hfiles count will not be added into
metric.");
{code}
Can more information be added to the above log so that user has more
information ?
> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at
> most {{replication.source.size.capacity}} size of data or
> {{replication.source.nb.capacity}} entries.
> The size is calculated by considering the cells size in each entry which will
> get calculated wrongly in case of bulk loaded data replication, in this case
> we need to consider the size of hfiles not cell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
