[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381163#comment-16381163
]
Hadoop QA commented on HBASE-19423:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1.4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
0s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
25s{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_162.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s{color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_162. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
25s{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_171.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s{color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_171. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
20s{color} | {color:red} hbase-server: The patch generated 1 new + 6 unchanged
- 0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 1m
31s{color} | {color:red} patch has 41 errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m
29s{color} | {color:red} The patch causes 41 errors with Hadoop v2.4.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
26s{color} | {color:red} The patch causes 41 errors with Hadoop v2.5.2. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m
19s{color} | {color:red} The patch causes 41 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
11s{color} | {color:red} The patch causes 41 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
32s{color} | {color:red} hbase-server-jdk1.8.0_162 with JDK v1.8.0_162
generated 6 new + 3 unchanged - 0 fixed = 9 total (was 3) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
37s{color} | {color:red} hbase-server-jdk1.7.0_171 with JDK v1.7.0_171
generated 6 new + 3 unchanged - 0 fixed = 9 total (was 3) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 25s{color}
| {color:red} hbase-server in the patch
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281401#comment-16281401
]
Anoop Sam John commented on HBASE-19423:
My concern was on the general approach.. What the test is doing in the CP hook.
{quote}
I have a scenario where in a user table some column families are system
generated and these families data need not to be replicated.
Because these are system generated column families it would not be appropriate
to expect user to disable replication for these families.
{quote}
Well the replication enable is at CF level. One can set on HCD. By default
the replication is disabled for any CF. ie REPLICATION_SCOPE_LOCAL
One can enable this by setting REPLICATION_SCOPE_GLOBAL on those CFs which has
to be replicated. So in ur case when the table is having some of the system
generated CFs the replication is OFF on them by default.. And this is what u
need. The other CFs on the user table, which the user created, the use can opt
to set replication scope on those only which he wants.
So in general I dont see any need NOT to set the replication global scope
enabled on any of the CFs of the table and doing the selective replication of
some of the CF data using the CP hook. That is very very strange usage for me.
Hope am making it clear. Ya seems in master branch, the place where the scope
been set on the WALKey seems different so there is no reset happening. But
setting the scope on the walkey, I really doubt whether that can be a work of
the CPs. This is some thing the core code has to do. Even on master, the
WALKey is still marked @InterfaceAudience.Private. Means we really dont expect
CPs to use all methods in that. This is an issue that we over expose things to
CPs. I agree.. So pls consider ur solution 1st I would say.
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 1.4.0, 1.3.2
>
> Attachments: HBASE-19423-branch-1.3-001.patch,
> HBASE-19423-branch-1.4-001.patch, HBASE-19423-master-001-test.patch
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280954#comment-16280954
]
Hadoop QA commented on HBASE-19423:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
2s{color} | {color:red} hbase-server: The patch generated 1 new + 0 unchanged -
0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
51m 33s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}102m
54s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19423 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900911/HBASE-19423-master-001-test.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7fe7dd2c429e 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / eabad8a91c |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/10266/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/10266/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10266/testReport/ |
| modules | C:
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280848#comment-16280848
]
Hadoop QA commented on HBASE-19423:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1.4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_161 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
4s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_161 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
16s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
21s{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_152.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s{color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_152. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
25s{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_161.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s{color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_161. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
16s{color} | {color:red} hbase-server: The patch generated 1 new + 6 unchanged
- 0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 1m
32s{color} | {color:red} patch has 41 errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m
30s{color} | {color:red} The patch causes 41 errors with Hadoop v2.4.0. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
22s{color} | {color:red} The patch causes 41 errors with Hadoop v2.4.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m
20s{color} | {color:red} The patch causes 41 errors with Hadoop v2.5.0. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
15s{color} | {color:red} The patch causes 41 errors with Hadoop v2.5.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
8s{color} | {color:red} The patch causes 41 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m
6s{color} | {color:red} The patch causes 41 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
1s{color} | {color:red} The patch causes 41 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
55s{color} | {color:red} The patch causes 41 errors with
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280730#comment-16280730
]
Mohammad Arshad commented on HBASE-19423:
-
submitted patch for branch-1.4
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 1.4.0, 1.3.2
>
> Attachments: HBASE-19423-branch-1.3-001.patch,
> HBASE-19423-branch-1.4-001.patch, HBASE-19423-master-001-test.patch
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280636#comment-16280636
]
Ted Yu commented on HBASE-19423:
Thanks for the contribution, Mohammad.
The 1.3 patch doesn't apply to branch-1.
Do you mind attaching patch for branch-1.4 ?
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 1.4.0, 1.3.2
>
> Attachments: HBASE-19423-branch-1.3-001.patch,
> HBASE-19423-master-001-test.patch
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280630#comment-16280630
]
Mohammad Arshad commented on HBASE-19423:
-
This issue does not exist in master branch. Submitting master branch test patch.
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-19423-branch-1.3-001.patch
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280619#comment-16280619
]
Mohammad Arshad commented on HBASE-19423:
-
Thanks [~yuzhih...@gmail.com], [~anoop.hbase] for your response.
bq. Not sure why taking that path. Can u pls explain that part?
I have a scenario where in a user table some column families are system
generated and these families data need not to be replicated.
Because these are system generated column families it would not be appropriate
to expect user to disable replication for these families. So filtering is done
at system level through coprocessors.
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-19423-branch-1.3-001.patch
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278583#comment-16278583
]
Anoop Sam John commented on HBASE-19423:
master patch would be always better.
So here the replication on HCD is set to be not replicating and changing the
same in WalKey in Observer. Not sure why taking that path. Can u pls explain
that part?
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-19423-branch-1.3-001.patch
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277804#comment-16277804
]
Hadoop QA commented on HBASE-19423:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1.3 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_161 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
44s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_161 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.7.0_161 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
17s{color} | {color:red} hbase-server: The patch generated 1 new + 6 unchanged
- 0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
25s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 4s{color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3.
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_161 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 56s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestEndToEndSplitTransaction |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:dca6535 |
| JIRA Issue | HBASE-19423 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277687#comment-16277687
]
Ted Yu commented on HBASE-19423:
Can you customize TestWALObserverAsReplicationFilter.java so that it compiles
against master branch ?
Thanks
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-19423-branch-1.3-001.patch
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277627#comment-16277627
]
Mohammad Arshad commented on HBASE-19423:
-
yes, i will submit it now
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277624#comment-16277624
]
Ted Yu commented on HBASE-19423:
Thanks for reporting.
Do you want to attach a patch ?
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
