[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-19 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297871#comment-16297871
 ] 

Hudson commented on HBASE-19494:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4257 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/4257/])
HBASE-19494 Create simple WALKey filter that can be plugged in on the (stack: 
rev 32f6fd41c274a955338b7c43ab80309b9adbba0d)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntrySinkFilter.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntrySinkFilter.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/WALEntryFilter.java


> Create simple WALKey filter that can be plugged in on the Replication Sink
> --
>
> Key: HBASE-19494
> URL: https://issues.apache.org/jira/browse/HBASE-19494
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 
> 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, 
> HBASE-19494.master.001.patch
>
>
> hbase-indexer used to look at WALKeys on the sink to see if their time of 
> creation was before the time at which the replication stream was enabled.
> In the parent redo, there is no means for doing this anymore (because WALKey 
> used to be Private and because to get at the WALKey in the Sink, you had to 
> override all of the Replication which meant importing a million Private 
> objects...).
> This issue is about adding a simple filter to Replication on the sink-side 
> that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently 
> made read-only).
> Assigned myself. Need to do this so hbase-indexer can move to hbase2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-19 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297481#comment-16297481
 ] 

stack commented on HBASE-19494:
---

We have a few timeouts going on. This time it was this:

[INFO] Results:
[INFO] 
[WARNING] Flakes: 
[WARNING] 
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient.testSkipFlushTableSnapshot(org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient)
[ERROR]   Run 1: TestFlushSnapshotFromClient.testSkipFlushTableSnapshot:201 
Expected snapshot to contain family 'fam', but it does not.
[INFO]   Run 2: PASS
[INFO] 
[INFO] 
[WARNING] Tests run: 3402, Failures: 0, Errors: 0, Skipped: 33, Flakes: 1


Need to fix.

Unrelated. Let me push this change.

> Create simple WALKey filter that can be plugged in on the Replication Sink
> --
>
> Key: HBASE-19494
> URL: https://issues.apache.org/jira/browse/HBASE-19494
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 
> 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, 
> HBASE-19494.master.001.patch
>
>
> hbase-indexer used to look at WALKeys on the sink to see if their time of 
> creation was before the time at which the replication stream was enabled.
> In the parent redo, there is no means for doing this anymore (because WALKey 
> used to be Private and because to get at the WALKey in the Sink, you had to 
> override all of the Replication which meant importing a million Private 
> objects...).
> This issue is about adding a simple filter to Replication on the sink-side 
> that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently 
> made read-only).
> Assigned myself. Need to do this so hbase-indexer can move to hbase2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-19 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297459#comment-16297459
 ] 

Hadoop QA commented on HBASE-19494:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
36s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m  
3s{color} | {color:red} hbase-server: The patch generated 22 new + 17 unchanged 
- 3 fixed = 39 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
34s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
18m 49s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 41s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m  5s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19494 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12902907/HBASE-19494.master.001.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 9e6be9603a28 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / dbe409e2c7 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HBASE-Build/10567/artifact/patchprocess/diff-checkstyle-hbase-server.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/10567/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/10567/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/10567/console |
| Powered by | Apache Yetus

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-19 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297392#comment-16297392
 ] 

stack commented on HBASE-19494:
---

Thanks for the feedback [~whoschek]

Will commit when hadoopqa comes back.


> Create simple WALKey filter that can be plugged in on the Replication Sink
> --
>
> Key: HBASE-19494
> URL: https://issues.apache.org/jira/browse/HBASE-19494
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 
> 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, 
> HBASE-19494.master.001.patch
>
>
> hbase-indexer used to look at WALKeys on the sink to see if their time of 
> creation was before the time at which the replication stream was enabled.
> In the parent redo, there is no means for doing this anymore (because WALKey 
> used to be Private and because to get at the WALKey in the Sink, you had to 
> override all of the Replication which meant importing a million Private 
> objects...).
> This issue is about adding a simple filter to Replication on the sink-side 
> that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently 
> made read-only).
> Assigned myself. Need to do this so hbase-indexer can move to hbase2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-19 Thread wolfgang hoschek (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297391#comment-16297391
 ] 

wolfgang hoschek commented on HBASE-19494:
--

Great. I think this works. Thanks!

> Create simple WALKey filter that can be plugged in on the Replication Sink
> --
>
> Key: HBASE-19494
> URL: https://issues.apache.org/jira/browse/HBASE-19494
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 
> 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, 
> HBASE-19494.master.001.patch
>
>
> hbase-indexer used to look at WALKeys on the sink to see if their time of 
> creation was before the time at which the replication stream was enabled.
> In the parent redo, there is no means for doing this anymore (because WALKey 
> used to be Private and because to get at the WALKey in the Sink, you had to 
> override all of the Replication which meant importing a million Private 
> objects...).
> This issue is about adding a simple filter to Replication on the sink-side 
> that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently 
> made read-only).
> Assigned myself. Need to do this so hbase-indexer can move to hbase2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-19 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297260#comment-16297260
 ] 

stack commented on HBASE-19494:
---

See 001 [~whoschek] when you get a chance sir. It passes the Filter the 
Connection in an init. I think this will work for you. Its ugly but less so 
than having Connection implement a Replication Filter. Thanks.

> Create simple WALKey filter that can be plugged in on the Replication Sink
> --
>
> Key: HBASE-19494
> URL: https://issues.apache.org/jira/browse/HBASE-19494
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 
> 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, 
> HBASE-19494.master.001.patch
>
>
> hbase-indexer used to look at WALKeys on the sink to see if their time of 
> creation was before the time at which the replication stream was enabled.
> In the parent redo, there is no means for doing this anymore (because WALKey 
> used to be Private and because to get at the WALKey in the Sink, you had to 
> override all of the Replication which meant importing a million Private 
> objects...).
> This issue is about adding a simple filter to Replication on the sink-side 
> that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently 
> made read-only).
> Assigned myself. Need to do this so hbase-indexer can move to hbase2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-19 Thread wolfgang hoschek (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297160#comment-16297160
 ] 

wolfgang hoschek commented on HBASE-19494:
--

bq. First cut. I don't think it enough. I think hbase-indexer is going to want 
some context passed

Yes, indeed. I don't think your proposed patch works. Instead, my idea (as I've 
already outlined above in 
https://issues.apache.org/jira/browse/HBASE-19494?focusedCommentId=16292858=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16292858)
 is for ReplicationSink. replicateEntries() to delegate the filter call to the 
Connection (if it implements that new filter interface method), which can 
maintain state on a per table basis, like so:

{code}
Connection conn = getConnection();
 if (conn instanceof WALEntrySinkFilter) { // this is a new Interface
   if (((WALEntrySinkFilter)conn).filter(table, 
entry.getKey().getWriteTime())) {
  continue; // ignore unwanted entry
   }
{code}

The connection can maintain state on a per table basis, and connection can look 
up the timestamp thresholds via the table name (again as already outlined above 
in 
https://issues.apache.org/jira/browse/HBASE-19494?focusedCommentId=16292858=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16292858)
 like so:

{code}
public class IndexerConnection implements Connection, WALEntryFilter { 
   private Map timestampThresholds = ...
   public boolean filter(TableName table, long writeTime) {
   long timestampThreshold = 
timestampThresholds.get(table.getNameAsString());
   return writeTime < timestampThreshold;
   }
   ... and so on
}
{code}

> Create simple WALKey filter that can be plugged in on the Replication Sink
> --
>
> Key: HBASE-19494
> URL: https://issues.apache.org/jira/browse/HBASE-19494
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 
> 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch
>
>
> hbase-indexer used to look at WALKeys on the sink to see if their time of 
> creation was before the time at which the replication stream was enabled.
> In the parent redo, there is no means for doing this anymore (because WALKey 
> used to be Private and because to get at the WALKey in the Sink, you had to 
> override all of the Replication which meant importing a million Private 
> objects...).
> This issue is about adding a simple filter to Replication on the sink-side 
> that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently 
> made read-only).
> Assigned myself. Need to do this so hbase-indexer can move to hbase2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-18 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295884#comment-16295884
 ] 

Hadoop QA commented on HBASE-19494:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
45s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m  
5s{color} | {color:red} hbase-server: The patch generated 21 new + 19 unchanged 
- 1 fixed = 40 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
45s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
20m 17s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 28s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 43s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.wal.TestLogRolling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19494 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12902707/0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 3c62bfac 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 9d0c7c6dfb |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HBASE-Build/10537/artifact/patchprocess/diff-checkstyle-hbase-server.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/10537/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/10537/testReport/ |
| modules | C: hbase-server

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-18 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295716#comment-16295716
 ] 

stack commented on HBASE-19494:
---

Yeah, [~whoschek], where you get the current enabled-replication timestamp from 
? We need to pass this into the filter on construction sir?

> Create simple WALKey filter that can be plugged in on the Replication Sink
> --
>
> Key: HBASE-19494
> URL: https://issues.apache.org/jira/browse/HBASE-19494
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 
> 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch
>
>
> hbase-indexer used to look at WALKeys on the sink to see if their time of 
> creation was before the time at which the replication stream was enabled.
> In the parent redo, there is no means for doing this anymore (because WALKey 
> used to be Private and because to get at the WALKey in the Sink, you had to 
> override all of the Replication which meant importing a million Private 
> objects...).
> This issue is about adding a simple filter to Replication on the sink-side 
> that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently 
> made read-only).
> Assigned myself. Need to do this so hbase-indexer can move to hbase2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-15 Thread wolfgang hoschek (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292858#comment-16292858
 ] 

wolfgang hoschek commented on HBASE-19494:
--

The configurable write time threshold for the filtering is per peer 
subscriptionId aka per table. 

Here is a proposal: Add a small interface (say "WALEntryFilter") that takes the 
entry's timestamp and table and returns whether the entry should be ignored, 
along these lines:

{code}
public interface WALEntryFilter { // this is a new Interface
   public boolean filter(TableName table, long writeTime);
}
{code}

Now modify ReplicationSink.replicateEntries slightly to call that filter if the 
connection implements it:

{code}
ReplicationSink.replicateEntries(...) {
  ...
  for (WALEntry entry : entries) {
 TableName table = 
TableName.valueOf(entry.getKey().getTableName().toByteArray());
 Connection conn = getConnection();
 if (conn instanceof WALEntryFilter) { // this is a new Interface
   if (((WALEntryFilter)conn).filter(table, entry.getKey().getWriteTime())) 
{
  continue; // ignore unwanted entry
   }
 }
 ... same as current code
   }
   ... same as current code   
}
{code}

Then the IndexerConnection implements not just Connection, but also implements 
the WALEntryFilter interface, along these lines:

{code}
public class IndexerConnection implements Connection, WALEntryFilter { 
   private Map timestampThresholds = ...
   public boolean filter(TableName table, long writeTime) {
   long timestampThreshold = timestampThresholds(table.getNameAsString());
   return writeTime < timestampThreshold;
   }
   ... and so on
}
{code}



> Create simple WALKey filter that can be plugged in on the Replication Sink
> --
>
> Key: HBASE-19494
> URL: https://issues.apache.org/jira/browse/HBASE-19494
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
>
> hbase-indexer used to look at WALKeys on the sink to see if their time of 
> creation was before the time at which the replication stream was enabled.
> In the parent redo, there is no means for doing this anymore (because WALKey 
> used to be Private and because to get at the WALKey in the Sink, you had to 
> override all of the Replication which meant importing a million Private 
> objects...).
> This issue is about adding a simple filter to Replication on the sink-side 
> that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently 
> made read-only).
> Assigned myself. Need to do this so hbase-indexer can move to hbase2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink

2017-12-14 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291857#comment-16291857
 ] 

stack commented on HBASE-19494:
---

This is follow-on from HBASE-18846.

It is awkward because replicating we operate on WALEntry and WALKey protobufs, 
not pojos. I can't pass a filter or CP pb WALEntry or WALKeys.  Asking 
hbase-indexer crew if I could just pass a few attributes from WALEntry as 
params on a simple filter.

> Create simple WALKey filter that can be plugged in on the Replication Sink
> --
>
> Key: HBASE-19494
> URL: https://issues.apache.org/jira/browse/HBASE-19494
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
>
> hbase-indexer used to look at WALKeys on the sink to see if their time of 
> creation was before the time at which the replication stream was enabled.
> In the parent redo, there is no means for doing this anymore (because WALKey 
> used to be Private and because to get at the WALKey in the Sink, you had to 
> override all of the Replication which meant importing a million Private 
> objects...).
> This issue is about adding a simple filter to Replication on the sink-side 
> that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently 
> made read-only).
> Assigned myself. Need to do this so hbase-indexer can move to hbase2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)