[jira] [Updated] (HBASE-22784) OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)
[ https://issues.apache.org/jira/browse/HBASE-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-22784: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-1 and branch-1.4. Thanks [~wchevreuil] > OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 > clusters) > - > > Key: HBASE-22784 > URL: https://issues.apache.org/jira/browse/HBASE-22784 > Project: HBase > Issue Type: Bug > Components: regionserver, Replication >Affects Versions: 1.4.9, 1.4.10 >Reporter: Solvannan R M >Assignee: Wellington Chevreuil >Priority: Blocker > Fix For: 1.5.0, 1.4.11 > > Attachments: HBASE-22784.branch-1.001.patch, > HBASE-22784.branch-1.002.patch, HBASE-22784.branch-1.003.patch, > HBASE-22784.branch-1.004.patch > > > When a cluster is passive (receiving edits only via replication) in a cyclic > replication setup of 2 clusters, OldWALs size keeps on growing. On analysing, > we observed the following behaviour. > # New entry is added to WAL (Edit replicated from other cluster). > # ReplicationSourceWALReaderThread(RSWALRT) reads and applies the configured > filters (due to cyclic replication setup, ClusterMarkingEntryFilter discards > new entry from other cluster). > # Entry is null, RSWALRT neither updates the batch stats > (WALEntryBatch.lastWalPosition) nor puts it in the entryBatchQueue. > # ReplicationSource thread is blocked in entryBachQueue.take(). > # So ReplicationSource#updateLogPosition has never invoked and WAL file is > never cleared from ReplicationQueue. > # Hence LogCleaner on the master, doesn't deletes the oldWAL files from > hadoop. > NOTE: When a new edit is added via hbase-client, ReplicationSource thread > process and clears the oldWAL files from replication queues and hence master > cleans up the WALs > Please provide us a solution > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22784) OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)
[ https://issues.apache.org/jira/browse/HBASE-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-22784: --- Fix Version/s: 1.4.11 > OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 > clusters) > - > > Key: HBASE-22784 > URL: https://issues.apache.org/jira/browse/HBASE-22784 > Project: HBase > Issue Type: Bug > Components: regionserver, Replication >Affects Versions: 1.4.9, 1.4.10 >Reporter: Solvannan R M >Assignee: Wellington Chevreuil >Priority: Blocker > Fix For: 1.5.0, 1.4.11 > > Attachments: HBASE-22784.branch-1.001.patch, > HBASE-22784.branch-1.002.patch, HBASE-22784.branch-1.003.patch, > HBASE-22784.branch-1.004.patch > > > When a cluster is passive (receiving edits only via replication) in a cyclic > replication setup of 2 clusters, OldWALs size keeps on growing. On analysing, > we observed the following behaviour. > # New entry is added to WAL (Edit replicated from other cluster). > # ReplicationSourceWALReaderThread(RSWALRT) reads and applies the configured > filters (due to cyclic replication setup, ClusterMarkingEntryFilter discards > new entry from other cluster). > # Entry is null, RSWALRT neither updates the batch stats > (WALEntryBatch.lastWalPosition) nor puts it in the entryBatchQueue. > # ReplicationSource thread is blocked in entryBachQueue.take(). > # So ReplicationSource#updateLogPosition has never invoked and WAL file is > never cleared from ReplicationQueue. > # Hence LogCleaner on the master, doesn't deletes the oldWAL files from > hadoop. > NOTE: When a new edit is added via hbase-client, ReplicationSource thread > process and clears the oldWAL files from replication queues and hence master > cleans up the WALs > Please provide us a solution > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22784) OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)
[ https://issues.apache.org/jira/browse/HBASE-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-22784: - Attachment: HBASE-22784.branch-1.004.patch > OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 > clusters) > - > > Key: HBASE-22784 > URL: https://issues.apache.org/jira/browse/HBASE-22784 > Project: HBase > Issue Type: Bug > Components: regionserver, Replication >Affects Versions: 1.4.9, 1.4.10 >Reporter: Solvannan R M >Assignee: Wellington Chevreuil >Priority: Blocker > Fix For: 1.5.0 > > Attachments: HBASE-22784.branch-1.001.patch, > HBASE-22784.branch-1.002.patch, HBASE-22784.branch-1.003.patch, > HBASE-22784.branch-1.004.patch > > > When a cluster is passive (receiving edits only via replication) in a cyclic > replication setup of 2 clusters, OldWALs size keeps on growing. On analysing, > we observed the following behaviour. > # New entry is added to WAL (Edit replicated from other cluster). > # ReplicationSourceWALReaderThread(RSWALRT) reads and applies the configured > filters (due to cyclic replication setup, ClusterMarkingEntryFilter discards > new entry from other cluster). > # Entry is null, RSWALRT neither updates the batch stats > (WALEntryBatch.lastWalPosition) nor puts it in the entryBatchQueue. > # ReplicationSource thread is blocked in entryBachQueue.take(). > # So ReplicationSource#updateLogPosition has never invoked and WAL file is > never cleared from ReplicationQueue. > # Hence LogCleaner on the master, doesn't deletes the oldWAL files from > hadoop. > NOTE: When a new edit is added via hbase-client, ReplicationSource thread > process and clears the oldWAL files from replication queues and hence master > cleans up the WALs > Please provide us a solution > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22784) OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)
[ https://issues.apache.org/jira/browse/HBASE-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-22784: - Attachment: HBASE-22784.branch-1.003.patch > OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 > clusters) > - > > Key: HBASE-22784 > URL: https://issues.apache.org/jira/browse/HBASE-22784 > Project: HBase > Issue Type: Bug > Components: regionserver, Replication >Affects Versions: 1.4.9, 1.4.10 >Reporter: Solvannan R M >Assignee: Wellington Chevreuil >Priority: Blocker > Fix For: 1.5.0 > > Attachments: HBASE-22784.branch-1.001.patch, > HBASE-22784.branch-1.002.patch, HBASE-22784.branch-1.003.patch > > > When a cluster is passive (receiving edits only via replication) in a cyclic > replication setup of 2 clusters, OldWALs size keeps on growing. On analysing, > we observed the following behaviour. > # New entry is added to WAL (Edit replicated from other cluster). > # ReplicationSourceWALReaderThread(RSWALRT) reads and applies the configured > filters (due to cyclic replication setup, ClusterMarkingEntryFilter discards > new entry from other cluster). > # Entry is null, RSWALRT neither updates the batch stats > (WALEntryBatch.lastWalPosition) nor puts it in the entryBatchQueue. > # ReplicationSource thread is blocked in entryBachQueue.take(). > # So ReplicationSource#updateLogPosition has never invoked and WAL file is > never cleared from ReplicationQueue. > # Hence LogCleaner on the master, doesn't deletes the oldWAL files from > hadoop. > NOTE: When a new edit is added via hbase-client, ReplicationSource thread > process and clears the oldWAL files from replication queues and hence master > cleans up the WALs > Please provide us a solution > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22784) OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)
[ https://issues.apache.org/jira/browse/HBASE-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-22784: - Status: Patch Available (was: In Progress) Got NPE while manually testing 1st patch. 2nd patch does work on my branch-1 test clusters. Tried active-active replication, set a table for replication on both, verified that replication is working both ways, then ran ltt for a new, not targeted to replication table, then verified oldWALs are getting deleted and log position reported by _DumpReplicationQueues_ is updated even when we have only edits not targeted for replication. > OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 > clusters) > - > > Key: HBASE-22784 > URL: https://issues.apache.org/jira/browse/HBASE-22784 > Project: HBase > Issue Type: Bug > Components: regionserver, Replication >Affects Versions: 1.4.10, 1.4.9 >Reporter: Solvannan R M >Assignee: Wellington Chevreuil >Priority: Blocker > Fix For: 1.5.0 > > Attachments: HBASE-22784.branch-1.001.patch, > HBASE-22784.branch-1.002.patch > > > When a cluster is passive (receiving edits only via replication) in a cyclic > replication setup of 2 clusters, OldWALs size keeps on growing. On analysing, > we observed the following behaviour. > # New entry is added to WAL (Edit replicated from other cluster). > # ReplicationSourceWALReaderThread(RSWALRT) reads and applies the configured > filters (due to cyclic replication setup, ClusterMarkingEntryFilter discards > new entry from other cluster). > # Entry is null, RSWALRT neither updates the batch stats > (WALEntryBatch.lastWalPosition) nor puts it in the entryBatchQueue. > # ReplicationSource thread is blocked in entryBachQueue.take(). > # So ReplicationSource#updateLogPosition has never invoked and WAL file is > never cleared from ReplicationQueue. > # Hence LogCleaner on the master, doesn't deletes the oldWAL files from > hadoop. > NOTE: When a new edit is added via hbase-client, ReplicationSource thread > process and clears the oldWAL files from replication queues and hence master > cleans up the WALs > Please provide us a solution > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22784) OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)
[ https://issues.apache.org/jira/browse/HBASE-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-22784: - Attachment: HBASE-22784.branch-1.002.patch > OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 > clusters) > - > > Key: HBASE-22784 > URL: https://issues.apache.org/jira/browse/HBASE-22784 > Project: HBase > Issue Type: Bug > Components: regionserver, Replication >Affects Versions: 1.4.9, 1.4.10 >Reporter: Solvannan R M >Assignee: Wellington Chevreuil >Priority: Blocker > Fix For: 1.5.0 > > Attachments: HBASE-22784.branch-1.001.patch, > HBASE-22784.branch-1.002.patch > > > When a cluster is passive (receiving edits only via replication) in a cyclic > replication setup of 2 clusters, OldWALs size keeps on growing. On analysing, > we observed the following behaviour. > # New entry is added to WAL (Edit replicated from other cluster). > # ReplicationSourceWALReaderThread(RSWALRT) reads and applies the configured > filters (due to cyclic replication setup, ClusterMarkingEntryFilter discards > new entry from other cluster). > # Entry is null, RSWALRT neither updates the batch stats > (WALEntryBatch.lastWalPosition) nor puts it in the entryBatchQueue. > # ReplicationSource thread is blocked in entryBachQueue.take(). > # So ReplicationSource#updateLogPosition has never invoked and WAL file is > never cleared from ReplicationQueue. > # Hence LogCleaner on the master, doesn't deletes the oldWAL files from > hadoop. > NOTE: When a new edit is added via hbase-client, ReplicationSource thread > process and clears the oldWAL files from replication queues and hence master > cleans up the WALs > Please provide us a solution > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22784) OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)
[ https://issues.apache.org/jira/browse/HBASE-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-22784: - Attachment: HBASE-22784.branch-1.001.patch > OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 > clusters) > - > > Key: HBASE-22784 > URL: https://issues.apache.org/jira/browse/HBASE-22784 > Project: HBase > Issue Type: Bug > Components: regionserver, Replication >Affects Versions: 1.4.9, 1.4.10 >Reporter: Solvannan R M >Assignee: Wellington Chevreuil >Priority: Blocker > Fix For: 1.5.0 > > Attachments: HBASE-22784.branch-1.001.patch > > > When a cluster is passive (receiving edits only via replication) in a cyclic > replication setup of 2 clusters, OldWALs size keeps on growing. On analysing, > we observed the following behaviour. > # New entry is added to WAL (Edit replicated from other cluster). > # ReplicationSourceWALReaderThread(RSWALRT) reads and applies the configured > filters (due to cyclic replication setup, ClusterMarkingEntryFilter discards > new entry from other cluster). > # Entry is null, RSWALRT neither updates the batch stats > (WALEntryBatch.lastWalPosition) nor puts it in the entryBatchQueue. > # ReplicationSource thread is blocked in entryBachQueue.take(). > # So ReplicationSource#updateLogPosition has never invoked and WAL file is > never cleared from ReplicationQueue. > # Hence LogCleaner on the master, doesn't deletes the oldWAL files from > hadoop. > NOTE: When a new edit is added via hbase-client, ReplicationSource thread > process and clears the oldWAL files from replication queues and hence master > cleans up the WALs > Please provide us a solution > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22784) OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)
[ https://issues.apache.org/jira/browse/HBASE-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-22784: --- Priority: Blocker (was: Major) Fix Version/s: 1.5.0 This at least impacts branch-1, which I want to immediately release 1.5.0 from (see my mail to dev@ today), so let me raise priority of this to blocker and set fix version to 1.5.0. Hopefully we can get a fix soon. > OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 > clusters) > - > > Key: HBASE-22784 > URL: https://issues.apache.org/jira/browse/HBASE-22784 > Project: HBase > Issue Type: Bug > Components: regionserver, Replication >Affects Versions: 1.4.9, 1.4.10 >Reporter: Solvannan R M >Assignee: Wellington Chevreuil >Priority: Blocker > Fix For: 1.5.0 > > > When a cluster is passive (receiving edits only via replication) in a cyclic > replication setup of 2 clusters, OldWALs size keeps on growing. On analysing, > we observed the following behaviour. > # New entry is added to WAL (Edit replicated from other cluster). > # ReplicationSourceWALReaderThread(RSWALRT) reads and applies the configured > filters (due to cyclic replication setup, ClusterMarkingEntryFilter discards > new entry from other cluster). > # Entry is null, RSWALRT neither updates the batch stats > (WALEntryBatch.lastWalPosition) nor puts it in the entryBatchQueue. > # ReplicationSource thread is blocked in entryBachQueue.take(). > # So ReplicationSource#updateLogPosition has never invoked and WAL file is > never cleared from ReplicationQueue. > # Hence LogCleaner on the master, doesn't deletes the oldWAL files from > hadoop. > NOTE: When a new edit is added via hbase-client, ReplicationSource thread > process and clears the oldWAL files from replication queues and hence master > cleans up the WALs > Please provide us a solution > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
