[jira] [Created] (TEPHRA-255) PruneUpperBoundWriterSupplierTest is flaky
Andreas Neumann created TEPHRA-255:
--
Summary: PruneUpperBoundWriterSupplierTest is flaky
Key: TEPHRA-255
URL: https://issues.apache.org/jira/browse/TEPHRA-255
Project: Tephra
Issue Type: Bug
Affects Versions: 0.13.0-incubating
Reporter: Andreas Neumann
Assignee: Poorna Chandra
The travis build, on rare occasions, fails with:
{noformat}
testSupplier(org.apache.tephra.hbase.txprune.PruneUpperBoundWriterSupplierTest)
Time elapsed: 2.599 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at
org.apache.tephra.hbase.txprune.PruneUpperBoundWriterSupplierTest.testSupplier(PruneUpperBoundWriterSupplierTest.java:120)
{noformat}
This looks like a race between the shutting down the writer and asserting that
it has shut down.
It may be caused by the fact that upon shutdown(), the PruneUpperBoundWriter
gives its flush thread only 1 ms to join. In slow test environments, that may
not be sufficient.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEPHRA-244) Invalid tx pruning does not handle deletion of tables well
[
https://issues.apache.org/jira/browse/TEPHRA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160738#comment-16160738
]
ASF GitHub Bot commented on TEPHRA-244:
---
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/55#discussion_r137981943
--- Diff: tephra-hbase-compat-1.1-base/src/test/resources/logback-test.xml
---
@@ -0,0 +1,39 @@
+
+
+
+
+
+
+ %d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
+
+
+
+
+
--- End diff --
I'll copy over this file to the other compat modules
> Invalid tx pruning does not handle deletion of tables well
> --
>
> Key: TEPHRA-244
> URL: https://issues.apache.org/jira/browse/TEPHRA-244
> Project: Tephra
> Issue Type: Bug
> Components: core, manager
>Affects Versions: 0.12.0-incubating
>Reporter: Andreas Neumann
>Assignee: Poorna Chandra
> Fix For: 0.13.0-incubating
>
>
> Suppose an application regularly creates and deletes tables (for example,
> temporary tables). In such a scenario, there will always be such a temporary
> table when pruning runs, and its regions will be recorded for that time.
> However, the region will be deleted before it ever compacts or flushes, and
> it will never record prune information. Because all prune times have such
> regions, there will never be a set of transactional regions that all have
> prune info, and pruning will never happen.
> The fix is to exclude deleted tables from the list of regions at each time.
> This is all regions of deleted tables (tables that do not exist any more),
> not deleted regions: a region may disappear due to a split and its data will
> be in new regions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] incubator-tephra pull request #55: TEPHRA-244 Remove regions of deleted tabl...
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/55#discussion_r137981943
--- Diff: tephra-hbase-compat-1.1-base/src/test/resources/logback-test.xml
---
@@ -0,0 +1,39 @@
+
+
+
+
+
+
+ %d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
+
+
+
+
+
--- End diff --
I'll copy over this file to the other compat modules
---
[jira] [Commented] (TEPHRA-244) Invalid tx pruning does not handle deletion of tables well
[
https://issues.apache.org/jira/browse/TEPHRA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160736#comment-16160736
]
ASF GitHub Bot commented on TEPHRA-244:
---
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/55#discussion_r137981906
--- Diff:
tephra-hbase-compat-1.1-base/src/test/java/org/apache/tephra/hbase/txprune/InvalidListPruneTest.java
---
@@ -384,7 +384,67 @@ public void testPruneEmptyTable() throws Exception {
hBaseAdmin.disableTable(txEmptyTable);
hBaseAdmin.deleteTable(txEmptyTable);
}
+ }
+
+ @Test
+ public void testPruneTransientTable() throws Exception {
+// Make sure that transient tables do not block the progress of pruning
+
+// Create a temp table
+TableName txTempTable = TableName.valueOf("tempTable");
+createTable(txTempTable.getName(), new byte[][]{family}, false,
+
Collections.singletonList(TestTransactionProcessor.class.getName()));
+
+TableName txDataTable2 = null;
+
+TransactionPruningPlugin transactionPruningPlugin = new
TestTransactionPruningPlugin();
+transactionPruningPlugin.initialize(conf);
+
+try {
+ long now1 = System.currentTimeMillis();
+ long inactiveTxTimeNow1 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
+ long noPruneUpperBound = -1;
+ long expectedPruneUpperBound1 = (now1 - 200) *
TxConstants.MAX_TX_PER_MS;
+ InMemoryTransactionStateCache.setTransactionSnapshot(
+new TransactionSnapshot(expectedPruneUpperBound1,
expectedPruneUpperBound1, expectedPruneUpperBound1,
+ImmutableSet.of(expectedPruneUpperBound1),
+ImmutableSortedMap.of()));
+
+ // fetch prune upper bound, there should be no prune upper bound
since nothing has been compacted yet.
+ // This run is only to store the initial set of regions
+ long pruneUpperBound1 =
transactionPruningPlugin.fetchPruneUpperBound(now1, inactiveTxTimeNow1);
+ Assert.assertEquals(noPruneUpperBound, pruneUpperBound1);
+ transactionPruningPlugin.pruneComplete(now1, noPruneUpperBound);
+
+ // Now delete the transient table
+ hBaseAdmin.disableTable(txTempTable);
+ hBaseAdmin.deleteTable(txTempTable);
+
+ // Compact the data table now
+ testUtil.compact(txDataTable1, true);
+ // Since the write to prune table happens async, we need to sleep a
bit before checking the state of the table
+ TimeUnit.SECONDS.sleep(2);
+
+ // Create a new table that will not be compacted
+ txDataTable2 = TableName.valueOf("invalidListPruneTestTable2");
+ createTable(txDataTable2.getName(), new byte[][]{family}, false,
+
Collections.singletonList(TestTransactionProcessor.class.getName()));
+
+ // fetch prune upper bound, there should be a prune upper bound even
though txTempTable does not exist anymore,
+ // and txDataTable2 has not been compacted/flushed yet
+ long now2 = System.currentTimeMillis();
+ long inactiveTxTimeNow2 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
+ long pruneUpperBound2 =
transactionPruningPlugin.fetchPruneUpperBound(now2, inactiveTxTimeNow2);
+ Assert.assertEquals(expectedPruneUpperBound1, pruneUpperBound2);
+ transactionPruningPlugin.pruneComplete(now2,
expectedPruneUpperBound1);
+} finally {
+ transactionPruningPlugin.destroy();
+ if (txDataTable2 != null) {
--- End diff --
👍
> Invalid tx pruning does not handle deletion of tables well
> --
>
> Key: TEPHRA-244
> URL: https://issues.apache.org/jira/browse/TEPHRA-244
> Project: Tephra
> Issue Type: Bug
> Components: core, manager
>Affects Versions: 0.12.0-incubating
>Reporter: Andreas Neumann
>Assignee: Poorna Chandra
> Fix For: 0.13.0-incubating
>
>
> Suppose an application regularly creates and deletes tables (for example,
> temporary tables). In such a scenario, there will always be such a temporary
> table when pruning runs, and its regions will be recorded for that time.
> However, the region will be deleted before it ever compacts or flushes, and
> it will never record prune information. Because all prune times have such
> regions, there will never be a set of transactional regions that all have
> prune info, and pruning will never happen.
> The fix is to exclude deleted tables from the list of regions at each time.
> This is all regions of deleted tables (tables that do not exist any
[GitHub] incubator-tephra pull request #55: TEPHRA-244 Remove regions of deleted tabl...
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/55#discussion_r137981906
--- Diff:
tephra-hbase-compat-1.1-base/src/test/java/org/apache/tephra/hbase/txprune/InvalidListPruneTest.java
---
@@ -384,7 +384,67 @@ public void testPruneEmptyTable() throws Exception {
hBaseAdmin.disableTable(txEmptyTable);
hBaseAdmin.deleteTable(txEmptyTable);
}
+ }
+
+ @Test
+ public void testPruneTransientTable() throws Exception {
+// Make sure that transient tables do not block the progress of pruning
+
+// Create a temp table
+TableName txTempTable = TableName.valueOf("tempTable");
+createTable(txTempTable.getName(), new byte[][]{family}, false,
+
Collections.singletonList(TestTransactionProcessor.class.getName()));
+
+TableName txDataTable2 = null;
+
+TransactionPruningPlugin transactionPruningPlugin = new
TestTransactionPruningPlugin();
+transactionPruningPlugin.initialize(conf);
+
+try {
+ long now1 = System.currentTimeMillis();
+ long inactiveTxTimeNow1 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
+ long noPruneUpperBound = -1;
+ long expectedPruneUpperBound1 = (now1 - 200) *
TxConstants.MAX_TX_PER_MS;
+ InMemoryTransactionStateCache.setTransactionSnapshot(
+new TransactionSnapshot(expectedPruneUpperBound1,
expectedPruneUpperBound1, expectedPruneUpperBound1,
+ImmutableSet.of(expectedPruneUpperBound1),
+ImmutableSortedMap.of()));
+
+ // fetch prune upper bound, there should be no prune upper bound
since nothing has been compacted yet.
+ // This run is only to store the initial set of regions
+ long pruneUpperBound1 =
transactionPruningPlugin.fetchPruneUpperBound(now1, inactiveTxTimeNow1);
+ Assert.assertEquals(noPruneUpperBound, pruneUpperBound1);
+ transactionPruningPlugin.pruneComplete(now1, noPruneUpperBound);
+
+ // Now delete the transient table
+ hBaseAdmin.disableTable(txTempTable);
+ hBaseAdmin.deleteTable(txTempTable);
+
+ // Compact the data table now
+ testUtil.compact(txDataTable1, true);
+ // Since the write to prune table happens async, we need to sleep a
bit before checking the state of the table
+ TimeUnit.SECONDS.sleep(2);
+
+ // Create a new table that will not be compacted
+ txDataTable2 = TableName.valueOf("invalidListPruneTestTable2");
+ createTable(txDataTable2.getName(), new byte[][]{family}, false,
+
Collections.singletonList(TestTransactionProcessor.class.getName()));
+
+ // fetch prune upper bound, there should be a prune upper bound even
though txTempTable does not exist anymore,
+ // and txDataTable2 has not been compacted/flushed yet
+ long now2 = System.currentTimeMillis();
+ long inactiveTxTimeNow2 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
+ long pruneUpperBound2 =
transactionPruningPlugin.fetchPruneUpperBound(now2, inactiveTxTimeNow2);
+ Assert.assertEquals(expectedPruneUpperBound1, pruneUpperBound2);
+ transactionPruningPlugin.pruneComplete(now2,
expectedPruneUpperBound1);
+} finally {
+ transactionPruningPlugin.destroy();
+ if (txDataTable2 != null) {
--- End diff --
ð
---
[jira] [Commented] (TEPHRA-244) Invalid tx pruning does not handle deletion of tables well
[
https://issues.apache.org/jira/browse/TEPHRA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160719#comment-16160719
]
ASF GitHub Bot commented on TEPHRA-244:
---
Github user anew commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/55#discussion_r137981190
--- Diff: tephra-hbase-compat-1.1-base/src/test/resources/logback-test.xml
---
@@ -0,0 +1,39 @@
+
+
+
+
+
+
+ %d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
+
+
+
+
+
--- End diff --
seems like this would be useful for all test cases... it removes a lot of
distracting log messages.
> Invalid tx pruning does not handle deletion of tables well
> --
>
> Key: TEPHRA-244
> URL: https://issues.apache.org/jira/browse/TEPHRA-244
> Project: Tephra
> Issue Type: Bug
> Components: core, manager
>Affects Versions: 0.12.0-incubating
>Reporter: Andreas Neumann
>Assignee: Poorna Chandra
> Fix For: 0.13.0-incubating
>
>
> Suppose an application regularly creates and deletes tables (for example,
> temporary tables). In such a scenario, there will always be such a temporary
> table when pruning runs, and its regions will be recorded for that time.
> However, the region will be deleted before it ever compacts or flushes, and
> it will never record prune information. Because all prune times have such
> regions, there will never be a set of transactional regions that all have
> prune info, and pruning will never happen.
> The fix is to exclude deleted tables from the list of regions at each time.
> This is all regions of deleted tables (tables that do not exist any more),
> not deleted regions: a region may disappear due to a split and its data will
> be in new regions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEPHRA-244) Invalid tx pruning does not handle deletion of tables well
[
https://issues.apache.org/jira/browse/TEPHRA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160718#comment-16160718
]
ASF GitHub Bot commented on TEPHRA-244:
---
Github user anew commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/55#discussion_r137981077
--- Diff:
tephra-hbase-compat-1.1-base/src/test/java/org/apache/tephra/hbase/txprune/InvalidListPruneTest.java
---
@@ -384,7 +384,67 @@ public void testPruneEmptyTable() throws Exception {
hBaseAdmin.disableTable(txEmptyTable);
hBaseAdmin.deleteTable(txEmptyTable);
}
+ }
+
+ @Test
+ public void testPruneTransientTable() throws Exception {
+// Make sure that transient tables do not block the progress of pruning
+
+// Create a temp table
+TableName txTempTable = TableName.valueOf("tempTable");
+createTable(txTempTable.getName(), new byte[][]{family}, false,
+
Collections.singletonList(TestTransactionProcessor.class.getName()));
+
+TableName txDataTable2 = null;
+
+TransactionPruningPlugin transactionPruningPlugin = new
TestTransactionPruningPlugin();
+transactionPruningPlugin.initialize(conf);
+
+try {
+ long now1 = System.currentTimeMillis();
+ long inactiveTxTimeNow1 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
+ long noPruneUpperBound = -1;
+ long expectedPruneUpperBound1 = (now1 - 200) *
TxConstants.MAX_TX_PER_MS;
+ InMemoryTransactionStateCache.setTransactionSnapshot(
+new TransactionSnapshot(expectedPruneUpperBound1,
expectedPruneUpperBound1, expectedPruneUpperBound1,
+ImmutableSet.of(expectedPruneUpperBound1),
+ImmutableSortedMap.of()));
+
+ // fetch prune upper bound, there should be no prune upper bound
since nothing has been compacted yet.
+ // This run is only to store the initial set of regions
+ long pruneUpperBound1 =
transactionPruningPlugin.fetchPruneUpperBound(now1, inactiveTxTimeNow1);
+ Assert.assertEquals(noPruneUpperBound, pruneUpperBound1);
+ transactionPruningPlugin.pruneComplete(now1, noPruneUpperBound);
+
+ // Now delete the transient table
+ hBaseAdmin.disableTable(txTempTable);
+ hBaseAdmin.deleteTable(txTempTable);
+
+ // Compact the data table now
+ testUtil.compact(txDataTable1, true);
+ // Since the write to prune table happens async, we need to sleep a
bit before checking the state of the table
+ TimeUnit.SECONDS.sleep(2);
+
+ // Create a new table that will not be compacted
+ txDataTable2 = TableName.valueOf("invalidListPruneTestTable2");
+ createTable(txDataTable2.getName(), new byte[][]{family}, false,
+
Collections.singletonList(TestTransactionProcessor.class.getName()));
+
+ // fetch prune upper bound, there should be a prune upper bound even
though txTempTable does not exist anymore,
+ // and txDataTable2 has not been compacted/flushed yet
+ long now2 = System.currentTimeMillis();
+ long inactiveTxTimeNow2 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
+ long pruneUpperBound2 =
transactionPruningPlugin.fetchPruneUpperBound(now2, inactiveTxTimeNow2);
+ Assert.assertEquals(expectedPruneUpperBound1, pruneUpperBound2);
+ transactionPruningPlugin.pruneComplete(now2,
expectedPruneUpperBound1);
+} finally {
+ transactionPruningPlugin.destroy();
+ if (txDataTable2 != null) {
--- End diff --
you should also do this for txTempTable. The test case deletes it, but if
it fails before that, the table never gets deleted.
> Invalid tx pruning does not handle deletion of tables well
> --
>
> Key: TEPHRA-244
> URL: https://issues.apache.org/jira/browse/TEPHRA-244
> Project: Tephra
> Issue Type: Bug
> Components: core, manager
>Affects Versions: 0.12.0-incubating
>Reporter: Andreas Neumann
>Assignee: Poorna Chandra
> Fix For: 0.13.0-incubating
>
>
> Suppose an application regularly creates and deletes tables (for example,
> temporary tables). In such a scenario, there will always be such a temporary
> table when pruning runs, and its regions will be recorded for that time.
> However, the region will be deleted before it ever compacts or flushes, and
> it will never record prune information. Because all prune times have such
> regions, there will never be a set of transactional regions that all have
> prune info, and pruning will never happen.
> The fix is to exclude deleted t
[jira] [Commented] (TEPHRA-244) Invalid tx pruning does not handle deletion of tables well
[ https://issues.apache.org/jira/browse/TEPHRA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160717#comment-16160717 ] ASF GitHub Bot commented on TEPHRA-244: --- Github user anew commented on a diff in the pull request: https://github.com/apache/incubator-tephra/pull/55#discussion_r137980868 --- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/HBaseTransactionPruningPlugin.java --- @@ -335,6 +351,25 @@ private long computePruneUpperBound(TimeRegions timeRegions) throws IOException return -1; } + private SortedSet filterDeletedRegions(final Set existingTables, --- End diff -- Better name this filterDeletedTables(), or filterDeletedTableRegions() - it does not filter out deleted regions - it does filter out all regions of deleted tables > Invalid tx pruning does not handle deletion of tables well > -- > > Key: TEPHRA-244 > URL: https://issues.apache.org/jira/browse/TEPHRA-244 > Project: Tephra > Issue Type: Bug > Components: core, manager >Affects Versions: 0.12.0-incubating >Reporter: Andreas Neumann >Assignee: Poorna Chandra > Fix For: 0.13.0-incubating > > > Suppose an application regularly creates and deletes tables (for example, > temporary tables). In such a scenario, there will always be such a temporary > table when pruning runs, and its regions will be recorded for that time. > However, the region will be deleted before it ever compacts or flushes, and > it will never record prune information. Because all prune times have such > regions, there will never be a set of transactional regions that all have > prune info, and pruning will never happen. > The fix is to exclude deleted tables from the list of regions at each time. > This is all regions of deleted tables (tables that do not exist any more), > not deleted regions: a region may disappear due to a split and its data will > be in new regions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] incubator-tephra pull request #55: TEPHRA-244 Remove regions of deleted tabl...
Github user anew commented on a diff in the pull request: https://github.com/apache/incubator-tephra/pull/55#discussion_r137980868 --- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/HBaseTransactionPruningPlugin.java --- @@ -335,6 +351,25 @@ private long computePruneUpperBound(TimeRegions timeRegions) throws IOException return -1; } + private SortedSet filterDeletedRegions(final Set existingTables, --- End diff -- Better name this filterDeletedTables(), or filterDeletedTableRegions() - it does not filter out deleted regions - it does filter out all regions of deleted tables ---
[GitHub] incubator-tephra pull request #55: TEPHRA-244 Remove regions of deleted tabl...
Github user anew commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/55#discussion_r137981190
--- Diff: tephra-hbase-compat-1.1-base/src/test/resources/logback-test.xml
---
@@ -0,0 +1,39 @@
+
+
+
+
+
+
+ %d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
+
+
+
+
+
--- End diff --
seems like this would be useful for all test cases... it removes a lot of
distracting log messages.
---
[GitHub] incubator-tephra pull request #55: TEPHRA-244 Remove regions of deleted tabl...
Github user anew commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/55#discussion_r137981077
--- Diff:
tephra-hbase-compat-1.1-base/src/test/java/org/apache/tephra/hbase/txprune/InvalidListPruneTest.java
---
@@ -384,7 +384,67 @@ public void testPruneEmptyTable() throws Exception {
hBaseAdmin.disableTable(txEmptyTable);
hBaseAdmin.deleteTable(txEmptyTable);
}
+ }
+
+ @Test
+ public void testPruneTransientTable() throws Exception {
+// Make sure that transient tables do not block the progress of pruning
+
+// Create a temp table
+TableName txTempTable = TableName.valueOf("tempTable");
+createTable(txTempTable.getName(), new byte[][]{family}, false,
+
Collections.singletonList(TestTransactionProcessor.class.getName()));
+
+TableName txDataTable2 = null;
+
+TransactionPruningPlugin transactionPruningPlugin = new
TestTransactionPruningPlugin();
+transactionPruningPlugin.initialize(conf);
+
+try {
+ long now1 = System.currentTimeMillis();
+ long inactiveTxTimeNow1 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
+ long noPruneUpperBound = -1;
+ long expectedPruneUpperBound1 = (now1 - 200) *
TxConstants.MAX_TX_PER_MS;
+ InMemoryTransactionStateCache.setTransactionSnapshot(
+new TransactionSnapshot(expectedPruneUpperBound1,
expectedPruneUpperBound1, expectedPruneUpperBound1,
+ImmutableSet.of(expectedPruneUpperBound1),
+ImmutableSortedMap.of()));
+
+ // fetch prune upper bound, there should be no prune upper bound
since nothing has been compacted yet.
+ // This run is only to store the initial set of regions
+ long pruneUpperBound1 =
transactionPruningPlugin.fetchPruneUpperBound(now1, inactiveTxTimeNow1);
+ Assert.assertEquals(noPruneUpperBound, pruneUpperBound1);
+ transactionPruningPlugin.pruneComplete(now1, noPruneUpperBound);
+
+ // Now delete the transient table
+ hBaseAdmin.disableTable(txTempTable);
+ hBaseAdmin.deleteTable(txTempTable);
+
+ // Compact the data table now
+ testUtil.compact(txDataTable1, true);
+ // Since the write to prune table happens async, we need to sleep a
bit before checking the state of the table
+ TimeUnit.SECONDS.sleep(2);
+
+ // Create a new table that will not be compacted
+ txDataTable2 = TableName.valueOf("invalidListPruneTestTable2");
+ createTable(txDataTable2.getName(), new byte[][]{family}, false,
+
Collections.singletonList(TestTransactionProcessor.class.getName()));
+
+ // fetch prune upper bound, there should be a prune upper bound even
though txTempTable does not exist anymore,
+ // and txDataTable2 has not been compacted/flushed yet
+ long now2 = System.currentTimeMillis();
+ long inactiveTxTimeNow2 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
+ long pruneUpperBound2 =
transactionPruningPlugin.fetchPruneUpperBound(now2, inactiveTxTimeNow2);
+ Assert.assertEquals(expectedPruneUpperBound1, pruneUpperBound2);
+ transactionPruningPlugin.pruneComplete(now2,
expectedPruneUpperBound1);
+} finally {
+ transactionPruningPlugin.destroy();
+ if (txDataTable2 != null) {
--- End diff --
you should also do this for txTempTable. The test case deletes it, but if
it fails before that, the table never gets deleted.
---
[jira] [Commented] (TEPHRA-244) Invalid tx pruning does not handle deletion of tables well
[ https://issues.apache.org/jira/browse/TEPHRA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160694#comment-16160694 ] Poorna Chandra commented on TEPHRA-244: --- https://github.com/apache/incubator-tephra/pull/55 - This PR just removes regions of deleted tables when computing prune upper bound. This ensures that transient tables do not block pruning. However, if a transient table is created/deleted again with the same name and never gets a chance to be compacted, then it has a potential to block pruning. Filed TEPHRA-254 to handle such cases. > Invalid tx pruning does not handle deletion of tables well > -- > > Key: TEPHRA-244 > URL: https://issues.apache.org/jira/browse/TEPHRA-244 > Project: Tephra > Issue Type: Bug > Components: core, manager >Affects Versions: 0.12.0-incubating >Reporter: Andreas Neumann >Assignee: Poorna Chandra > Fix For: 0.13.0-incubating > > > Suppose an application regularly creates and deletes tables (for example, > temporary tables). In such a scenario, there will always be such a temporary > table when pruning runs, and its regions will be recorded for that time. > However, the region will be deleted before it ever compacts or flushes, and > it will never record prune information. Because all prune times have such > regions, there will never be a set of transactional regions that all have > prune info, and pruning will never happen. > The fix is to exclude deleted tables from the list of regions at each time. > This is all regions of deleted tables (tables that do not exist any more), > not deleted regions: a region may disappear due to a split and its data will > be in new regions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEPHRA-254) Handle transient tables that get recreated multiple times during pruning
Poorna Chandra created TEPHRA-254: - Summary: Handle transient tables that get recreated multiple times during pruning Key: TEPHRA-254 URL: https://issues.apache.org/jira/browse/TEPHRA-254 Project: Tephra Issue Type: Bug Reporter: Poorna Chandra Assignee: Poorna Chandra TEPHRA-244 handles temporary tables during pruning by removing regions of deleted tables when computing prune upper bound. This ensures that transient tables do not block pruning. However, if a transient table is created/deleted again with the same name and never gets a chance to be compacted, then it has a potential to block pruning. One way to handle this would be to keep track of creation time for every table and use that information to remove regions of previous iterations of a transient table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEPHRA-244) Invalid tx pruning does not handle deletion of tables well
[ https://issues.apache.org/jira/browse/TEPHRA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160669#comment-16160669 ] ASF GitHub Bot commented on TEPHRA-244: --- GitHub user poornachandra opened a pull request: https://github.com/apache/incubator-tephra/pull/55 TEPHRA-244 Remove regions of deleted tables when computing prune upper bound JIRA - https://issues.apache.org/jira/browse/TEPHRA-244 This PR just removes regions of deleted tables when computing prune upper bound. This ensures that transient tables do not block pruning. However, if a transient table is created/deleted again with the same name and never gets a chance to be compacted, then it has a potential to block pruning. You can merge this pull request into a Git repository by running: $ git pull https://github.com/poornachandra/incubator-tephra feature/prune-transient-tables Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-tephra/pull/55.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #55 commit ddfed1bf9f764cf2a29d900ed55f283b45861d02 Author: poorna Date: 2017-09-11T04:38:08Z TEPHRA-244 Remove regions of deleted tables when computing prune upper bound > Invalid tx pruning does not handle deletion of tables well > -- > > Key: TEPHRA-244 > URL: https://issues.apache.org/jira/browse/TEPHRA-244 > Project: Tephra > Issue Type: Bug > Components: core, manager >Affects Versions: 0.12.0-incubating >Reporter: Andreas Neumann >Assignee: Poorna Chandra > Fix For: 0.13.0-incubating > > > Suppose an application regularly creates and deletes tables (for example, > temporary tables). In such a scenario, there will always be such a temporary > table when pruning runs, and its regions will be recorded for that time. > However, the region will be deleted before it ever compacts or flushes, and > it will never record prune information. Because all prune times have such > regions, there will never be a set of transactional regions that all have > prune info, and pruning will never happen. > The fix is to exclude deleted tables from the list of regions at each time. > This is all regions of deleted tables (tables that do not exist any more), > not deleted regions: a region may disappear due to a split and its data will > be in new regions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] incubator-tephra pull request #55: TEPHRA-244 Remove regions of deleted tabl...
GitHub user poornachandra opened a pull request: https://github.com/apache/incubator-tephra/pull/55 TEPHRA-244 Remove regions of deleted tables when computing prune upper bound JIRA - https://issues.apache.org/jira/browse/TEPHRA-244 This PR just removes regions of deleted tables when computing prune upper bound. This ensures that transient tables do not block pruning. However, if a transient table is created/deleted again with the same name and never gets a chance to be compacted, then it has a potential to block pruning. You can merge this pull request into a Git repository by running: $ git pull https://github.com/poornachandra/incubator-tephra feature/prune-transient-tables Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-tephra/pull/55.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #55 commit ddfed1bf9f764cf2a29d900ed55f283b45861d02 Author: poorna Date: 2017-09-11T04:38:08Z TEPHRA-244 Remove regions of deleted tables when computing prune upper bound ---
[jira] [Resolved] (TEPHRA-253) TransactionProcessorTest is sometimes flaky
[
https://issues.apache.org/jira/browse/TEPHRA-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andreas Neumann resolved TEPHRA-253.
Resolution: Fixed
> TransactionProcessorTest is sometimes flaky
> ---
>
> Key: TEPHRA-253
> URL: https://issues.apache.org/jira/browse/TEPHRA-253
> Project: Tephra
> Issue Type: Bug
>Affects Versions: 0.12.0-incubating
>Reporter: Andreas Neumann
>Assignee: Andreas Neumann
> Fix For: 0.13.0-incubating
>
>
> The test sometimes fails as follows:
> {noformat}
> Running org.apache.tephra.hbase.coprocessor.TransactionProcessorTest
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.741 sec <<<
> FAILURE!
> testFamilyDeleteTimestamp(org.apache.tephra.hbase.coprocessor.TransactionProcessorTest)
> Time elapsed: 1.526 sec
> testTransactionStateCache(org.apache.tephra.hbase.coprocessor.TransactionProcessorTest)
> Time elapsed: 0.053 sec
> testDataJanitorRegionScanner(org.apache.tephra.hbase.coprocessor.TransactionProcessorTest)
> Time elapsed: 0.288 sec <<< FAILURE!
> org.junit.internal.ArrayComparisonFailure: arrays first differed at element
> [3]; expected:<4> but was:<1>
> at
> org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
> at org.junit.Assert.internalArrayEquals(Assert.java:473)
> at org.junit.Assert.assertArrayEquals(Assert.java:294)
> at org.junit.Assert.assertArrayEquals(Assert.java:305)
> at
> org.apache.tephra.hbase.coprocessor.TransactionProcessorTest.assertKeyValueMatches(TransactionProcessorTest.java:593)
> at
> org.apache.tephra.hbase.coprocessor.TransactionProcessorTest.assertKeyValueMatches(TransactionProcessorTest.java:585)
> at
> org.apache.tephra.hbase.coprocessor.TransactionProcessorTest.testDataJanitorRegionScanner(TransactionProcessorTest.java:190)
> {noformat}
> It is not clear what is causing this, most likely the region server did not
> have an up-to-date transaction state snapshot at the time of the lfush (that
> might be due to TEPHRA-239 orTEPHRA-249, or it might be a condition where
> flush() has no effect because the region is already flushing,
> Let's observe this and gather more information when/if it happens again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] incubator-tephra pull request #54: (TEPHRA-253) Fix flaky TransactionProcess...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-tephra/pull/54 ---
[jira] [Commented] (TEPHRA-253) TransactionProcessorTest is sometimes flaky
[
https://issues.apache.org/jira/browse/TEPHRA-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160643#comment-16160643
]
ASF GitHub Bot commented on TEPHRA-253:
---
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-tephra/pull/54
> TransactionProcessorTest is sometimes flaky
> ---
>
> Key: TEPHRA-253
> URL: https://issues.apache.org/jira/browse/TEPHRA-253
> Project: Tephra
> Issue Type: Bug
>Affects Versions: 0.12.0-incubating
>Reporter: Andreas Neumann
>Assignee: Andreas Neumann
> Fix For: 0.13.0-incubating
>
>
> The test sometimes fails as follows:
> {noformat}
> Running org.apache.tephra.hbase.coprocessor.TransactionProcessorTest
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.741 sec <<<
> FAILURE!
> testFamilyDeleteTimestamp(org.apache.tephra.hbase.coprocessor.TransactionProcessorTest)
> Time elapsed: 1.526 sec
> testTransactionStateCache(org.apache.tephra.hbase.coprocessor.TransactionProcessorTest)
> Time elapsed: 0.053 sec
> testDataJanitorRegionScanner(org.apache.tephra.hbase.coprocessor.TransactionProcessorTest)
> Time elapsed: 0.288 sec <<< FAILURE!
> org.junit.internal.ArrayComparisonFailure: arrays first differed at element
> [3]; expected:<4> but was:<1>
> at
> org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
> at org.junit.Assert.internalArrayEquals(Assert.java:473)
> at org.junit.Assert.assertArrayEquals(Assert.java:294)
> at org.junit.Assert.assertArrayEquals(Assert.java:305)
> at
> org.apache.tephra.hbase.coprocessor.TransactionProcessorTest.assertKeyValueMatches(TransactionProcessorTest.java:593)
> at
> org.apache.tephra.hbase.coprocessor.TransactionProcessorTest.assertKeyValueMatches(TransactionProcessorTest.java:585)
> at
> org.apache.tephra.hbase.coprocessor.TransactionProcessorTest.testDataJanitorRegionScanner(TransactionProcessorTest.java:190)
> {noformat}
> It is not clear what is causing this, most likely the region server did not
> have an up-to-date transaction state snapshot at the time of the lfush (that
> might be due to TEPHRA-239 orTEPHRA-249, or it might be a condition where
> flush() has no effect because the region is already flushing,
> Let's observe this and gather more information when/if it happens again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
