[jira] [Commented] (BOOKKEEPER-792) Split bookkeeper client into a separate maven module
[
https://issues.apache.org/jira/browse/BOOKKEEPER-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195546#comment-14195546
]
Hadoop QA commented on BOOKKEEPER-792:
--
Testing JIRA BOOKKEEPER-792
Patch
[0001-Split-the-bookkeeper-client-from-the-server.patch|https://issues.apache.org/jira/secure/attachment/12674744/0001-Split-the-bookkeeper-client-from-the-server.patch]
downloaded at Tue Nov 4 01:13:45 UTC 2014
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
120
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 FINDBUGS{color}
.{color:green}+1{color} the patch does not seem to introduce new Findbugs
warnings
{color:red}-1 TESTS{color}
.Tests run: 929
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
. testSimpleChat(org.apache.hedwig.jms.BasicJMSTest)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/bookkeeper-trunk-precommit-build/808/
> Split bookkeeper client into a separate maven module
>
>
> Key: BOOKKEEPER-792
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-792
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
> Fix For: 4.4.0
>
> Attachments: 0001-Split-the-bookkeeper-client-from-the-server.patch,
> 0001-Split-the-bookkeeper-client-from-the-server.patch,
> 0001-Split-the-bookkeeper-client-from-the-server.patch
>
>
> Right now, to include bookkeeper as a dependency, you need to include
> org.apache.bookkeeper:bookkeeper-server
> This pulls in all the bookkeeper-server code, and all the server dependencies
> as well as the client. The server dependencies are larger than the client
> dependencies. For one thing, it pulls in jna and log4j which should never be
> pulled in from the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-795) Race condition causes writes to hang if ledger is fences
[
https://issues.apache.org/jira/browse/BOOKKEEPER-795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195514#comment-14195514
]
Hadoop QA commented on BOOKKEEPER-795:
--
Testing JIRA BOOKKEEPER-795
Patch
[0001-Made-ledger-metadata-immutable.patch|https://issues.apache.org/jira/secure/attachment/12678958/0001-Made-ledger-metadata-immutable.patch]
downloaded at Tue Nov 4 00:37:32 UTC 2014
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
120
.{color:green}+1{color} the patch does adds/modifies 20 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1 FINDBUGS{color}
.{color:red}-1{color} the patch seems to introduce 2 new Findbugs
warning(s) in module(s) [bookkeeper-server]
{color:green}+1 TESTS{color}
.Tests run: 930
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/bookkeeper-trunk-precommit-build/807/
> Race condition causes writes to hang if ledger is fences
>
>
> Key: BOOKKEEPER-795
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-795
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Blocker
> Fix For: 4.4.0
>
> Attachments: 0001-Demonstrate-race-condition.patch,
> 0001-Made-ledger-metadata-immutable.patch,
> TEST-org.apache.bookkeeper.client.LedgerCloseTest.xml
>
>
> If a ledger is fenced while the write is still writing to it, some of the
> writes will fail to ever complete.
> I've attached the log of this happening along with a test case that will
> trigger the behaviour.
> What appears to be happening is that when the fence occurs, the first write
> after the fence gets an unrecoverable error, so tries to close the ledger.
> Closing the ledger sets the closed flag on the ledger metadata, and tries to
> write it, which fails as the metadata in zookeeper was modified by the
> fencing operation, so the close op fails, resets the closed status for a
> moment, a write operation gets through, which then fails with a fencing
> error, so we try to close the ledger, but the other close operation has since
> closed the ledger in our metadata, so nothing happens, and the write hangs
> forever.
> There's a number of issues here, but foremost, the ledger metadata that the
> handle is using should only ever represent what is actually in zookeeper.
> Having various parts of the code flipping bits just explodes the state space.
> The LedgerMetadata object itself should be immutable, and should only be
> modified, as a local variable, using a builder, before writing to zookeeper.
> Only when the zookeeper operation succeeds should we update the reference
> which LedgerHandle has access to.
> There's also a problem in how we handle pendingaddops when we close. Really
> it shouldn't be possible for a write op to get through after a closure, but
> we should be defensive here and error out anything that has gotten through,
> adding a big old log message to alert us that this cases that shouldn't
> happen, is happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-795) Race condition causes writes to hang if ledger is fences
[
https://issues.apache.org/jira/browse/BOOKKEEPER-795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195462#comment-14195462
]
Hadoop QA commented on BOOKKEEPER-795:
--
Testing JIRA BOOKKEEPER-795
Patch
[0001-Made-ledger-metadata-immutable.patch|https://issues.apache.org/jira/secure/attachment/12678958/0001-Made-ledger-metadata-immutable.patch]
downloaded at Tue Nov 4 00:01:05 UTC 2014
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
120
.{color:green}+1{color} the patch does adds/modifies 20 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1 FINDBUGS{color}
.{color:red}-1{color} the patch seems to introduce 2 new Findbugs
warning(s) in module(s) [bookkeeper-server]
{color:green}+1 TESTS{color}
.Tests run: 930
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/bookkeeper-trunk-precommit-build/806/
> Race condition causes writes to hang if ledger is fences
>
>
> Key: BOOKKEEPER-795
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-795
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Blocker
> Fix For: 4.4.0
>
> Attachments: 0001-Demonstrate-race-condition.patch,
> 0001-Made-ledger-metadata-immutable.patch,
> TEST-org.apache.bookkeeper.client.LedgerCloseTest.xml
>
>
> If a ledger is fenced while the write is still writing to it, some of the
> writes will fail to ever complete.
> I've attached the log of this happening along with a test case that will
> trigger the behaviour.
> What appears to be happening is that when the fence occurs, the first write
> after the fence gets an unrecoverable error, so tries to close the ledger.
> Closing the ledger sets the closed flag on the ledger metadata, and tries to
> write it, which fails as the metadata in zookeeper was modified by the
> fencing operation, so the close op fails, resets the closed status for a
> moment, a write operation gets through, which then fails with a fencing
> error, so we try to close the ledger, but the other close operation has since
> closed the ledger in our metadata, so nothing happens, and the write hangs
> forever.
> There's a number of issues here, but foremost, the ledger metadata that the
> handle is using should only ever represent what is actually in zookeeper.
> Having various parts of the code flipping bits just explodes the state space.
> The LedgerMetadata object itself should be immutable, and should only be
> modified, as a local variable, using a builder, before writing to zookeeper.
> Only when the zookeeper operation succeeds should we update the reference
> which LedgerHandle has access to.
> There's also a problem in how we handle pendingaddops when we close. Really
> it shouldn't be possible for a write op to get through after a closure, but
> we should be defensive here and error out anything that has gotten through,
> adding a big old log message to alert us that this cases that shouldn't
> happen, is happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-795) Race condition causes writes to hang if ledger is fences
[
https://issues.apache.org/jira/browse/BOOKKEEPER-795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195381#comment-14195381
]
Hadoop QA commented on BOOKKEEPER-795:
--
Testing JIRA BOOKKEEPER-795
Patch
[0001-Made-ledger-metadata-immutable.patch|https://issues.apache.org/jira/secure/attachment/12678958/0001-Made-ledger-metadata-immutable.patch]
downloaded at Mon Nov 3 23:10:17 UTC 2014
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
120
.{color:green}+1{color} the patch does adds/modifies 20 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1 FINDBUGS{color}
.{color:red}-1{color} the patch seems to introduce 2 new Findbugs
warning(s) in module(s) [bookkeeper-server]
{color:green}+1 TESTS{color}
.Tests run: 930
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/bookkeeper-trunk-precommit-build/805/
> Race condition causes writes to hang if ledger is fences
>
>
> Key: BOOKKEEPER-795
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-795
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Blocker
> Fix For: 4.4.0
>
> Attachments: 0001-Demonstrate-race-condition.patch,
> 0001-Made-ledger-metadata-immutable.patch,
> TEST-org.apache.bookkeeper.client.LedgerCloseTest.xml
>
>
> If a ledger is fenced while the write is still writing to it, some of the
> writes will fail to ever complete.
> I've attached the log of this happening along with a test case that will
> trigger the behaviour.
> What appears to be happening is that when the fence occurs, the first write
> after the fence gets an unrecoverable error, so tries to close the ledger.
> Closing the ledger sets the closed flag on the ledger metadata, and tries to
> write it, which fails as the metadata in zookeeper was modified by the
> fencing operation, so the close op fails, resets the closed status for a
> moment, a write operation gets through, which then fails with a fencing
> error, so we try to close the ledger, but the other close operation has since
> closed the ledger in our metadata, so nothing happens, and the write hangs
> forever.
> There's a number of issues here, but foremost, the ledger metadata that the
> handle is using should only ever represent what is actually in zookeeper.
> Having various parts of the code flipping bits just explodes the state space.
> The LedgerMetadata object itself should be immutable, and should only be
> modified, as a local variable, using a builder, before writing to zookeeper.
> Only when the zookeeper operation succeeds should we update the reference
> which LedgerHandle has access to.
> There's also a problem in how we handle pendingaddops when we close. Really
> it shouldn't be possible for a write op to get through after a closure, but
> we should be defensive here and error out anything that has gotten through,
> adding a big old log message to alert us that this cases that shouldn't
> happen, is happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-795) Race condition causes writes to hang if ledger is fences
[
https://issues.apache.org/jira/browse/BOOKKEEPER-795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194783#comment-14194783
]
Hadoop QA commented on BOOKKEEPER-795:
--
Testing JIRA BOOKKEEPER-795
Patch
[0001-Made-ledger-metadata-immutable.patch|https://issues.apache.org/jira/secure/attachment/12678958/0001-Made-ledger-metadata-immutable.patch]
downloaded at Mon Nov 3 17:10:17 UTC 2014
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
120
.{color:green}+1{color} the patch does adds/modifies 20 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1 FINDBUGS{color}
.{color:red}-1{color} the patch seems to introduce 2 new Findbugs
warning(s) in module(s) [bookkeeper-server]
{color:red}-1 TESTS{color}
.Tests run: 930
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
. testSimpleChat(org.apache.hedwig.jms.BasicJMSTest)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/bookkeeper-trunk-precommit-build/804/
> Race condition causes writes to hang if ledger is fences
>
>
> Key: BOOKKEEPER-795
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-795
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Blocker
> Fix For: 4.4.0
>
> Attachments: 0001-Demonstrate-race-condition.patch,
> 0001-Made-ledger-metadata-immutable.patch,
> TEST-org.apache.bookkeeper.client.LedgerCloseTest.xml
>
>
> If a ledger is fenced while the write is still writing to it, some of the
> writes will fail to ever complete.
> I've attached the log of this happening along with a test case that will
> trigger the behaviour.
> What appears to be happening is that when the fence occurs, the first write
> after the fence gets an unrecoverable error, so tries to close the ledger.
> Closing the ledger sets the closed flag on the ledger metadata, and tries to
> write it, which fails as the metadata in zookeeper was modified by the
> fencing operation, so the close op fails, resets the closed status for a
> moment, a write operation gets through, which then fails with a fencing
> error, so we try to close the ledger, but the other close operation has since
> closed the ledger in our metadata, so nothing happens, and the write hangs
> forever.
> There's a number of issues here, but foremost, the ledger metadata that the
> handle is using should only ever represent what is actually in zookeeper.
> Having various parts of the code flipping bits just explodes the state space.
> The LedgerMetadata object itself should be immutable, and should only be
> modified, as a local variable, using a builder, before writing to zookeeper.
> Only when the zookeeper operation succeeds should we update the reference
> which LedgerHandle has access to.
> There's also a problem in how we handle pendingaddops when we close. Really
> it shouldn't be possible for a write op to get through after a closure, but
> we should be defensive here and error out anything that has gotten through,
> adding a big old log message to alert us that this cases that shouldn't
> happen, is happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-795) Race condition causes writes to hang if ledger is fences
[ https://issues.apache.org/jira/browse/BOOKKEEPER-795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194734#comment-14194734 ] Ivan Kelly commented on BOOKKEEPER-795: --- Review board: https://reviews.apache.org/r/27529/ > Race condition causes writes to hang if ledger is fences > > > Key: BOOKKEEPER-795 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-795 > Project: Bookkeeper > Issue Type: Bug >Reporter: Ivan Kelly >Assignee: Ivan Kelly >Priority: Blocker > Fix For: 4.4.0 > > Attachments: 0001-Demonstrate-race-condition.patch, > 0001-Made-ledger-metadata-immutable.patch, > TEST-org.apache.bookkeeper.client.LedgerCloseTest.xml > > > If a ledger is fenced while the write is still writing to it, some of the > writes will fail to ever complete. > I've attached the log of this happening along with a test case that will > trigger the behaviour. > What appears to be happening is that when the fence occurs, the first write > after the fence gets an unrecoverable error, so tries to close the ledger. > Closing the ledger sets the closed flag on the ledger metadata, and tries to > write it, which fails as the metadata in zookeeper was modified by the > fencing operation, so the close op fails, resets the closed status for a > moment, a write operation gets through, which then fails with a fencing > error, so we try to close the ledger, but the other close operation has since > closed the ledger in our metadata, so nothing happens, and the write hangs > forever. > There's a number of issues here, but foremost, the ledger metadata that the > handle is using should only ever represent what is actually in zookeeper. > Having various parts of the code flipping bits just explodes the state space. > The LedgerMetadata object itself should be immutable, and should only be > modified, as a local variable, using a builder, before writing to zookeeper. > Only when the zookeeper operation succeeds should we update the reference > which LedgerHandle has access to. > There's also a problem in how we handle pendingaddops when we close. Really > it shouldn't be possible for a write op to get through after a closure, but > we should be defensive here and error out anything that has gotten through, > adding a big old log message to alert us that this cases that shouldn't > happen, is happening. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 27529: BOOKKEEPER-795 Race condition causes writes to hang if ledger is fences
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27529/ --- Review request for bookkeeper. Bugs: BOOKKEEPER-795 https://issues.apache.org/jira/browse/BOOKKEEPER-795 Repository: bookkeeper-git Description --- Made ledger metadata immutable Ensembles are now ImmutableMap>. The LedgerManager manager interface has changed to avoid implicitly updating the version of a LedgerMetadata object. Recovery, Closing and handling bookie failures have now changed, as there are no longer merge operations. If a write to zookeeper fails, then recovery/closing/failureHandling are retried with the new metadata, provided that assumptions are not broken. The interesting changes are in LedgerHandle and LedgerMetadata. There are a lot of changes from ArrayList -> ImmutableList,List Also a lot of changes from currentEnsemble -> getCurrentEnsemble. One test had to change. With the old code, if someone fenced a ledger, closing would fail. This isn't always the case now. If there's lac of the writing ledger matches the lastEntryId of the new metadata, then close can succeed. ConditionalSetTest has changed to reflect this. Diffs - bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java 18a801c bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java 3f2580f bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerCreateOp.java fe223af bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerFragment.java 6aadb8a bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerFragmentReplicator.java 4501524 bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java 7204d6c bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerMetadata.java a20f34a bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerRecoveryOp.java 7ed7aa2 bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java 1b92d09 bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingReadOp.java e548f3d bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedOp.java af21f44 bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadOnlyLedgerHandle.java 8de4092 bookkeeper-server/src/main/java/org/apache/bookkeeper/client/TryReadLastConfirmedOp.java 01b81c9 bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/AbstractZkLedgerManager.java 0fc8afe bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/CleanupLedgerManager.java a873112 bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManager.java 2bc4258 bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManager.java 7f2df73 bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManager.java 7229028 bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/MSLedgerManagerFactory.java 2510b89 bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/BookieLedgerIndexer.java 1b4efaf bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/ReplicationWorker.java 02154e5 bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/CompactionTest.java ef4cea8 bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java f18e159 bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieWriteLedgerTest.java 7b77c48 bookkeeper-server/src/test/java/org/apache/bookkeeper/client/ClientUtil.java dc43b2c bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LedgerCloseTest.java eef56b0 bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LedgerHandleAdapter.java 9af527a bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LedgerRecoveryTest.java f54cde1 bookkeeper-server/src/test/java/org/apache/bookkeeper/client/SlowBookieTest.java 521d1e3 bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestLedgerChecker.java eb61c21 bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestLedgerFragmentReplication.java e4f744f bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestSpeculativeRead.java 2a6c71d bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestTryReadLastConfirmed.java ac0afb6 bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestWatchEnsembleChange.java eb833a3 bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java 19aab44 bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/AuditorPeriodicBookieCheckTest.java 91aae77 bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/BookieAutoRecoveryTest.j
[jira] [Updated] (BOOKKEEPER-795) Race condition causes writes to hang if ledger is fences
[ https://issues.apache.org/jira/browse/BOOKKEEPER-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated BOOKKEEPER-795: -- Attachment: 0001-Made-ledger-metadata-immutable.patch Patch makes LedgerMetadata immutable. Ensembles are now ImmutableMap>. The LedgerManager manager interface has changed to avoid implicitly updating the version of a LedgerMetadata object. Recovery, Closing and handling bookie failures have now changed, as there are no longer merge operations. If a write to zookeeper fails, then recovery/closing/failureHandling are retried with the new metadata, provided that assumptions are not broken. The interesting changes are in LedgerHandle and LedgerMetadata. There are a lot of changes from ArrayList -> ImmutableList,List Also a lot of changes from currentEnsemble -> getCurrentEnsemble. One test had to change. With the old code, if someone fenced a ledger, closing would fail. This isn't always the case now. If there's lac of the writing ledger matches the lastEntryId of the new metadata, then close can succeed. ConditionalSetTest has changed to reflect this. > Race condition causes writes to hang if ledger is fences > > > Key: BOOKKEEPER-795 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-795 > Project: Bookkeeper > Issue Type: Bug >Reporter: Ivan Kelly >Priority: Blocker > Fix For: 4.4.0 > > Attachments: 0001-Demonstrate-race-condition.patch, > 0001-Made-ledger-metadata-immutable.patch, > TEST-org.apache.bookkeeper.client.LedgerCloseTest.xml > > > If a ledger is fenced while the write is still writing to it, some of the > writes will fail to ever complete. > I've attached the log of this happening along with a test case that will > trigger the behaviour. > What appears to be happening is that when the fence occurs, the first write > after the fence gets an unrecoverable error, so tries to close the ledger. > Closing the ledger sets the closed flag on the ledger metadata, and tries to > write it, which fails as the metadata in zookeeper was modified by the > fencing operation, so the close op fails, resets the closed status for a > moment, a write operation gets through, which then fails with a fencing > error, so we try to close the ledger, but the other close operation has since > closed the ledger in our metadata, so nothing happens, and the write hangs > forever. > There's a number of issues here, but foremost, the ledger metadata that the > handle is using should only ever represent what is actually in zookeeper. > Having various parts of the code flipping bits just explodes the state space. > The LedgerMetadata object itself should be immutable, and should only be > modified, as a local variable, using a builder, before writing to zookeeper. > Only when the zookeeper operation succeeds should we update the reference > which LedgerHandle has access to. > There's also a problem in how we handle pendingaddops when we close. Really > it shouldn't be possible for a write op to get through after a closure, but > we should be defensive here and error out anything that has gotten through, > adding a big old log message to alert us that this cases that shouldn't > happen, is happening. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-788) Provide release inspector script
[ https://issues.apache.org/jira/browse/BOOKKEEPER-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194727#comment-14194727 ] Ivan Kelly commented on BOOKKEEPER-788: --- [~jiannan] [~fpj] I was referring to the QA scripts, bin/test-*. Jiannan, I don't have any automated scripts for testing perf, since it means spinning up a cluster. > Provide release inspector script > > > Key: BOOKKEEPER-788 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-788 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jiannan Wang >Assignee: Jiannan Wang >Priority: Trivial > Fix For: 4.4.0 > > Attachments: BOOKKEEPER-788.patch > > > For each release, we need to type a sequence of commands to validate release > candidate tag. This process could be achieved by a single script. > Here is the behavior of the script in my mind: >1. Collecting build environment info (Operation System, JDK, Maven) >2. Run 'mvn install -DskipTests' >3. Checking code (findbugs, rat). >4. Running unit test >5. Start standalone Bookie and Hedwig service >6. Run simple Bookie test: bookkeeper-server/bin/bookkeeper simpletest > -ensemble 3 -writeQuorum 3 >7. Run simple Hedwig test: hedwig-server/bin/hedwig console; pubsub > topic subscriber 10 message >8. Tar all above logs when error/exit > In addition, this script could also be provided to user which could collect > related information to diagnose compile/UT failure issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
