Re: [PR] fix dead lock issue in ReadOnlyLedgerHandle#handleBookieFailure [bookkeeper]

via GitHub Sat, 07 Sep 2024 00:39:36 -0700


chenhongSZ commented on PR #4467:
URL: https://github.com/apache/bookkeeper/pull/4467#issuecomment-2335105543


   We also faced this bug (version:4.16.5), where two 
`BookKeeperClientWorker-OrderedExecutor` threads are deadlocked.  I guess 
Moving the `unsetSuccessAndSendWriteRequestoutside` out of the metadata lock 
scope is better.  because the `metadataLock` is only used to protect the 
`newEnsemblesFromRecovery` map.
   
   <img width="1067" alt="image" 
src="https://github.com/user-attachments/assets/a7c76b28-bb3e-4c2f-8256-83e78bb14341";>
   
   just like org.apache.bookkeeper.client.LedgerHandle#ensembleChangeLoop
   
   
![image](https://github.com/user-attachments/assets/c9042f92-164d-4e72-b7f9-774999d013ce)
   
   
   the stack snippet is below:
   
   ```
        Found one Java-level deadlock:
   =============================
   "BookKeeperClientWorker-OrderedExecutor-0-0":
     waiting to lock monitor 0x00007f77ac006160 (object 0x00000004080b9600, a 
org.apache.bookkeeper.client.PendingAddOp),
     which is held by "BookKeeperClientWorker-OrderedExecutor-8-0"
   
   "BookKeeperClientWorker-OrderedExecutor-8-0":
     waiting to lock monitor 0x00007f778c001230 (object 0x00000004080bf600, a 
java.lang.Object),
     which is held by "BookKeeperClientWorker-OrderedExecutor-0-0"
   
   Java stack information for the threads listed above:
   ===================================================
   "BookKeeperClientWorker-OrderedExecutor-0-0":
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:264)
        - waiting to lock <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9858> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf600> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b99d0> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.initiate(PendingAddOp.java:258)
        - locked <0x00000004080b99d0> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.doAsyncAddEntry(LedgerHandle.java:1358)
        at 
org.apache.bookkeeper.client.LedgerHandle.asyncRecoveryAddEntry(LedgerHandle.java:1212)
        at 
org.apache.bookkeeper.client.LedgerRecoveryOp.onEntryComplete(LedgerRecoveryOp.java:204)
        at 
org.apache.bookkeeper.client.ListenerBasedPendingReadOp.submitCallback(ListenerBasedPendingReadOp.java:67)
        at 
org.apache.bookkeeper.client.PendingReadOp.readEntryComplete(PendingReadOp.java:608)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion$1.readEntryComplete(PerChannelBookieClient.java:1885)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.handleReadResponse(PerChannelBookieClient.java:1967)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.handleV2Response(PerChannelBookieClient.java:1919)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.run(PerChannelBookieClient.java:1397)
        at 
org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137)
        at 
org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113)
        at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run([email protected]/Thread.java:840)
   "BookKeeperClientWorker-OrderedExecutor-8-0":
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:215)
        - waiting to lock <0x00000004080bf600> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080bf708> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:396)
        at 
org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:356)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2548)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2453)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:334)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:543)
        at 
org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:677)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120)
        at 
org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:309)
        at 
org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150)
        at 
org.apache.bookkeeper.client.PendingAddOp.unsetSuccessAndSendWriteRequest(PendingAddOp.java:225)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.client.LedgerHandle.unsetSuccessAndSendWriteRequest(LedgerHandle.java:2007)
        at 
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:227)
        - locked <0x00000004080bf9b8> (a java.lang.Object)
        at 
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353)
        - locked <0x00000004080b9600> (a 
org.apache.bookkeeper.client.PendingAddOp)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:284)
        at 
org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:78)
   
   Found 1 deadlock.
   ``` 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [PR] fix dead lock issue in ReadOnlyLedgerHandle#handleBookieFailure [bookkeeper]

Reply via email to