codelipenghui commented on code in PR #4924:
URL: https://github.com/apache/bookkeeper/pull/4924#discussion_r4460159958


##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java:
##########
@@ -188,56 +189,31 @@ synchronized void timeoutQuorumWait() {
         
lh.handleUnrecoverableErrorDuringAdd(BKException.Code.AddEntryQuorumTimeoutException);
     }
 
-    synchronized void unsetSuccessAndSendWriteRequest(List<BookieId> ensemble, 
int bookieIndex) {
-        // update the ensemble
-        this.ensemble = ensemble;
-
-        if (toSend == null) {
-            // this addOp hasn't yet had its mac computed. When the mac is
-            // computed, its write requests will be sent, so no need to send it
-            // now
-            return;
+    synchronized Set<Integer> prepareEnsembleChange(List<BookieId> target, 
Set<Integer> changed) {
+        Set<Integer> resend = new HashSet<>(changed);
+        resend.addAll(EnsembleUtils.diffEnsemble(ensemble, target));
+        List<BookieId> previous = ensemble;
+        ensemble = target;
+        if (toSend == null || callbackTriggered) {
+            return Collections.emptySet();
         }
-        // Suppose that unset doesn't happen on the write set of an entry. In 
this
-        // case we don't need to resend the write request upon an ensemble 
change.
-        // We do need to invoke #sendAddSuccessCallbacks() for such entries 
because
-        // they may have already completed, but they are just waiting for the 
ensemble
-        // to change.
-        // E.g.
-        // ensemble (A, B, C, D), entry k is written to (A, B, D). An ensemble 
change
-        // happens to replace C with E. Entry k does not complete until C is
-        // replaced with E successfully. When the ensemble change completes, 
it tries
-        // to unset entry k. C however is not in k's write set, so no entry is 
written
-        // again, and no one triggers #sendAddSuccessCallbacks. Consequently, 
k never
-        // completes.
-        //
-        // We call sendAddSuccessCallback when unsetting t cover this case.
-        if (!lh.distributionSchedule.hasEntry(entryId, bookieIndex)) {
-            lh.sendAddSuccessCallbacks();
-            return;
+        resend.removeIf(index -> !lh.distributionSchedule.hasEntry(entryId, 
index));
+        for (int index : resend) {
+            addEntrySuccessBookies.remove(previous.get(index));
+            if (!ackSet.removeBookieAndCheck(index)) {
+                completed = false;
+            }

Review Comment:
   **P1 — Revalidate fault-domain requirements after ACK removal.** With 
fault-domain enforcement enabled, consider ensemble/write quorum 3 and ACK 
quorum 2: A/B acknowledge in rack 1 and C acknowledges in rack 2 while 
callbacks are held for an ensemble change. Replacing C removes its ACK, but A/B 
still satisfy the numeric quorum, so `completed` remains true. The final 
`sendAddSuccessCallbacks()` pass then reports success and advances LAC before 
the replacement acknowledges, although the remaining successful bookies violate 
the required rack count.
   
   After all removals, please also clear `completed` when the enabled 
placement-policy check rejects `addEntrySuccessBookies`. Add a two-rack 
regression that holds the replacement response and asserts that neither the 
callback nor LAC advances prematurely.



##########
bookkeeper-server/src/test/java/org/apache/bookkeeper/client/PendingAddOpTest.java:
##########
@@ -73,6 +86,173 @@ public void setup() {
         payload.writerIndex(data.length);
     }
 
+    @Test(timeout = 30000)
+    public void testReconcileStaleEnsembleOnNoOpChange() throws Exception {
+        checkStaleEnsembleReconciled(Collections.emptySet());
+    }
+
+    @Test(timeout = 30000)
+    public void testReconcileStaleEnsembleWhenOnlyOtherIndexChanges() throws 
Exception {
+        checkStaleEnsembleReconciled(Collections.singleton(1));
+    }
+
+    private void checkStaleEnsembleReconciled(Set<Integer> changedIndices) 
throws Exception {
+        BookieId removed = BookieId.parse("removed:3181");
+        BookieId replacement = BookieId.parse("replacement:3181");
+        BookieId other = BookieId.parse("other:3181");
+        BookieId newOther = BookieId.parse("new-other:3181");
+        List<BookieId> target = Arrays.asList(replacement,
+                changedIndices.isEmpty() ? other : newOther);
+        MockClientContext client = MockClientContext.create();
+        CompletableFuture<Void> oldWrite = new CompletableFuture<>();
+        try {
+            Versioned<LedgerMetadata> metadata = 
ClientUtil.setupLedger(client, 10L,
+                    
LedgerMetadataBuilder.create().withEnsembleSize(2).withWriteQuorumSize(2).withAckQuorumSize(2)
+                            .newEnsembleEntry(0L, target));
+            LedgerHandle handle = new LedgerHandle(client, 10L, metadata, 
BookKeeper.DigestType.CRC32C,
+                    ClientUtil.PASSWD, WriteFlag.NONE);
+            client.getMockBookieClient().setPreWriteHook((bookie, ledgerId, 
entryId) ->
+                    bookie.equals(removed) ? oldWrite : 
CompletableFuture.completedFuture(null));
+            CompletableFuture<Integer> first = new CompletableFuture<>();
+            // Reproduce the interleaving deterministically: metadata is 
already updated,
+            // but the operation captured the old ensemble before it entered 
the queue.
+            PendingAddOp stale = PendingAddOp.create(handle, client, 
Arrays.asList(removed, other),
+                    Unpooled.wrappedBuffer(new byte[] {1}), WriteFlag.NONE,
+                    (rc, lh, entryId, latency, ctx) -> first.complete(rc), 
null);
+            handle.doAsyncAddEntry(stale);
+            CompletableFuture<Integer> second = new CompletableFuture<>();
+            handle.asyncAddEntry(new byte[] {2}, (rc, lh, entryId, ctx) -> 
second.complete(rc), null);
+            handle.executor.submit(() -> { }).get();
+            assertFalse(first.isDone());
+            assertFalse(second.isDone());
+            handle.executor.submit(() -> 
handle.unsetSuccessAndSendWriteRequest(target, changedIndices)).get();
+            assertEquals(Integer.valueOf(Code.OK), first.get(10, 
TimeUnit.SECONDS));
+            assertEquals(Integer.valueOf(Code.OK), second.get(10, 
TimeUnit.SECONDS));
+            assertEquals(1L, handle.getLastAddConfirmed());
+        } finally {
+            oldWrite.complete(null);
+            client.getMainWorkerPool().shutdown();
+            client.getScheduler().shutdown();
+        }
+    }
+
+    @Test(timeout = 30000)
+    public void testMultiIndexAckCleanupAndLateResponsesAcrossChanges() throws 
Exception {
+        MockClientContext client = MockClientContext.create();
+        // Control every response explicitly, including replies from older 
generations.
+        client.setBookieClient(mock(BookieClient.class));
+        List<BookieId> original = Arrays.asList(BookieId.parse("a:3181"),
+                BookieId.parse("b:3181"), BookieId.parse("c:3181"));
+        List<BookieId> next = Arrays.asList(BookieId.parse("d:3181"),
+                BookieId.parse("e:3181"), BookieId.parse("f:3181"));
+        List<BookieId> last = Arrays.asList(BookieId.parse("g:3181"),
+                BookieId.parse("h:3181"), BookieId.parse("i:3181"));
+        try {
+            Versioned<LedgerMetadata> metadata = 
ClientUtil.setupLedger(client, 11L,
+                    
LedgerMetadataBuilder.create().withEnsembleSize(3).withWriteQuorumSize(2).withAckQuorumSize(2)
+                            .newEnsembleEntry(0L, original));
+            LedgerHandle handle = new LedgerHandle(client, 11L, metadata, 
BookKeeper.DigestType.CRC32C,
+                    ClientUtil.PASSWD, WriteFlag.NONE);
+            AtomicInteger callbacks = new AtomicInteger();
+            handle.executor.submit(() -> {
+                PendingAddOp[] ops = new PendingAddOp[2];
+                handle.changingEnsemble = true;
+                handle.pendingAddsSequenceHead = 0;
+                for (int entry = 0; entry < ops.length; entry++) {
+                    PendingAddOp op = PendingAddOp.create(handle, client, 
original,
+                            Unpooled.wrappedBuffer(new byte[] {1}), 
WriteFlag.NONE,
+                            (rc, lh, eid, latency, ctx) -> {
+                                assertEquals(Code.OK, rc);
+                                callbacks.incrementAndGet();
+                            }, null);
+                    ops[entry] = op;
+                    op.entryId = entry + 1;
+                    op.toSend = op.payload;
+                    handle.pendingAddOps.add(op);
+                    for (int index = 0; index < 3; index++) {
+                        if (handle.distributionSchedule.hasEntry(op.entryId, 
index)) {
+                            op.pendingWriteRequests++;
+                            op.writeComplete(Code.OK, 11L, op.entryId, 
original.get(index), index);
+                        }
+                    }
+                    assertTrue(op.completed);
+                }
+                handle.changingEnsemble = false;
+                Set<Integer> changed = new LinkedHashSet<>(Arrays.asList(2, 0, 
1));

Review Comment:
   **Regression coverage:** Index 2 belongs to both entries' write sets 
(`{1,2}` and `{2,0}`). Processing it first breaks their ACK quorums before a 
write-set-external index can trigger callbacks, so the callback assertions do 
not isolate the premature-callback ordering regression.
   
   Please process index 0 before index 2, for example `{0,2,1}`: index 0 is 
outside the head entry's write set and exposes callbacks running before the 
affected ACKs are invalidated. Also cover a resend callback that could drain 
later pending entries.



##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java:
##########
@@ -188,56 +189,31 @@ synchronized void timeoutQuorumWait() {
         
lh.handleUnrecoverableErrorDuringAdd(BKException.Code.AddEntryQuorumTimeoutException);
     }
 
-    synchronized void unsetSuccessAndSendWriteRequest(List<BookieId> ensemble, 
int bookieIndex) {
-        // update the ensemble
-        this.ensemble = ensemble;
-
-        if (toSend == null) {
-            // this addOp hasn't yet had its mac computed. When the mac is
-            // computed, its write requests will be sent, so no need to send it
-            // now
-            return;
+    synchronized Set<Integer> prepareEnsembleChange(List<BookieId> target, 
Set<Integer> changed) {
+        Set<Integer> resend = new HashSet<>(changed);
+        resend.addAll(EnsembleUtils.diffEnsemble(ensemble, target));
+        List<BookieId> previous = ensemble;
+        ensemble = target;
+        if (toSend == null || callbackTriggered) {
+            return Collections.emptySet();
         }
-        // Suppose that unset doesn't happen on the write set of an entry. In 
this
-        // case we don't need to resend the write request upon an ensemble 
change.
-        // We do need to invoke #sendAddSuccessCallbacks() for such entries 
because
-        // they may have already completed, but they are just waiting for the 
ensemble
-        // to change.
-        // E.g.
-        // ensemble (A, B, C, D), entry k is written to (A, B, D). An ensemble 
change
-        // happens to replace C with E. Entry k does not complete until C is
-        // replaced with E successfully. When the ensemble change completes, 
it tries
-        // to unset entry k. C however is not in k's write set, so no entry is 
written
-        // again, and no one triggers #sendAddSuccessCallbacks. Consequently, 
k never
-        // completes.
-        //
-        // We call sendAddSuccessCallback when unsetting t cover this case.
-        if (!lh.distributionSchedule.hasEntry(entryId, bookieIndex)) {
-            lh.sendAddSuccessCallbacks();
-            return;
+        resend.removeIf(index -> !lh.distributionSchedule.hasEntry(entryId, 
index));
+        for (int index : resend) {
+            addEntrySuccessBookies.remove(previous.get(index));
+            if (!ackSet.removeBookieAndCheck(index)) {
+                completed = false;
+            }
         }
+        return resend;
+    }
 
-        if (callbackTriggered) {
+    synchronized void resendEnsembleChange(Set<Integer> indices) {
+        if (toSend == null || callbackTriggered) {
             return;
         }
-
-
-        log.debug()
-                .ctx(lh.log)
-                .attr("entryId", entryId)
-                .attr("bookieIndex", bookieIndex)
-                .log("Unsetting success");
-
-
-        // if we had already heard a success from this array index, need to
-        // increment our number of responses that are pending, since we are
-        // going to unset this success
-        if (!ackSet.removeBookieAndCheck(bookieIndex)) {
-            // unset completed if this results in loss of ack quorum
-            completed = false;
+        for (int index : indices) {
+            sendWriteRequest(ensemble, index);

Review Comment:
   **P2 — Recheck operation liveness before every resend.** 
`BookieClientImpl.addEntry()` invokes `writeComplete()` inline when client-pool 
lookup fails. If the entry retains ACK quorum after preparation, that callback 
can deliver its success callback and clear `toSend` before this loop reaches 
the next index. For example, with write quorum 4 and ACK quorum 2, retained 
ACKs at indices 2/3 and retries at 0/1, a synchronous pool-creation failure at 
0 can clear the buffer before a cached client for index 1 reaches 
`toSend.retain()`.
   
   The previous per-index method checked liveness before each send; this loop 
checks it only once. Please repeat the `toSend`/`callbackTriggered` guard 
inside the loop and add a regression using an initiated operation and a 
synchronous callback on the first of multiple retries. The operation monitor 
does not prevent same-thread reentrancy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to