spetz opened a new pull request, #2734:
URL: https://github.com/apache/iggy/pull/2734
When consumers join a group sequentially, partitions were
reassigned instantly — but the old consumer already polled
and was mid-processing. The new consumer would poll the
same uncommitted offset, causing duplicate processing.
Implements cooperative rebalancing: instead of moving
partitions immediately on join, excess partitions are
marked as "pending revocation." The old consumer stops
receiving new messages from revoked partitions, and the
transfer completes only when the offset is committed.
Key mechanics:
- pending_revocations on member metadata tracks revoked
partitions and their target members
- get_next_member_partition_id fast path (zero alloc)
when no revocations, linear scan slow path otherwise
- record_polled_offset tracks last offset per CG member
via Arc<papaya::HashMap> for lock-free reads
- CompletePartitionRevocation control-plane op fired on
offset commit (explicit or auto-commit)
- find_completable_revocations reads last_polled_offsets
from left-right reader copy to avoid Arc divergence
- member_id guard on CompletePartitionRevocation prevents
stale ops after slab slot reuse
- Never-polled partitions transfer immediately on join
- Member leave triggers full rebalance, clearing all
pending revocations (at-least-once delivery preserved)
Tests cover: no-reshuffle on join, no duplicates during
rebalance, auto-commit completion, never-polled transfer,
member leave with pending revocations, target member
leave, manual commit blocking, concurrent joins, stale
client cleanup. All run across Tcp/WebSocket/Quic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
