Abhishek Pal created RATIS-2632:
-----------------------------------
Summary: install-snapshot chunk loop send lacks isReady()
backpressure
Key: RATIS-2632
URL: https://issues.apache.org/jira/browse/RATIS-2632
Project: Ratis
Issue Type: Improvement
Components: gRPC
Reporter: Abhishek Pal
Assignee: Abhishek Pal
When the leader streams a snapshot to a lagging follower, the chunk send loop
in
[GrpcLogAppender.installSnapshot()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L775-L783]
calls `*snapshotRequestObserver.onNext(request)*` for every chunk without ever
checking `*CallStreamObserver.isReady()*`.
The append-entries path already [checks
isReady()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L366],
but the snapshot path does not, so it is possible that chunks are enqueued
faster than the network can drain them.
This can cause:
- Unbounded outbound buffering of snapshot chunks in the gRPC/Netty write queue
when the follower is slow.
- Leader OOM / direct-memory exhaustion during large-snapshot catch-up, which
can crash the leader and impact the whole group.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
