oscerd opened a new pull request, #25898:
URL: https://github.com/apache/camel/pull/25898
The last open item of the camel-google audit: the realtime half of
CAMEL-24347, deferred at the time
because it needed an overflow-policy decision.
### The problem
`GoogleFirestoreConsumer` buffers the document changes reported by the
snapshot listener and drains that
buffer on the scheduled poll:
```java
private final Queue<Exchange> pendingExchanges = new
ConcurrentLinkedQueue<>();
```
The listener callback runs on a Firestore client thread and is never
throttled by the route, so whenever
changes arrive faster than the poll consumes them the buffer grows without
limit — every element being a
full `Exchange` holding the document data. It also keeps growing while the
route is suspended, or when the
consumer is configured with a long poll delay, so a burst of writes on a
busy collection can exhaust the
heap.
### The change
A new `maxPendingChanges` consumer option bounds the buffer:
* **the default is unbounded**, so existing routes behave exactly as before
and nothing changes on upgrade;
* when a limit is configured and reached, the **oldest** buffered change is
discarded and a warning is
logged (the first one, then every hundredth), which leaves the route with
the most recent state of the
collection;
* discarded exchanges are released back to the exchange pool, and so is
whatever is still buffered when the
consumer stops, or left over in a batch that the stop interrupted.
Blocking the listener callback is deliberately not offered as a policy: that
thread belongs to the Firestore
gRPC client, and stalling it would stall the watch stream itself. Dropping
is the only backpressure a
snapshot listener can apply without hurting the client.
The buffer is now a `LinkedBlockingQueue`, which gives an O(1) `size()`, a
capacity, and a `drainTo` for the
poll — `ConcurrentLinkedQueue.size()` walks the whole queue, which is
exactly the wrong cost on this path.
Main only: this adds an option, so it is not a candidate for the maintenance
branches.
### Verification
* `mvn clean install` on `components/camel-google/camel-google-firestore`,
green — 11 unit tests, including
three new ones covering the unbounded default, the drop-oldest behaviour,
and a buffer of one.
* The tests build the consumer directly instead of resolving a started
endpoint, so they need no Google
credentials.
* Full reactor `mvn clean install -DskipTests -Dquickly`, green. The
endpoint-DSL and component-DSL
factories were regenerated separately (`mvn generate-sources -pl
camel-componentdsl,camel-endpointdsl`),
since the `regen` profile that produces them is disabled by `-Dquickly`;
re-running it after the rebase
reports no further changes.
---
_Claude Code on behalf of oscerd_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
