igorraspopovic opened a new issue, #41251: URL: https://github.com/apache/beam/issues/41251
### What would you like to happen? `SpannerIO.readChangeStream().withLowLatency()` (added in #37718) sets three things as a fixed macro: `heartbeatMillis = 100`, `cancelQueryOnHeartbeat = true` and `realTimeCheckpointInterval = 1s` (`LOW_LATENCY_REAL_TIME_CHECKPOINT_INTERVAL`). The real-time checkpoint interval can't be set on its own. `ReadChangeStream.Builder#setRealTimeCheckpointInterval` and `toBuilder()` are package-private, so pipeline authors can only choose between the 2 minute default and the fixed 1 second value of `withLowLatency()`. Why the interval matters: heartbeats only fire when a partition has had no writes for `heartbeatMillis`. Under a steady write load (a write every few ms per partition) heartbeats never fire, so `cancelQueryOnHeartbeat` never triggers, and the only thing that ends the reader's open query is the real-time checkpoint interval. Each partition therefore hands its records downstream once per interval, and the added latency is roughly uniform in [0, interval]. With the fixed 1 s we measured about 0.5 s median and 0.98 s P95 reader-to-publish latency under a steady burst on Dataflow, and downstream consumers see 1 s waves of messages. Proposal: add a public method, for example ```java /** Sets how long a partition query stays open before the reader checkpoints and hands over records. */ public ReadChangeStream withRealTimeCheckpointInterval(Duration interval) ``` with the same validation as the other builder options (positive, and probably a sensible minimum such as the heartbeat interval). `withLowLatency()` would keep its current behaviour, and the new method would override the 1 s value when called after it. The tradeoff to document: a shorter interval means more change stream queries and metadata table row updates per partition (about `1000 / intervalMillis` per second during sustained writes), so it is a trade between latency and Spanner load. We currently work around this with a helper class declared in `org.apache.beam.sdk.io.gcp.spanner` in our own jar, which calls the package-private builder. That works on 2.76.0 but is not something we can rely on across Beam upgrades. Happy to send a PR if the approach is acceptable. ### Issue Priority Priority: 3 (nice-to-have improvement) ### Issue Components - [ ] Component: Python SDK - [ ] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [x] Component: IO connector -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
