stankiewicz opened a new pull request, #37580:
URL: https://github.com/apache/beam/pull/37580
Background / Goal: The default polling intervals and heartbeat
configurations for Spanner change streams (2-minute time increments and
2-second heartbeats) are too slow for pipelines aiming for sub-second
latencies. Furthermore, when the change stream only emits heartbeat records
during a checkpoint interval, the pipeline processing can get stuck. This PR
introduces a low-latency mode and improves robustness for continuous change
streams reading.
Changes Made:
- Added Low-Latency Config Macros: Replaced implicit low-latency flags with
explicit cdcTimeIncrement and heartbeatMillis states in
SpannerIO.ReadChangeStream.
withLowLatency()
- Configuration: Now acts as a macro that configures the change stream
pipeline with a 1-second CDC time increment and a 100ms heartbeat interval
(defaults are 2 minutes and 2000ms, respectively).
- Renamed Variables for Clarity: Renamed legacy configurations like
endTimestampDuration to cdcTimeIncrement throughout the action classes and
DoFns (QueryChangeStreamAction, ActionFactory, ReadChangeStreamPartitionDoFn).
- Resolved Pipeline Stalls on Heartbeats: Modified HeartbeatRecordAction to
return ProcessContinuation.resume() instead of Optional.empty() upon successful
heartbeat processing, ensuring the change stream doesn't get stuck doing
nothing in Dataflow when there is no data in the checkpoint interval.
Testing: Updated HeartbeatRecordActionTest
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
