lostluck opened a new pull request, #33542:
URL: https://github.com/apache/beam/pull/33542
One thing lead to another with this so this PR does a few related things.
* Exports the windowing strategy struct to configure aggregation stages in
the element manager.
* This is prep-work for fully supporting the rest of the windowing
strategies + triggers.
* Uses the above to support AllowedLateness.
* In doing so, it was noted that incorrect conversions from integers
(representing durations of nanosecond precision) to `mtime.Time` which is
strictly a millisecond duration. This fix revealed two issues with Session
Windows.
* Fixes to Session Window Merging
* Merging wasn't correctly happening per key. Windows were being treated
"globally".
* We were incorrectly using the GapSize in prism. GapSize was already
accounted for in the generation of the SessionWindow's interval Window
representation.
* These fixes resolved the issue that #32085 was filed for, so the
re-written Python test could be removed.
* Late data is now only dropped before aggregations, when it matters most.
* This avoids a correctness issue dropping data in a Reshuffle.
* This will be re-evaluated as windowing strategies are implemented, but
it's better to have more tests enabled, so they can provide a signal.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
