bvolpato opened a new pull request, #39993: URL: https://github.com/apache/beam/pull/39993
`WriteToFiles` currently catches every `BeamIOError` from final renames and emits successful `FileResult`s even when the destination files are absent. A genuine I/O failure can therefore leave data in temporary files while the pipeline reports success. Propagate rename failures so the bundle can retry. Before renaming, skip a file only when its temporary source is absent and its final destination exists, preserving retries after complete or partial finalization. An existing destination does not suppress errors while the source remains. Five regression tests cover failed renames without successful output, already-completed retries, partial moves followed by retry, stale destinations, and missing files. Against the unfixed implementation, four fail and the completed-retry compatibility test passes. ## Testing From `sdks/python`, with the SDK and test dependencies installed: ```sh python -m pytest apache_beam/io/fileio_test.py -q ``` Python 3.12: 49 passed, 2 skipped. YAPF 0.43.0, Ruff 0.15.22, and `git diff --check` pass for the changed files. ## Downsides Finalization adds one existence lookup per temporary file, plus a destination lookup when that source is absent. Remote filesystems incur these metadata requests before the existing batched rename. Genuine finalization errors now fail the bundle instead of being suppressed. ------------------------ - [x] Describe the bug and include reproducible regression tests; no separate Python issue is linked. - [x] Update `CHANGES.md` with the behavior change. - [ ] Apache Individual Contributor License Agreement, if required for this contribution. See the [Contributor Guide](https://beam.apache.org/contribute) and [CI documentation](https://github.com/apache/beam/blob/master/CI.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
