developer-rpai commented on PR #40293: URL: https://github.com/apache/beam/pull/40293#issuecomment-5925350540
Thanks for the thorough review, @claudevdm. Good catch on the sink -- fixed as suggested: `open()` now returns `(writer, file_handle)` and the handle travels with the writer instead of living on the sink, so each window's writer closes its own file and we no longer get Parquet files with missing footers. Also added the end-to-end `test_write_streaming_rows_land_in_their_own_window` test to `WriteStreamingTest` (one bundle spanning 3 windows, each row verified in its own window's file). The new `RowDictionariesToArrowTableTest` unit tests cover the DoFn-level windowing behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
