ODriscollBoys opened a new pull request, #40499:
URL: https://github.com/apache/beam/pull/40499
`fileio.WriteToFiles` writes to temporary files and then renames them to
`path/<name>`, where `<name>` comes from `file_naming`. Before the rename it
only created `path`, so if `file_naming` returns a name with a directory in it
(for example `'subdir/file.txt'` or `'%s/shard-%d' % (destination,
shard_index)`), the rename fails on file systems that need the parent directory
to exist, such as the local file system. That failure is logged only at debug
level because it is expected when a bundle is retried. The records were
therefore silently left in the temporary directory and the only visible sign
was the "Some files may be left orphaned" warning.
The directory of every final file is now created before the rename. For
names without a directory this is the same `path` as before, and file systems
where `mkdirs` is a no-op (GCS, S3) behave as before.
Added a test that writes each destination to its own subdirectory and reads
the files back. It fails on master and passes with the change.
`fileio_test.py`, `localfilesystem_test.py` and `filesystems_test.py` pass. I
also re-ran the example from the issue on the DirectRunner and checked that
`out/subdir/file.txt` is now written.
fixes #21313
------------------------
- [x] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [x] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
