[
https://issues.apache.org/jira/browse/BEAM-6821?focusedWorklogId=213002&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-213002
]
ASF GitHub Bot logged work on BEAM-6821:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Mar/19 08:56
Start Date: 14/Mar/19 08:56
Worklog Time Spent: 10m
Work Description: gkovelman commented on pull request #8054: [BEAM-6821]
FileBasedSink improper paths
URL: https://github.com/apache/beam/pull/8054
File path generated in open_writer method is not according to target
filesystem, because os.path.join is used and not FileSystems.join.
This created incompatibilities between, for example, Windows and GCS.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
<br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
| --- | --- | ---
See
[.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md)
for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 213002)
Time Spent: 10m
Remaining Estimate: 0h
> FileBasedSink is not creating file paths according to target filesystem
> -----------------------------------------------------------------------
>
> Key: BEAM-6821
> URL: https://issues.apache.org/jira/browse/BEAM-6821
> Project: Beam
> Issue Type: Bug
> Components: io-java-text
> Affects Versions: 2.11.0
> Environment: Windows 10
> Reporter: Gregory Kovelman
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> File path generated in _open_writer_ method is not according to target
> filesystem, because
> os.path.join is used and not FileSystems.join.
> apache_beam\io\filebasedsink.py extract:
>
> {code:java}
> def _create_temp_dir(self, file_path_prefix):
> base_path, last_component = FileSystems.split(file_path_prefix)
> if not last_component:
> # Trying to re-split the base_path to check if it's a root.
> new_base_path, _ = FileSystems.split(base_path)
> if base_path == new_base_path:
> raise ValueError('Cannot create a temporary directory for root path '
> 'prefix %s. Please specify a file path prefix with '
> 'at least two components.' % file_path_prefix)
> path_components = [base_path,
> 'beam-temp-' + last_component + '-' + uuid.uuid1().hex]
> return FileSystems.join(*path_components)
> @check_accessible(['file_path_prefix', 'file_name_suffix'])
> def open_writer(self, init_result, uid):
> # A proper suffix is needed for AUTO compression detection.
> # We also ensure there will be no collisions with uid and a
> # (possibly unsharded) file_path_prefix and a (possibly empty)
> # file_name_suffix.
> file_path_prefix = self.file_path_prefix.get()
> file_name_suffix = self.file_name_suffix.get()
> suffix = (
> '.' + os.path.basename(file_path_prefix) + file_name_suffix)
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> {code}
>
>
> This created incompatibilities between, for example, Windows and GCS.
> Expected: gs://bucket/beam-temp-result-uuid\\uid.result
> Actual: gs://bucket/beam-temp-result-uuid/uid.result
> Replacing os.path.join with FileSystems.join fixes the issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
