[
https://issues.apache.org/jira/browse/BEAM-4711?focusedWorklogId=134038&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-134038
]
ASF GitHub Bot logged work on BEAM-4711:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Aug/18 01:40
Start Date: 13/Aug/18 01:40
Worklog Time Spent: 10m
Work Description: ryan-williams commented on a change in pull request
#5863: [BEAM-4711] fix globbing in LocalFileSystem.delete
URL: https://github.com/apache/beam/pull/5863#discussion_r209477356
##########
File path: sdks/python/apache_beam/io/localfilesystem.py
##########
@@ -295,11 +296,20 @@ def _delete_path(path):
raise IOError(err)
exceptions = {}
- for path in paths:
+
+ def try_delete(path):
try:
_delete_path(path)
except Exception as e: # pylint: disable=broad-except
exceptions[path] = e
+ for path in paths:
+ expanded = glob(path)
+ if not expanded:
+ try_delete(path)
+ else:
+ for p in expanded:
Review comment:
good idea, done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 134038)
Time Spent: 1h (was: 50m)
> LocalFileSystem.delete doesn't support globbing
> -----------------------------------------------
>
> Key: BEAM-4711
> URL: https://issues.apache.org/jira/browse/BEAM-4711
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
> Affects Versions: 2.5.0
> Reporter: Ryan Williams
> Assignee: Ryan Williams
> Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I attempted to run {{wordcount_it_test:WordCountIT.test_wordcount_it}}
> locally with {{DirectRunner}}:
> {code}
> python setup.py nosetests \
> --tests
> apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
> --test-pipeline-options="--output=foo"
> {code}
> It failed in [the {{delete_files}} cleanup
> command|https://github.com/apache/beam/blob/a58f1ffaafb0e2ebcc73a1c5abfb05a15ec6a84b/sdks/python/apache_beam/examples/wordcount_it_test.py#L64]:
> {code}
> root: WARNING: Retry with exponential backoff: waiting for 11.1454450937
> seconds before retrying delete_files because we caught exception:
> BeamIOError: Delete operation failed with exceptions
> {'foo/1530557644/results*': IOError(OSError(2, 'No such file or directory'),)}
> Traceback for above exception (most recent call last):
> File "/Users/ryan/c/beam/sdks/python/apache_beam/utils/retry.py", line 184,
> in wrapper
> return fun(*args, **kwargs)
> File "/Users/ryan/c/beam/sdks/python/apache_beam/testing/test_utils.py",
> line 136, in delete_files
> FileSystems.delete(file_paths)
> File "/Users/ryan/c/beam/sdks/python/apache_beam/io/filesystems.py", line
> 282, in delete
> return filesystem.delete(paths)
> File "/Users/ryan/c/beam/sdks/python/apache_beam/io/localfilesystem.py",
> line 304, in delete
> raise BeamIOError("Delete operation failed", exceptions)
> {code}
> The line:
> {code}
> self.addCleanup(delete_files, [output + '*'])
> {code}
> works as expected in GCS, and deletes a test's output-directory, but it fails
> in on the local-filesystem, which doesn't expand globs before attempting to
> delete paths.
> It would be good to make these consistent, presumably by adding glob-support
> to {{LocalFileSystem}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
