Joar Wandborg created BEAM-5417:
-----------------------------------
Summary: FileSystems.match behaviour diff between GCS and local
file system
Key: BEAM-5417
URL: https://issues.apache.org/jira/browse/BEAM-5417
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Affects Versions: 2.6.0
Reporter: Joar Wandborg
Assignee: Ahmet Altay
Given the directory structure:
{noformat}
.
├── filesystem-match-test
│ ├── a
│ │ └── file.txt
│ └── b
│ └── file.txt
└── filesystem-match-test.py
{noformat}
Where {{filesystem-match-test.py}} contains:
{code:python}
from __future__ import print_function
import os
import posixpath
from apache_beam.io.filesystem import MatchResult
from apache_beam.io.filesystems import FileSystems
BASES = [
os.path.join(os.path.dirname(__file__), "./"),
"gs://my-bucket/test/",
]
pattern = "filesystem-match-test/*/file.txt"
for base_path in BASES:
full_pattern = posixpath.join(base_path, pattern)
print("full_pattern: {}".format(full_pattern))
match_result = FileSystems.match([full_pattern])[0] # type: MatchResult
print("metadata list: {}".format(match_result.metadata_list))
{code}
Running {{python filesystem-match-test.py}} does not match any files locally,
but does match files on GCS:
{noformat}
full_pattern: ./filesystem-match-test/*/file.txt
metadata list: []
full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
metadata list:
[FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6),
FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
{noformat}
The expected result is that a/file.txt and b/file.txt should be matched for
both patterns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
