Chamikara Madhusanka Jayalath created BEAM-8399:
---------------------------------------------------
Summary: Python HDFS implementation should support filenames of
the format "hdfs://namenodehost/parent/child"
Key: BEAM-8399
URL: https://issues.apache.org/jira/browse/BEAM-8399
Project: Beam
Issue Type: Improvement
Components: sdk-py-core
Reporter: Chamikara Madhusanka Jayalath
"hdfs://namenodehost/parent/child" and "/parent/child" seems to be the correct
filename formats for HDFS based on [1] but we currently support format
"hdfs://parent/child".
To not break existing users, we have to either (1) somehow support both
versions by default (based on [2] seems like HDFS does not allow colons in file
path so this might be possible) (2) make "hdfs://namenodehost/parent/child"
optional for now and change it to default after few versions.
We should also make sure that Beam Java and Python HDFS file-system
implementations are consistent in this regard.
[1][https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html]
[2] https://issues.apache.org/jira/browse/HDFS-13
cc: [~udim]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
