[
https://issues.apache.org/jira/browse/FLINK-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501958#comment-16501958
]
ASF GitHub Bot commented on FLINK-9525:
---------------------------------------
Github user yew1eb commented on the issue:
https://github.com/apache/flink/pull/6118
yes, this is a hadoop-file-system discovery issue (similar case:
https://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file),
but if flink-job dependency this `hadoop-common`, and flink-cluster uses
hdfs to store checkpoint, job will throw error when init filesystem for
checkpoint.
i think we should improve `load file system factories` part.
see `org.apache.flink.core.fs.FileSystem` code snippets：
```
/** All available file system factories. */
private static final List<FileSystemFactory> RAW_FACTORIES =
loadFileSystems();
/** Mapping of file system schemes to the corresponding factories,
* populated in {@link FileSystem#initialize(Configuration)}. */
private static final HashMap<String, FileSystemFactory> FS_FACTORIES =
new HashMap<>();
/** The default factory that is used when no scheme matches. */
private static final FileSystemFactory FALLBACK_FACTORY =
loadHadoopFsFactory();
```
@StephanEwen , what do you think about this?
> Missing META-INF/services/*FileSystemFactory in flink-hadoop-fs module
> ----------------------------------------------------------------------
>
> Key: FLINK-9525
> URL: https://issues.apache.org/jira/browse/FLINK-9525
> Project: Flink
> Issue Type: Bug
> Components: FileSystem
> Affects Versions: 1.4.0, 1.5.0, 1.6.0
> Reporter: Hai Zhou
> Assignee: Hai Zhou
> Priority: Blocker
> Fix For: 1.6.0, 1.5.1
>
> Attachments: [email protected]
>
>
> if flink job dependencies includes `hadoop-common` and `hadoop-hdfs`, will
> throw runtime error.
> like this case:
> [https://stackoverflow.com/questions/47890596/java-util-serviceconfigurationerror-org-apache-hadoop-fs-filesystem-provider-o].
> the root cause:
> see {{org.apache.flink.core.fs.FileSystem}}
> This class will load all available file system factories via
> {{ServiceLoader.load(FileSystemFactory.class)}}.
> Since {{ META-INF / services / org.apache.flink.core.fs.FileSystemFactory }}
> file in the classpath does not have an
> `org.apache.flink.runtime.fs.hdfs.HadoopFsFactory`,
> and finaly only loaded one available {{LocalFileSystemFactory}} .
> more error messages see this screenshot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
