Josh Forman-Gornall created FLINK-4115:
------------------------------------------
Summary: FsStateBackend filesystem verification can cause
classpath exceptions
Key: FLINK-4115
URL: https://issues.apache.org/jira/browse/FLINK-4115
Project: Flink
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Reporter: Josh Forman-Gornall
Priority: Minor
In the constructor of FsStateBackend, the FileSystem for the checkpoint
directory is initialised and it is verified that the checkpoint path exists.
This verification happens in the Flink client program when submitting a job and
can cause classpath issues if classes required to access the file system are
not available in the client's classpath.
For example, if we run Flink on YARN over AWS EMR using RocksDBStateBackend and
an s3:// checkpoint directory, we get the below ClassNotFoundException. This is
because the jars needed to use the EMR file system are available only in the
YARN context and not when submitting the job via the Flink client.
```
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
at
org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
at
org.apache.flink.core.fs.FileSystem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:280)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.validateAndNormalizeUri(FsStateBackend.java:383)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:175)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:144)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:205)
```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
