What image are you using? Sent from my iPhone
> On Dec 19, 2018, at 9:44 AM, Avi Levi <avi.l...@bluevoyant.com> wrote: > > Hi Chesnay, > What do you mean? I am creating a fat jar with all dependencies (using sbt > assembly). which jar I should place in the /lib directory ? > >> On Wed, Dec 19, 2018 at 4:44 PM Chesnay Schepler <ches...@apache.org> wrote: >> Are you including the filesystems in your jar? Filesystem jars must be >> placed in the /lib directory of the flink distribution. >> >>> On 19.12.2018 15:03, Avi Levi wrote: >>> Hi, >>> I am trying to set the backend state to hdfs >>> val stateUri = "hdfs/path_to_dir" >>> val backend: RocksDBStateBackend = new RocksDBStateBackend(stateUri, true) >>> env.setStateBackend(backend) >>> >>> I am running with flink 1.7.0 with the following dependencies (tried them >>> with different combinations) : >>> "org.apache.flink" %% "flink-connector-filesystem" % flinkV >>> "org.apache.flink" % "flink-hadoop-fs" % flinkV >>> "org.apache.hadoop" % "hadoop-hdfs" % >>> hadoopVersion >>> "org.apache.hadoop" % "hadoop-common" % >>> hadoopVersion >>> >>> however when running the jar I am getting this error: >>> >>> Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: >>> Could not find a file system implementation for scheme 'hdfs'. The scheme >>> is not directly supported by Flink and no Hadoop file system to support >>> this scheme could be loaded. >>> at >>> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403) >>> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318) >>> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298) >>> at >>> org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58) >>> at >>> org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444) >>> at >>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createCheckpointStorage(RocksDBStateBackend.java:407) >>> at >>> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249) >>> ... 17 more >>> Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: >>> Hadoop is not in the classpath/dependencies. >>> at >>> org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64) >>> at >>> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399) >>> ... 23 more >>> >>> any help will be greatly appreciated >>> >>
