Harshvardhan created FLINK-23477: ------------------------------------ Summary: wasbs/s3 FileSystem support Key: FLINK-23477 URL: https://issues.apache.org/jira/browse/FLINK-23477 Project: Flink Issue Type: Bug Components: Deployment / Kubernetes Affects Versions: 1.13.0 Reporter: Harshvardhan
Flink is unable to identify Azure and AWS FileSystem plugins independent of Hadoop APIs on Native Kubernetes on Azure. Plugins added as libraries are kept under /opt/flink/plugins/flink-s3-fs-hadoop.jar/flink-s3-fs-hadoop.jar The Flink is unable to read the plugins. Same issue even after enabling dependencies in YAML file as 'ENABLE_BUILT_IN_PLUGINS'. Reference Documentation : [here | https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/plugins/] Exception: {{}} {{Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported byFlink through the following plugins: flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory.See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:513)at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407)at org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.run(ContinuousFileMonitoringFunction.java:214)at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)