raphaelauv created FLINK-38282:
----------------------------------
Summary: K8S operator : S3 jarURI
Key: FLINK-38282
URL: https://issues.apache.org/jira/browse/FLINK-38282
Project: Flink
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: raphaelauv
Attachments: flink-kubernetes-operator.log
Using the flink operator 1.8.2
with flink 2.0.0
I can't submit FlinkSessionJob that have a jarURI on a s3
```yaml
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: basic-example
spec:
image:
flinkVersion: v2_0
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
s3.endpoint: http://minio:9000
s3.path.style.access: "true"
s3.access-key: admin
s3.secret-key: adminadmin
s3.region: eu-west-3
serviceAccount: flink
jobManager:
resource:
memory: "2048m"
cpu: 1
taskManager:
resource:
memory: "2048m"
cpu: 1
podTemplate:
spec:
containers:
- name: flink-main-container
env:
- name: ENABLE_BUILT_IN_PLUGINS
value: "flink-s3-fs-hadoop-2.0.0.jar"
```
```yaml
apiVersion: flink.apache.org/v1beta1
kind: FlinkSessionJob
metadata:
name: my-flink-job
spec:
deploymentName: basic-example
job:
jarURI: s3://test-bucket/myjar-0.0.1.jar
parallelism: 1
upgradeMode: stateless
```
the FlinkSessionJob fail with :
```
Could not find a file system implementation for scheme 's3a'. The scheme is
directly supported by Flink through the following plugin(s):
flink-s3-fs-hadoop. Please ensure that each plugin resides within its own
subfolder within the plugins directory. See
https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/
for more information. If you want to use a Hadoop file system for that scheme,
please add the scheme to the configuration fs.allowed-fallback-filesystems. For
a full list of supported file systems, please see
https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
```
and in the operator pod the logs are ->
```
java.util.ServiceConfigurationError:
org.apache.flink.core.fs.FileSystemFactory:
org.apache.flink.fs.s3hadoop.S3FileSystemFactory Unable to get public no-arg
constructor
at java.base/java.util.ServiceLoader.fail(Unknown Source)
at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
Source)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
Source)
at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
at
org.apache.flink.core.plugin.PluginLoader$ContextClassLoaderSettingIterator.hasNext(PluginLoader.java:130)
at
org.apache.flink.shaded.guava31.com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1391)
at
org.apache.flink.shaded.guava31.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at
org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1088)
at
org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1071)
at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:340)
at
org.apache.flink.kubernetes.operator.FlinkOperator.<init>(FlinkOperator.java:112)
at
org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:281)
Caused by: java.lang.NoClassDefFoundError:
org/apache/flink/core/fs/PathsCopyingFileSystem
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
at java.base/java.net.URLClassLoader.defineClass(Unknown Source)
at java.base/java.net.URLClassLoader$1.run(Unknown Source)
at java.base/java.net.URLClassLoader$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.net.URLClassLoader.findClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at
org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:150)
at
org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromOwnerFirst(ComponentClassLoader.java:172)
at
org.apache.flink.core.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:107)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
at java.base/java.lang.Class.getConstructor0(Unknown Source)
at java.base/java.lang.Class.getConstructor(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
... 13 more
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
