[
https://issues.apache.org/jira/browse/FLINK-38282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
raphaelauv updated FLINK-38282:
-------------------------------
Description:
Using the flink operator 1.8.2
with flink 2.0.0
I can't submit FlinkSessionJob that have a jarURI on a s3
{code:java}
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: basic-example
spec:
image:
flinkVersion: v2_0
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
s3.endpoint: http://minio:9000
s3.path.style.access: "true"
s3.access-key: admin
s3.secret-key: adminadmin
s3.region: eu-west-3
serviceAccount: flink
jobManager:
resource:
memory: "2048m"
cpu: 1
taskManager:
resource:
memory: "2048m"
cpu: 1
podTemplate:
spec:
containers:
- name: flink-main-container
env:
- name: ENABLE_BUILT_IN_PLUGINS
value: "flink-s3-fs-hadoop-2.0.0.jar" {code}
{code:java}
apiVersion: flink.apache.org/v1beta1
kind: FlinkSessionJob
metadata:
name: my-flink-job
spec:
deploymentName: basic-example
job:
jarURI: s3://test-bucket/myjar-0.0.1.jar
parallelism: 1
upgradeMode: stateless {code}
the FlinkSessionJob fail with :
{code:java}
Could not find a file system implementation for scheme 's3a'. The scheme is
directly supported by Flink through the following plugin(s):
flink-s3-fs-hadoop. Please ensure that each plugin resides within its own
subfolder within the plugins directory. See
https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/
for more information. If you want to use a Hadoop file system for that scheme,
please add the scheme to the configuration fs.allowed-fallback-filesystems. For
a full list of supported file systems, please see
https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. {code}
and in the operator pod the logs are ->
{code:java}
java.util.ServiceConfigurationError:
org.apache.flink.core.fs.FileSystemFactory:
org.apache.flink.fs.s3hadoop.S3FileSystemFactory Unable to get public no-arg
constructor
at java.base/java.util.ServiceLoader.fail(Unknown Source)
at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
Source)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
Source)
at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
at
org.apache.flink.core.plugin.PluginLoader$ContextClassLoaderSettingIterator.hasNext(PluginLoader.java:130)
at
org.apache.flink.shaded.guava31.com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1391)
at
org.apache.flink.shaded.guava31.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at
org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1088)
at
org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1071)
at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:340)
at
org.apache.flink.kubernetes.operator.FlinkOperator.<init>(FlinkOperator.java:112)
at
org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:281)
Caused by: java.lang.NoClassDefFoundError:
org/apache/flink/core/fs/PathsCopyingFileSystem
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
at java.base/java.net.URLClassLoader.defineClass(Unknown Source)
at java.base/java.net.URLClassLoader$1.run(Unknown Source)
at java.base/java.net.URLClassLoader$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.net.URLClassLoader.findClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at
org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:150)
at
org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromOwnerFirst(ComponentClassLoader.java:172)
at
org.apache.flink.core.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:107)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
at java.base/java.lang.Class.getConstructor0(Unknown Source)
at java.base/java.lang.Class.getConstructor(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
... 13 more
{code}
was:
Using the flink operator 1.8.2
with flink 2.0.0
I can't submit FlinkSessionJob that have a jarURI on a s3
```yaml
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: basic-example
spec:
image:
flinkVersion: v2_0
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
s3.endpoint: http://minio:9000
s3.path.style.access: "true"
s3.access-key: admin
s3.secret-key: adminadmin
s3.region: eu-west-3
serviceAccount: flink
jobManager:
resource:
memory: "2048m"
cpu: 1
taskManager:
resource:
memory: "2048m"
cpu: 1
podTemplate:
spec:
containers:
- name: flink-main-container
env:
- name: ENABLE_BUILT_IN_PLUGINS
value: "flink-s3-fs-hadoop-2.0.0.jar"
```
```yaml
apiVersion: flink.apache.org/v1beta1
kind: FlinkSessionJob
metadata:
name: my-flink-job
spec:
deploymentName: basic-example
job:
jarURI: s3://test-bucket/myjar-0.0.1.jar
parallelism: 1
upgradeMode: stateless
```
the FlinkSessionJob fail with :
```
Could not find a file system implementation for scheme 's3a'. The scheme is
directly supported by Flink through the following plugin(s):
flink-s3-fs-hadoop. Please ensure that each plugin resides within its own
subfolder within the plugins directory. See
https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/
for more information. If you want to use a Hadoop file system for that scheme,
please add the scheme to the configuration fs.allowed-fallback-filesystems. For
a full list of supported file systems, please see
https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
```
and in the operator pod the logs are ->
```
java.util.ServiceConfigurationError:
org.apache.flink.core.fs.FileSystemFactory:
org.apache.flink.fs.s3hadoop.S3FileSystemFactory Unable to get public no-arg
constructor
at java.base/java.util.ServiceLoader.fail(Unknown Source)
at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
Source)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
Source)
at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
at
org.apache.flink.core.plugin.PluginLoader$ContextClassLoaderSettingIterator.hasNext(PluginLoader.java:130)
at
org.apache.flink.shaded.guava31.com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1391)
at
org.apache.flink.shaded.guava31.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at
org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1088)
at
org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1071)
at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:340)
at
org.apache.flink.kubernetes.operator.FlinkOperator.<init>(FlinkOperator.java:112)
at
org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:281)
Caused by: java.lang.NoClassDefFoundError:
org/apache/flink/core/fs/PathsCopyingFileSystem
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
at java.base/java.net.URLClassLoader.defineClass(Unknown Source)
at java.base/java.net.URLClassLoader$1.run(Unknown Source)
at java.base/java.net.URLClassLoader$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.net.URLClassLoader.findClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at
org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:150)
at
org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromOwnerFirst(ComponentClassLoader.java:172)
at
org.apache.flink.core.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:107)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
at java.base/java.lang.Class.getConstructor0(Unknown Source)
at java.base/java.lang.Class.getConstructor(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
... 13 more
```
> K8S operator : S3 jarURI
> -------------------------
>
> Key: FLINK-38282
> URL: https://issues.apache.org/jira/browse/FLINK-38282
> Project: Flink
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: raphaelauv
> Priority: Major
> Attachments: flink-kubernetes-operator.log
>
>
> Using the flink operator 1.8.2
> with flink 2.0.0
> I can't submit FlinkSessionJob that have a jarURI on a s3
> {code:java}
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkDeployment
> metadata:
> name: basic-example
> spec:
> image:
> flinkVersion: v2_0
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "2"
> s3.endpoint: http://minio:9000
> s3.path.style.access: "true"
> s3.access-key: admin
> s3.secret-key: adminadmin
> s3.region: eu-west-3
> serviceAccount: flink
> jobManager:
> resource:
> memory: "2048m"
> cpu: 1
> taskManager:
> resource:
> memory: "2048m"
> cpu: 1
> podTemplate:
> spec:
> containers:
> - name: flink-main-container
> env:
> - name: ENABLE_BUILT_IN_PLUGINS
> value: "flink-s3-fs-hadoop-2.0.0.jar" {code}
> {code:java}
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkSessionJob
> metadata:
> name: my-flink-job
> spec:
> deploymentName: basic-example
> job:
> jarURI: s3://test-bucket/myjar-0.0.1.jar
> parallelism: 1
> upgradeMode: stateless {code}
>
>
> the FlinkSessionJob fail with :
> {code:java}
> Could not find a file system implementation for scheme 's3a'. The scheme is
> directly supported by Flink through the following plugin(s):
> flink-s3-fs-hadoop. Please ensure that each plugin resides within its own
> subfolder within the plugins directory. See
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/
> for more information. If you want to use a Hadoop file system for that
> scheme, please add the scheme to the configuration
> fs.allowed-fallback-filesystems. For a full list of supported file systems,
> please see
> https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. {code}
> and in the operator pod the logs are ->
>
> {code:java}
> java.util.ServiceConfigurationError:
> org.apache.flink.core.fs.FileSystemFactory:
> org.apache.flink.fs.s3hadoop.S3FileSystemFactory Unable to get public no-arg
> constructor
> at java.base/java.util.ServiceLoader.fail(Unknown Source)
> at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
> at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
> Source)
> at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
> Source)
> at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
> at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
> at
> org.apache.flink.core.plugin.PluginLoader$ContextClassLoaderSettingIterator.hasNext(PluginLoader.java:130)
> at
> org.apache.flink.shaded.guava31.com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1391)
> at
> org.apache.flink.shaded.guava31.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
> at
> org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1088)
> at
> org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1071)
> at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:340)
> at
> org.apache.flink.kubernetes.operator.FlinkOperator.<init>(FlinkOperator.java:112)
> at
> org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:281)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/core/fs/PathsCopyingFileSystem
> at java.base/java.lang.ClassLoader.defineClass1(Native Method)
> at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
> at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
> at java.base/java.net.URLClassLoader.defineClass(Unknown Source)
> at java.base/java.net.URLClassLoader$1.run(Unknown Source)
> at java.base/java.net.URLClassLoader$1.run(Unknown Source)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/java.net.URLClassLoader.findClass(Unknown Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at
> org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:150)
> at
> org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromOwnerFirst(ComponentClassLoader.java:172)
> at
> org.apache.flink.core.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:107)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
> at java.base/java.lang.Class.getConstructor0(Unknown Source)
> at java.base/java.lang.Class.getConstructor(Unknown Source)
> at java.base/java.util.ServiceLoader$1.run(Unknown Source)
> at java.base/java.util.ServiceLoader$1.run(Unknown Source)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> ... 13 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
