I also noticed that Flink s3-hadoop plugin has Hadoop common dependency. I' trying this.
>From the logs, the plugin is enabled: Enabling required built-in plugins Linking flink-s3-fs-hadoop-1.12-SNAPSHOT.jar to plugin directory Successfully enabled flink-s3-fs-hadoop-1.12-SNAPSHOT.jar But I also got: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configurable at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:63) So it looks like user code cannot use plugin's classes (e.g. Hadoop classes)? I don't see hadoop-common is shaded at the plugin. On 2021/08/22 18:24:24, L. C. Hsieh <vii...@apache.org> wrote: > > As I know, flink-shaded-hadoop is not officially supported since Flink 1.11 > (https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html). > > Anyway, I installed Hadoop common package into the docker images to make > Flink happy. I marked the hadoop dependencies in the iceberg-test application > as "provided". Looks okay as it won't throw the previous LinkageError anymore. > > But a new issue is, the application submitted cannot be run because the task > manager pod is failed to run. There is warning message by describing pod > status: > > Warning FailedMount 4m4s kubelet Unable to > attach or mount volumes: unmounted volumes=[hadoop-config-volume], unattached > volumes=[flink-config-volume kube-api-access-772x5 hadoop-config-volume]: > timed out waiting for the condition > Warning FailedMount 108s (x2 over 6m22s) kubelet Unable to > attach or mount volumes: unmounted volumes=[hadoop-config-volume], unattached > volumes=[hadoop-config-volume flink-config-volume kube-api-access-772x5]: > timed out waiting for the condition > Warning FailedMount 11s (x12 over 8m25s) kubelet > MountVolume.SetUp failed for volume "hadoop-config-volume" : configmap > "hadoop-config-my-first-flink-cluster" not found > > Seems it cannot mount "hadoop-config-volume". > > From the doc > (https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/), > it looks like that Flink will add some internal ConfigMap volumes > automatically. So again, I am not sure what is wrong in above steps... > > > On 2021/08/22 10:01:25, Manong Karl <abc549...@gmail.com> wrote: > > I prefer using flink bundled hadoop, such as > > https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar. > > May help. > > > > L. C. Hsieh <vii...@apache.org> 于2021年8月22日周日 上午1:40写道： > > > > > > > > BTW, I checked dependency tree, the flink-iceberg demo only has one Hadoop > > > common dependency. So I'm not sure why Flink throws such exception. Based > > > on Flink doc, I suppose that Flink binary doesn't include Hadoop > > > dependencies, right? > > > > > > Based on the exception, looks like when FlinkCatalogFactory (from Iceberg) > > > calls HadoopUtils.getHadoopConfiguration (from Flink), their classloaders > > > are different and referring to different Hadoop Configuration Class > > > objects. > > > > > > I'm not familiar with Flink. So I'm wondering what step is wrong during > > > the testing? It is a pretty simple test to verify Iceberg and Flink. > > > > > > On 2021/08/21 08:50:05, L. C. Hsieh <vii...@apache.org> wrote: > > > > > > > > Thanks for replying. > > > > > > > > I'm using Flink 1.12.x. And I think Iceberg 0.12 uses Flink 1.12 > > > actually. > > > > > > > > Once I upgraded the Iceberg from 0.11.0 to 0.12.0 for the Java > > > application. I got new exception as below: > > > > > > > > java.lang.LinkageError: loader constraint violation: when resolving > > > method > > > "org.apache.flink.runtime.util.HadoopUtils.getHadoopConfiguration(Lorg/apache/flink/configuration/Configuration;)Lorg/apache/hadoop/conf/Configuration;" > > > the class loader (instance of org/apache/flink/util/ChildFirstClassLoader) > > > of the current class, org/apache/iceberg/flink/FlinkCatalogFactory, and > > > the > > > class loader (instance of sun/misc/Launcher$AppClassLoader) for the > > > method's defining class, org/apache/flink/runtime/util/HadoopUtils, have > > > different Class objects for the type org/apache/hadoop/conf/Configuration > > > used in the signature at > > > org.apache.iceberg.flink.FlinkCatalogFactory.clusterHadoopConf(FlinkCatalogFactory.java:152) > > > > > > > > > > > > On 2021/08/21 08:11:33, Manong Karl <abc549...@gmail.com> wrote: > > > > > Iceberg v0.11 or v0.12 not capable with flink v1.13.x. > > > > > > > > > > L. C. Hsieh <vii...@apache.org> 于2021年8月21日周六 下午3:52写道： > > > > > > > > > > > Hi, I'm testing using Flink to write Iceberg table. I run Flink > > > native K8S > > > > > > cluster locally and submit a simple Java program that writes out > > > Iceberg > > > > > > table (https://github.com/spancer/flink-iceberg-demo). But got an > > > > > > exception: > > > > > > > > > > > > java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration > > > at > > > > > > > > > org.apache.iceberg.flink.FlinkCatalogFactory.clusterHadoopConf(FlinkCatalogFactory.java:148) > > > > > > at > > > > > > > > > org.apache.iceberg.flink.TableLoader.fromHadoopTable(TableLoader.java:49) > > > > > > at > > > > > > > > > com.coomia.iceberg.test.IcebergReadWriteTest.main(IcebergReadWriteTest.java:89) > > > > > > > > > > > > The uploaded is a fat jar. I also checked the uploaded application > > > jar. It > > > > > > has the Configuration class. So I don't know what is wrong there. > > > Any idea > > > > > > or suggestion? Thanks. > > > > > > > > > > > > > > > > > > > > >
