On Thu, Jun 02, 2022 at 06:23:20PM +0800, Qingsheng Ren wrote: > Thanks for the input ChangZhuo. > > Could you check if the configuration "classloader.resolve-order” is > set to “parent-first” in your Flink 1.14 cluster? I didn’t notice any > changes related to the user code classloader in Flink 1.15. If my > assumption is correct, you package the gcs-connector into your job JAR > but the Hadoop FS dependencies are not included, so > org.apache.hadoop.fs.FileSystem is loaded by app classloader from > flink-s3-fs-hadoop.jar under the lib of Flink, but > GoogleHadoopFileSystem is loaded by user code classloader from job > JAR. Setting the resolve order to "parent-first" could bypass the > issue [1] so I assume you have this config in 1.14 but not in 1.15. > Please forgive me if I understand incorrectly!
No, we do not config classloader.resolve-order in both 1.14, and 1.15
setup. We will check if "parent-first" can solve the problem, thanks for
the advise.
Also, in 1.14, we include the following jars into /opt/flink/lib to
support GCS:
* flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar
* gcs-connector-hadoop3-2.2.2-shaded.jar
In 1.15, we add flink-gs-fs-hadoop-1.15.0.jar to /opt/flink/lib to
support GCS. Maybe this different causes problem?
--
ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org
http://czchen.info/
Key fingerprint = BA04 346D C2E1 FE63 C790 8793 CC65 B0CD EC27 5D5B
signature.asc
Description: PGP signature
