[
https://issues.apache.org/jira/browse/FLINK-30318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646412#comment-17646412
]
Samrat Deb commented on FLINK-30318:
------------------------------------
Thank you [~luoyuxia] , for spending time and looking at it .
https://issues.apache.org/jira/browse/FLINK-15635 explained same, as you have
suggested
There are 4 type of classloader introduced
# ComponentClassloader
# AppClassloader
# ClientWrapperClassloader
# FlinkUserCodeClassloader
We have created a ticket for the fix in sync
https://issues.apache.org/jira/browse/FLINK-30377
I will add a patch for it by today , please help review the same .
i will close this ticket and create different one for the show jars unexpected
behaviour in 1.16.0 onwards .
> sql-client failed to load jar passed with -j args in flink 1.16.0
> ------------------------------------------------------------------
>
> Key: FLINK-30318
> URL: https://issues.apache.org/jira/browse/FLINK-30318
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
> Affects Versions: 1.16.0
> Reporter: Samrat Deb
> Priority: Major
>
> In Flink 1.16.0 , /usr/lib/flink/bin/sql-client.sh embedded -j
> hudi-flink-bundle_2.12-0.10.1.jar failed to load jar passed through argument
>
> {code:java}
> /usr/lib/flink/bin/sql-client.sh embedded -j
> hudi-flink-bundle_2.12-0.10.1.jar
> Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
> Using the result of 'hadoop classpath' to augment the Hadoop classpath:
> /etc/hadoop/conf:/usr/lib/hadoop/lib/:/usr/lib/hadoop/.//:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/:/usr/lib/hadoop-hdfs/.//:/usr/lib/hadoop-mapreduce/.//:/usr/lib/hadoop-yarn/lib/:/usr/lib/hadoop-yarn/.//:/etc/tez/conf:/usr/lib/tez/hadoop-shim-0.10.2-amzn-0.jar:/usr/lib/tez/hadoop-shim-2.8-0.10.2-amzn-0.jar:/usr/lib/tez/lib:/usr/lib/tez/LICENSE:/usr/lib/tez/LICENSE-BSD-3clause:/usr/lib/tez/LICENSE-CDDLv1.1-GPLv2_withCPE:/usr/lib/tez/LICENSE-MIT:/usr/lib/tez/LICENSE-SIL_OpenFontLicense-v1.1:/usr/lib/tez/NOTICE:/usr/lib/tez/tez-api-0.10.2-amzn-0.jar:/usr/lib/tez/tez-aux-services-0.10.2-amzn-0.jar:/usr/lib/tez/tez-build-tools-0.10.2-amzn-0.jar:/usr/lib/tez/tez-common-0.10.2-amzn-0.jar:/usr/lib/tez/tez-dag-0.10.2-amzn-0.jar:/usr/lib/tez/tez-examples-0.10.2-amzn-0.jar:/usr/lib/tez/tez-history-parser-0.10.2-amzn-0.jar:/usr/lib/tez/tez-javadoc-tools-0.10.2-amzn-0.jar:/usr/lib/tez/tez-job-analyzer-0.10.2-amzn-0.jar:/usr/lib/tez/tez-mapreduce-0.10.2-amzn-0.jar:/usr/lib/tez/tez-protobuf-history-plugin-0.10.2-amzn-0.jar:/usr/lib/tez/tez-runtime-internals-0.10.2-amzn-0.jar:/usr/lib/tez/tez-runtime-library-0.10.2-amzn-0.jar:/usr/lib/tez/tez-tests-0.10.2-amzn-0.jar:/usr/lib/tez/tez-ui-0.10.2-amzn-0.war:/usr/lib/tez/tez-yarn-timeline-cache-plugin-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-with-acls-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-with-fs-0.10.2-amzn-0.jar:/usr/lib/tez/lib/async-http-client-2.12.3.jar:/usr/lib/tez/lib/commons-cli-1.2.jar:/usr/lib/tez/lib/commons-codec-1.11.jar:/usr/lib/tez/lib/commons-collections4-4.1.jar:/usr/lib/tez/lib/commons-io-2.8.0.jar:/usr/lib/tez/lib/commons-lang-2.6.jar:/usr/lib/tez/lib/guava-31.1-jre.jar:/usr/lib/tez/lib/hadoop-annotations.jar:/usr/lib/tez/lib/hadoop-auth.jar:/usr/lib/tez/lib/hadoop-hdfs-client-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-core-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-yarn-server-web-proxy.jar:/usr/lib/tez/lib/hhadoop-mapreduce-client-core.jar:/usr/lib/tez/lib/javax.servlet-api-3.1.0.jar:/usr/lib/tez/lib/jersey-client-1.19.jar:/usr/lib/tez/lib/jersey-json-1.19.jar:/usr/lib/tez/lib/jettison-1.3.4.jar:/usr/lib/tez/lib/jsr305-3.0.0.jar:/usr/lib/tez/lib/metrics-core-3.1.0.jar:/usr/lib/tez/lib/netty-all-4.1.72.Final.jar:/usr/lib/tez/lib/protobuf-java-2.5.0.jar:/usr/lib/tez/lib/RoaringBitmap-0.7.45.jar:/usr/lib/tez/lib/slf4j-api-1.7.36.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/native:/usr/share/aws/aws-java-sdk/aws-java-sdk-bundle-1.12.331.jar:/usr/share/aws/aws-java-sdk/LICENSE.txt:/usr/share/aws/aws-java-sdk/NOTICE.txt:/usr/share/aws/aws-java-sdk/README.md:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/animal-sniffer-annotations-1.14.jar:/usr/share/aws/emr/emrfs/lib/annotations-16.0.2.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-ext-jdk15on-1.66.jar:/usr/share/aws/emr/emrfs/lib/checker-qual-2.5.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.54.0.jar:/usr/share/aws/emr/emrfs/lib/error_prone_annotations-2.1.3.jar:/usr/share/aws/emr/emrfs/lib/findbugs-annotations-3.0.1.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/j2objc-annotations-1.1.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.12.331.jar:/usr/share/aws/emr/emrfs/lib/jsr305-3.0.2.jar:/usr/share/aws/emr/emrfs/auxlib/:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-2.3.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/lib/flink/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> 2022-12-07 03:17:05,554 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Found Yarn properties file under
> /var/lib/flink/yarn/.yarn-properties-hadoop.
> 2022-12-07 03:17:05,554 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Found Yarn properties file under
> /var/lib/flink/yarn/.yarn-properties-hadoop.
> ????????
> ????????????????
> ??????? ??????? ?
> ???? ????????? ?????
> ??? ??????? ?????
> ??? ??? ?????
> ?? ???????????????
> ?? ? ??? ?????? ?????
> ????? ???? ????? ?????
> ??????? ??? ??????? ???
> ????????? ?? ?? ??????????
> ???????? ?? ? ?? ???????
> ???? ??? ? ?? ???????? ?????
> ???? ? ?? ? ?? ???????? ???? ??
> ???? ???? ?????????? ??? ?? ????
> ???? ?? ??? ??????????? ???? ? ? ???
> ??? ?? ??? ????????? ???? ???
> ?? ? ??????? ???????? ??? ??
> ??? ??? ???????????????????? ???? ?
> ????? ??? ?????? ???????? ???? ??
> ???????? ??????????????? ??
> ?? ???? ??????? ??? ?????? ?? ???
> ??? ??? ??? ??????? ???? ?????????????
> ??? ????? ???? ?? ?? ???? ???
> ?? ??? ? ?? ?? ??
> ?? ?? ?? ?? ????????
> ?? ????? ?? ??????????? ??
> ?? ???? ? ??????? ??
> ??? ????? ?? ???????????
> ???? ???? ??????? ????????
> ????? ?? ???? ?????
> ????????????????????????????????? ?????
> ______ _ _ _ _____ ____ _ _____ _ _ _
> BETA
> | ___| () | | / ___|/ __ | | / ____| () | |
> | |__ | |_ _ __ | | __ | (___ | | | | | | | | |_ ___ _ __ | |_
> | _| | | | ' | |/ / ___ | | | | | | | | | |/ _ \ '_ | __|
> | | | | | | | | < ___) | || | |___ | |___| | | / | | | |
> || |||| |||_\ |____/ ______| ___|||_|| ||_|
> Welcome! Enter 'HELP;' to list all available commands. 'QUIT;' to
> exit.
> Command history file path: /home/hadoop/.flink-sql-history
> Flink SQL> show jars;
> Empty set
> {code}
> This works only when keeping the hudi-flink bundle jar in /flink/lib/ path
>
> But this works with flink 1.15.2
> {code:java}
> /usr/lib/flink/bin/sql-client.sh embedded -j hudi-flink-bundle_2.12-0.10.1.jar
> Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
> Using the result of 'hadoop classpath' to augment the Hadoop classpath:
> /etc/hadoop/conf:/usr/lib/hadoop/lib/:/usr/lib/hadoop/.//:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/:/usr/lib/hadoop-hdfs/.//:/usr/lib/hadoop-mapreduce/.//:/usr/lib/hadoop-yarn/lib/:/usr/lib/hadoop-yarn/.//:/etc/tez/conf:/usr/lib/tez/hadoop-shim-0.10.2-amzn-0.jar:/usr/lib/tez/hadoop-shim-2.8-0.10.2-amzn-0.jar:/usr/lib/tez/lib:/usr/lib/tez/LICENSE:/usr/lib/tez/LICENSE-BSD-3clause:/usr/lib/tez/LICENSE-CDDLv1.1-GPLv2_withCPE:/usr/lib/tez/LICENSE-MIT:/usr/lib/tez/LICENSE-SIL_OpenFontLicense-v1.1:/usr/lib/tez/NOTICE:/usr/lib/tez/tez-api-0.10.2-amzn-0.jar:/usr/lib/tez/tez-aux-services-0.10.2-amzn-0.jar:/usr/lib/tez/tez-build-tools-0.10.2-amzn-0.jar:/usr/lib/tez/tez-common-0.10.2-amzn-0.jar:/usr/lib/tez/tez-dag-0.10.2-amzn-0.jar:/usr/lib/tez/tez-examples-0.10.2-amzn-0.jar:/usr/lib/tez/tez-history-parser-0.10.2-amzn-0.jar:/usr/lib/tez/tez-javadoc-tools-0.10.2-amzn-0.jar:/usr/lib/tez/tez-job-analyzer-0.10.2-amzn-0.jar:/usr/lib/tez/tez-mapreduce-0.10.2-amzn-0.jar:/usr/lib/tez/tez-protobuf-history-plugin-0.10.2-amzn-0.jar:/usr/lib/tez/tez-runtime-internals-0.10.2-amzn-0.jar:/usr/lib/tez/tez-runtime-library-0.10.2-amzn-0.jar:/usr/lib/tez/tez-tests-0.10.2-amzn-0.jar:/usr/lib/tez/tez-ui-0.10.2-amzn-0.war:/usr/lib/tez/tez-yarn-timeline-cache-plugin-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-with-acls-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-with-fs-0.10.2-amzn-0.jar:/usr/lib/tez/lib/async-http-client-2.12.3.jar:/usr/lib/tez/lib/commons-cli-1.2.jar:/usr/lib/tez/lib/commons-codec-1.11.jar:/usr/lib/tez/lib/commons-collections4-4.1.jar:/usr/lib/tez/lib/commons-io-2.8.0.jar:/usr/lib/tez/lib/commons-lang-2.6.jar:/usr/lib/tez/lib/guava-31.1-jre.jar:/usr/lib/tez/lib/hadoop-annotations.jar:/usr/lib/tez/lib/hadoop-auth.jar:/usr/lib/tez/lib/hadoop-hdfs-client-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-core-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-yarn-server-web-proxy.jar:/usr/lib/tez/lib/hhadoop-mapreduce-client-core.jar:/usr/lib/tez/lib/javax.servlet-api-3.1.0.jar:/usr/lib/tez/lib/jersey-client-1.19.jar:/usr/lib/tez/lib/jersey-json-1.19.jar:/usr/lib/tez/lib/jettison-1.3.4.jar:/usr/lib/tez/lib/jsr305-3.0.0.jar:/usr/lib/tez/lib/metrics-core-3.1.0.jar:/usr/lib/tez/lib/netty-all-4.1.72.Final.jar:/usr/lib/tez/lib/protobuf-java-2.5.0.jar:/usr/lib/tez/lib/RoaringBitmap-0.7.45.jar:/usr/lib/tez/lib/slf4j-api-1.7.36.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/native:/usr/share/aws/aws-java-sdk/aws-java-sdk-bundle-1.12.331.jar:/usr/share/aws/aws-java-sdk/LICENSE.txt:/usr/share/aws/aws-java-sdk/NOTICE.txt:/usr/share/aws/aws-java-sdk/README.md:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/animal-sniffer-annotations-1.14.jar:/usr/share/aws/emr/emrfs/lib/annotations-16.0.2.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-ext-jdk15on-1.66.jar:/usr/share/aws/emr/emrfs/lib/checker-qual-2.5.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.54.0.jar:/usr/share/aws/emr/emrfs/lib/error_prone_annotations-2.1.3.jar:/usr/share/aws/emr/emrfs/lib/findbugs-annotations-3.0.1.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/j2objc-annotations-1.1.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.12.331.jar:/usr/share/aws/emr/emrfs/lib/jsr305-3.0.2.jar:/usr/share/aws/emr/emrfs/auxlib/:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-2.3.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/lib/flink/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> 2022-12-07 03:19:37,344 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Found Yarn properties file under
> /var/lib/flink/yarn/.yarn-properties-hadoop.
> 2022-12-07 03:19:37,344 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Found Yarn properties file under
> /var/lib/flink/yarn/.yarn-properties-hadoop.
> ????????
> ????????????????
> ??????? ??????? ?
> ???? ????????? ?????
> ??? ??????? ?????
> ??? ??? ?????
> ?? ???????????????
> ?? ? ??? ?????? ?????
> ????? ???? ????? ?????
> ??????? ??? ??????? ???
> ????????? ?? ?? ??????????
> ???????? ?? ? ?? ???????
> ???? ??? ? ?? ???????? ?????
> ???? ? ?? ? ?? ???????? ???? ??
> ???? ???? ?????????? ??? ?? ????
> ???? ?? ??? ??????????? ???? ? ? ???
> ??? ?? ??? ????????? ???? ???
> ?? ? ??????? ???????? ??? ??
> ??? ??? ???????????????????? ???? ?
> ????? ??? ?????? ???????? ???? ??
> ???????? ??????????????? ??
> ?? ???? ??????? ??? ?????? ?? ???
> ??? ??? ??? ??????? ???? ?????????????
> ??? ????? ???? ?? ?? ???? ???
> ?? ??? ? ?? ?? ??
> ?? ?? ?? ?? ????????
> ?? ????? ?? ??????????? ??
> ?? ???? ? ??????? ??
> ??? ????? ?? ???????????
> ???? ???? ??????? ????????
> ????? ?? ???? ?????
> ????????????????????????????????? ?????
> ______ _ _ _ _____ ____ _ _____ _ _ _
> BETA
> | ___| () | | / ___|/ __ | | / ____| () | |
> | |__ | |_ _ __ | | __ | (___ | | | | | | | | |_ ___ _ __ | |_
> | _| | | | ' | |/ / ___ | | | | | | | | | |/ _ \ '_ | __|
> | | | | | | | | < ___) | || | |___ | |___| | | / | | | |
> || |||| |||_\ |____/ ______| ___|||_|| ||_|
> Welcome! Enter 'HELP;' to list all available commands. 'QUIT;' to
> exit.
> Command history file path: /home/hadoop/.flink-sql-history
> Flink SQL> show jars;
> /usr/lib/flink/opt/flink-sql-client-1.15.2.jar
> /usr/lib/flink/opt/flink-python_2.12-1.15.2.jar
> /home/hadoop/hudi-flink-bundle_2.12-0.10.1.jar
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
