Re: yarnship option

[Mikhail Pryakhin]

Hi Robert! Thanks a lot for your reply! >Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ? The jar does contain the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file is listed below) > Also, were there any previous Kafka09 related exceptions in the log??     No, it was the very first exception… You can find log file attached.  the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following: root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar Archive:  job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar   Length      Date    Time    Name ---------  ---------- -----   ----         0  2017-04-11 01:59   META-INF/       443  2017-04-11 01:59   META-INF/MANIFEST.MF      1451  2017-04-11 01:59   META-INF/DEPENDENCIES     11358  2017-04-11 01:59   META-INF/LICENSE       182  2017-04-11 01:59   META-INF/NOTICE         0  2017-04-11 01:59   org/         0  2017-04-11 01:59   org/apache/         0  2017-04-11 01:59   org/apache/flink/         0  2017-04-11 01:59   org/apache/flink/streaming/         0  2017-04-11 01:59   org/apache/flink/streaming/connectors/         0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/     11554  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class      2603  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class      4463  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class      2693  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class         0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/      1125  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class       570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class     10268  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class       570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class      3018  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover.class     11854  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class      2182  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class       311  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class      2329  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class         0  2017-04-11 01:59   META-INF/maven/         0  2017-04-11 01:59   META-INF/maven/org.apache.flink/         0  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/      6039  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml       131  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties         0  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/      3285  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.xml       114  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.properties ---------                     -------     76543                     33 files

flink-root-client-dmpkit-dev-dn1.log
Description: Binary data

------ Mike Pryakhin On 22 May 2017, at 22:14, Robert Metzger <rmetz...@apache.org> wrote: Hi, this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ? Also, were there any previous Kafka09 related exceptions in the log?? From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695 On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.prya...@cleverdata.ru> wrote: Hi all! I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out. The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option. Here is the way I run the job: export HADOOP_USER_NAME=hdfs; \ export HADOOP_CONF_DIR=/etc/hadoop/conf/; \ /opt/flink-1.2.1/bin/flink run \         -yst \         -yt /home/user/job-libs  \         -m yarn-cluster \         -yn 3 \         -c com.flink.Test \         flink-test_2.11-1.0.0-SNAPSHOT.jar Finally the job fails complaing that it can't find the class: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath: 2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner             -  Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...> Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath ---- Mike Pryakhin