Hi Robert! You were right.. it was my fault, I didn’t copy all required dependencies for the flink-connector-kafka. That made impossible to load the class.
Everything works as expected but only if I put required dependencies inside the flink lib folder, but when dependencies are located in a different folder which is passed in the yarnship command line option then they seem not to be put on the TM's classpath, and the job fails with ClassNotFoundException. Is there anything else I need to provide to make libraries in that folder to be on the TM's classpath? Many thanks, —— Mike Pryakhin > On 22 May 2017, at 23:06, Mikhail Pryakhin <m.prya...@cleverdata.ru> wrote: > > Hi Robert! > Thanks a lot for your reply! > > >Can you double check if the > >job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains > >org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ? > > The jar does contain the class > org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class (The > contents of the jar file is listed below) > > > Also, were there any previous Kafka09 related exceptions in the log?? > No, it was the very first exception… You can find log file attached. > > > the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following: > > root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar > Archive: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar > Length Date Time Name > --------- ---------- ----- ---- > 0 2017-04-11 01:59 META-INF/ > 443 2017-04-11 01:59 META-INF/MANIFEST.MF > 1451 2017-04-11 01:59 META-INF/DEPENDENCIES > 11358 2017-04-11 01:59 META-INF/LICENSE > 182 2017-04-11 01:59 META-INF/NOTICE > 0 2017-04-11 01:59 org/ > 0 2017-04-11 01:59 org/apache/ > 0 2017-04-11 01:59 org/apache/flink/ > 0 2017-04-11 01:59 org/apache/flink/streaming/ > 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/ > 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/ > 11554 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class > 2603 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class > 4463 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class > 2693 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class > 0 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/internal/ > 1125 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class > 570 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class > 10268 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class > 570 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class > 3018 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/internal/Handover.class > 11854 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class > 2182 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class > 311 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class > 2329 2017-04-11 01:59 > org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class > 0 2017-04-11 01:59 META-INF/maven/ > 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/ > 0 2017-04-11 01:59 > META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/ > 6039 2017-04-11 01:59 > META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml > 131 2017-04-11 01:59 > META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties > 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/ > 3285 2017-04-11 01:59 > META-INF/maven/org.apache.flink/force-shading/pom.xml > 114 2017-04-11 01:59 > META-INF/maven/org.apache.flink/force-shading/pom.properties > --------- ------- > 76543 33 files > > > > > <flink-root-client-dmpkit-dev-dn1.log> > > > ------ > Mike Pryakhin > > > >> On 22 May 2017, at 22:14, Robert Metzger <rmetz...@apache.org >> <mailto:rmetz...@apache.org>> wrote: >> >> Hi, >> >> this issue is unexpected :) Can you double check if the >> job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains >> org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ? >> Also, were there any previous Kafka09 related exceptions in the log?? >> >> From this SO answer, it seems that this is not really the classical >> classNotFoundException, but a bit differenT: >> https://stackoverflow.com/a/5756989/568695 >> <https://stackoverflow.com/a/5756989/568695> >> >> On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.prya...@cleverdata.ru >> <mailto:m.prya...@cleverdata.ru>> wrote: >> Hi all! >> >> I'm playing with flink streaming job on yarn cluster. The job consumes >> events from kafka and prints them to the standard out. >> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed >> via the --yarnship option. >> Here is the way I run the job: >> >> export HADOOP_USER_NAME=hdfs; \ >> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \ >> /opt/flink-1.2.1/bin/flink run \ >> -yst \ >> -yt /home/user/job-libs \ >> -m yarn-cluster \ >> -yn 3 \ >> -c com.flink.Test \ >> flink-test_2.11-1.0.0-SNAPSHOT.jar >> >> Finally the job fails complaing that it can't find the class: >> java.lang.NoClassDefFoundError: >> org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 >> >> I looked through the jobmanager.log and found that the >> flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath: >> >> 2017-05-22 17:41:59,637 INFO >> org.apache.flink.yarn.YarnApplicationMasterRunner - Classpath: >> job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...> >> >> Could please help to figure out why the class >> org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be >> loaded inspite of the jar containing this class was added to the classpath >> >> ---- >> Mike Pryakhin >> >
