Hi all!
I'm playing with flink streaming job on yarn cluster. The job consumes events
from kafka and prints them to the standard out.
The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed
via the --yarnship option.
Here is the way I run the job:
export HADOOP_USER_NAME=hdfs; \
export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
/opt/flink-1.2.1/bin/flink run \
-yst \
-yt /home/user/job-libs \
-m yarn-cluster \
-yn 3 \
-c com.flink.Test \
flink-test_2.11-1.0.0-SNAPSHOT.jar
Finally the job fails complaing that it can't find the class:
java.lang.NoClassDefFoundError:
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
I looked through the jobmanager.log and found that the
flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:
2017-05-22 17:41:59,637 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Classpath:
job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>
Could please help to figure out why the class
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be
loaded inspite of the jar containing this class was added to the classpath
----
Mike Pryakhin
