Hi Arvid, Alexander Preuß has already replied to me and I also found a discussion on https://stackoverflow.com/questions/51479657/flinkkafkaconsumer011-not-found-on-flink-cluster.
So by following https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/project-configuration/#adding-connector-and-library-dependencies, that is to remove "provided" dependency scope and let Maven include it in fat jar, it can work. So, if we do not put the connector jar under Flink /lib, but include it in the fat jar of the application. Everything is fine. I have replied to Alexander a further question as the follows: But I am also curious about how to explain it? Actually the information you pointed out said it is recommended to do it, but not said "cannot put into /lib". As I mentioned at the beginning, Flink does "child-first" resolution on classes, why Flink cannot find Kafka classes in the application jar if the connector jar is under /lib? It is pretty confusing from a user's point of view. On Mon, Oct 18, 2021 at 2:01 AM Arvid Heise <ar...@apache.org> wrote: > > This looks very odd. How do you create the fat jar? What's your Flink version? > > I don't think this is a general Flink issue or else no one would be able to > read from Kafka at all. > > On Fri, Oct 15, 2021 at 4:16 AM L. C. Hsieh <vii...@apache.org> wrote: >> >> Hi, Flink developers, >> >> Does anyone encounter the following error? >> >> java.lang.NoClassDefFoundError: >> org/apache/kafka/common/serialization/ByteArrayDeserializer >> at >> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.setDeserializer(FlinkKafkaConsumer.java:322) >> at >> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:223) >> at >> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:154) >> at >> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:139) >> at >> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:108) >> >> If I put kafka-clients jar into Flink's lib/, Flink can find it. But if I >> assembly it into the application jar, Flink cannot find it. But based on >> what I read from Flink doc, Flink does "child-first" resolution on classes. >> Why it cannot find kafka classes if they are in application jar?? >> >> I examined the application jar content. It includes these kafka classes >> actually. >> >> I tested it with K8S session and job clusters on Flink built from current >> source. Both have the same error.
