Hi Alex, Yes, starting from Beam 2.20.0, "beam-sdks-java-io-kafka” requires an additional dependency “kafka-avro-serializer” from external repository (https://packages.confluent.io/maven/).
This is reflected in published POM file: https://search.maven.org/artifact/org.apache.beam/beam-sdks-java-io-kafka/2.31.0/jar Did it work for you before version 2.30.0? Could you share your build.gradle file? — Alexey > On 9 Jul 2021, at 11:23, Alex Van Boxel <a...@vanboxel.be> wrote: > > Hi all, > > I've been building for years via gradle. The dependency management is > probably a bit different from that of maven, but it seems that dataflow now > requires Confluent Kafka dependencies. They are not available in Maven > Central. This feels wrong for an Apache project. > > - > file:/Users/alex.vanboxel/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom > - > https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom > > <https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom> > - > https://repository.apache.org/content/repositories/releases/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom > > <https://repository.apache.org/content/repositories/releases/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom> > > Excluding the dependencies "exclude module: 'beam-sdks-java-io-kafka'" > doesn't work with: > > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/beam/sdk/io/kafka/KafkaIO$Read > at > org.apache.beam.runners.dataflow.DataflowRunner.getOverrides(DataflowRunner.java:522) > at > org.apache.beam.runners.dataflow.DataflowRunner.replaceV1Transforms(DataflowRunner.java:1337) > at > org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:967) > at > org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:196) > > This happens from version 2.30 onwards. Is this intended?! > > _/ > _/ Alex Van Boxel
