I agree that we should make this optional. What would be the best way to it with gradle?
> On 11 Jul 2021, at 16:40, Jan Lukavský <je...@seznam.cz> wrote: > > I'd be +1 to making it optional as well. Looks really like an overhead for > users not using avro. > > On 7/11/21 10:36 AM, Alex Van Boxel wrote: >> It worked before 2.30. It's fine for when you're using Confluent Kafka, but >> feels like a hard dependency for non-Kafka users. Certainly the requirement >> for including an extra repo. Certain companies have to go through a lengthy >> process to include extra repo's. Feels like a strange requirement, for.... >> nothing. Isn't it a bug in the DataflowRunner? >> >> dependencies { >> compile("org.apache.beam:beam-sdks-java-core:$beamVersion") >> compile("org.apache.beam:beam-runners-direct-java:$beamVersion") >> compile("org.apache.beam:beam-runners-google-cloud-dataflow-java:$beamVersion") >> // waiting for response on mailinglist (2.30 onwards), dataflow runner fails >> // { >> // exclude module: 'beam-sdks-java-io-kafka' >> // } >> >> compile("org.apache.beam:beam-sdks-java-io-elasticsearch:$beamVersion") >> compile("org.apache.beam:beam-sdks-java-io-jdbc:$beamVersion") >> >> compile("org.apache.beam:beam-sdks-java-extensions-protobuf:$beamVersion") >> compile("org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:$beamVersion") >> compile("org.apache.beam:beam-sdks-java-extensions-sql:$beamVersion") >> compile("org.apache.beam:beam-sdks-java-extensions-zetasketch:$beamVersion") >> compile("org.apache.beam:beam-sdks-java-extensions-json-jackson:$beamVersion") >> >> compile("org.apache.beam:beam-sdks-java-io-google-cloud-platform:$beamVersion") >> compile("org.apache.beam:beam-sdks-java-io-common:$beamVersion") >> >> // force to the JRE, as the android version is auto resolved >> compile("com.google.guava:guava:30.1.1-jre") >> >> compile("org.slf4j:slf4j-log4j12:1.7.30") >> compile("commons-io:commons-io:2.8.0") >> >> compile("io.opentelemetry:opentelemetry-proto:1.3.0-alpha") >> >> compile("com.microsoft.sqlserver:mssql-jdbc:9.1.0.jre8-preview") >> >> compile("io.swagger.parser.v3:swagger-parser:2.0.24") >> >> testCompile("org.hamcrest:hamcrest-all:1.3") >> testCompile("org.assertj:assertj-core:3.4.1") >> testCompile("junit:junit:4.12") >> } >> >> >> _/ >> _/ Alex Van Boxel >> >> >> On Fri, Jul 9, 2021 at 1:55 PM Alexey Romanenko <aromanenko....@gmail.com >> <mailto:aromanenko....@gmail.com>> wrote: >> Hi Alex, >> >> Yes, starting from Beam 2.20.0, "beam-sdks-java-io-kafka” requires an >> additional dependency “kafka-avro-serializer” from external repository >> (https://packages.confluent.io/maven/ >> <https://packages.confluent.io/maven/>). >> >> This is reflected in published POM file: >> https://search.maven.org/artifact/org.apache.beam/beam-sdks-java-io-kafka/2.31.0/jar >> >> <https://search.maven.org/artifact/org.apache.beam/beam-sdks-java-io-kafka/2.31.0/jar> >> >> Did it work for you before version 2.30.0? >> Could you share your build.gradle file? >> >> — >> Alexey >> >> >> >>> On 9 Jul 2021, at 11:23, Alex Van Boxel <a...@vanboxel.be >>> <mailto:a...@vanboxel.be>> wrote: >>> >>> Hi all, >>> >>> I've been building for years via gradle. The dependency management is >>> probably a bit different from that of maven, but it seems that dataflow now >>> requires Confluent Kafka dependencies. They are not available in Maven >>> Central. This feels wrong for an Apache project. >>> >>> - >>> file:/Users/alex.vanboxel/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom >>> >>> <file:///Users/alex.vanboxel/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom> >>> - >>> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom >>> >>> <https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom> >>> - >>> https://repository.apache.org/content/repositories/releases/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom >>> >>> <https://repository.apache.org/content/repositories/releases/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom> >>> >>> Excluding the dependencies "exclude module: 'beam-sdks-java-io-kafka'" >>> doesn't work with: >>> >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/apache/beam/sdk/io/kafka/KafkaIO$Read >>> at >>> org.apache.beam.runners.dataflow.DataflowRunner.getOverrides(DataflowRunner.java:522) >>> at >>> org.apache.beam.runners.dataflow.DataflowRunner.replaceV1Transforms(DataflowRunner.java:1337) >>> at >>> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:967) >>> at >>> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:196) >>> >>> This happens from version 2.30 onwards. Is this intended?! >>> >>> _/ >>> _/ Alex Van Boxel >>