Hi John, This old Avro version in Beam is a very long story. Briefly, since initially it was toughly integrated into Java SDK “core” module then it was not possible to upgrade an Avro version without breaking changes for users (because of some Avro incompatible changes, as you have noticed before). So, we decided to extract Avro-related classes from Beam “core” to a dedicated Avro extension [2] that supports and actually is tested with different Avro versions. More details on this work are here [1]
Regarding auto-generated classes. Initially, we used a Gradle plugin for that but it’s limited with only one Avro version per instance of this plugin, so it was not possible to generate these classes with different Avro versions. So, we do this with a special Gradle task (“JavaExec") that executes “org.apache.avro.tool.Main” and generate Avro classes per every tested Avro version [3]. We still keep an old Avro version 1.8.2. as a default dependency version but it will be overwritten if users have a more recent one as a project dependency in their classpath. I think we need to completely remove Avro Gradle plugin (use “JavaExec” task to generate Avro classes with a provided Avro version instead) and update the default Avro version to the more recent one since now it’s not part of Java “core”. Any thoughts? — Alexey [1] https://github.com/apache/beam/issues/24292 [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro [3] https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135 > On 10 Nov 2023, at 18:05, John Casey via dev <dev@beam.apache.org> wrote: > > Hi All, > > There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying to > upgrade to avro 1.11.3. > > Unfortunately, it seems that our auto-generated Avro test classes aren't > being generated properly with this new version. I've updated our avro > generation plugin as well, but for whatever reason, it seems that the > generated AvroTest file is being generated with references to classes that > did exist in 1.8.2, but no longer exist in 1.11.3. > > It seems like our autogeneration is being run with the wrong avro version, > but I can't seem to find where that would be configured. > > Here is the PR with my changes so far: > https://github.com/apache/beam/pull/29390 > > Is anyone familiar with what might be misconfigured here? > > John
