I guess I'm a bit confused as to why specifically generateTestAvroJava seems to use the wrong version. I see our version specific generated code, but this action appears to be inherited from the plugin, and is configured with whichever avro version is provided. Given that I tried to just change to 1.11.3, I'm confused as to why its generating invalid java files for the provided avro version.
Unlike the classes generated out of the JavaExec you referenced, this appears to only generate one version of the files. It may be that we don't need this action, but it still seems to run, as we depend on it in the applyAvroNature() action. We could remove this entirely. The java exec only generates versions for pre-configured test versions anyways On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko <aromanenko....@gmail.com> wrote: > Hi John, > > This old Avro version in Beam is a very long story. Briefly, since > initially it was toughly integrated into Java SDK “core” module then it was > not possible to upgrade an Avro version without breaking changes for users > (because of some Avro incompatible changes, as you have noticed before). > So, we decided to extract Avro-related classes from Beam “core” to a > dedicated Avro extension [2] that supports and actually is tested with > different Avro versions. More details on this work are here [1] > > Regarding auto-generated classes. Initially, we used a Gradle plugin for > that but it’s limited with only one Avro version per instance of this > plugin, so it was not possible to generate these classes with different > Avro versions. So, we do this with a special Gradle task (“JavaExec") that > executes “org.apache.avro.tool.Main” and generate Avro classes per every > tested Avro version [3]. > > We still keep an old Avro version 1.8.2. as a default dependency version > but it will be overwritten if users have a more recent one as a project > dependency in their classpath. > > I think we need to completely remove Avro Gradle plugin (use “JavaExec” > task to generate Avro classes with a provided Avro version instead) and > update the default Avro version to the more recent one since now it’s not > part of Java “core”. > > Any thoughts? > > — > Alexey > > > [1] https://github.com/apache/beam/issues/24292 > [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro > [3] > https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135 > > > > On 10 Nov 2023, at 18:05, John Casey via dev <dev@beam.apache.org> wrote: > > Hi All, > > There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying to > upgrade to avro 1.11.3. > > Unfortunately, it seems that our auto-generated Avro test classes aren't > being generated properly with this new version. I've updated our avro > generation plugin as well, but for whatever reason, it seems that the > generated AvroTest file is being generated with references to classes that > did exist in 1.8.2, but no longer exist in 1.11.3. > > It seems like our autogeneration is being run with the wrong avro version, > but I can't seem to find where that would be configured. > > Here is the PR with my changes so far: > https://github.com/apache/beam/pull/29390 > > Is anyone familiar with what might be misconfigured here? > > John > > >
