The vulnerability said to upgrade to 1.11.3, so I think that would be my starting point.
On Mon, Nov 13, 2023 at 12:23 PM Alexey Romanenko <aromanenko....@gmail.com> wrote: > > > On 10 Nov 2023, at 19:23, John Casey <theotherj...@google.com> wrote: > > I guess I'm a bit confused as to why specifically generateTestAvroJava > seems to use the wrong version. I see our version specific generated code, > but this action appears to be inherited from the plugin, and is configured > with whichever avro version is provided. Given that I tried to just change > to 1.11.3, I'm confused as to why its generating invalid java files for the > provided avro version. > > Unlike the classes generated out of the JavaExec you referenced, this > appears to only generate one version of the files. > > > It was supposed to generate files with a specific Avro version every time > to run the same tests again this specific Avro version. > > It may be that we don't need this action, but it still seems to run, as we > depend on it in the applyAvroNature() action. > > > I started to think if we really still need this action. > > We could remove this entirely. The java exec only generates versions for > pre-configured test versions anyways > > > Right. The point is in how many places in Beam we need to generate these > files and which version(s) of Avro to use? > > — > Alexey > > > On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko < > aromanenko....@gmail.com> wrote: > >> Hi John, >> >> This old Avro version in Beam is a very long story. Briefly, since >> initially it was toughly integrated into Java SDK “core” module then it was >> not possible to upgrade an Avro version without breaking changes for users >> (because of some Avro incompatible changes, as you have noticed before). >> So, we decided to extract Avro-related classes from Beam “core” to a >> dedicated Avro extension [2] that supports and actually is tested with >> different Avro versions. More details on this work are here [1] >> >> Regarding auto-generated classes. Initially, we used a Gradle plugin for >> that but it’s limited with only one Avro version per instance of this >> plugin, so it was not possible to generate these classes with different >> Avro versions. So, we do this with a special Gradle task (“JavaExec") that >> executes “org.apache.avro.tool.Main” and generate Avro classes per every >> tested Avro version [3]. >> >> We still keep an old Avro version 1.8.2. as a default dependency version >> but it will be overwritten if users have a more recent one as a project >> dependency in their classpath. >> >> I think we need to completely remove Avro Gradle plugin (use “JavaExec” >> task to generate Avro classes with a provided Avro version instead) and >> update the default Avro version to the more recent one since now it’s not >> part of Java “core”. >> >> Any thoughts? >> >> — >> Alexey >> >> >> [1] https://github.com/apache/beam/issues/24292 >> [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro >> [3] >> https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135 >> >> >> >> On 10 Nov 2023, at 18:05, John Casey via dev <dev@beam.apache.org> wrote: >> >> Hi All, >> >> There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying >> to upgrade to avro 1.11.3. >> >> Unfortunately, it seems that our auto-generated Avro test classes aren't >> being generated properly with this new version. I've updated our avro >> generation plugin as well, but for whatever reason, it seems that the >> generated AvroTest file is being generated with references to classes that >> did exist in 1.8.2, but no longer exist in 1.11.3. >> >> It seems like our autogeneration is being run with the wrong avro >> version, but I can't seem to find where that would be configured. >> >> Here is the PR with my changes so far: >> https://github.com/apache/beam/pull/29390 >> >> Is anyone familiar with what might be misconfigured here? >> >> John >> >> >> >
