Re: Upgrading Avro dependencies

[Alexey Romanenko]

As I mentioned before, we always can generate version-dependent Avro classes by 
running “org.apache.avro.tool.Main" directly with “JavaExec” Gradle task.

Please, see this implementation in Avro extension: 
https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135

In this case, we don’t depend on any 3rd party plugin, that can be run only 
with one Avro version by the whole project iirc.

—
Alexey 


 
> On 15 Nov 2023, at 17:14, John Casey  wrote:
> 
> Alright, it looks like something was broken with my setup. I think a 
> straightforward upgrade is actually possible, so I'm going to continue on that
> 
> On Wed, Nov 15, 2023 at 10:17 AM John Casey  > wrote:
>> So, thats the thing. I've upgraded to 1.11.3, but the plugin still seems to 
>> be generating Avro code that isn't compatible with 1.11.3.
>> 
>> Looking at the PR https://github.com/apache/beam/pull/29390, it looks like 
>> it generates avro code based on 1.9.2, which ends up being incompatible. 
>> 
>> I don't think we can continue to support every avro version with our current 
>> setup.
>> 
>> John
>> 
>> On Tue, Nov 14, 2023 at 4:20 PM Alexey Romanenko > > wrote:
>>> Thanks! Please, let me know if you need any help on this.
>>> 
>>> —
>>> Alexey
>>> 
 On 14 Nov 2023, at 17:52, John Casey >>> > wrote:
 
 The vulnerability said to upgrade to 1.11.3, so I think that would be my 
 starting point.
 
 
 On Mon, Nov 13, 2023 at 12:23 PM Alexey Romanenko 
 mailto:aromanenko@gmail.com>> wrote:
> 
> 
>> On 10 Nov 2023, at 19:23, John Casey > > wrote:
>> 
>> I guess I'm a bit confused as to why specifically generateTestAvroJava 
>> seems to use the wrong version. I see our version specific generated 
>> code, but this action appears to be inherited from the plugin, and is 
>> configured with whichever avro version is provided. Given that I tried 
>> to just change to 1.11.3, I'm confused as to why its generating invalid 
>> java files for the provided avro version.
>> 
>> Unlike the classes generated out of the JavaExec you referenced, this 
>> appears to only generate one version of the files.
> 
> It was supposed to generate files with a specific Avro version every time 
> to run the same tests again this specific Avro version. 
> 
>> It may be that we don't need this action, but it still seems to run, as 
>> we depend on it in the applyAvroNature() action.
> 
> I started to think if we really still need this action.
> 
>> We could remove this entirely. The java exec only generates versions for 
>> pre-configured test versions anyways
> 
> Right. The point is in how many places in Beam we need to generate these 
> files and which version(s) of Avro to use?
> 
> —
> Alexey
> 
>> 
>> On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko 
>> mailto:aromanenko@gmail.com>> wrote:
>>> Hi John,
>>> 
>>> This old Avro version in Beam is a very long story. Briefly, since 
>>> initially it was toughly integrated into Java SDK “core” module then it 
>>> was not possible to upgrade an Avro version without breaking changes 
>>> for users (because of some Avro incompatible changes, as you have 
>>> noticed before). So, we decided to extract Avro-related classes from 
>>> Beam “core” to a dedicated Avro extension [2] that supports and 
>>> actually is tested with different Avro versions. More details on this 
>>> work are here [1]
>>> 
>>> Regarding auto-generated classes. Initially, we used a Gradle plugin 
>>> for that but it’s limited with only one Avro version per instance of 
>>> this plugin, so it was not possible to generate these classes with 
>>> different Avro versions. So, we do this with a special Gradle task 
>>> (“JavaExec") that executes “org.apache.avro.tool.Main” and generate 
>>> Avro classes per every tested Avro version [3].
>>> 
>>> We still keep an old Avro version 1.8.2. as a default dependency 
>>> version but it will be overwritten if users have a more recent one as a 
>>> project dependency in their classpath.
>>> 
>>> I think we need to completely remove Avro Gradle plugin (use “JavaExec” 
>>> task to generate Avro classes with a provided Avro version instead) and 
>>> update the default Avro version to the more recent one since now it’s 
>>> not part of Java “core”.
>>> 
>>> Any thoughts?
>>> 
>>> —
>>> Alexey
>>>  
>>> 
>>> [1] https://github.com/apache/beam/issues/24292
>>> [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro
>>> [3] 
>>> https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f73568

Re: Upgrading Avro dependencies

[John Casey via dev]

Alright, it looks like something was broken with my setup. I think a
straightforward upgrade is actually possible, so I'm going to continue on
that

On Wed, Nov 15, 2023 at 10:17 AM John Casey  wrote:

> So, thats the thing. I've upgraded to 1.11.3, but the plugin still seems
> to be generating Avro code that isn't compatible with 1.11.3.
>
> Looking at the PR https://github.com/apache/beam/pull/29390, it looks
> like it generates avro code based on 1.9.2, which ends up being
> incompatible.
>
> I don't think we can continue to support every avro version with our
> current setup.
>
> John
>
> On Tue, Nov 14, 2023 at 4:20 PM Alexey Romanenko 
> wrote:
>
>> Thanks! Please, let me know if you need any help on this.
>>
>> —
>> Alexey
>>
>> On 14 Nov 2023, at 17:52, John Casey  wrote:
>>
>> The vulnerability said to upgrade to 1.11.3, so I think that would be my
>> starting point.
>>
>>
>> On Mon, Nov 13, 2023 at 12:23 PM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>>
>>>
>>> On 10 Nov 2023, at 19:23, John Casey  wrote:
>>>
>>> I guess I'm a bit confused as to why specifically generateTestAvroJava
>>> seems to use the wrong version. I see our version specific generated code,
>>> but this action appears to be inherited from the plugin, and is configured
>>> with whichever avro version is provided. Given that I tried to just change
>>> to 1.11.3, I'm confused as to why its generating invalid java files for the
>>> provided avro version.
>>>
>>> Unlike the classes generated out of the JavaExec you referenced, this
>>> appears to only generate one version of the files.
>>>
>>>
>>> It was supposed to generate files with a specific Avro version every
>>> time to run the same tests again this specific Avro version.
>>>
>>> It may be that we don't need this action, but it still seems to run, as
>>> we depend on it in the applyAvroNature() action.
>>>
>>>
>>> I started to think if we really still need this action.
>>>
>>> We could remove this entirely. The java exec only generates versions for
>>> pre-configured test versions anyways
>>>
>>>
>>> Right. The point is in how many places in Beam we need to generate these
>>> files and which version(s) of Avro to use?
>>>
>>> —
>>> Alexey
>>>
>>>
>>> On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko <
>>> aromanenko@gmail.com> wrote:
>>>
 Hi John,

 This old Avro version in Beam is a very long story. Briefly, since
 initially it was toughly integrated into Java SDK “core” module then it was
 not possible to upgrade an Avro version without breaking changes for users
 (because of some Avro incompatible changes, as you have noticed before).
 So, we decided to extract Avro-related classes from Beam “core” to a
 dedicated Avro extension [2] that supports and actually is tested with
 different Avro versions. More details on this work are here [1]

 Regarding auto-generated classes. Initially, we used a Gradle plugin
 for that but it’s limited with only one Avro version per instance of this
 plugin, so it was not possible to generate these classes with different
 Avro versions. So, we do this with a special Gradle task (“JavaExec") that
 executes “org.apache.avro.tool.Main” and generate Avro classes per every
 tested Avro version [3].

 We still keep an old Avro version 1.8.2. as a default dependency
 version but it will be overwritten if users have a more recent one as a
 project dependency in their classpath.

 I think we need to completely remove Avro Gradle plugin (use “JavaExec”
 task to generate Avro classes with a provided Avro version instead) and
 update the default Avro version to the more recent one since now it’s not
 part of Java “core”.

 Any thoughts?

 —
 Alexey


 [1] https://github.com/apache/beam/issues/24292
 [2]
 https://github.com/apache/beam/tree/master/sdks/java/extensions/avro
 [3]
 https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135



 On 10 Nov 2023, at 18:05, John Casey via dev 
 wrote:

 Hi All,

 There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying
 to upgrade to avro 1.11.3.

 Unfortunately, it seems that our auto-generated Avro test classes
 aren't being generated properly with this new version. I've updated our
 avro generation plugin as well, but for whatever reason, it seems that the
 generated AvroTest file is being generated with references to classes that
 did exist in 1.8.2, but no longer exist in 1.11.3.

 It seems like our autogeneration is being run with the wrong avro
 version, but I can't seem to find where that would be configured.

 Here is the PR with my changes so far:
 https://github.com/apache/beam/pull/29390

 Is anyone familiar with what might be misconfigured here?

 John

Re: Upgrading Avro dependencies

[John Casey via dev]

So, thats the thing. I've upgraded to 1.11.3, but the plugin still seems to
be generating Avro code that isn't compatible with 1.11.3.

Looking at the PR https://github.com/apache/beam/pull/29390, it looks like
it generates avro code based on 1.9.2, which ends up being incompatible.

I don't think we can continue to support every avro version with our
current setup.

John

On Tue, Nov 14, 2023 at 4:20 PM Alexey Romanenko 
wrote:

> Thanks! Please, let me know if you need any help on this.
>
> —
> Alexey
>
> On 14 Nov 2023, at 17:52, John Casey  wrote:
>
> The vulnerability said to upgrade to 1.11.3, so I think that would be my
> starting point.
>
>
> On Mon, Nov 13, 2023 at 12:23 PM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>>
>>
>> On 10 Nov 2023, at 19:23, John Casey  wrote:
>>
>> I guess I'm a bit confused as to why specifically generateTestAvroJava
>> seems to use the wrong version. I see our version specific generated code,
>> but this action appears to be inherited from the plugin, and is configured
>> with whichever avro version is provided. Given that I tried to just change
>> to 1.11.3, I'm confused as to why its generating invalid java files for the
>> provided avro version.
>>
>> Unlike the classes generated out of the JavaExec you referenced, this
>> appears to only generate one version of the files.
>>
>>
>> It was supposed to generate files with a specific Avro version every time
>> to run the same tests again this specific Avro version.
>>
>> It may be that we don't need this action, but it still seems to run, as
>> we depend on it in the applyAvroNature() action.
>>
>>
>> I started to think if we really still need this action.
>>
>> We could remove this entirely. The java exec only generates versions for
>> pre-configured test versions anyways
>>
>>
>> Right. The point is in how many places in Beam we need to generate these
>> files and which version(s) of Avro to use?
>>
>> —
>> Alexey
>>
>>
>> On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Hi John,
>>>
>>> This old Avro version in Beam is a very long story. Briefly, since
>>> initially it was toughly integrated into Java SDK “core” module then it was
>>> not possible to upgrade an Avro version without breaking changes for users
>>> (because of some Avro incompatible changes, as you have noticed before).
>>> So, we decided to extract Avro-related classes from Beam “core” to a
>>> dedicated Avro extension [2] that supports and actually is tested with
>>> different Avro versions. More details on this work are here [1]
>>>
>>> Regarding auto-generated classes. Initially, we used a Gradle plugin for
>>> that but it’s limited with only one Avro version per instance of this
>>> plugin, so it was not possible to generate these classes with different
>>> Avro versions. So, we do this with a special Gradle task (“JavaExec") that
>>> executes “org.apache.avro.tool.Main” and generate Avro classes per every
>>> tested Avro version [3].
>>>
>>> We still keep an old Avro version 1.8.2. as a default dependency version
>>> but it will be overwritten if users have a more recent one as a project
>>> dependency in their classpath.
>>>
>>> I think we need to completely remove Avro Gradle plugin (use “JavaExec”
>>> task to generate Avro classes with a provided Avro version instead) and
>>> update the default Avro version to the more recent one since now it’s not
>>> part of Java “core”.
>>>
>>> Any thoughts?
>>>
>>> —
>>> Alexey
>>>
>>>
>>> [1] https://github.com/apache/beam/issues/24292
>>> [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro
>>> [3]
>>> https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135
>>>
>>>
>>>
>>> On 10 Nov 2023, at 18:05, John Casey via dev 
>>> wrote:
>>>
>>> Hi All,
>>>
>>> There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying
>>> to upgrade to avro 1.11.3.
>>>
>>> Unfortunately, it seems that our auto-generated Avro test classes aren't
>>> being generated properly with this new version. I've updated our avro
>>> generation plugin as well, but for whatever reason, it seems that the
>>> generated AvroTest file is being generated with references to classes that
>>> did exist in 1.8.2, but no longer exist in 1.11.3.
>>>
>>> It seems like our autogeneration is being run with the wrong avro
>>> version, but I can't seem to find where that would be configured.
>>>
>>> Here is the PR with my changes so far:
>>> https://github.com/apache/beam/pull/29390
>>>
>>> Is anyone familiar with what might be misconfigured here?
>>>
>>> John
>>>
>>>
>>>
>>
>

Re: Upgrading Avro dependencies

[Alexey Romanenko]

Thanks! Please, let me know if you need any help on this.

—
Alexey

> On 14 Nov 2023, at 17:52, John Casey  wrote:
> 
> The vulnerability said to upgrade to 1.11.3, so I think that would be my 
> starting point.
> 
> 
> On Mon, Nov 13, 2023 at 12:23 PM Alexey Romanenko  > wrote:
>> 
>> 
>>> On 10 Nov 2023, at 19:23, John Casey >> > wrote:
>>> 
>>> I guess I'm a bit confused as to why specifically generateTestAvroJava 
>>> seems to use the wrong version. I see our version specific generated code, 
>>> but this action appears to be inherited from the plugin, and is configured 
>>> with whichever avro version is provided. Given that I tried to just change 
>>> to 1.11.3, I'm confused as to why its generating invalid java files for the 
>>> provided avro version.
>>> 
>>> Unlike the classes generated out of the JavaExec you referenced, this 
>>> appears to only generate one version of the files.
>> 
>> It was supposed to generate files with a specific Avro version every time to 
>> run the same tests again this specific Avro version. 
>> 
>>> It may be that we don't need this action, but it still seems to run, as we 
>>> depend on it in the applyAvroNature() action.
>> 
>> I started to think if we really still need this action.
>> 
>>> We could remove this entirely. The java exec only generates versions for 
>>> pre-configured test versions anyways
>> 
>> Right. The point is in how many places in Beam we need to generate these 
>> files and which version(s) of Avro to use?
>> 
>> —
>> Alexey
>> 
>>> 
>>> On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko >> > wrote:
 Hi John,
 
 This old Avro version in Beam is a very long story. Briefly, since 
 initially it was toughly integrated into Java SDK “core” module then it 
 was not possible to upgrade an Avro version without breaking changes for 
 users (because of some Avro incompatible changes, as you have noticed 
 before). So, we decided to extract Avro-related classes from Beam “core” 
 to a dedicated Avro extension [2] that supports and actually is tested 
 with different Avro versions. More details on this work are here [1]
 
 Regarding auto-generated classes. Initially, we used a Gradle plugin for 
 that but it’s limited with only one Avro version per instance of this 
 plugin, so it was not possible to generate these classes with different 
 Avro versions. So, we do this with a special Gradle task (“JavaExec") that 
 executes “org.apache.avro.tool.Main” and generate Avro classes per every 
 tested Avro version [3].
 
 We still keep an old Avro version 1.8.2. as a default dependency version 
 but it will be overwritten if users have a more recent one as a project 
 dependency in their classpath.
 
 I think we need to completely remove Avro Gradle plugin (use “JavaExec” 
 task to generate Avro classes with a provided Avro version instead) and 
 update the default Avro version to the more recent one since now it’s not 
 part of Java “core”.
 
 Any thoughts?
 
 —
 Alexey
  
 
 [1] https://github.com/apache/beam/issues/24292
 [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro
 [3] 
 https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135
 
 
 
> On 10 Nov 2023, at 18:05, John Casey via dev  > wrote:
> 
> Hi All,
> 
> There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying to 
> upgrade to avro 1.11.3.
> 
> Unfortunately, it seems that our auto-generated Avro test classes aren't 
> being generated properly with this new version. I've updated our avro 
> generation plugin as well, but for whatever reason, it seems that the 
> generated AvroTest file is being generated with references to classes 
> that did exist in 1.8.2, but no longer exist in 1.11.3.
> 
> It seems like our autogeneration is being run with the wrong avro 
> version, but I can't seem to find where that would be configured.
> 
> Here is the PR with my changes so far: 
> https://github.com/apache/beam/pull/29390
> 
> Is anyone familiar with what might be misconfigured here?
> 
> John
 
>> 

Re: Upgrading Avro dependencies

[John Casey via dev]

The vulnerability said to upgrade to 1.11.3, so I think that would be my
starting point.


On Mon, Nov 13, 2023 at 12:23 PM Alexey Romanenko 
wrote:

>
>
> On 10 Nov 2023, at 19:23, John Casey  wrote:
>
> I guess I'm a bit confused as to why specifically generateTestAvroJava
> seems to use the wrong version. I see our version specific generated code,
> but this action appears to be inherited from the plugin, and is configured
> with whichever avro version is provided. Given that I tried to just change
> to 1.11.3, I'm confused as to why its generating invalid java files for the
> provided avro version.
>
> Unlike the classes generated out of the JavaExec you referenced, this
> appears to only generate one version of the files.
>
>
> It was supposed to generate files with a specific Avro version every time
> to run the same tests again this specific Avro version.
>
> It may be that we don't need this action, but it still seems to run, as we
> depend on it in the applyAvroNature() action.
>
>
> I started to think if we really still need this action.
>
> We could remove this entirely. The java exec only generates versions for
> pre-configured test versions anyways
>
>
> Right. The point is in how many places in Beam we need to generate these
> files and which version(s) of Avro to use?
>
> —
> Alexey
>
>
> On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> Hi John,
>>
>> This old Avro version in Beam is a very long story. Briefly, since
>> initially it was toughly integrated into Java SDK “core” module then it was
>> not possible to upgrade an Avro version without breaking changes for users
>> (because of some Avro incompatible changes, as you have noticed before).
>> So, we decided to extract Avro-related classes from Beam “core” to a
>> dedicated Avro extension [2] that supports and actually is tested with
>> different Avro versions. More details on this work are here [1]
>>
>> Regarding auto-generated classes. Initially, we used a Gradle plugin for
>> that but it’s limited with only one Avro version per instance of this
>> plugin, so it was not possible to generate these classes with different
>> Avro versions. So, we do this with a special Gradle task (“JavaExec") that
>> executes “org.apache.avro.tool.Main” and generate Avro classes per every
>> tested Avro version [3].
>>
>> We still keep an old Avro version 1.8.2. as a default dependency version
>> but it will be overwritten if users have a more recent one as a project
>> dependency in their classpath.
>>
>> I think we need to completely remove Avro Gradle plugin (use “JavaExec”
>> task to generate Avro classes with a provided Avro version instead) and
>> update the default Avro version to the more recent one since now it’s not
>> part of Java “core”.
>>
>> Any thoughts?
>>
>> —
>> Alexey
>>
>>
>> [1] https://github.com/apache/beam/issues/24292
>> [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro
>> [3]
>> https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135
>>
>>
>>
>> On 10 Nov 2023, at 18:05, John Casey via dev  wrote:
>>
>> Hi All,
>>
>> There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying
>> to upgrade to avro 1.11.3.
>>
>> Unfortunately, it seems that our auto-generated Avro test classes aren't
>> being generated properly with this new version. I've updated our avro
>> generation plugin as well, but for whatever reason, it seems that the
>> generated AvroTest file is being generated with references to classes that
>> did exist in 1.8.2, but no longer exist in 1.11.3.
>>
>> It seems like our autogeneration is being run with the wrong avro
>> version, but I can't seem to find where that would be configured.
>>
>> Here is the PR with my changes so far:
>> https://github.com/apache/beam/pull/29390
>>
>> Is anyone familiar with what might be misconfigured here?
>>
>> John
>>
>>
>>
>

Re: Upgrading Avro dependencies

[Alexey Romanenko]

> On 10 Nov 2023, at 19:23, John Casey  wrote:
> 
> I guess I'm a bit confused as to why specifically generateTestAvroJava seems 
> to use the wrong version. I see our version specific generated code, but this 
> action appears to be inherited from the plugin, and is configured with 
> whichever avro version is provided. Given that I tried to just change to 
> 1.11.3, I'm confused as to why its generating invalid java files for the 
> provided avro version.
> 
> Unlike the classes generated out of the JavaExec you referenced, this appears 
> to only generate one version of the files.

It was supposed to generate files with a specific Avro version every time to 
run the same tests again this specific Avro version. 

> It may be that we don't need this action, but it still seems to run, as we 
> depend on it in the applyAvroNature() action.

I started to think if we really still need this action.

> We could remove this entirely. The java exec only generates versions for 
> pre-configured test versions anyways

Right. The point is in how many places in Beam we need to generate these files 
and which version(s) of Avro to use?

—
Alexey

> 
> On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko  > wrote:
>> Hi John,
>> 
>> This old Avro version in Beam is a very long story. Briefly, since initially 
>> it was toughly integrated into Java SDK “core” module then it was not 
>> possible to upgrade an Avro version without breaking changes for users 
>> (because of some Avro incompatible changes, as you have noticed before). So, 
>> we decided to extract Avro-related classes from Beam “core” to a dedicated 
>> Avro extension [2] that supports and actually is tested with different Avro 
>> versions. More details on this work are here [1]
>> 
>> Regarding auto-generated classes. Initially, we used a Gradle plugin for 
>> that but it’s limited with only one Avro version per instance of this 
>> plugin, so it was not possible to generate these classes with different Avro 
>> versions. So, we do this with a special Gradle task (“JavaExec") that 
>> executes “org.apache.avro.tool.Main” and generate Avro classes per every 
>> tested Avro version [3].
>> 
>> We still keep an old Avro version 1.8.2. as a default dependency version but 
>> it will be overwritten if users have a more recent one as a project 
>> dependency in their classpath.
>> 
>> I think we need to completely remove Avro Gradle plugin (use “JavaExec” task 
>> to generate Avro classes with a provided Avro version instead) and update 
>> the default Avro version to the more recent one since now it’s not part of 
>> Java “core”.
>> 
>> Any thoughts?
>> 
>> —
>> Alexey
>>  
>> 
>> [1] https://github.com/apache/beam/issues/24292
>> [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro
>> [3] 
>> https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135
>> 
>> 
>> 
>>> On 10 Nov 2023, at 18:05, John Casey via dev >> > wrote:
>>> 
>>> Hi All,
>>> 
>>> There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying to 
>>> upgrade to avro 1.11.3.
>>> 
>>> Unfortunately, it seems that our auto-generated Avro test classes aren't 
>>> being generated properly with this new version. I've updated our avro 
>>> generation plugin as well, but for whatever reason, it seems that the 
>>> generated AvroTest file is being generated with references to classes that 
>>> did exist in 1.8.2, but no longer exist in 1.11.3.
>>> 
>>> It seems like our autogeneration is being run with the wrong avro version, 
>>> but I can't seem to find where that would be configured.
>>> 
>>> Here is the PR with my changes so far: 
>>> https://github.com/apache/beam/pull/29390
>>> 
>>> Is anyone familiar with what might be misconfigured here?
>>> 
>>> John
>> 

Re: Upgrading Avro dependencies

[John Casey via dev]

I guess I'm a bit confused as to why specifically generateTestAvroJava
seems to use the wrong version. I see our version specific generated code,
but this action appears to be inherited from the plugin, and is configured
with whichever avro version is provided. Given that I tried to just change
to 1.11.3, I'm confused as to why its generating invalid java files for the
provided avro version.

Unlike the classes generated out of the JavaExec you referenced, this
appears to only generate one version of the files.
It may be that we don't need this action, but it still seems to run, as we
depend on it in the applyAvroNature() action.

We could remove this entirely. The java exec only generates versions for
pre-configured test versions anyways

On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko 
wrote:

> Hi John,
>
> This old Avro version in Beam is a very long story. Briefly, since
> initially it was toughly integrated into Java SDK “core” module then it was
> not possible to upgrade an Avro version without breaking changes for users
> (because of some Avro incompatible changes, as you have noticed before).
> So, we decided to extract Avro-related classes from Beam “core” to a
> dedicated Avro extension [2] that supports and actually is tested with
> different Avro versions. More details on this work are here [1]
>
> Regarding auto-generated classes. Initially, we used a Gradle plugin for
> that but it’s limited with only one Avro version per instance of this
> plugin, so it was not possible to generate these classes with different
> Avro versions. So, we do this with a special Gradle task (“JavaExec") that
> executes “org.apache.avro.tool.Main” and generate Avro classes per every
> tested Avro version [3].
>
> We still keep an old Avro version 1.8.2. as a default dependency version
> but it will be overwritten if users have a more recent one as a project
> dependency in their classpath.
>
> I think we need to completely remove Avro Gradle plugin (use “JavaExec”
> task to generate Avro classes with a provided Avro version instead) and
> update the default Avro version to the more recent one since now it’s not
> part of Java “core”.
>
> Any thoughts?
>
> —
> Alexey
>
>
> [1] https://github.com/apache/beam/issues/24292
> [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro
> [3]
> https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135
>
>
>
> On 10 Nov 2023, at 18:05, John Casey via dev  wrote:
>
> Hi All,
>
> There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying to
> upgrade to avro 1.11.3.
>
> Unfortunately, it seems that our auto-generated Avro test classes aren't
> being generated properly with this new version. I've updated our avro
> generation plugin as well, but for whatever reason, it seems that the
> generated AvroTest file is being generated with references to classes that
> did exist in 1.8.2, but no longer exist in 1.11.3.
>
> It seems like our autogeneration is being run with the wrong avro version,
> but I can't seem to find where that would be configured.
>
> Here is the PR with my changes so far:
> https://github.com/apache/beam/pull/29390
>
> Is anyone familiar with what might be misconfigured here?
>
> John
>
>
>

Re: Upgrading Avro dependencies

[Alexey Romanenko]

Hi John,

This old Avro version in Beam is a very long story. Briefly, since initially it 
was toughly integrated into Java SDK “core” module then it was not possible to 
upgrade an Avro version without breaking changes for users (because of some 
Avro incompatible changes, as you have noticed before). So, we decided to 
extract Avro-related classes from Beam “core” to a dedicated Avro extension [2] 
that supports and actually is tested with different Avro versions. More details 
on this work are here [1]

Regarding auto-generated classes. Initially, we used a Gradle plugin for that 
but it’s limited with only one Avro version per instance of this plugin, so it 
was not possible to generate these classes with different Avro versions. So, we 
do this with a special Gradle task (“JavaExec") that executes 
“org.apache.avro.tool.Main” and generate Avro classes per every tested Avro 
version [3].

We still keep an old Avro version 1.8.2. as a default dependency version but it 
will be overwritten if users have a more recent one as a project dependency in 
their classpath.

I think we need to completely remove Avro Gradle plugin (use “JavaExec” task to 
generate Avro classes with a provided Avro version instead) and update the 
default Avro version to the more recent one since now it’s not part of Java 
“core”.

Any thoughts?

—
Alexey
 

[1] https://github.com/apache/beam/issues/24292
[2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro
[3] 
https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135



> On 10 Nov 2023, at 18:05, John Casey via dev  wrote:
> 
> Hi All,
> 
> There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying to 
> upgrade to avro 1.11.3.
> 
> Unfortunately, it seems that our auto-generated Avro test classes aren't 
> being generated properly with this new version. I've updated our avro 
> generation plugin as well, but for whatever reason, it seems that the 
> generated AvroTest file is being generated with references to classes that 
> did exist in 1.8.2, but no longer exist in 1.11.3.
> 
> It seems like our autogeneration is being run with the wrong avro version, 
> but I can't seem to find where that would be configured.
> 
> Here is the PR with my changes so far: 
> https://github.com/apache/beam/pull/29390
> 
> Is anyone familiar with what might be misconfigured here?
> 
> John

Upgrading Avro dependencies

[John Casey via dev]

Hi All,

There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying to
upgrade to avro 1.11.3.

Unfortunately, it seems that our auto-generated Avro test classes aren't
being generated properly with this new version. I've updated our avro
generation plugin as well, but for whatever reason, it seems that the
generated AvroTest file is being generated with references to classes that
did exist in 1.8.2, but no longer exist in 1.11.3.

It seems like our autogeneration is being run with the wrong avro version,
but I can't seem to find where that would be configured.

Here is the PR with my changes so far:
https://github.com/apache/beam/pull/29390

Is anyone familiar with what might be misconfigured here?

John