Looks like there's some strategy to get to the right solution here and that it may likely involve breaking compatibility.
One option for myself would be to strip the Beam JAR of AvroCoder and combine with the old AvroCoder from Beam 2.16 -- this would allow me to upgrade Beam but of course is rather hacky. On second thought, was the breaking change from Beam 2.16->2.17 really necessary? If not, could AvroCoder be restored to a 1.9.x "compatible" implementation and kept this way for the Beam 2.1x version lineage? This seems like a somewhat fair ask given the way that I'm suddenly blocked --- however I do realize this is somewhat of a technicality; ie, Beam 2.16-'s compatibility with my usage of Avro 1.9.x was incidental. But, still, if the changes to AvroCoder weren't necessary, restoring back would unblock me and anyone else using Avro 1.9.x (surely I'm not the only one!?) On Thu, Jan 16, 2020 at 12:22 PM Elliotte Rusty Harold <elh...@ibiblio.org> wrote: > Avro does not follow semver. They update the major version when the > serialization format changes and the minor version when the API > changes in a backwards incompatible way. See > https://issues.apache.org/jira/browse/AVRO-2687 > > On Thu, Jan 16, 2020 at 12:50 PM Luke Cwik <lc...@google.com> wrote: > > > > Does avro not follow semantic versioning and upgrading to 1.9 should > have been backwards compatible or does our usage reach into the internals > of avro? > > > > On Thu, Jan 16, 2020 at 6:16 AM Ismaël Mejía <ieme...@gmail.com> wrote: > >> > >> I forgot to explain why the most obvious path (just upgrade Avro to > version > >> 1.9.x) is not a valid long term solution. Other systems Beam runs on > top of > >> (e.g. Spark!) also leak Avro into their core so in the moment Spark > moves up > >> to Avro 1.9.x Spark runner users will be in a really fragile position > where > >> things will work until they don't (similar to Aaron's case) so a > stronger reason > >> to getAvro out of Beam core. > >> > >> > >> On Thu, Jan 16, 2020 at 1:59 PM Elliotte Rusty Harold < > elh...@ibiblio.org> wrote: > >>> > >>> Shading should be a last resort: > >>> > >>> https://jlbp.dev/JLBP-18.html > >>> > >>> It tends to cause more problems than it solves. At best it's a stopgap > >>> measure when you don't have the resources to fix the real problem. In > >>> this case it sounds like the real issue is that AVRO is not stable. > >>> There are at least three other solutions in a case like this: > >>> > >>> 1. Fix Avro at the root. > >>> 2. Fork Avro and then fix it. > >>> 3. Stop depending on Avro. > >>> > >>> None of these are trivial which is why shading gets considered. > >>> However shading doesn't fix the underlying problems, and ultimately > >>> makes a product as unreliable as its least reliable dependency. :-( > >>> > >>> On Thu, Jan 16, 2020 at 2:01 AM jincheng sun <sunjincheng...@gmail.com> > wrote: > >>> > > >>> > I found that there are several dependencies shaded and planned to > made as vendored artifacts in [1]. I'm not sure why Avro is not shaded > before. From my point of view, it's a good idea to shade Avro and make it a > vendored artifact if there are no special reasons blocking us to do that. > Regarding to how to create a vendored artifact, you can refer to [2] for > more details. > >>> > > >>> > Best, > >>> > Jincheng > >>> > > >>> > [1] https://issues.apache.org/jira/browse/BEAM-5819 > >>> > [2] https://github.com/apache/beam/blob/master/vendor/README.md > >>> > > >>> > > >>> > Tomo Suzuki <suzt...@google.com> 于2020年1月16日周四 下午1:18写道： > >>> >> > >>> >> I've been upgrading dependencies around gRPC. This Avro-problem is > >>> >> interesting to me. > >>> >> I'll study BEAM-8388 more tomorrow. > >>> >> > >>> >> On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik <lc...@google.com> > wrote: > >>> >> > > >>> >> > +Tomo Suzuki +jincheng sun > >>> >> > There have been a few contributors upgrading the dependencies and > validating things not breaking by running the majority of the post commit > integration tests and also using the linkage checker to show that we aren't > worse off with respect to our dependency tree. Reaching out to them to help > your is your best bet of getting these upgrades through. > >>> >> > > >>> >> > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon <atdi...@gmail.com> > wrote: > >>> >> >> > >>> >> >> I meant to mention that we must use Avro 1.9.x as we rely on > some schema resolution fixes not present in 1.8.x - so am indeed blocked. > >>> >> >> > >>> >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon <atdi...@gmail.com> > wrote: > >>> >> >>> > >>> >> >>> It looks like Avro version dependency from Beam has come up in > the past [1, 2]. > >>> >> >>> > >>> >> >>> I'm currently on Beam 2.16.0, which has been compatible with my > usage of Avro 1.9.x. > >>> >> >>> > >>> >> >>> But upgrading to Beam 2.17.0 is not possible for us now that > 2.17.0 has some dependencies on Avro classes only available in 1.8.x. > >>> >> >>> > >>> >> >>> Wondering if anyone else is similar blocked and what it would > take to prioritize Beam upgrading to 1.9.x or better using a shaded version > so that clients can use their own Avro version for their own coding > purposes. (Eg, I parse Avro messages from a KafkaIO source and need 1.9.x > for this but am perfectly happy if Beam's Avro coding facilities used a > shaded other version.) > >>> >> >>> > >>> >> >>> I've made a comment on BEAM-8388 [1] to this effect. But > polling community for discussion. > >>> >> >>> > >>> >> >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 > >>> >> >>> [2] https://github.com/apache/beam/pull/9779 > >>> >> >>> > >>> >> > >>> >> > >>> >> -- > >>> >> Regards, > >>> >> Tomo > >>> > >>> > >>> > >>> -- > >>> Elliotte Rusty Harold > >>> elh...@ibiblio.org > > > > -- > Elliotte Rusty Harold > elh...@ibiblio.org >
