I forgot to explain why the most obvious path (just upgrade Avro to version 1.9.x) is not a valid long term solution. Other systems Beam runs on top of (e.g. Spark!) also leak Avro into their core so in the moment Spark moves up to Avro 1.9.x Spark runner users will be in a really fragile position where things will work until they don't (similar to Aaron's case) so a stronger reason to getAvro out of Beam core.
On Thu, Jan 16, 2020 at 1:59 PM Elliotte Rusty Harold <elh...@ibiblio.org> wrote: > Shading should be a last resort: > > https://jlbp.dev/JLBP-18.html > > It tends to cause more problems than it solves. At best it's a stopgap > measure when you don't have the resources to fix the real problem. In > this case it sounds like the real issue is that AVRO is not stable. > There are at least three other solutions in a case like this: > > 1. Fix Avro at the root. > 2. Fork Avro and then fix it. > 3. Stop depending on Avro. > > None of these are trivial which is why shading gets considered. > However shading doesn't fix the underlying problems, and ultimately > makes a product as unreliable as its least reliable dependency. :-( > > On Thu, Jan 16, 2020 at 2:01 AM jincheng sun <sunjincheng...@gmail.com> > wrote: > > > > I found that there are several dependencies shaded and planned to made > as vendored artifacts in [1]. I'm not sure why Avro is not shaded before. > From my point of view, it's a good idea to shade Avro and make it a > vendored artifact if there are no special reasons blocking us to do that. > Regarding to how to create a vendored artifact, you can refer to [2] for > more details. > > > > Best, > > Jincheng > > > > [1] https://issues.apache.org/jira/browse/BEAM-5819 > > [2] https://github.com/apache/beam/blob/master/vendor/README.md > > > > > > Tomo Suzuki <suzt...@google.com> 于2020年1月16日周四 下午1:18写道： > >> > >> I've been upgrading dependencies around gRPC. This Avro-problem is > >> interesting to me. > >> I'll study BEAM-8388 more tomorrow. > >> > >> On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik <lc...@google.com> wrote: > >> > > >> > +Tomo Suzuki +jincheng sun > >> > There have been a few contributors upgrading the dependencies and > validating things not breaking by running the majority of the post commit > integration tests and also using the linkage checker to show that we aren't > worse off with respect to our dependency tree. Reaching out to them to help > your is your best bet of getting these upgrades through. > >> > > >> > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon <atdi...@gmail.com> > wrote: > >> >> > >> >> I meant to mention that we must use Avro 1.9.x as we rely on some > schema resolution fixes not present in 1.8.x - so am indeed blocked. > >> >> > >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon <atdi...@gmail.com> > wrote: > >> >>> > >> >>> It looks like Avro version dependency from Beam has come up in the > past [1, 2]. > >> >>> > >> >>> I'm currently on Beam 2.16.0, which has been compatible with my > usage of Avro 1.9.x. > >> >>> > >> >>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 > has some dependencies on Avro classes only available in 1.8.x. > >> >>> > >> >>> Wondering if anyone else is similar blocked and what it would take > to prioritize Beam upgrading to 1.9.x or better using a shaded version so > that clients can use their own Avro version for their own coding purposes. > (Eg, I parse Avro messages from a KafkaIO source and need 1.9.x for this > but am perfectly happy if Beam's Avro coding facilities used a shaded other > version.) > >> >>> > >> >>> I've made a comment on BEAM-8388 [1] to this effect. But polling > community for discussion. > >> >>> > >> >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 > >> >>> [2] https://github.com/apache/beam/pull/9779 > >> >>> > >> > >> > >> -- > >> Regards, > >> Tomo > > > > -- > Elliotte Rusty Harold > elh...@ibiblio.org >
