Hi Vinay, Thanks for the clarification.
Do you have a timeline about all you described works w.r.t. the compatibility will be completed? I'm asking this is because we need to release 3.3.0 earlier if possible since there're 1k+ of patches in 3.3.0 already, we should get it out earlier. If the PB work will take more time, do you think if we should create a branch for 3.3, revert PB changes from branch-3.3, and keep on working on PB for the next minor release? (or major release if we do see some compatibility issue in the future). Just my $0.02 Thanks, Wangda On Mon, Sep 23, 2019 at 5:43 AM Steve Loughran <ste...@cloudera.com.invalid> wrote: > aah, that makes sense > > On Sun, Sep 22, 2019 at 6:11 PM Vinayakumar B <vinayakum...@apache.org> > wrote: > > > Thanks Steve. > > > > Idea is not to shade all artifacts. > > Instead maintain one artifact ( hadoop-thirdparty) which have all such > > dependencies ( com.google.* may be), add this artifact as dependency in > > hadoop modules. Use shaded classes directly in the code of hadoop modules > > instead of shading at package phase. > > > > Hbase, ozone and ratis already following this way. The artifact ( > > hadoop-thirdparty) with shaded dependencies can be maintained in a > separate > > repo as suggested by stack on HADOOP-13363 or could be maintained as a > > separate module in Hadoop repo. If maintained in separate repo, need to > > build this only when there are changes related to shaded dependencies. > > > > > > -Vinay > > > > On Sun, 22 Sep 2019, 10:11 pm Steve Loughran, <ste...@cloudera.com> > wrote: > > > > > > > > > > > On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B <vinayakum...@apache.org > > > > > wrote: > > > > > >> Protobuf provides Wire compatibility between releases.. but not > > >> guarantees the source compatibility in generated sources. There will > be > > a > > >> problem in compatibility if anyone uses generated protobuf message > > outside > > >> of Hadoop modules. Which ideally shouldn't be as generated sources are > > not > > >> public APIs. > > >> > > >> There should not be any compatibility problems between releases in > > >> terms > > >> of communication provided both uses same syntax (proto2) of proto > > message. > > >> This I have verified by communication between protobuf 2.5.0 client > with > > >> protobuf 3.7.1 server. > > >> > > >> To avoid the downstream transitive dependency classpath problem, > who > > >> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 > > classes > > >> and its usages in all hadoop modules.. and keep 2.5.0 jar back in > hadoop > > >> classpath. > > >> > > >> Hope I have answered your question. > > >> > > >> -Vinay > > >> > > >> > > > While I support the move and CP isolation, this is going to (finally) > > > force us to make shaded versions of all artifacts which we publish with > > the > > > intent of them being loaded on the classpath of other applications > > > > > >
