Thanks Steve. Idea is not to shade all artifacts. Instead maintain one artifact ( hadoop-thirdparty) which have all such dependencies ( com.google.* may be), add this artifact as dependency in hadoop modules. Use shaded classes directly in the code of hadoop modules instead of shading at package phase.
Hbase, ozone and ratis already following this way. The artifact ( hadoop-thirdparty) with shaded dependencies can be maintained in a separate repo as suggested by stack on HADOOP-13363 or could be maintained as a separate module in Hadoop repo. If maintained in separate repo, need to build this only when there are changes related to shaded dependencies. -Vinay On Sun, 22 Sep 2019, 10:11 pm Steve Loughran, <ste...@cloudera.com> wrote: > > > On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B <vinayakum...@apache.org> > wrote: > >> Protobuf provides Wire compatibility between releases.. but not >> guarantees the source compatibility in generated sources. There will be a >> problem in compatibility if anyone uses generated protobuf message outside >> of Hadoop modules. Which ideally shouldn't be as generated sources are not >> public APIs. >> >> There should not be any compatibility problems between releases in >> terms >> of communication provided both uses same syntax (proto2) of proto message. >> This I have verified by communication between protobuf 2.5.0 client with >> protobuf 3.7.1 server. >> >> To avoid the downstream transitive dependency classpath problem, who >> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 classes >> and its usages in all hadoop modules.. and keep 2.5.0 jar back in hadoop >> classpath. >> >> Hope I have answered your question. >> >> -Vinay >> >> > While I support the move and CP isolation, this is going to (finally) > force us to make shaded versions of all artifacts which we publish with the > intent of them being loaded on the classpath of other applications >
