Related: very relevant concerns have been raised in the comments over on HADOOP-11656. See Christopher Tubbs's recent remarks.
On Thu, Apr 13, 2017 at 4:46 PM, Josh Elser <[email protected]> wrote: > > > Stack wrote: > >> On Wed, Apr 12, 2017 at 8:22 AM, Josh Elser<[email protected]> wrote: >> .... >> >> This makes me wonder if we could construct source jars just the same as >>> we're creating shaded jars. Google has lead me to [2][3], but I've never >>> tried either. The latter option seems to be acknowledging that the source >>> might not actually compile, but the package names would at least be >>> correct. >>> >>> I think this would be a good early experiment which, if it does work out, >>> removes the only acknowledged "hole" in the current plan. >>> >>> >>> Mighty Josh. >> >> As it happens, the 'createSources' you cite is currently in use by our >> hbase-protocol-shaded module for the specific purpose of keeping our >> IDE'rs >> happy; we relocate protobuf itself so we can have pb3.2 and pb2.5 on our >> CLASSPATH but IDEs don't complain because the src for the relocated pb is >> checked-in to our codebase. >> > > Aha! Wizards are ahead of me on the path already :) > > > Here's more if interested: >> >> All modules that make use of protos by convention require a 'pre-build' >> step. Generally, the pre-build is required if you add or mod .proto files. >> At pre-build we generate class files from the amended .protos and then >> check-in the product. See the README in each of our proto-carrying >> modules. >> >> In the hbase-protocol-shaded case, we abuse the pre-build notion so we can >> house an internal pb version, one that does not agree w/ hadoops nor w/ >> that used by our CPs describing Endpoint services. We do as follows (the >> order may not be exact in below): >> >> * Generate class files from protos >> * Shade the built artifact (with the createSource flag set). >> * Unjar the artifact which has generated proto and protobuf src classes >> in >> it >> * Apply a few patches to protobuf to support offheap work (stuff we need >> to push back up to pb) >> * Overlay our current protobuf src w/ the new version in the src tree >> -- a >> big no-no (smile). >> >> You then check it all in.... >> >> It has been suggested we undo these hokey pre-build steps and just >> generate >> classes from protobuf inline w/ the main build. A plugin we just figured >> makes this possible since it provides the platform-appropriate protoc >> (org.xolstice.maven.plugins). We could do the simplification currently for >> all but the hbase-protocol-shaded module because of the above jujitsu. One >> thought is that if we had a pre-build artifact as is being suggested in >> this thread, I could move the patched pb there and purge the pre-build >> step >> everywhere. >> > > The xolstice plugin has been working well for me elsewhere so far. +1 > > I think we're speaking the same language. My realization was that, > hopefully, we could automatically create that pre-built artifact via Maven > magic instead of having a human copy source files around. > > I think pushing this part forward with some code is the next logical step. > Seems to be consensus about taking our known internal dependencies and > performing this shade magic. > > Don't want to stomp on your worries, Nick. I think your worries are more > about the presentation to downstream and we're in agreement about isolating > our internal deps with the described approach? > > > Thanks, >> St.Ack >> >> >> >> We (The HBase PMC) will have to make releases of this new artifact and >>> vote >>> >>>> on them. I think it will be a relatively rare event. >>>> >>>> I'd be up for doing the first cut if folks are game. >>>> >>>> St.Ack >>>> >>>> >>>> 1. URL via Sean but for committers to view only: >>>> https://reporeq.apache.org/ >>>> >>>> >>>> [2] https://maven.apache.org/plugins/maven-shade-plugin/shade- >>> mojo.html#createSourcesJar >>> [3] https://maven.apache.org/plugins/maven-shade-plugin/shade- >>> mojo.html#shadeSourcesContent >>> >>> >>
