Mike Yep - that will need to get addressed/improved as well. First step is get the root classpath cleaned up again (NIFI-2971) and the next step is to understand and address the excessive declaration of dependencies which leads to duplication on such a scale.
Thanks Joe On Tue, Nov 1, 2016 at 12:15 PM, Michael Moser <moser...@gmail.com> wrote: > I would like to point out that some very common dependencies have been > placed into the nifi-standard-services-api-nar. This nar is a > Nar-Dependency-Id of several other nars. Many of these other nars also > include the same dependencies, which would not be used by the nar class > loader. > > In particular, nifi-standard-services-api-nar contains nifi-utils, > nifi-processor-utils, and nifi-security-utils. It also contains the > bcprov-jdk15on and bcpkix-jdk15on jars. > > Then looking at nifi-standard-nar, nifi-aws-nar, nifi-kafka-0-10-nar and > more, they also contain these same jars in their bundled-dependencies. > Their Nar-Dependency-Id is nifi-standard-services-api-nar. > > -- Mike > > > On Tue, Nov 1, 2016 at 1:50 PM, Aldrin Piri <aldrinp...@gmail.com> wrote: > >> I think in light of the fact that the bcprov item was an accidental >> inclusion and not explicitly meant to be handled as such in the framework >> reorganizing seems like the best option as well. The original intent of >> the PR was to minimize that footprint where we were double dipping given >> the inclusion of nifi-processor-utils. However, if we manage to shuffle >> items around a bit, the net size reduction could ultimately be the same. >> Most importantly, from a maintenance perspective, bundles will only include >> the library in the NARs that need it which allows that dependency graph to >> be significantly more comprehensible than the scope tweaking originally >> presented. >> >> Will this also work to adjust the dependencies of nifi-processor-utils (and >> potentially nifi-security-utils)? Currently, the archetype also provides >> nifi-processor-utils, which could lead to similar conflicts for folks that >> were looking to make use of different versions of JARs within a given NAR. >> >> I think the listing Joe provided in the ticket is great and we should be >> mindful of any deltas while performing each release. For the reasons >> highlighted in this chain and on that issue, one jar inclusion could be >> quite detrimental and cause some difficult issues to those that have >> created extensions. >> >> On Mon, Oct 31, 2016 at 9:39 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >> > So...I might be a bit of a flip flopper... >> > >> > Had some off-list discussions with a couple people and they really >> > disliked this idea. They, rightfully, pointed out how this >> > establishes a problematic precedent, how it is really only being done >> > to save space, and that while perhaps not able to pair down to 200MB >> > we can likely do more analysis of the dependency graph to improve the >> > situation. The real issue that kicked this off was the finding that >> > bcprov (and commons-lang3) had made their way into the root of our >> > classloader (which impacts all classloaders) and this was introduced >> > with the important NIFI-1831 work. So I've created [1] to address >> > that and return the application classloader to the minimal set of jars >> > necessary which should basically be the runtime application launcher, >> > the logging libraries, and the nifi-api. >> > >> > The ultimate solution to addressing the distribution size concerns >> > will be to address it via the extension registry. >> > >> > [1] https://issues.apache.org/jira/browse/NIFI-2971 >> > >> > Thanks >> > Joe >> > >> > On Mon, Oct 31, 2016 at 8:51 PM, Andre <andre-li...@fucs.org> wrote: >> > > +1 as well. >> > > >> > > On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt <joe.w...@gmail.com> wrote: >> > > >> > >> Team, >> > >> >> > >> In the 1.x line bcprov made its way into the root of the classpath/lib >> > >> folder to support the encrypted sensitive properties features. This >> > >> can be reworked to isolate it a bit more if we need to. >> > >> >> > >> However, in [1] it has been observed that we have bcprov dependencies >> > >> in about 53 places with each instance taking about 4MB of space for a >> > >> total hit of about 200MB of bcprov. >> > >> >> > >> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the >> > >> core provided list of things right along side the >> > >> logback/slf4j/logging interfaces. This means all things will have >> > >> access to these going forward and it means it impacts our version >> > >> compatibility. Given that it would be a standard/provided thing if we >> > >> want to upgrade versions or swap it out with something else we could >> > >> break extensions that then chose to depend on it. We are not in >> > >> control of the bcprov api just like we're not in control of the >> > >> logging APIs. If there was some important security related fix we >> > >> needed from bcprov but changing also pulled in api changes for them it >> > >> could break our extensions. >> > >> >> > >> Even with all this said, given the nature, importance, and size >> > >> benefit, I am in favor of NIFI-2954. But, would like to highlight >> > >> this in case others have perspective they'd like to share. >> > >> >> > >> [1] https://issues.apache.org/jira/browse/NIFI-2954 >> > >> >> > >> Thanks >> > >> Joe >> > >> >> > >>
