Could we potentially look at removing the embedded external dependences for the nars and have them resolve on first start up using an embed manger like what spark jobs and some of the OSGI frameworks do?
Edward On Thu, 22 Aug 2019, 04:48 Bryan Bende, <bbe...@gmail.com> wrote: > I would vote for doing the reorganization after this release because on its > own it’s doesn’t really solver this specific problem. We would be moving > all the NARs to another git repo, but still need a way to distribute them. > Some of the options would be... > > A) Pick and choose only some of them to be included in the standard > distribution (same thing we are doing now and doesn’t require > reorganization). > > B) Create a couple of different NAR assemblies that each on their own don’t > exceed the size limit, and could easily be dropped into the base > distribution (In previous discussions no one could agree on how to do > this). > > C) Release each NAR independently to a central extension registry to be > easily installed to anyone’s NiFi (ultimate goal but doesn’t exist yet). > > On Wed, Aug 21, 2019 at 10:13 PM Jeff <jtsw...@gmail.com> wrote: > > > Realistically we'll have to do the reorganization and removal of optional > > NARs. Regardless of the reorganization, the convenience binary has grown > > too large. Given that, it probably makes the most sense to pull out an > > initial set of optional NARs from the convenience binary this release, > and > > implement the reorganization in the following release. > > > > On Wed, Aug 21, 2019 at 10:06 PM Mike Thomsen <mikerthom...@gmail.com> > > wrote: > > > > > My impression was that the reorganization initiative has enough steps > > that > > > it might actually be wise for us to at least take on a chunk of them > now > > so > > > we don't end up with a release that suddenly feels to NiFi users the > way > > > Java 9 felt to Java 7 and 8 users. > > > > > > On Wed, Aug 21, 2019 at 9:59 PM Jeff <jtsw...@gmail.com> wrote: > > > > > > > How motivated could we be to do the reorganization of the NiFi > > repository > > > > before the 1.10.0 release? It sounds like we have a few paths to get > > the > > > > resulting convenience binary down below the size limit. If we don't > do > > > the > > > > reorganization for this upcoming release, we should make it a top > > > priority > > > > for the following release. > > > > > > > > On Wed, Aug 21, 2019 at 6:22 PM Mike Thomsen <mikerthom...@gmail.com > > > > > > wrote: > > > > > > > > > Another factor on why that would be a good idea: I might soon have > to > > > > pivot > > > > > and do the R&D on adding dgraph support to graph bundle. So it's > not > > > > > altogether unlikely that it might need to be refactored to make > room > > > for > > > > > other graph tech. > > > > > > > > > > On Wed, Aug 21, 2019 at 6:20 PM Mike Thomsen < > mikerthom...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > Go ahead and remove the whole graph bundle from the assembly. I > > would > > > > > > recommend cutting a release of it separately and putting it up on > > > > > GitHub's > > > > > > releases listing if that's a possibility for us/INFRA w/ GitHub. > > Most > > > > of > > > > > > our potential graph users are savvy enough that if add a few > > steps, I > > > > > don't > > > > > > see it causing any grief on them getting it stood up and giving > us > > > > > feedback. > > > > > > > > > > > > Might be a good idea also to add a "full-build" profile to the > > > assembly > > > > > so > > > > > > that we can throw the whole kitchen sink into an unofficial build > > if > > > we > > > > > > build it ourselves for someone else. > > > > > > > > > > > > On Wed, Aug 21, 2019 at 3:09 PM Joe Witt <joe.w...@gmail.com> > > wrote: > > > > > > > > > > > >> Bryan > > > > > >> > > > > > >> I agree with all of that. What does that get us to? > > > > > >> > > > > > >> Thanks > > > > > >> > > > > > >> On Wed, Aug 21, 2019 at 3:03 PM Bryan Bende <bbe...@gmail.com> > > > wrote: > > > > > >> > > > > > >> > I would vote to make nifi-flume-nar optional, and it looks > like > > > > > >> > nifi-other-graph-services-nar might be new since last release, > > so > > > > > >> > since that is in the top 10 and not released yet, it might > also > > > be a > > > > > >> > good candidate (not downplaying the usefulness of anything in > > that > > > > > >> > NAR). > > > > > >> > > > > > > >> > I would also think we could consider the nifi-kafka-0-8-nar > > since > > > > > >> > Kafka 0.8 is quite old at this point, and we already have > other > > > > Kafka > > > > > >> > NARs for 0.9, 0.10, 0.11, 1.0, and 2.0. Might even consider > > > > dropping a > > > > > >> > few more versions from default assembly. > > > > > >> > > > > > > >> > On Wed, Aug 21, 2019 at 2:45 PM Aldrin Piri < > ald...@apache.org> > > > > > wrote: > > > > > >> > > > > > > > >> > > Hi folks, > > > > > >> > > > > > > > >> > > Doing a recent PR review and build, it seems that master has > > > > amassed > > > > > >> some > > > > > >> > > additional size since our 1.9.2 release approaching 200MB. > > > > > >> > > > > > > > >> > > Unfortunately, this is problematic and needs to be addressed > > in > > > > > >> advance > > > > > >> > of > > > > > >> > > our 1.10 release. INFRA has been more than helpful making > one > > > off > > > > > >> > > exceptions [1][2] for the larger assembly to get published > to > > > the > > > > > ASF > > > > > >> > > repository and its associated mirrors, but another release > > that > > > is > > > > > >> even > > > > > >> > > larger is not something we can allow. In a Linux > environment, > > > the > > > > > >> master > > > > > >> > > build reports in at 1575671276 which puts us over the hard > > limit > > > > > >> > > highlighted in [2]. > > > > > >> > > > > > > > >> > > We had a prior community discussion [3] about splitting the > > > > > framework > > > > > >> and > > > > > >> > > extension repos and I am hoping to revive that discussion, > in > > > > part. > > > > > >> We > > > > > >> > > certainly know what our longer term goals and ambitions are > > but > > > > > need a > > > > > >> > fix > > > > > >> > > in the interim. In the current state, we will not be able > to > > > make > > > > > our > > > > > >> > > convenience binaries available at the conclusion of the > > release > > > > > >> process. > > > > > >> > > > > > > > >> > > At minimum we should evaluate which bundles are eligible to > > get > > > > > >> treated > > > > > >> > as > > > > > >> > > optional dependencies and only enabled via profile, much > like > > > the > > > > > work > > > > > >> > that > > > > > >> > > has occurred surrounding some of our other, hefty NARs. [4] > A > > > > > listing > > > > > >> of > > > > > >> > > the top 50 largest NARs, excluding framework and standard, > is > > > > > >> available > > > > > >> > in > > > > > >> > > a gist [5]. The nifi-media-nar looks to be a good initial > > > > candidate > > > > > >> for > > > > > >> > > exclusion. > > > > > >> > > > > > > > >> > > Thanks for your consideration! > > > > > >> > > > > > > > >> > > --aldrin > > > > > >> > > > > > > > >> > > [1] https://issues.apache.org/jira/browse/INFRA-11252 > > > > > >> > > [2] https://issues.apache.org/jira/browse/INFRA-15816 > > > > > >> > > [3] > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/939a7630a2e32594cd10444e48b7a1321fd9ce51834d911a8c04b6a9@ > > > > > >> > <dev.nifi.apache.org> > > > > > >> > > [4] > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://github.com/apache/nifi/blob/master/nifi-assembly/pom.xml#L807-L875 > > > > > >> > > [5] > > > > https://gist.github.com/apiri/4d9a02f9f6b46867b601956df83b6d8c > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > Sent from Gmail Mobile >
