Thanks Kevin, this looks really promising. Updating the link here as I think the page may have moved: https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring <https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring>
Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 10, 2019, at 12:08 PM, Kevin Doran <kdo...@apache.org> wrote: > > Hi NiFi Dev Community, > > Jeff Storck, Bryan Bende, and I have been collaborating back and forth > on a proposal for how to restructure the NiFi source code into smaller > Maven projects and repositories based on the discussion that took > place awhile back on this thread. I'm reviving this older thread in > order to share that proposal with the community and generate farther > discussion about at solidifying a destination and a plan for how to > get there. > > Specifically, the proposal we've started working on has three parts: > > 1. Goals (more or less a summary of the earlier discussion that took > place on this thread) > 2. Proposed end state of the new Maven project and repository structure > 3. Proposed approach for how to get from where we are today to the > desired end state > > The proposal is on the Apache NiFi Wiki [1], so that we can all > collaborate on it or leave comments there. > > [1] > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring > > Thanks, > Kevin, Jeff, and Bryan > > On Thu, May 30, 2019 at 1:31 PM Kevin Doran <kdo...@apache.org> wrote: >> >> I am also in favor of splitting the nifi maven project up into smaller >> projects with independent release cycles in order to decouple >> development at well defined boundaries/interfaces and also to >> facilitate code reuse. >> >> In anticipation of eventually working towards a NiFi 2.0 that >> introduces bigger changes for developers and users, I've started work >> on a nifi-commons project in which I've extracted out some of the code >> that originally got ported from NiFi -> NiFi Registry, and now exists >> as similar code in both projects, into a standalone modular library. >> That premilinary work is here on my personal github account for now >> [1]. >> >> So far, it only contains some security code in a submodule, and is a >> WIP (more work coming when I have time), but the idea is nifi-commons >> could have several libraries/modules and would be released >> periodically to use across nifi and registry. If we are talking about >> spliting the nifi project into framework and extensions, then >> nifi-commons might be a good home for code that needs to be shared >> across those two sub projects as well, such as the nifi-api bits Joe >> mentioned. >> >> As part of this larger effort, I would be happy to help get a >> nifi-commons repository started in Apache where we can move shared >> code such as nifi-api to prepare for splitting nifi-framework and >> nifi-extensions. It also occurs to me that if nifi-framework and >> nifi-extensions are being released independently, nifi-assembly should >> probably just become a project that pulls in and assembles the latest >> releases of framework and extensions. >> >> Overall, I think this would be beneficial for most of the work going >> on in Apache NiFi, which would not have to cut across these different >> project and therefore would be easier to code, test, build, and >> release. However, the level of difficulty will increase for changes >> that will need to span multiple projects, though those are fewer in >> number, so overall I think it would be a net win for the dev >> community. >> >> [1] https://github.com/kevdoran/nifi-commons >> >> Thanks, >> Kevin >> >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto <alopre...@apache.org> wrote: >>> >>> I am a strong +1 on the separation and reducing the build time. With that >>> in mind, I think the process I brought up yesterday [1] of signing our >>> artifacts with GPG as part of the Maven build is paramount, because we >>> would now be consuming core code across multiple projects/repositories, so >>> there is even less guarantee the code is coming from “us”. >>> >>> [1] >>> https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E >>> >>> <https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E> >>> >>> Andy LoPresto >>> alopre...@apache.org >>> alopresto.apa...@gmail.com >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries <b...@jhu.edu> wrote: >>>> >>>> In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would >>>> also be a good time to look at more clearly defining the separation between >>>> the UI and the framework. Where nifi-api is the contract between the >>>> extensions and the framework, the NiFi Rest api is the contract between the >>>> UI and framework... These pieces could potentially be built / deployed / >>>> updated independently. >>>> >>>> On Thu, May 30, 2019 at 11:39 AM Jeff <jtsw...@gmail.com> wrote: >>>> >>>>> In the same category of challenges that Peter pointed out, it might be >>>>> difficult for Travis to build the "framework" and "extensions" projects if >>>>> there are changes in a PR that affect both projects. >>>>> >>>>> Is there a good way in Travis to have the workspace/maven repo shared >>>>> between projects in a single build? >>>>> >>>>> It's probably always in the direction of the extensions project needing >>>>> something new to be added to the framework project rather than the other >>>>> way around, but it'll be tricky to get that working right in Travis if >>>>> it's >>>>> not possible to set up the Travis build to know it needs to deploy the >>>>> framework project artifacts into a maven repo that the extension project >>>>> will use. >>>>> >>>>> One way might be to make sure that changes to the framework project must >>>>> be >>>>> in master before the extensions project can make use of them, but that >>>>> would require a "default master" build for the framework project which >>>>> builds master after each commit, and deploys the build artifacts to a >>>>> persistent maven repo that the extension project builds can access. It >>>>> also makes project-spanning change-sets take longer to review and get >>>>> fully >>>>> committed to master. >>>>> >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) <pwi...@micron.com> >>>>> wrote: >>>>> >>>>>> One more "not awesome" would be that core changes that affect extensions >>>>>> will be a little harder to test. If I make a core change that changes the >>>>>> signature of an interface/etc... I'll need to do some extra work to make >>>>>> sure I don't break extensions that use it. >>>>>> >>>>>> Still worth it, just one more thing to mention. >>>>>> >>>>>> -----Original Message----- >>>>>> From: Joe Witt <joew...@apache.org> >>>>>> Sent: Thursday, May 30, 2019 9:19 AM >>>>>> To: dev@nifi.apache.org >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and >>>>>> releases >>>>>> >>>>>> Team, >>>>>> >>>>>> We've discussed this a bit over the years in various forms but it again >>>>>> seems time to progress this topic and enough has changed I think to >>>>> warrant >>>>>> it. >>>>>> >>>>>> Tensions: >>>>>> 1) Our build times take too long. In travis-ci for instance it takes 40 >>>>>> minutes when it works. >>>>>> 2) The number of builds we do has increased. We do us/jp/fr builds on >>>>>> open and oracle JDKs. That is 6 builds. >>>>>> 3) We want to add Java 11 support such that one could build with 8 or 11 >>>>>> and the above still apply. The becomes 6 builds. >>>>>> 4) With the progress in NiFi registry we can now load artifacts there and >>>>>> could pull them into NiFi. And this integration will only get better. >>>>>> 5) The NiFi build is too huge and cannot grow any longer or else we >>>>> cannot >>>>>> upload convenience binaries. >>>>>> >>>>>> We cannot solve all the things just yet but we can make progress. I >>>>>> suggest we split apart the NiFi 'framework/application' in its own >>>>> release >>>>>> cycle and code repository from the 'nifi extensions' into its own >>>>>> repository and release cycle. The NiFi release would still pull in a >>>>>> specific set of extension bundles so to our end users at this time there >>>>> is >>>>>> no change. In the future we could also just stop including the extensions >>>>>> in nifi the application and they could be sourced at runtime as needed >>>>> from >>>>>> the registry (call that a NiFi 2.x thing). >>>>>> >>>>>> Why does this help? >>>>>> - Builds would only take as long as just extensions take or just core/app >>>>>> takes. This reduces time for each change cycle and reduces load on >>>>>> travis-ci which runs the same tests over and over and over for each pull >>>>>> request/push regardless of whether it was an extension or core. >>>>>> >>>>>> - It moves us toward the direction we're heading anyway whereby >>>>> extensions >>>>>> can have their own lifecycle from the framework/app itself. >>>>>> >>>>>> How is this not awesome: >>>>>> - Doesn't yet solve for the large builds problem. I think we'll get >>>>> there >>>>>> with a NiFi 2.x release which fully leverages nifi-registry for retrieval >>>>>> of all extensions. >>>>>> - Adds another 'thing we need to do a release cycle for'. This is >>>>>> generally unpleasant but it is paid for once a release cycle and it does >>>>>> allow us to release independently for new cool extensions/fixes apart >>>>> from >>>>>> the framework itself. >>>>>> >>>>>> Would be great to hear others thoughts if they too feel it is time to >>>>> make >>>>>> this happen. >>>>>> >>>>>> Thanks >>>>>> Joe >>>>>> >>>>> >>>
