In the same category of challenges that Peter pointed out, it might be difficult for Travis to build the "framework" and "extensions" projects if there are changes in a PR that affect both projects.
Is there a good way in Travis to have the workspace/maven repo shared between projects in a single build? It's probably always in the direction of the extensions project needing something new to be added to the framework project rather than the other way around, but it'll be tricky to get that working right in Travis if it's not possible to set up the Travis build to know it needs to deploy the framework project artifacts into a maven repo that the extension project will use. One way might be to make sure that changes to the framework project must be in master before the extensions project can make use of them, but that would require a "default master" build for the framework project which builds master after each commit, and deploys the build artifacts to a persistent maven repo that the extension project builds can access. It also makes project-spanning change-sets take longer to review and get fully committed to master. On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) <pwi...@micron.com> wrote: > One more "not awesome" would be that core changes that affect extensions > will be a little harder to test. If I make a core change that changes the > signature of an interface/etc... I'll need to do some extra work to make > sure I don't break extensions that use it. > > Still worth it, just one more thing to mention. > > -----Original Message----- > From: Joe Witt <joew...@apache.org> > Sent: Thursday, May 30, 2019 9:19 AM > To: dev@nifi.apache.org > Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and > releases > > Team, > > We've discussed this a bit over the years in various forms but it again > seems time to progress this topic and enough has changed I think to warrant > it. > > Tensions: > 1) Our build times take too long. In travis-ci for instance it takes 40 > minutes when it works. > 2) The number of builds we do has increased. We do us/jp/fr builds on > open and oracle JDKs. That is 6 builds. > 3) We want to add Java 11 support such that one could build with 8 or 11 > and the above still apply. The becomes 6 builds. > 4) With the progress in NiFi registry we can now load artifacts there and > could pull them into NiFi. And this integration will only get better. > 5) The NiFi build is too huge and cannot grow any longer or else we cannot > upload convenience binaries. > > We cannot solve all the things just yet but we can make progress. I > suggest we split apart the NiFi 'framework/application' in its own release > cycle and code repository from the 'nifi extensions' into its own > repository and release cycle. The NiFi release would still pull in a > specific set of extension bundles so to our end users at this time there is > no change. In the future we could also just stop including the extensions > in nifi the application and they could be sourced at runtime as needed from > the registry (call that a NiFi 2.x thing). > > Why does this help? > - Builds would only take as long as just extensions take or just core/app > takes. This reduces time for each change cycle and reduces load on > travis-ci which runs the same tests over and over and over for each pull > request/push regardless of whether it was an extension or core. > > - It moves us toward the direction we're heading anyway whereby extensions > can have their own lifecycle from the framework/app itself. > > How is this not awesome: > - Doesn't yet solve for the large builds problem. I think we'll get there > with a NiFi 2.x release which fully leverages nifi-registry for retrieval > of all extensions. > - Adds another 'thing we need to do a release cycle for'. This is > generally unpleasant but it is paid for once a release cycle and it does > allow us to release independently for new cool extensions/fixes apart from > the framework itself. > > Would be great to hear others thoughts if they too feel it is time to make > this happen. > > Thanks > Joe >
