RE: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
One more "not awesome" would be that core changes that affect extensions will be a little harder to test. If I make a core change that changes the signature of an interface/etc... I'll need to do some extra work to make sure I don't break extensions that use it. Still worth it, just one more thing to mention. -Original Message- From: Joe Witt Sent: Thursday, May 30, 2019 9:19 AM To: dev@nifi.apache.org Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and releases Team, We've discussed this a bit over the years in various forms but it again seems time to progress this topic and enough has changed I think to warrant it. Tensions: 1) Our build times take too long. In travis-ci for instance it takes 40 minutes when it works. 2) The number of builds we do has increased. We do us/jp/fr builds on open and oracle JDKs. That is 6 builds. 3) We want to add Java 11 support such that one could build with 8 or 11 and the above still apply. The becomes 6 builds. 4) With the progress in NiFi registry we can now load artifacts there and could pull them into NiFi. And this integration will only get better. 5) The NiFi build is too huge and cannot grow any longer or else we cannot upload convenience binaries. We cannot solve all the things just yet but we can make progress. I suggest we split apart the NiFi 'framework/application' in its own release cycle and code repository from the 'nifi extensions' into its own repository and release cycle. The NiFi release would still pull in a specific set of extension bundles so to our end users at this time there is no change. In the future we could also just stop including the extensions in nifi the application and they could be sourced at runtime as needed from the registry (call that a NiFi 2.x thing). Why does this help? - Builds would only take as long as just extensions take or just core/app takes. This reduces time for each change cycle and reduces load on travis-ci which runs the same tests over and over and over for each pull request/push regardless of whether it was an extension or core. - It moves us toward the direction we're heading anyway whereby extensions can have their own lifecycle from the framework/app itself. How is this not awesome: - Doesn't yet solve for the large builds problem. I think we'll get there with a NiFi 2.x release which fully leverages nifi-registry for retrieval of all extensions. - Adds another 'thing we need to do a release cycle for'. This is generally unpleasant but it is paid for once a release cycle and it does allow us to release independently for new cool extensions/fixes apart from the framework itself. Would be great to hear others thoughts if they too feel it is time to make this happen. Thanks Joe
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
yeah that is a good point. We 'could' also split out the 'nifi-api' which is the only 'contract' between extensions and core. Not sure the juice is worth the squeeze but then again we rarely change that thing. On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) wrote: > One more "not awesome" would be that core changes that affect extensions > will be a little harder to test. If I make a core change that changes the > signature of an interface/etc... I'll need to do some extra work to make > sure I don't break extensions that use it. > > Still worth it, just one more thing to mention. > > -Original Message- > From: Joe Witt > Sent: Thursday, May 30, 2019 9:19 AM > To: dev@nifi.apache.org > Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and > releases > > Team, > > We've discussed this a bit over the years in various forms but it again > seems time to progress this topic and enough has changed I think to warrant > it. > > Tensions: > 1) Our build times take too long. In travis-ci for instance it takes 40 > minutes when it works. > 2) The number of builds we do has increased. We do us/jp/fr builds on > open and oracle JDKs. That is 6 builds. > 3) We want to add Java 11 support such that one could build with 8 or 11 > and the above still apply. The becomes 6 builds. > 4) With the progress in NiFi registry we can now load artifacts there and > could pull them into NiFi. And this integration will only get better. > 5) The NiFi build is too huge and cannot grow any longer or else we cannot > upload convenience binaries. > > We cannot solve all the things just yet but we can make progress. I > suggest we split apart the NiFi 'framework/application' in its own release > cycle and code repository from the 'nifi extensions' into its own > repository and release cycle. The NiFi release would still pull in a > specific set of extension bundles so to our end users at this time there is > no change. In the future we could also just stop including the extensions > in nifi the application and they could be sourced at runtime as needed from > the registry (call that a NiFi 2.x thing). > > Why does this help? > - Builds would only take as long as just extensions take or just core/app > takes. This reduces time for each change cycle and reduces load on > travis-ci which runs the same tests over and over and over for each pull > request/push regardless of whether it was an extension or core. > > - It moves us toward the direction we're heading anyway whereby extensions > can have their own lifecycle from the framework/app itself. > > How is this not awesome: > - Doesn't yet solve for the large builds problem. I think we'll get there > with a NiFi 2.x release which fully leverages nifi-registry for retrieval > of all extensions. > - Adds another 'thing we need to do a release cycle for'. This is > generally unpleasant but it is paid for once a release cycle and it does > allow us to release independently for new cool extensions/fixes apart from > the framework itself. > > Would be great to hear others thoughts if they too feel it is time to make > this happen. > > Thanks > Joe >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
In the same category of challenges that Peter pointed out, it might be difficult for Travis to build the "framework" and "extensions" projects if there are changes in a PR that affect both projects. Is there a good way in Travis to have the workspace/maven repo shared between projects in a single build? It's probably always in the direction of the extensions project needing something new to be added to the framework project rather than the other way around, but it'll be tricky to get that working right in Travis if it's not possible to set up the Travis build to know it needs to deploy the framework project artifacts into a maven repo that the extension project will use. One way might be to make sure that changes to the framework project must be in master before the extensions project can make use of them, but that would require a "default master" build for the framework project which builds master after each commit, and deploys the build artifacts to a persistent maven repo that the extension project builds can access. It also makes project-spanning change-sets take longer to review and get fully committed to master. On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) wrote: > One more "not awesome" would be that core changes that affect extensions > will be a little harder to test. If I make a core change that changes the > signature of an interface/etc... I'll need to do some extra work to make > sure I don't break extensions that use it. > > Still worth it, just one more thing to mention. > > -Original Message- > From: Joe Witt > Sent: Thursday, May 30, 2019 9:19 AM > To: dev@nifi.apache.org > Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and > releases > > Team, > > We've discussed this a bit over the years in various forms but it again > seems time to progress this topic and enough has changed I think to warrant > it. > > Tensions: > 1) Our build times take too long. In travis-ci for instance it takes 40 > minutes when it works. > 2) The number of builds we do has increased. We do us/jp/fr builds on > open and oracle JDKs. That is 6 builds. > 3) We want to add Java 11 support such that one could build with 8 or 11 > and the above still apply. The becomes 6 builds. > 4) With the progress in NiFi registry we can now load artifacts there and > could pull them into NiFi. And this integration will only get better. > 5) The NiFi build is too huge and cannot grow any longer or else we cannot > upload convenience binaries. > > We cannot solve all the things just yet but we can make progress. I > suggest we split apart the NiFi 'framework/application' in its own release > cycle and code repository from the 'nifi extensions' into its own > repository and release cycle. The NiFi release would still pull in a > specific set of extension bundles so to our end users at this time there is > no change. In the future we could also just stop including the extensions > in nifi the application and they could be sourced at runtime as needed from > the registry (call that a NiFi 2.x thing). > > Why does this help? > - Builds would only take as long as just extensions take or just core/app > takes. This reduces time for each change cycle and reduces load on > travis-ci which runs the same tests over and over and over for each pull > request/push regardless of whether it was an extension or core. > > - It moves us toward the direction we're heading anyway whereby extensions > can have their own lifecycle from the framework/app itself. > > How is this not awesome: > - Doesn't yet solve for the large builds problem. I think we'll get there > with a NiFi 2.x release which fully leverages nifi-registry for retrieval > of all extensions. > - Adds another 'thing we need to do a release cycle for'. This is > generally unpleasant but it is paid for once a release cycle and it does > allow us to release independently for new cool extensions/fixes apart from > the framework itself. > > Would be great to hear others thoughts if they too feel it is time to make > this happen. > > Thanks > Joe >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would also be a good time to look at more clearly defining the separation between the UI and the framework. Where nifi-api is the contract between the extensions and the framework, the NiFi Rest api is the contract between the UI and framework... These pieces could potentially be built / deployed / updated independently. On Thu, May 30, 2019 at 11:39 AM Jeff wrote: > In the same category of challenges that Peter pointed out, it might be > difficult for Travis to build the "framework" and "extensions" projects if > there are changes in a PR that affect both projects. > > Is there a good way in Travis to have the workspace/maven repo shared > between projects in a single build? > > It's probably always in the direction of the extensions project needing > something new to be added to the framework project rather than the other > way around, but it'll be tricky to get that working right in Travis if it's > not possible to set up the Travis build to know it needs to deploy the > framework project artifacts into a maven repo that the extension project > will use. > > One way might be to make sure that changes to the framework project must be > in master before the extensions project can make use of them, but that > would require a "default master" build for the framework project which > builds master after each commit, and deploys the build artifacts to a > persistent maven repo that the extension project builds can access. It > also makes project-spanning change-sets take longer to review and get fully > committed to master. > > On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) > wrote: > > > One more "not awesome" would be that core changes that affect extensions > > will be a little harder to test. If I make a core change that changes the > > signature of an interface/etc... I'll need to do some extra work to make > > sure I don't break extensions that use it. > > > > Still worth it, just one more thing to mention. > > > > -Original Message- > > From: Joe Witt > > Sent: Thursday, May 30, 2019 9:19 AM > > To: dev@nifi.apache.org > > Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and > > releases > > > > Team, > > > > We've discussed this a bit over the years in various forms but it again > > seems time to progress this topic and enough has changed I think to > warrant > > it. > > > > Tensions: > > 1) Our build times take too long. In travis-ci for instance it takes 40 > > minutes when it works. > > 2) The number of builds we do has increased. We do us/jp/fr builds on > > open and oracle JDKs. That is 6 builds. > > 3) We want to add Java 11 support such that one could build with 8 or 11 > > and the above still apply. The becomes 6 builds. > > 4) With the progress in NiFi registry we can now load artifacts there and > > could pull them into NiFi. And this integration will only get better. > > 5) The NiFi build is too huge and cannot grow any longer or else we > cannot > > upload convenience binaries. > > > > We cannot solve all the things just yet but we can make progress. I > > suggest we split apart the NiFi 'framework/application' in its own > release > > cycle and code repository from the 'nifi extensions' into its own > > repository and release cycle. The NiFi release would still pull in a > > specific set of extension bundles so to our end users at this time there > is > > no change. In the future we could also just stop including the extensions > > in nifi the application and they could be sourced at runtime as needed > from > > the registry (call that a NiFi 2.x thing). > > > > Why does this help? > > - Builds would only take as long as just extensions take or just core/app > > takes. This reduces time for each change cycle and reduces load on > > travis-ci which runs the same tests over and over and over for each pull > > request/push regardless of whether it was an extension or core. > > > > - It moves us toward the direction we're heading anyway whereby > extensions > > can have their own lifecycle from the framework/app itself. > > > > How is this not awesome: > > - Doesn't yet solve for the large builds problem. I think we'll get > there > > with a NiFi 2.x release which fully leverages nifi-registry for retrieval > > of all extensions. > > - Adds another 'thing we need to do a release cycle for'. This is > > generally unpleasant but it is paid for once a release cycle and it does > > allow us to release independently for new cool extensions/fixes apart > from > > the framework itself. > > > > Would be great to hear others thoughts if they too feel it is time to > make > > this happen. > > > > Thanks > > Joe > > >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
I am a strong +1 on the separation and reducing the build time. With that in mind, I think the process I brought up yesterday [1] of signing our artifacts with GPG as part of the Maven build is paramount, because we would now be consuming core code across multiple projects/repositories, so there is even less guarantee the code is coming from “us”. [1] https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E <https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 30, 2019, at 9:15 AM, Brandon DeVries wrote: > > In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would > also be a good time to look at more clearly defining the separation between > the UI and the framework. Where nifi-api is the contract between the > extensions and the framework, the NiFi Rest api is the contract between the > UI and framework... These pieces could potentially be built / deployed / > updated independently. > > On Thu, May 30, 2019 at 11:39 AM Jeff wrote: > >> In the same category of challenges that Peter pointed out, it might be >> difficult for Travis to build the "framework" and "extensions" projects if >> there are changes in a PR that affect both projects. >> >> Is there a good way in Travis to have the workspace/maven repo shared >> between projects in a single build? >> >> It's probably always in the direction of the extensions project needing >> something new to be added to the framework project rather than the other >> way around, but it'll be tricky to get that working right in Travis if it's >> not possible to set up the Travis build to know it needs to deploy the >> framework project artifacts into a maven repo that the extension project >> will use. >> >> One way might be to make sure that changes to the framework project must be >> in master before the extensions project can make use of them, but that >> would require a "default master" build for the framework project which >> builds master after each commit, and deploys the build artifacts to a >> persistent maven repo that the extension project builds can access. It >> also makes project-spanning change-sets take longer to review and get fully >> committed to master. >> >> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) >> wrote: >> >>> One more "not awesome" would be that core changes that affect extensions >>> will be a little harder to test. If I make a core change that changes the >>> signature of an interface/etc... I'll need to do some extra work to make >>> sure I don't break extensions that use it. >>> >>> Still worth it, just one more thing to mention. >>> >>> -Original Message- >>> From: Joe Witt >>> Sent: Thursday, May 30, 2019 9:19 AM >>> To: dev@nifi.apache.org >>> Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and >>> releases >>> >>> Team, >>> >>> We've discussed this a bit over the years in various forms but it again >>> seems time to progress this topic and enough has changed I think to >> warrant >>> it. >>> >>> Tensions: >>> 1) Our build times take too long. In travis-ci for instance it takes 40 >>> minutes when it works. >>> 2) The number of builds we do has increased. We do us/jp/fr builds on >>> open and oracle JDKs. That is 6 builds. >>> 3) We want to add Java 11 support such that one could build with 8 or 11 >>> and the above still apply. The becomes 6 builds. >>> 4) With the progress in NiFi registry we can now load artifacts there and >>> could pull them into NiFi. And this integration will only get better. >>> 5) The NiFi build is too huge and cannot grow any longer or else we >> cannot >>> upload convenience binaries. >>> >>> We cannot solve all the things just yet but we can make progress. I >>> suggest we split apart the NiFi 'framework/application' in its own >> release >>> cycle and code repository from the 'nifi extensions' into its own >>> repository and release cycle. The NiFi release would still pull in a >>> specific set of extension bundles so to our end users at this time there >> is >>> no
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
t; >> committed to master. > >> > >> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) > >> wrote: > >> > >>> One more "not awesome" would be that core changes that affect extensions > >>> will be a little harder to test. If I make a core change that changes the > >>> signature of an interface/etc... I'll need to do some extra work to make > >>> sure I don't break extensions that use it. > >>> > >>> Still worth it, just one more thing to mention. > >>> > >>> -Original Message- > >>> From: Joe Witt > >>> Sent: Thursday, May 30, 2019 9:19 AM > >>> To: dev@nifi.apache.org > >>> Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and > >>> releases > >>> > >>> Team, > >>> > >>> We've discussed this a bit over the years in various forms but it again > >>> seems time to progress this topic and enough has changed I think to > >> warrant > >>> it. > >>> > >>> Tensions: > >>> 1) Our build times take too long. In travis-ci for instance it takes 40 > >>> minutes when it works. > >>> 2) The number of builds we do has increased. We do us/jp/fr builds on > >>> open and oracle JDKs. That is 6 builds. > >>> 3) We want to add Java 11 support such that one could build with 8 or 11 > >>> and the above still apply. The becomes 6 builds. > >>> 4) With the progress in NiFi registry we can now load artifacts there and > >>> could pull them into NiFi. And this integration will only get better. > >>> 5) The NiFi build is too huge and cannot grow any longer or else we > >> cannot > >>> upload convenience binaries. > >>> > >>> We cannot solve all the things just yet but we can make progress. I > >>> suggest we split apart the NiFi 'framework/application' in its own > >> release > >>> cycle and code repository from the 'nifi extensions' into its own > >>> repository and release cycle. The NiFi release would still pull in a > >>> specific set of extension bundles so to our end users at this time there > >> is > >>> no change. In the future we could also just stop including the extensions > >>> in nifi the application and they could be sourced at runtime as needed > >> from > >>> the registry (call that a NiFi 2.x thing). > >>> > >>> Why does this help? > >>> - Builds would only take as long as just extensions take or just core/app > >>> takes. This reduces time for each change cycle and reduces load on > >>> travis-ci which runs the same tests over and over and over for each pull > >>> request/push regardless of whether it was an extension or core. > >>> > >>> - It moves us toward the direction we're heading anyway whereby > >> extensions > >>> can have their own lifecycle from the framework/app itself. > >>> > >>> How is this not awesome: > >>> - Doesn't yet solve for the large builds problem. I think we'll get > >> there > >>> with a NiFi 2.x release which fully leverages nifi-registry for retrieval > >>> of all extensions. > >>> - Adds another 'thing we need to do a release cycle for'. This is > >>> generally unpleasant but it is paid for once a release cycle and it does > >>> allow us to release independently for new cool extensions/fixes apart > >> from > >>> the framework itself. > >>> > >>> Would be great to hear others thoughts if they too feel it is time to > >> make > >>> this happen. > >>> > >>> Thanks > >>> Joe > >>> > >> >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
9 AM Jeff wrote: > > > > > >> In the same category of challenges that Peter pointed out, it might be > > >> difficult for Travis to build the "framework" and "extensions" projects > > >> if > > >> there are changes in a PR that affect both projects. > > >> > > >> Is there a good way in Travis to have the workspace/maven repo shared > > >> between projects in a single build? > > >> > > >> It's probably always in the direction of the extensions project needing > > >> something new to be added to the framework project rather than the other > > >> way around, but it'll be tricky to get that working right in Travis if > > >> it's > > >> not possible to set up the Travis build to know it needs to deploy the > > >> framework project artifacts into a maven repo that the extension project > > >> will use. > > >> > > >> One way might be to make sure that changes to the framework project must > > >> be > > >> in master before the extensions project can make use of them, but that > > >> would require a "default master" build for the framework project which > > >> builds master after each commit, and deploys the build artifacts to a > > >> persistent maven repo that the extension project builds can access. It > > >> also makes project-spanning change-sets take longer to review and get > > >> fully > > >> committed to master. > > >> > > >> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) > > >> wrote: > > >> > > >>> One more "not awesome" would be that core changes that affect extensions > > >>> will be a little harder to test. If I make a core change that changes > > >>> the > > >>> signature of an interface/etc... I'll need to do some extra work to make > > >>> sure I don't break extensions that use it. > > >>> > > >>> Still worth it, just one more thing to mention. > > >>> > > >>> -Original Message- > > >>> From: Joe Witt > > >>> Sent: Thursday, May 30, 2019 9:19 AM > > >>> To: dev@nifi.apache.org > > >>> Subject: [EXT] [discuss] Splitting NiFi framework and extension repos > > >>> and > > >>> releases > > >>> > > >>> Team, > > >>> > > >>> We've discussed this a bit over the years in various forms but it again > > >>> seems time to progress this topic and enough has changed I think to > > >> warrant > > >>> it. > > >>> > > >>> Tensions: > > >>> 1) Our build times take too long. In travis-ci for instance it takes 40 > > >>> minutes when it works. > > >>> 2) The number of builds we do has increased. We do us/jp/fr builds on > > >>> open and oracle JDKs. That is 6 builds. > > >>> 3) We want to add Java 11 support such that one could build with 8 or 11 > > >>> and the above still apply. The becomes 6 builds. > > >>> 4) With the progress in NiFi registry we can now load artifacts there > > >>> and > > >>> could pull them into NiFi. And this integration will only get better. > > >>> 5) The NiFi build is too huge and cannot grow any longer or else we > > >> cannot > > >>> upload convenience binaries. > > >>> > > >>> We cannot solve all the things just yet but we can make progress. I > > >>> suggest we split apart the NiFi 'framework/application' in its own > > >> release > > >>> cycle and code repository from the 'nifi extensions' into its own > > >>> repository and release cycle. The NiFi release would still pull in a > > >>> specific set of extension bundles so to our end users at this time there > > >> is > > >>> no change. In the future we could also just stop including the > > >>> extensions > > >>> in nifi the application and they could be sourced at runtime as needed > > >> from > > >>> the registry (call that a NiFi 2.x thing). > > >>> > > >>> Why does this help? > > >>> - Builds would only take as long as just extensions take or just > > >>> core/app > > >>> takes. This reduces time for each change cycle and reduces load on > > >>> travis-ci which runs the same tests over and over and over for each pull > > >>> request/push regardless of whether it was an extension or core. > > >>> > > >>> - It moves us toward the direction we're heading anyway whereby > > >> extensions > > >>> can have their own lifecycle from the framework/app itself. > > >>> > > >>> How is this not awesome: > > >>> - Doesn't yet solve for the large builds problem. I think we'll get > > >> there > > >>> with a NiFi 2.x release which fully leverages nifi-registry for > > >>> retrieval > > >>> of all extensions. > > >>> - Adds another 'thing we need to do a release cycle for'. This is > > >>> generally unpleasant but it is paid for once a release cycle and it does > > >>> allow us to release independently for new cool extensions/fixes apart > > >> from > > >>> the framework itself. > > >>> > > >>> Would be great to hear others thoughts if they too feel it is time to > > >> make > > >>> this happen. > > >>> > > >>> Thanks > > >>> Joe > > >>> > > >> > >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
t;>> alopre...@apache.org >>> alopresto.apa...@gmail.com >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries wrote: >>>> >>>> In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would >>>> also be a good time to look at more clearly defining the separation between >>>> the UI and the framework. Where nifi-api is the contract between the >>>> extensions and the framework, the NiFi Rest api is the contract between the >>>> UI and framework... These pieces could potentially be built / deployed / >>>> updated independently. >>>> >>>> On Thu, May 30, 2019 at 11:39 AM Jeff wrote: >>>> >>>>> In the same category of challenges that Peter pointed out, it might be >>>>> difficult for Travis to build the "framework" and "extensions" projects if >>>>> there are changes in a PR that affect both projects. >>>>> >>>>> Is there a good way in Travis to have the workspace/maven repo shared >>>>> between projects in a single build? >>>>> >>>>> It's probably always in the direction of the extensions project needing >>>>> something new to be added to the framework project rather than the other >>>>> way around, but it'll be tricky to get that working right in Travis if >>>>> it's >>>>> not possible to set up the Travis build to know it needs to deploy the >>>>> framework project artifacts into a maven repo that the extension project >>>>> will use. >>>>> >>>>> One way might be to make sure that changes to the framework project must >>>>> be >>>>> in master before the extensions project can make use of them, but that >>>>> would require a "default master" build for the framework project which >>>>> builds master after each commit, and deploys the build artifacts to a >>>>> persistent maven repo that the extension project builds can access. It >>>>> also makes project-spanning change-sets take longer to review and get >>>>> fully >>>>> committed to master. >>>>> >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) >>>>> wrote: >>>>> >>>>>> One more "not awesome" would be that core changes that affect extensions >>>>>> will be a little harder to test. If I make a core change that changes the >>>>>> signature of an interface/etc... I'll need to do some extra work to make >>>>>> sure I don't break extensions that use it. >>>>>> >>>>>> Still worth it, just one more thing to mention. >>>>>> >>>>>> -Original Message- >>>>>> From: Joe Witt >>>>>> Sent: Thursday, May 30, 2019 9:19 AM >>>>>> To: dev@nifi.apache.org >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and >>>>>> releases >>>>>> >>>>>> Team, >>>>>> >>>>>> We've discussed this a bit over the years in various forms but it again >>>>>> seems time to progress this topic and enough has changed I think to >>>>> warrant >>>>>> it. >>>>>> >>>>>> Tensions: >>>>>> 1) Our build times take too long. In travis-ci for instance it takes 40 >>>>>> minutes when it works. >>>>>> 2) The number of builds we do has increased. We do us/jp/fr builds on >>>>>> open and oracle JDKs. That is 6 builds. >>>>>> 3) We want to add Java 11 support such that one could build with 8 or 11 >>>>>> and the above still apply. The becomes 6 builds. >>>>>> 4) With the progress in NiFi registry we can now load artifacts there and >>>>>> could pull them into NiFi. And this integration will only get better. >>>>>> 5) The NiFi build is too huge and cannot grow any longer or else we >>>>> cannot >>>>>> upload convenience binaries. >>>>>> >>>>>> We cannot solve all the things just yet but we can make progress. I >>>>>> suggest we split apart the NiFi 'framework/application' in its own >>>>> release >>>>>> cycle and code repository from the 'nifi extensions' into its own >>>>>> repository and release cycle. The NiFi release would still pull in a >>>>>> specific set of extension bundles so to our end users at this time there >>>>> is >>>>>> no change. In the future we could also just stop including the extensions >>>>>> in nifi the application and they could be sourced at runtime as needed >>>>> from >>>>>> the registry (call that a NiFi 2.x thing). >>>>>> >>>>>> Why does this help? >>>>>> - Builds would only take as long as just extensions take or just core/app >>>>>> takes. This reduces time for each change cycle and reduces load on >>>>>> travis-ci which runs the same tests over and over and over for each pull >>>>>> request/push regardless of whether it was an extension or core. >>>>>> >>>>>> - It moves us toward the direction we're heading anyway whereby >>>>> extensions >>>>>> can have their own lifecycle from the framework/app itself. >>>>>> >>>>>> How is this not awesome: >>>>>> - Doesn't yet solve for the large builds problem. I think we'll get >>>>> there >>>>>> with a NiFi 2.x release which fully leverages nifi-registry for retrieval >>>>>> of all extensions. >>>>>> - Adds another 'thing we need to do a release cycle for'. This is >>>>>> generally unpleasant but it is paid for once a release cycle and it does >>>>>> allow us to release independently for new cool extensions/fixes apart >>>>> from >>>>>> the framework itself. >>>>>> >>>>>> Would be great to hear others thoughts if they too feel it is time to >>>>> make >>>>>> this happen. >>>>>> >>>>>> Thanks >>>>>> Joe >>>>>> >>>>> >>>
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
> >> Kevin > >> > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto > wrote: > >>> > >>> I am a strong +1 on the separation and reducing the build time. With > that in mind, I think the process I brought up yesterday [1] of signing our > artifacts with GPG as part of the Maven build is paramount, because we > would now be consuming core code across multiple projects/repositories, so > there is even less guarantee the code is coming from “us”. > >>> > >>> [1] > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > < > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > >>> > >>> Andy LoPresto > >>> alopre...@apache.org > >>> alopresto.apa...@gmail.com > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >>> > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries wrote: > >>>> > >>>> In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 > would > >>>> also be a good time to look at more clearly defining the separation > between > >>>> the UI and the framework. Where nifi-api is the contract between the > >>>> extensions and the framework, the NiFi Rest api is the contract > between the > >>>> UI and framework... These pieces could potentially be built / > deployed / > >>>> updated independently. > >>>> > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff wrote: > >>>> > >>>>> In the same category of challenges that Peter pointed out, it might > be > >>>>> difficult for Travis to build the "framework" and "extensions" > projects if > >>>>> there are changes in a PR that affect both projects. > >>>>> > >>>>> Is there a good way in Travis to have the workspace/maven repo shared > >>>>> between projects in a single build? > >>>>> > >>>>> It's probably always in the direction of the extensions project > needing > >>>>> something new to be added to the framework project rather than the > other > >>>>> way around, but it'll be tricky to get that working right in Travis > if it's > >>>>> not possible to set up the Travis build to know it needs to deploy > the > >>>>> framework project artifacts into a maven repo that the extension > project > >>>>> will use. > >>>>> > >>>>> One way might be to make sure that changes to the framework project > must be > >>>>> in master before the extensions project can make use of them, but > that > >>>>> would require a "default master" build for the framework project > which > >>>>> builds master after each commit, and deploys the build artifacts to a > >>>>> persistent maven repo that the extension project builds can access. > It > >>>>> also makes project-spanning change-sets take longer to review and > get fully > >>>>> committed to master. > >>>>> > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) < > pwi...@micron.com> > >>>>> wrote: > >>>>> > >>>>>> One more "not awesome" would be that core changes that affect > extensions > >>>>>> will be a little harder to test. If I make a core change that > changes the > >>>>>> signature of an interface/etc... I'll need to do some extra work to > make > >>>>>> sure I don't break extensions that use it. > >>>>>> > >>>>>> Still worth it, just one more thing to mention. > >>>>>> > >>>>>> -Original Message- > >>>>>> From: Joe Witt > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > >>>>>> To: dev@nifi.apache.org > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension > repos and > >>>>>> releases > >>>>>> > >>>>>> Team, > >>>>>> > >>>>>> We've discussed this a bit over the years in various forms but it > again > >>>>>> seems time to progress this
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
t; >> that will need to span multiple projects, though those are fewer in > >> number, so overall I think it would be a net win for the dev > >> community. > >> > >> [1] https://github.com/kevdoran/nifi-commons > >> > >> Thanks, > >> Kevin > >> > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto > wrote: > >>> > >>> I am a strong +1 on the separation and reducing the build time. With > that in mind, I think the process I brought up yesterday [1] of signing our > artifacts with GPG as part of the Maven build is paramount, because we > would now be consuming core code across multiple projects/repositories, so > there is even less guarantee the code is coming from “us”. > >>> > >>> [1] > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > < > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > >>> > >>> Andy LoPresto > >>> alopre...@apache.org > >>> alopresto.apa...@gmail.com > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >>> > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries wrote: > >>>> > >>>> In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 > would > >>>> also be a good time to look at more clearly defining the separation > between > >>>> the UI and the framework. Where nifi-api is the contract between the > >>>> extensions and the framework, the NiFi Rest api is the contract > between the > >>>> UI and framework... These pieces could potentially be built / > deployed / > >>>> updated independently. > >>>> > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff wrote: > >>>> > >>>>> In the same category of challenges that Peter pointed out, it might > be > >>>>> difficult for Travis to build the "framework" and "extensions" > projects if > >>>>> there are changes in a PR that affect both projects. > >>>>> > >>>>> Is there a good way in Travis to have the workspace/maven repo shared > >>>>> between projects in a single build? > >>>>> > >>>>> It's probably always in the direction of the extensions project > needing > >>>>> something new to be added to the framework project rather than the > other > >>>>> way around, but it'll be tricky to get that working right in Travis > if it's > >>>>> not possible to set up the Travis build to know it needs to deploy > the > >>>>> framework project artifacts into a maven repo that the extension > project > >>>>> will use. > >>>>> > >>>>> One way might be to make sure that changes to the framework project > must be > >>>>> in master before the extensions project can make use of them, but > that > >>>>> would require a "default master" build for the framework project > which > >>>>> builds master after each commit, and deploys the build artifacts to a > >>>>> persistent maven repo that the extension project builds can access. > It > >>>>> also makes project-spanning change-sets take longer to review and > get fully > >>>>> committed to master. > >>>>> > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) < > pwi...@micron.com> > >>>>> wrote: > >>>>> > >>>>>> One more "not awesome" would be that core changes that affect > extensions > >>>>>> will be a little harder to test. If I make a core change that > changes the > >>>>>> signature of an interface/etc... I'll need to do some extra work to > make > >>>>>> sure I don't break extensions that use it. > >>>>>> > >>>>>> Still worth it, just one more thing to mention. > >>>>>> > >>>>>> -Original Message- > >>>>>> From: Joe Witt > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > >>>>>> To: dev@nifi.apache.org > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension > repos
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
hat affect both projects. > > >>>>> > > >>>>> Is there a good way in Travis to have the workspace/maven repo > shared > > >>>>> between projects in a single build? > > >>>>> > > >>>>> It's probably always in the direction of the extensions project > > needing > > >>>>> something new to be added to the framework project rather than the > > other > > >>>>> way around, but it'll be tricky to get that working right in Travis > > if it's > > >>>>> not possible to set up the Travis build to know it needs to deploy > > the > > >>>>> framework project artifacts into a maven repo that the extension > > project > > >>>>> will use. > > >>>>> > > >>>>> One way might be to make sure that changes to the framework project > > must be > > >>>>> in master before the extensions project can make use of them, but > > that > > >>>>> would require a "default master" build for the framework project > > which > > >>>>> builds master after each commit, and deploys the build artifacts to > a > > >>>>> persistent maven repo that the extension project builds can access. > > It > > >>>>> also makes project-spanning change-sets take longer to review and > > get fully > > >>>>> committed to master. > > >>>>> > > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) < > > pwi...@micron.com> > > >>>>> wrote: > > >>>>> > > >>>>>> One more "not awesome" would be that core changes that affect > > extensions > > >>>>>> will be a little harder to test. If I make a core change that > > changes the > > >>>>>> signature of an interface/etc... I'll need to do some extra work > to > > make > > >>>>>> sure I don't break extensions that use it. > > >>>>>> > > >>>>>> Still worth it, just one more thing to mention. > > >>>>>> > > >>>>>> -Original Message- > > >>>>>> From: Joe Witt > > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > > >>>>>> To: dev@nifi.apache.org > > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension > > repos and > > >>>>>> releases > > >>>>>> > > >>>>>> Team, > > >>>>>> > > >>>>>> We've discussed this a bit over the years in various forms but it > > again > > >>>>>> seems time to progress this topic and enough has changed I think > to > > >>>>> warrant > > >>>>>> it. > > >>>>>> > > >>>>>> Tensions: > > >>>>>> 1) Our build times take too long. In travis-ci for instance it > > takes 40 > > >>>>>> minutes when it works. > > >>>>>> 2) The number of builds we do has increased. We do us/jp/fr builds > > on > > >>>>>> open and oracle JDKs. That is 6 builds. > > >>>>>> 3) We want to add Java 11 support such that one could build with 8 > > or 11 > > >>>>>> and the above still apply. The becomes 6 builds. > > >>>>>> 4) With the progress in NiFi registry we can now load artifacts > > there and > > >>>>>> could pull them into NiFi. And this integration will only get > > better. > > >>>>>> 5) The NiFi build is too huge and cannot grow any longer or else > we > > >>>>> cannot > > >>>>>> upload convenience binaries. > > >>>>>> > > >>>>>> We cannot solve all the things just yet but we can make progress. > I > > >>>>>> suggest we split apart the NiFi 'framework/application' in its own > > >>>>> release > > >>>>>> cycle and code repository from the 'nifi extensions' into its own > > >>>>>> repository and release cycle. The NiFi release would still pull in > > a > > >>>>>> specific set of extension bundles so to our end users at this time > > there > > >>>>> is > > >>>>>> no change. In the future we could also just stop including the > > extensions > > >>>>>> in nifi the application and they could be sourced at runtime as > > needed > > >>>>> from > > >>>>>> the registry (call that a NiFi 2.x thing). > > >>>>>> > > >>>>>> Why does this help? > > >>>>>> - Builds would only take as long as just extensions take or just > > core/app > > >>>>>> takes. This reduces time for each change cycle and reduces load on > > >>>>>> travis-ci which runs the same tests over and over and over for > each > > pull > > >>>>>> request/push regardless of whether it was an extension or core. > > >>>>>> > > >>>>>> - It moves us toward the direction we're heading anyway whereby > > >>>>> extensions > > >>>>>> can have their own lifecycle from the framework/app itself. > > >>>>>> > > >>>>>> How is this not awesome: > > >>>>>> - Doesn't yet solve for the large builds problem. I think we'll > get > > >>>>> there > > >>>>>> with a NiFi 2.x release which fully leverages nifi-registry for > > retrieval > > >>>>>> of all extensions. > > >>>>>> - Adds another 'thing we need to do a release cycle for'. This is > > >>>>>> generally unpleasant but it is paid for once a release cycle and > it > > does > > >>>>>> allow us to release independently for new cool extensions/fixes > > apart > > >>>>> from > > >>>>>> the framework itself. > > >>>>>> > > >>>>>> Would be great to hear others thoughts if they too feel it is time > > to > > >>>>> make > > >>>>>> this happen. > > >>>>>> > > >>>>>> Thanks > > >>>>>> Joe > > >>>>>> > > >>>>> > > >>> > > > > >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > >>> > > > >>> Andy LoPresto > > > >>> alopre...@apache.org > > > >>> alopresto.apa...@gmail.com > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > >>> > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries wrote: > > > >>>> > > > >>>> In regards to "We 'could' also split out the 'nifi-api'...", NiFi > > 2.0 > > > would > > > >>>> also be a good time to look at more clearly defining the > separation > > > between > > > >>>> the UI and the framework. Where nifi-api is the contract between > the > > > >>>> extensions and the framework, the NiFi Rest api is the contract > > > between the > > > >>>> UI and framework... These pieces could potentially be built / > > > deployed / > > > >>>> updated independently. > > > >>>> > > > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff wrote: > > > >>>> > > > >>>>> In the same category of challenges that Peter pointed out, it > might > > > be > > > >>>>> difficult for Travis to build the "framework" and "extensions" > > > projects if > > > >>>>> there are changes in a PR that affect both projects. > > > >>>>> > > > >>>>> Is there a good way in Travis to have the workspace/maven repo > > shared > > > >>>>> between projects in a single build? > > > >>>>> > > > >>>>> It's probably always in the direction of the extensions project > > > needing > > > >>>>> something new to be added to the framework project rather than > the > > > other > > > >>>>> way around, but it'll be tricky to get that working right in > Travis > > > if it's > > > >>>>> not possible to set up the Travis build to know it needs to > deploy > > > the > > > >>>>> framework project artifacts into a maven repo that the extension > > > project > > > >>>>> will use. > > > >>>>> > > > >>>>> One way might be to make sure that changes to the framework > project > > > must be > > > >>>>> in master before the extensions project can make use of them, but > > > that > > > >>>>> would require a "default master" build for the framework project > > > which > > > >>>>> builds master after each commit, and deploys the build artifacts > to > > a > > > >>>>> persistent maven repo that the extension project builds can > access. > > > It > > > >>>>> also makes project-spanning change-sets take longer to review and > > > get fully > > > >>>>> committed to master. > > > >>>>> > > > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) < > > > pwi...@micron.com> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> One more "not awesome" would be that core changes that affect > > > extensions > > > >>>>>> will be a little harder to test. If I make a core change that > > > changes the > > > >>>>>> signature of an interface/etc... I'll need to do some extra work > > to > > > make > > > >>>>>> sure I don't break extensions that use it. > > > >>>>>> > > > >>>>>> Still worth it, just one more thing to mention. > > > >>>>>> > > > >>>>>> -Original Message- > > > >>>>>> From: Joe Witt > > > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > > > >>>>>> To: dev@nifi.apache.org > > > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension > > > repos and > > > >>>>>> releases > > > >>>>>> > > > >>>>>> Team, > > > >>>>>> > > > >>>>>> We've discussed this a bit over the years in various forms but > it > > > again > > > >>>>>> seems time to progress this topic and enough has changed I think > > to > > > >>>>> warrant > > > >>>>>> it. > > > >>>>>> > > > >>>>>> Tensions: > > > >>>>>> 1) Our build times take too long. In travis-ci for instance it > > > takes 40 > > > >>>>>> minutes when it works. > > > >>>>>> 2) The number of builds we do has increased. We do us/jp/fr > builds > > > on > > > >>>>>> open and oracle JDKs. That is 6 builds. > > > >>>>>> 3) We want to add Java 11 support such that one could build > with 8 > > > or 11 > > > >>>>>> and the above still apply. The becomes 6 builds. > > > >>>>>> 4) With the progress in NiFi registry we can now load artifacts > > > there and > > > >>>>>> could pull them into NiFi. And this integration will only get > > > better. > > > >>>>>> 5) The NiFi build is too huge and cannot grow any longer or else > > we > > > >>>>> cannot > > > >>>>>> upload convenience binaries. > > > >>>>>> > > > >>>>>> We cannot solve all the things just yet but we can make > progress. > > I > > > >>>>>> suggest we split apart the NiFi 'framework/application' in its > own > > > >>>>> release > > > >>>>>> cycle and code repository from the 'nifi extensions' into its > own > > > >>>>>> repository and release cycle. The NiFi release would still pull > in > > > a > > > >>>>>> specific set of extension bundles so to our end users at this > time > > > there > > > >>>>> is > > > >>>>>> no change. In the future we could also just stop including the > > > extensions > > > >>>>>> in nifi the application and they could be sourced at runtime as > > > needed > > > >>>>> from > > > >>>>>> the registry (call that a NiFi 2.x thing). > > > >>>>>> > > > >>>>>> Why does this help? > > > >>>>>> - Builds would only take as long as just extensions take or just > > > core/app > > > >>>>>> takes. This reduces time for each change cycle and reduces load > on > > > >>>>>> travis-ci which runs the same tests over and over and over for > > each > > > pull > > > >>>>>> request/push regardless of whether it was an extension or core. > > > >>>>>> > > > >>>>>> - It moves us toward the direction we're heading anyway whereby > > > >>>>> extensions > > > >>>>>> can have their own lifecycle from the framework/app itself. > > > >>>>>> > > > >>>>>> How is this not awesome: > > > >>>>>> - Doesn't yet solve for the large builds problem. I think we'll > > get > > > >>>>> there > > > >>>>>> with a NiFi 2.x release which fully leverages nifi-registry for > > > retrieval > > > >>>>>> of all extensions. > > > >>>>>> - Adds another 'thing we need to do a release cycle for'. This > is > > > >>>>>> generally unpleasant but it is paid for once a release cycle and > > it > > > does > > > >>>>>> allow us to release independently for new cool extensions/fixes > > > apart > > > >>>>> from > > > >>>>>> the framework itself. > > > >>>>>> > > > >>>>>> Would be great to hear others thoughts if they too feel it is > time > > > to > > > >>>>> make > > > >>>>>> this happen. > > > >>>>>> > > > >>>>>> Thanks > > > >>>>>> Joe > > > >>>>>> > > > >>>>> > > > >>> > > > > > > > > >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
;> releases of framework and extensions. > > > > >> > > > > >> Overall, I think this would be beneficial for most of the work > going > > > > >> on in Apache NiFi, which would not have to cut across these > > different > > > > >> project and therefore would be easier to code, test, build, and > > > > >> release. However, the level of difficulty will increase for > changes > > > > >> that will need to span multiple projects, though those are fewer > in > > > > >> number, so overall I think it would be a net win for the dev > > > > >> community. > > > > >> > > > > >> [1] https://github.com/kevdoran/nifi-commons > > > > >> > > > > >> Thanks, > > > > >> Kevin > > > > >> > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > > alopre...@apache.org> > > > > wrote: > > > > >>> > > > > >>> I am a strong +1 on the separation and reducing the build time. > > With > > > > that in mind, I think the process I brought up yesterday [1] of > signing > > > our > > > > artifacts with GPG as part of the Maven build is paramount, because > we > > > > would now be consuming core code across multiple > projects/repositories, > > > so > > > > there is even less guarantee the code is coming from “us”. > > > > >>> > > > > >>> [1] > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > < > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > > > >>> > > > > >>> Andy LoPresto > > > > >>> alopre...@apache.org > > > > >>> alopresto.apa...@gmail.com > > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > EF69 > > > > >>> > > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries > wrote: > > > > >>>> > > > > >>>> In regards to "We 'could' also split out the 'nifi-api'...", > NiFi > > > 2.0 > > > > would > > > > >>>> also be a good time to look at more clearly defining the > > separation > > > > between > > > > >>>> the UI and the framework. Where nifi-api is the contract between > > the > > > > >>>> extensions and the framework, the NiFi Rest api is the contract > > > > between the > > > > >>>> UI and framework... These pieces could potentially be built / > > > > deployed / > > > > >>>> updated independently. > > > > >>>> > > > > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff > wrote: > > > > >>>> > > > > >>>>> In the same category of challenges that Peter pointed out, it > > might > > > > be > > > > >>>>> difficult for Travis to build the "framework" and "extensions" > > > > projects if > > > > >>>>> there are changes in a PR that affect both projects. > > > > >>>>> > > > > >>>>> Is there a good way in Travis to have the workspace/maven repo > > > shared > > > > >>>>> between projects in a single build? > > > > >>>>> > > > > >>>>> It's probably always in the direction of the extensions project > > > > needing > > > > >>>>> something new to be added to the framework project rather than > > the > > > > other > > > > >>>>> way around, but it'll be tricky to get that working right in > > Travis > > > > if it's > > > > >>>>> not possible to set up the Travis build to know it needs to > > deploy > > > > the > > > > >>>>> framework project artifacts into a maven repo that the > extension > > > > project > > > > >>>>> will use. > > > > >>>>>
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
exists > > > > > >> as similar code in both projects, into a standalone modular > > library. > > > > > >> That premilinary work is here on my personal github account for > > now > > > > > >> [1]. > > > > > >> > > > > > >> So far, it only contains some security code in a submodule, and > > is a > > > > > >> WIP (more work coming when I have time), but the idea is > > > nifi-commons > > > > > >> could have several libraries/modules and would be released > > > > > >> periodically to use across nifi and registry. If we are talking > > > about > > > > > >> spliting the nifi project into framework and extensions, then > > > > > >> nifi-commons might be a good home for code that needs to be > shared > > > > > >> across those two sub projects as well, such as the nifi-api bits > > Joe > > > > > >> mentioned. > > > > > >> > > > > > >> As part of this larger effort, I would be happy to help get a > > > > > >> nifi-commons repository started in Apache where we can move > shared > > > > > >> code such as nifi-api to prepare for splitting nifi-framework > and > > > > > >> nifi-extensions. It also occurs to me that if nifi-framework and > > > > > >> nifi-extensions are being released independently, nifi-assembly > > > should > > > > > >> probably just become a project that pulls in and assembles the > > > latest > > > > > >> releases of framework and extensions. > > > > > >> > > > > > >> Overall, I think this would be beneficial for most of the work > > going > > > > > >> on in Apache NiFi, which would not have to cut across these > > > different > > > > > >> project and therefore would be easier to code, test, build, and > > > > > >> release. However, the level of difficulty will increase for > > changes > > > > > >> that will need to span multiple projects, though those are fewer > > in > > > > > >> number, so overall I think it would be a net win for the dev > > > > > >> community. > > > > > >> > > > > > >> [1] https://github.com/kevdoran/nifi-commons > > > > > >> > > > > > >> Thanks, > > > > > >> Kevin > > > > > >> > > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > > > alopre...@apache.org> > > > > > wrote: > > > > > >>> > > > > > >>> I am a strong +1 on the separation and reducing the build time. > > > With > > > > > that in mind, I think the process I brought up yesterday [1] of > > signing > > > > our > > > > > artifacts with GPG as part of the Maven build is paramount, because > > we > > > > > would now be consuming core code across multiple > > projects/repositories, > > > > so > > > > > there is even less guarantee the code is coming from “us”. > > > > > >>> > > > > > >>> [1] > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > < > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > > > > > >>> > > > > > >>> Andy LoPresto > > > > > >>> alopre...@apache.org > > > > > >>> alopresto.apa...@gmail.com > > > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > > EF69 > > > > > >>> > > > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries > > wrote: > > > > > >>>> > > > > > >>>> In regards to "We 'could' also split out the 'nifi-api'...", > > NiFi > > > > 2.0 > > > > > would > > > > > >>>> also be a good time to look at more clearly defining the > > > separ
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
test > > > > > > >> releases of framework and extensions. > > > > > > >> > > > > > > >> Overall, I think this would be beneficial for most of the work > > > going > > > > > > >> on in Apache NiFi, which would not have to cut across these > > > > different > > > > > > >> project and therefore would be easier to code, test, build, > and > > > > > > >> release. However, the level of difficulty will increase for > > > changes > > > > > > >> that will need to span multiple projects, though those are > fewer > > > in > > > > > > >> number, so overall I think it would be a net win for the dev > > > > > > >> community. > > > > > > >> > > > > > > >> [1] https://github.com/kevdoran/nifi-commons > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Kevin > > > > > > >> > > > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > > > > alopre...@apache.org> > > > > > > wrote: > > > > > > >>> > > > > > > >>> I am a strong +1 on the separation and reducing the build > time. > > > > With > > > > > > that in mind, I think the process I brought up yesterday [1] of > > > signing > > > > > our > > > > > > artifacts with GPG as part of the Maven build is paramount, > because > > > we > > > > > > would now be consuming core code across multiple > > > projects/repositories, > > > > > so > > > > > > there is even less guarantee the code is coming from “us”. > > > > > > >>> > > > > > > >>> [1] > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > > > > > > > >>> > > > > > > >>> Andy LoPresto > > > > > > >>> alopre...@apache.org > > > > > > >>> alopresto.apa...@gmail.com > > > > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > > > EF69 > > > > > > >>> > > > > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries > > > wrote: > > > > > > >>>> > > > > > > >>>> In regards to "We 'could' also split out the 'nifi-api'...", > > > NiFi > > > > > 2.0 > > > > > > would > > > > > > >>>> also be a good time to look at more clearly defining the > > > > separation > > > > > > between > > > > > > >>>> the UI and the framework. Where nifi-api is the contract > > between > > > > the > > > > > > >>>> extensions and the framework, the NiFi Rest api is the > > contract > > > > > > between the > > > > > > >>>> UI and framework... These pieces could potentially be built > / > > > > > > deployed / > > > > > > >>>> updated independently. > > > > > > >>>> > > > > > > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff > > > wrote: > > > > > > >>>> > > > > > > >>>>> In the same category of challenges that Peter pointed out, > it > > > > might > > > > > > be > > > > > > >>>>> difficult for Travis to build the "framework" and > > "extensions" > > > > > > projects if > > > > > > >>>>> there are changes in a PR that affect both projects. > > > > > > >>>>> > > > > > > >>>>> Is there a good way in Travis to have the worksp
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
t; Thanks, > > > > > > >> Kevin > > > > > > >> > > > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > > > > alopre...@apache.org> > > > > > > wrote: > > > > > > >>> > > > > > > >>> I am a strong +1 on the separation and reducing the build > time. > > > > With > > > > > > that in mind, I think the process I brought up yesterday [1] of > > > signing > > > > > our > > > > > > artifacts with GPG as part of the Maven build is paramount, > because > > > we > > > > > > would now be consuming core code across multiple > > > projects/repositories, > > > > > so > > > > > > there is even less guarantee the code is coming from “us”. > > > > > > >>> > > > > > > >>> [1] > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > > > > > > > >>> > > > > > > >>> Andy LoPresto > > > > > > >>> alopre...@apache.org > > > > > > >>> alopresto.apa...@gmail.com > > > > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > > > EF69 > > > > > > >>> > > > > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries > > > wrote: > > > > > > >>>> > > > > > > >>>> In regards to "We 'could' also split out the 'nifi-api'...", > > > NiFi > > > > > 2.0 > > > > > > would > > > > > > >>>> also be a good time to look at more clearly defining the > > > > separation > > > > > > between > > > > > > >>>> the UI and the framework. Where nifi-api is the contract > > between > > > > the > > > > > > >>>> extensions and the framework, the NiFi Rest api is the > > contract > > > > > > between the > > > > > > >>>> UI and framework... These pieces could potentially be built > / > > > > > > deployed / > > > > > > >>>> updated independently. > > > > > > >>>> > > > > > > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff > > > wrote: > > > > > > >>>> > > > > > > >>>>> In the same category of challenges that Peter pointed out, > it > > > > might > > > > > > be > > > > > > >>>>> difficult for Travis to build the "framework" and > > "extensions" > > > > > > projects if > > > > > > >>>>> there are changes in a PR that affect both projects. > > > > > > >>>>> > > > > > > >>>>> Is there a good way in Travis to have the workspace/maven > > repo > > > > > shared > > > > > > >>>>> between projects in a single build? > > > > > > >>>>> > > > > > > >>>>> It's probably always in the direction of the extensions > > project > > > > > > needing > > > > > > >>>>> something new to be added to the framework project rather > > than > > > > the > > > > > > other > > > > > > >>>>> way around, but it'll be tricky to get that working right > in > > > > Travis > > > > > > if it's > > > > > > >>>>> not possible to set up the Travis build to know it needs to > > > > deploy > > > > > > the > > > > > > >>>>> framework project artifacts into a maven repo that the > > > extension > > > > > > project > > > > > > >>>>> will use. > > > > > > >>>>> > > > > > > >>>>> One way might be to make sure that changes to the framework > > > > project > > > > > > must be > > > > > > >>>>> in master before the extensions project can make use of > them, > > > but > > > > > > that > > > > > > >>>>> would require a "default master" build for the framework > > > project > > > > > > which > > > > > > >>>>> builds master after each commit, and deploys the build > > > artifacts > > > > to > > > > > a > > > > > > >>>>> persistent maven repo that the extension project builds can > > > > access. > > > > > > It > > > > > > >>>>> also makes project-spanning change-sets take longer to > review > > > and > > > > > > get fully > > > > > > >>>>> committed to master. > > > > > > >>>>> > > > > > > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) < > > > > > > pwi...@micron.com> > > > > > > >>>>> wrote: > > > > > > >>>>> > > > > > > >>>>>> One more "not awesome" would be that core changes that > > affect > > > > > > extensions > > > > > > >>>>>> will be a little harder to test. If I make a core change > > that > > > > > > changes the > > > > > > >>>>>> signature of an interface/etc... I'll need to do some > extra > > > work > > > > > to > > > > > > make > > > > > > >>>>>> sure I don't break extensions that use it. > > > > > > >>>>>> > > > > > > >>>>>> Still worth it, just one more thing to mention. > > > > > > >>>>>> > > > > > > >>>>>> -Original Message- > > > > > > >>>>>> From: Joe Witt > > > > > > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > > > > > > >>>>>> To: dev@nifi.apache.org > > > > > > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and > > > extension > > > > > > repos and > > > > > > >>>>>> releases > > > > > > >>>>>> > > > > > > >>>>>> Team, > > > > > > >>>>>> > > > > > > >>>>>> We've discussed this a bit over the years in various forms > > but > > > > it > > > > > > again > > > > > > >>>>>> seems time to progress this topic and enough has changed I > > > think > > > > > to > > > > > > >>>>> warrant > > > > > > >>>>>> it. > > > > > > >>>>>> > > > > > > >>>>>> Tensions: > > > > > > >>>>>> 1) Our build times take too long. In travis-ci for > instance > > it > > > > > > takes 40 > > > > > > >>>>>> minutes when it works. > > > > > > >>>>>> 2) The number of builds we do has increased. We do > us/jp/fr > > > > builds > > > > > > on > > > > > > >>>>>> open and oracle JDKs. That is 6 builds. > > > > > > >>>>>> 3) We want to add Java 11 support such that one could > build > > > > with 8 > > > > > > or 11 > > > > > > >>>>>> and the above still apply. The becomes 6 builds. > > > > > > >>>>>> 4) With the progress in NiFi registry we can now load > > > artifacts > > > > > > there and > > > > > > >>>>>> could pull them into NiFi. And this integration will only > > get > > > > > > better. > > > > > > >>>>>> 5) The NiFi build is too huge and cannot grow any longer > or > > > else > > > > > we > > > > > > >>>>> cannot > > > > > > >>>>>> upload convenience binaries. > > > > > > >>>>>> > > > > > > >>>>>> We cannot solve all the things just yet but we can make > > > > progress. > > > > > I > > > > > > >>>>>> suggest we split apart the NiFi 'framework/application' in > > its > > > > own > > > > > > >>>>> release > > > > > > >>>>>> cycle and code repository from the 'nifi extensions' into > > its > > > > own > > > > > > >>>>>> repository and release cycle. The NiFi release would still > > > pull > > > > in > > > > > > a > > > > > > >>>>>> specific set of extension bundles so to our end users at > > this > > > > time > > > > > > there > > > > > > >>>>> is > > > > > > >>>>>> no change. In the future we could also just stop including > > the > > > > > > extensions > > > > > > >>>>>> in nifi the application and they could be sourced at > runtime > > > as > > > > > > needed > > > > > > >>>>> from > > > > > > >>>>>> the registry (call that a NiFi 2.x thing). > > > > > > >>>>>> > > > > > > >>>>>> Why does this help? > > > > > > >>>>>> - Builds would only take as long as just extensions take > or > > > just > > > > > > core/app > > > > > > >>>>>> takes. This reduces time for each change cycle and reduces > > > load > > > > on > > > > > > >>>>>> travis-ci which runs the same tests over and over and over > > for > > > > > each > > > > > > pull > > > > > > >>>>>> request/push regardless of whether it was an extension or > > > core. > > > > > > >>>>>> > > > > > > >>>>>> - It moves us toward the direction we're heading anyway > > > whereby > > > > > > >>>>> extensions > > > > > > >>>>>> can have their own lifecycle from the framework/app > itself. > > > > > > >>>>>> > > > > > > >>>>>> How is this not awesome: > > > > > > >>>>>> - Doesn't yet solve for the large builds problem. I think > > > we'll > > > > > get > > > > > > >>>>> there > > > > > > >>>>>> with a NiFi 2.x release which fully leverages > nifi-registry > > > for > > > > > > retrieval > > > > > > >>>>>> of all extensions. > > > > > > >>>>>> - Adds another 'thing we need to do a release cycle for'. > > This > > > > is > > > > > > >>>>>> generally unpleasant but it is paid for once a release > cycle > > > and > > > > > it > > > > > > does > > > > > > >>>>>> allow us to release independently for new cool > > > extensions/fixes > > > > > > apart > > > > > > >>>>> from > > > > > > >>>>>> the framework itself. > > > > > > >>>>>> > > > > > > >>>>>> Would be great to hear others thoughts if they too feel it > > is > > > > time > > > > > > to > > > > > > >>>>> make > > > > > > >>>>>> this happen. > > > > > > >>>>>> > > > > > > >>>>>> Thanks > > > > > > >>>>>> Joe > > > > > > >>>>>> > > > > > > >>>>> > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
> > > >> development at well defined boundaries/interfaces and also to > > > > > > > >> facilitate code reuse. > > > > > > > >> > > > > > > > >> In anticipation of eventually working towards a NiFi 2.0 that > > > > > > > >> introduces bigger changes for developers and users, I've > > started > > > > > work > > > > > > > >> on a nifi-commons project in which I've extracted out some of > > > the > > > > > code > > > > > > > >> that originally got ported from NiFi -> NiFi Registry, and now > > > > > exists > > > > > > > >> as similar code in both projects, into a standalone modular > > > > library. > > > > > > > >> That premilinary work is here on my personal github account > > for > > > > now > > > > > > > >> [1]. > > > > > > > >> > > > > > > > >> So far, it only contains some security code in a submodule, > > and > > > > is a > > > > > > > >> WIP (more work coming when I have time), but the idea is > > > > > nifi-commons > > > > > > > >> could have several libraries/modules and would be released > > > > > > > >> periodically to use across nifi and registry. If we are > > talking > > > > > about > > > > > > > >> spliting the nifi project into framework and extensions, then > > > > > > > >> nifi-commons might be a good home for code that needs to be > > > shared > > > > > > > >> across those two sub projects as well, such as the nifi-api > > bits > > > > Joe > > > > > > > >> mentioned. > > > > > > > >> > > > > > > > >> As part of this larger effort, I would be happy to help get a > > > > > > > >> nifi-commons repository started in Apache where we can move > > > shared > > > > > > > >> code such as nifi-api to prepare for splitting nifi-framework > > > and > > > > > > > >> nifi-extensions. It also occurs to me that if nifi-framework > > and > > > > > > > >> nifi-extensions are being released independently, > > nifi-assembly > > > > > should > > > > > > > >> probably just become a project that pulls in and assembles the > > > > > latest > > > > > > > >> releases of framework and extensions. > > > > > > > >> > > > > > > > >> Overall, I think this would be beneficial for most of the work > > > > going > > > > > > > >> on in Apache NiFi, which would not have to cut across these > > > > > different > > > > > > > >> project and therefore would be easier to code, test, build, > > and > > > > > > > >> release. However, the level of difficulty will increase for > > > > changes > > > > > > > >> that will need to span multiple projects, though those are > > fewer > > > > in > > > > > > > >> number, so overall I think it would be a net win for the dev > > > > > > > >> community. > > > > > > > >> > > > > > > > >> [1] https://github.com/kevdoran/nifi-commons > > > > > > > >> > > > > > > > >> Thanks, > > > > > > > >> Kevin > > > > > > > >> > > > > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > > > > > alopre...@apache.org> > > > > > > > wrote: > > > > > > > >>> > > > > > > > >>> I am a strong +1 on the separation and reducing the build > > time. > > > > > With > > > > > > > that in mind, I think the process I brought up yesterday [1] of > > > > signing > > > > > > our > > > > > > > artifacts with GPG as part of the Maven build is paramount, > > because > > > > we > > > > > > > would now be consuming core code across multiple > > > > projects/repositories, > > > &g
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
> > > > >> That premilinary work is here on my personal github account > > > for > > > > > now > > > > > > > > >> [1]. > > > > > > > > >> > > > > > > > > >> So far, it only contains some security code in a submodule, > > > and > > > > > is a > > > > > > > > >> WIP (more work coming when I have time), but the idea is > > > > > > nifi-commons > > > > > > > > >> could have several libraries/modules and would be released > > > > > > > > >> periodically to use across nifi and registry. If we are > > > talking > > > > > > about > > > > > > > > >> spliting the nifi project into framework and extensions, then > > > > > > > > >> nifi-commons might be a good home for code that needs to be > > > > shared > > > > > > > > >> across those two sub projects as well, such as the nifi-api > > > bits > > > > > Joe > > > > > > > > >> mentioned. > > > > > > > > >> > > > > > > > > >> As part of this larger effort, I would be happy to help get a > > > > > > > > >> nifi-commons repository started in Apache where we can move > > > > shared > > > > > > > > >> code such as nifi-api to prepare for splitting nifi-framework > > > > and > > > > > > > > >> nifi-extensions. It also occurs to me that if nifi-framework > > > and > > > > > > > > >> nifi-extensions are being released independently, > > > nifi-assembly > > > > > > should > > > > > > > > >> probably just become a project that pulls in and assembles > > > > > > > > >> the > > > > > > latest > > > > > > > > >> releases of framework and extensions. > > > > > > > > >> > > > > > > > > >> Overall, I think this would be beneficial for most of the > > > > > > > > >> work > > > > > going > > > > > > > > >> on in Apache NiFi, which would not have to cut across these > > > > > > different > > > > > > > > >> project and therefore would be easier to code, test, build, > > > and > > > > > > > > >> release. However, the level of difficulty will increase for > > > > > changes > > > > > > > > >> that will need to span multiple projects, though those are > > > fewer > > > > > in > > > > > > > > >> number, so overall I think it would be a net win for the dev > > > > > > > > >> community. > > > > > > > > >> > > > > > > > > >> [1] https://github.com/kevdoran/nifi-commons > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> Kevin > > > > > > > > >> > > > > > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > > > > > > alopre...@apache.org> > > > > > > > > wrote: > > > > > > > > >>> > > > > > > > > >>> I am a strong +1 on the separation and reducing the build > > > time. > > > > > > With > > > > > > > > that in mind, I think the process I brought up yesterday [1] of > > > > > signing > > > > > > > our > > > > > > > > artifacts with GPG as part of the Maven build is paramount, > > > because > > > > > we > > > > > > > > would now be consuming core code across multiple > > > > > projects/repositories, > > > > > > > so > > > > > > > > there is even less guarantee the code is coming from “us”. > > > > > > > > >>> > > > > > > > > >>> [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thr
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
> > >> code such as nifi-api to prepare for splitting > nifi-framework > > > > > and > > > > > > > > > >> nifi-extensions. It also occurs to me that if > nifi-framework > > > > and > > > > > > > > > >> nifi-extensions are being released independently, > > > > nifi-assembly > > > > > > > should > > > > > > > > > >> probably just become a project that pulls in and > assembles the > > > > > > > latest > > > > > > > > > >> releases of framework and extensions. > > > > > > > > > >> > > > > > > > > > >> Overall, I think this would be beneficial for most of > the work > > > > > > going > > > > > > > > > >> on in Apache NiFi, which would not have to cut across > these > > > > > > > different > > > > > > > > > >> project and therefore would be easier to code, test, > build, > > > > and > > > > > > > > > >> release. However, the level of difficulty will increase > for > > > > > > changes > > > > > > > > > >> that will need to span multiple projects, though those > are > > > > fewer > > > > > > in > > > > > > > > > >> number, so overall I think it would be a net win for > the dev > > > > > > > > > >> community. > > > > > > > > > >> > > > > > > > > > >> [1] https://github.com/kevdoran/nifi-commons > > > > > > > > > >> > > > > > > > > > >> Thanks, > > > > > > > > > >> Kevin > > > > > > > > > >> > > > > > > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > > > > > > > alopre...@apache.org> > > > > > > > > > wrote: > > > > > > > > > >>> > > > > > > > > > >>> I am a strong +1 on the separation and reducing the > build > > > > time. > > > > > > > With > > > > > > > > > that in mind, I think the process I brought up yesterday > [1] of > > > > > > signing > > > > > > > > our > > > > > > > > > artifacts with GPG as part of the Maven build is paramount, > > > > because > > > > > > we > > > > > > > > > would now be consuming core code across multiple > > > > > > projects/repositories, > > > > > > > > so > > > > > > > > > there is even less guarantee the code is coming from “us”. > > > > > > > > > >>> > > > > > > > > > >>> [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > >>> Andy LoPresto > > > > > > > > > >>> alopre...@apache.org > > > > > > > > > >>> alopresto.apa...@gmail.com > > > > > > > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E > F65B 2F7D > > > > > > EF69 > > > > > > > > > >>> > > > > > > > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries < > b...@jhu.edu> > > > > > > wrote: > > > > > > > > > >>>> > > > > > > > > > >>>> In regards to "We 'could' also split out the > 'nifi-api'...", > > > > > > NiFi > > > > > > > > 2.0 > > > > > > > > > would > > > > > > > > > >>>> also be a good time to look at more clearly defining > the > > > > > > > separation > > > > > > > > > between > > > > > > > > > >>>> the UI and the framework. Where nifi-api is the > contract > > > > > between > > > > > > > the > > > > > > > > > >>>> extensions and the framework, the NiFi Rest api is the > > > > > contract > > > > > > > > > between the > > > > > > > > > >>>> UI and framework... These pieces could potentially be > built > > > > / > > > > > > > > > deployed / > > > > > > > > > >>>> updated independently. > > > > > > > > > >>>> > > > > > > > > > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff < > jtsw...@gmail.com> > > > > > > wrote: > > > > > > > > > >>>> > > > > > > > > > >>>>> In the same category of challenges that Peter > pointed out, > > > > it > > > > > > > might > > > > > > > > > be > > > > > > > > > >>>>> difficult for Travis to build the "framework" and > > > > > "extensions" > > > > > > > > > projects if > > > > > > > > > >>>>> there are changes in a PR that affect both projects. > > > > > > > > > >>>>> > > > > > > > > > >>>>> Is there a good way in Travis to have the > workspace/maven > > > > > repo > > > > > > > > shared > > > > > > > > > >>>>> between projects in a single build? > > > > > > > > > >>>>> > > > > > > > > > >>>>> It's probably always in the direction of the > extensions > > > > > project > > > > > > > > > needing > > > > > > > > > >>>>> something new to be added to the framework project > rather > > > > > than > > > > > > > the > > > > > > > > > other > > > > > > > > > >>>>> way around, but it'll be tricky to get that working > right > > > > in > > > > > > > Travis > > > > > > > > > if it's > > > > > > > > > >>>>> not possible to set up the Travis build to know it > needs to > > > > > > > deploy > > > > > > > > > the > > > > > > > > > >>>>> framework project artifacts into a maven repo that > the > > > > > > extension > > > > > > > > > project > > > > > > > > > >>>>> will use. > > > > > > > > > >>>>> > > > > > > > > > >>>>> One way might be to make sure that changes to the > framework > > > > > > > project > > > > > > > > > must be > > > > > > > > > >>>>> in master before the extensions project can make use > of > > > > them, > > > > > > but > > > > > > > > > that > > > > > > > > > >>>>> would require a "default master" build for the > framework > > > > > > project > > > > > > > > > which > > > > > > > > > >>>>> builds master after each commit, and deploys the > build > > > > > > artifacts > > > > > > > to > > > > > > > > a > > > > > > > > > >>>>> persistent maven repo that the extension project > builds can > > > > > > > access. > > > > > > > > > It > > > > > > > > > >>>>> also makes project-spanning change-sets take longer > to > > > > review > > > > > > and > > > > > > > > > get fully > > > > > > > > > >>>>> committed to master. > > > > > > > > > >>>>> > > > > > > > > > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks > (pwicks) < > > > > > > > > > pwi...@micron.com> > > > > > > > > > >>>>> wrote: > > > > > > > > > >>>>> > > > > > > > > > >>>>>> One more "not awesome" would be that core changes > that > > > > > affect > > > > > > > > > extensions > > > > > > > > > >>>>>> will be a little harder to test. If I make a core > change > > > > > that > > > > > > > > > changes the > > > > > > > > > >>>>>> signature of an interface/etc... I'll need to do > some > > > > extra > > > > > > work > > > > > > > > to > > > > > > > > > make > > > > > > > > > >>>>>> sure I don't break extensions that use it. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Still worth it, just one more thing to mention. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> -Original Message- > > > > > > > > > >>>>>> From: Joe Witt > > > > > > > > > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > > > > > > > > > >>>>>> To: dev@nifi.apache.org > > > > > > > > > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework > and > > > > > > extension > > > > > > > > > repos and > > > > > > > > > >>>>>> releases > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Team, > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> We've discussed this a bit over the years in > various forms > > > > > but > > > > > > > it > > > > > > > > > again > > > > > > > > > >>>>>> seems time to progress this topic and enough has > changed I > > > > > > think > > > > > > > > to > > > > > > > > > >>>>> warrant > > > > > > > > > >>>>>> it. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Tensions: > > > > > > > > > >>>>>> 1) Our build times take too long. In travis-ci for > > > > instance > > > > > it > > > > > > > > > takes 40 > > > > > > > > > >>>>>> minutes when it works. > > > > > > > > > >>>>>> 2) The number of builds we do has increased. We do > > > > us/jp/fr > > > > > > > builds > > > > > > > > > on > > > > > > > > > >>>>>> open and oracle JDKs. That is 6 builds. > > > > > > > > > >>>>>> 3) We want to add Java 11 support such that one > could > > > > build > > > > > > > with 8 > > > > > > > > > or 11 > > > > > > > > > >>>>>> and the above still apply. The becomes 6 builds. > > > > > > > > > >>>>>> 4) With the progress in NiFi registry we can now > load > > > > > > artifacts > > > > > > > > > there and > > > > > > > > > >>>>>> could pull them into NiFi. And this integration > will only > > > > > get > > > > > > > > > better. > > > > > > > > > >>>>>> 5) The NiFi build is too huge and cannot grow any > longer > > > > or > > > > > > else > > > > > > > > we > > > > > > > > > >>>>> cannot > > > > > > > > > >>>>>> upload convenience binaries. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> We cannot solve all the things just yet but we can > make > > > > > > > progress. > > > > > > > > I > > > > > > > > > >>>>>> suggest we split apart the NiFi > 'framework/application' in > > > > > its > > > > > > > own > > > > > > > > > >>>>> release > > > > > > > > > >>>>>> cycle and code repository from the 'nifi > extensions' into > > > > > its > > > > > > > own > > > > > > > > > >>>>>> repository and release cycle. The NiFi release > would still > > > > > > pull > > > > > > > in > > > > > > > > > a > > > > > > > > > >>>>>> specific set of extension bundles so to our end > users at > > > > > this > > > > > > > time > > > > > > > > > there > > > > > > > > > >>>>> is > > > > > > > > > >>>>>> no change. In the future we could also just stop > including > > > > > the > > > > > > > > > extensions > > > > > > > > > >>>>>> in nifi the application and they could be sourced at > > > > runtime > > > > > > as > > > > > > > > > needed > > > > > > > > > >>>>> from > > > > > > > > > >>>>>> the registry (call that a NiFi 2.x thing). > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Why does this help? > > > > > > > > > >>>>>> - Builds would only take as long as just extensions > take > > > > or > > > > > > just > > > > > > > > > core/app > > > > > > > > > >>>>>> takes. This reduces time for each change cycle and > reduces > > > > > > load > > > > > > > on > > > > > > > > > >>>>>> travis-ci which runs the same tests over and over > and over > > > > > for > > > > > > > > each > > > > > > > > > pull > > > > > > > > > >>>>>> request/push regardless of whether it was an > extension or > > > > > > core. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> - It moves us toward the direction we're heading > anyway > > > > > > whereby > > > > > > > > > >>>>> extensions > > > > > > > > > >>>>>> can have their own lifecycle from the framework/app > > > > itself. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> How is this not awesome: > > > > > > > > > >>>>>> - Doesn't yet solve for the large builds problem. I > think > > > > > > we'll > > > > > > > > get > > > > > > > > > >>>>> there > > > > > > > > > >>>>>> with a NiFi 2.x release which fully leverages > > > > nifi-registry > > > > > > for > > > > > > > > > retrieval > > > > > > > > > >>>>>> of all extensions. > > > > > > > > > >>>>>> - Adds another 'thing we need to do a release cycle > for'. > > > > > This > > > > > > > is > > > > > > > > > >>>>>> generally unpleasant but it is paid for once a > release > > > > cycle > > > > > > and > > > > > > > > it > > > > > > > > > does > > > > > > > > > >>>>>> allow us to release independently for new cool > > > > > > extensions/fixes > > > > > > > > > apart > > > > > > > > > >>>>> from > > > > > > > > > >>>>>> the framework itself. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Would be great to hear others thoughts if they too > feel it > > > > > is > > > > > > > time > > > > > > > > > to > > > > > > > > > >>>>> make > > > > > > > > > >>>>>> this happen. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Thanks > > > > > > > > > >>>>>> Joe > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
>>>>>>>> >>>>>>>>>> Andy LoPresto >>>>>>>>>> alopre...@apache.org >>>>>>>>>> alopresto.apa...@gmail.com >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B >> 2F7D >>>>> EF69 >>>>>>>>>> >>>>>>>>>>> On Jul 10, 2019, at 12:08 PM, Kevin Doran < >> kdo...@apache.org> >>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi NiFi Dev Community, >>>>>>>>>>> >>>>>>>>>>> Jeff Storck, Bryan Bende, and I have been collaborating >> back >>>>> and >>>>>>>> forth >>>>>>>>>>> on a proposal for how to restructure the NiFi source >> code into >>>>>>>> smaller >>>>>>>>>>> Maven projects and repositories based on the discussion >> that >>>>> took >>>>>>>>>>> place awhile back on this thread. I'm reviving this older >>>>> thread >>>>>> in >>>>>>>>>>> order to share that proposal with the community and >> generate >>>>>>> farther >>>>>>>>>>> discussion about at solidifying a destination and a plan >> for >>>>> how >>>>>> to >>>>>>>>>>> get there. >>>>>>>>>>> >>>>>>>>>>> Specifically, the proposal we've started working on has >> three >>>>>>> parts: >>>>>>>>>>> >>>>>>>>>>> 1. Goals (more or less a summary of the earlier >> discussion that >>>>>>> took >>>>>>>>>>> place on this thread) >>>>>>>>>>> 2. Proposed end state of the new Maven project and >> repository >>>>>>>> structure >>>>>>>>>>> 3. Proposed approach for how to get from where we are >> today to >>>>>> the >>>>>>>>>>> desired end state >>>>>>>>>>> >>>>>>>>>>> The proposal is on the Apache NiFi Wiki [1], so that we >> can all >>>>>>>>>>> collaborate on it or leave comments there. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Kevin, Jeff, and Bryan >>>>>>>>>>> >>>>>>>>>>> On Thu, May 30, 2019 at 1:31 PM Kevin Doran < >> kdo...@apache.org >>>>>> >>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I am also in favor of splitting the nifi maven project >> up into >>>>>>>> smaller >>>>>>>>>>>> projects with independent release cycles in order to >> decouple >>>>>>>>>>>> development at well defined boundaries/interfaces and >> also to >>>>>>>>>>>> facilitate code reuse. >>>>>>>>>>>> >>>>>>>>>>>> In anticipation of eventually working towards a NiFi >> 2.0 that >>>>>>>>>>>> introduces bigger changes for developers and users, I've >>>>> started >>>>>>>> work >>>>>>>>>>>> on a nifi-commons project in which I've extracted out >> some of >>>>>> the >>>>>>>> code >>>>>>>>>>>> that originally got ported from NiFi -> NiFi Registry, >> and now >>>>>>>> exists >>>>>>>>>>>> as similar code in both projects, into a standalone >> modular &
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
gt; the > > > > > > > > > > > desired end state > > > > > > > > > > > > > > > > > > > > > > The proposal is on the Apache NiFi Wiki [1], so that we > > can all > > > > > > > > > > > collaborate on it or leave comments there. > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Kevin, Jeff, and Bryan > > > > > > > > > > > > > > > > > > > > > > On Thu, May 30, 2019 at 1:31 PM Kevin Doran < > > kdo...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > > > > > > > > > > >> I am also in favor of splitting the nifi maven project > > up into > > > > > > > > smaller > > > > > > > > > > >> projects with independent release cycles in order to > > decouple > > > > > > > > > > >> development at well defined boundaries/interfaces and > > also to > > > > > > > > > > >> facilitate code reuse. > > > > > > > > > > >> > > > > > > > > > > >> In anticipation of eventually working towards a NiFi > > 2.0 that > > > > > > > > > > >> introduces bigger changes for developers and users, > I've > > > > > started > > > > > > > > work > > > > > > > > > > >> on a nifi-commons project in which I've extracted out > > some of > > > > > > the > > > > > > > > code > > > > > > > > > > >> that originally got ported from NiFi -> NiFi Registry, > > and now > > > > > > > > exists > > > > > > > > > > >> as similar code in both projects, into a standalone > > modular > > > > > > > library. > > > > > > > > > > >> That premilinary work is here on my personal github > > account > > > > > for > > > > > > > now > > > > > > > > > > >> [1]. > > > > > > > > > > >> > > > > > > > > > > >> So far, it only contains some security code in a > > submodule, > > > > > and > > > > > > > is a > > > > > > > > > > >> WIP (more work coming when I have time), but the idea > is > > > > > > > > nifi-commons > > > > > > > > > > >> could have several libraries/modules and would be > > released > > > > > > > > > > >> periodically to use across nifi and registry. If we > are > > > > > talking > > > > > > > > about > > > > > > > > > > >> spliting the nifi project into framework and > > extensions, then > > > > > > > > > > >> nifi-commons might be a good home for code that needs > > to be > > > > > > shared > > > > > > > > > > >> across those two sub projects as well, such as the > > nifi-api > > > > > bits > > > > > > > Joe > > > > > > > > > > >> mentioned. > > > > > > > > > > >> > > > > > > > > > > >> As part of this larger effort, I would be happy to > help > > get a > > > > > > > > > > >> nifi-commons repository started in Apache where we can > > move > > > > > > shared > > > > > > > > > > >> code such as nifi-api to prepare for splitting > > nifi-framework > > > > > > and > > > > > > > > >
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
> >> for > >>>>> how > >>>>>> to > >>>>>>>>>>> get there. > >>>>>>>>>>> > >>>>>>>>>>> Specifically, the proposal we've started working on has > >> three > >>>>>>> parts: > >>>>>>>>>>> > >>>>>>>>>>> 1. Goals (more or less a summary of the earlier > >> discussion that > >>>>>>> took > >>>>>>>>>>> place on this thread) > >>>>>>>>>>> 2. Proposed end state of the new Maven project and > >> repository > >>>>>>>> structure > >>>>>>>>>>> 3. Proposed approach for how to get from where we are > >> today to > >>>>>> the > >>>>>>>>>>> desired end state > >>>>>>>>>>> > >>>>>>>>>>> The proposal is on the Apache NiFi Wiki [1], so that we > >> can all > >>>>>>>>>>> collaborate on it or leave comments there. > >>>>>>>>>>> > >>>>>>>>>>> [1] > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Kevin, Jeff, and Bryan > >>>>>>>>>>> > >>>>>>>>>>> On Thu, May 30, 2019 at 1:31 PM Kevin Doran < > >> kdo...@apache.org > >>>>>> > >>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> I am also in favor of splitting the nifi maven project > >> up into > >>>>>>>> smaller > >>>>>>>>>>>> projects with independent release cycles in order to > >> decouple > >>>>>>>>>>>> development at well defined boundaries/interfaces and > >> also to > >>>>>>>>>>>> facilitate code reuse. > >>>>>>>>>>>> > >>>>>>>>>>>> In anticipation of eventually working towards a NiFi > >> 2.0 that > >>>>>>>>>>>> introduces bigger changes for developers and users, I've > >>>>> started > >>>>>>>> work > >>>>>>>>>>>> on a nifi-commons project in which I've extracted out > >> some of > >>>>>> the > >>>>>>>> code > >>>>>>>>>>>> that originally got ported from NiFi -> NiFi Registry, > >> and now > >>>>>>>> exists > >>>>>>>>>>>> as similar code in both projects, into a standalone > >> modular > >>>>>>> library. > >>>>>>>>>>>> That premilinary work is here on my personal github > >> account > >>>>> for > >>>>>>> now > >>>>>>>>>>>> [1]. > >>>>>>>>>>>> > >>>>>>>>>>>> So far, it only contains some security code in a > >> submodule, > >>>>> and > >>>>>>> is a > >>>>>>>>>>>> WIP (more work coming when I have time), but the idea is > >>>>>>>> nifi-commons > >>>>>>>>>>>> could have several libraries/modules and would be > >> released > >>>>>>>>>>>> periodically to use across nifi and registry. If we are > >>>>> talking > >>>>>>>> about > >>>>>>>>>>>> spliting the nifi project into framework and > >> extensions, then > >>>>>>>>>>>> nifi-commons might be a good home for code that needs > >> to be > >>&
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
3. Proposed approach for how to get from where we are > > >> today to > > >>>>>> the > > >>>>>>>>>>> desired end state > > >>>>>>>>>>> > > >>>>>>>>>>> The proposal is on the Apache NiFi Wiki [1], so that we > > >> can all > > >>>>>>>>>>> collaborate on it or leave comments there. > > >>>>>>>>>>> > > >>>>>>>>>>> [1] > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >> > > > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks, > > >>>>>>>>>>> Kevin, Jeff, and Bryan > > >>>>>>>>>>> > > >>>>>>>>>>> On Thu, May 30, 2019 at 1:31 PM Kevin Doran < > > >> kdo...@apache.org > > >>>>>> > > >>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>> I am also in favor of splitting the nifi maven project > > >> up into > > >>>>>>>> smaller > > >>>>>>>>>>>> projects with independent release cycles in order to > > >> decouple > > >>>>>>>>>>>> development at well defined boundaries/interfaces and > > >> also to > > >>>>>>>>>>>> facilitate code reuse. > > >>>>>>>>>>>> > > >>>>>>>>>>>> In anticipation of eventually working towards a NiFi > > >> 2.0 that > > >>>>>>>>>>>> introduces bigger changes for developers and users, I've > > >>>>> started > > >>>>>>>> work > > >>>>>>>>>>>> on a nifi-commons project in which I've extracted out > > >> some of > > >>>>>> the > > >>>>>>>> code > > >>>>>>>>>>>> that originally got ported from NiFi -> NiFi Registry, > > >> and now > > >>>>>>>> exists > > >>>>>>>>>>>> as similar code in both projects, into a standalone > > >> modular > > >>>>>>> library. > > >>>>>>>>>>>> That premilinary work is here on my personal github > > >> account > > >>>>> for > > >>>>>>> now > > >>>>>>>>>>>> [1]. > > >>>>>>>>>>>> > > >>>>>>>>>>>> So far, it only contains some security code in a > > >> submodule, > > >>>>> and > > >>>>>>> is a > > >>>>>>>>>>>> WIP (more work coming when I have time), but the idea is > > >>>>>>>> nifi-commons > > >>>>>>>>>>>> could have several libraries/modules and would be > > >> released > > >>>>>>>>>>>> periodically to use across nifi and registry. If we are > > >>>>> talking > > >>>>>>>> about > > >>>>>>>>>>>> spliting the nifi project into framework and > > >> extensions, then > > >>>>>>>>>>>> nifi-commons might be a good home for code that needs > > >> to be > > >>>>>> shared > > >>>>>>>>>>>> across those two sub projects as well, such as the > > >> nifi-api > > >>>>> bits > > >>>>>>> Joe > > >>>>>>>>>>>> mentioned. > > >>>>>>>>&
Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
nt: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B >>>>> 2F7D >>>>>>>> EF69 >>>>>>>>>>>>> >>>>>>>>>>>>>> On Jul 10, 2019, at 12:08 PM, Kevin Doran < >>>>> kdo...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi NiFi Dev Community, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jeff Storck, Bryan Bende, and I have been collaborating >>>>> back >>>>>>>> and >>>>>>>>>>> forth >>>>>>>>>>>>>> on a proposal for how to restructure the NiFi source >>>>> code into >>>>>>>>>>> smaller >>>>>>>>>>>>>> Maven projects and repositories based on the discussion >>>>> that >>>>>>>> took >>>>>>>>>>>>>> place awhile back on this thread. I'm reviving this older >>>>>>>> thread >>>>>>>>> in >>>>>>>>>>>>>> order to share that proposal with the community and >>>>> generate >>>>>>>>>> farther >>>>>>>>>>>>>> discussion about at solidifying a destination and a plan >>>>> for >>>>>>>> how >>>>>>>>> to >>>>>>>>>>>>>> get there. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Specifically, the proposal we've started working on has >>>>> three >>>>>>>>>> parts: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Goals (more or less a summary of the earlier >>>>> discussion that >>>>>>>>>> took >>>>>>>>>>>>>> place on this thread) >>>>>>>>>>>>>> 2. Proposed end state of the new Maven project and >>>>> repository >>>>>>>>>>> structure >>>>>>>>>>>>>> 3. Proposed approach for how to get from where we are >>>>> today to >>>>>>>>> the >>>>>>>>>>>>>> desired end state >>>>>>>>>>>>>> >>>>>>>>>>>>>> The proposal is on the Apache NiFi Wiki [1], so that we >>>>> can all >>>>>>>>>>>>>> collaborate on it or leave comments there. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>> >> https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Kevin, Jeff, and Bryan >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, May 30, 2019 at 1:31 PM Kevin Doran < >>>>> kdo...@apache.org >>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am also in favor of splitting the nifi maven project >>>>> up into >>>>>>>>>>> smaller >>>>>>>>>>>>>>> projects with independent release cycles in order to >>>>> decouple >>>>>>>>>>>>>>> development at well defined boundaries/interfaces and >>>>> also to >>>>>>>>>>>>>>> facilitate code reuse. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In anticipation of eventual
