Ah I agree the JIRA thing would be too heavy handed. A single JIRA with well defined components tied to 'repos' is good.
As far as separate code repos we're talking about different releasable artifacts for which we as a PMC are responsible for the meaning/etc.. As a many time RM I definitely dislike the mono repo construct as I understand it to function. I prefer repos per source release artifact where all source in that artifact is a function of the release. I am ok with different convenience binaries resulting from a single source release artifact though. Thanks On Fri, Jul 12, 2019 at 12:26 PM Adam Taft <a...@adamtaft.com> wrote: > I think the concerns around user management are valid, are they not? > Overhead in JIRA goes up (assigning rights to users in JIRA is > multiplied). Risk to new contributors is high, because each isolated > repository has its own life and code contribution styles. Maybe the actual > apache infra involvement is low, but the negative effects of community and > source code bifurcation goes up. > > Tagging in mono-repos is done by prefixing the name of the component in the > tag name. Your release sources are still generated from the component > folder (not from the root). > > Modularization (as being proposed) is a good thing, but can be done in a > single repository. It's not a requirement to split up the git project to > get the benefits of modularization. That's the point I'm hoping is seen in > this. > > > > On Fri, Jul 12, 2019 at 10:08 AM Joe Witt <joe.w...@gmail.com> wrote: > > > to clarify user management for infra is not a prob. it is an ldap group. > > > > repo creation is self service as well amd group access is tied to that. > > > > release artifact is the source we produce. this is typically correlated > to > > a tag of the repo. if we have all source in one repo it isnt clear to me > > how we can maintain that. > > > > in any event im not making a statement of whether to do many repos or > not. > > just correcting some potentially misleading claims. > > > > thanks > > > > On Fri, Jul 12, 2019, 12:01 PM Adam Taft <a...@adamtaft.com> wrote: > > > > > Just as a point of discussion, I'm not entirely sure that splitting > into > > > multiple physical git repositories is actually adding any value. I > think > > > it's worth consideration that all the (good) changes being proposed are > > > done under a single mono-repository model. > > > > > > If we split into multiple repositories, you have substantially > increased > > > the infra surface area. User account management overhead goes up. > Support > > > from the infra team goes up. JIRA issue management goes up, > > > misfiled/miscategorized issues become common. It becomes harder for > > > community members to interact and engage with the project, steeper > > learning > > > curve for new contributors. There are more "side channel" conversations > > and > > > less transparency into the project as a whole. Git history is much > harder > > > (or impossible) to follow across the entire project. Tracking down bugs > > and > > > performing git blame or git bisect becomes hard. > > > > > > There's nothing really stopping all of these changes from occurring in > > the > > > existing repo, we don't have to have a maven pom.xml in the root of the > > > project repository. It's much easier for contributors to just clone a > > > single repository, read the README at the root, and get oriented to the > > > project layout. Output artifacts can still be versioned differently > (api > > > can have a different version from extensions). "Splitting out" modules > > can > > > still happen in the mono-repository. Jenkins and friends can be taught > > the > > > project layout. > > > > > > tl;dr - The changes being proposed can be done in a single repository. > > > Splitting into multiple repositories is adding overhead on multiple > > levels, > > > which might be a sneaky form of muda. [1] > > > > > > Thanks for reading, > > > Adam > > > > > > [1] https://dzone.com/articles/seven-wastes-software > > > > > > > > > On Thu, Jul 11, 2019 at 11:01 AM Otto Fowler <ottobackwa...@gmail.com> > > > wrote: > > > > > > > I agree that this looks great. I think Mike’s idea is worth > considering > > > as > > > > well. I would hope, that as part of this effort some thought will be > > > given > > > > to enhancing the developer documentation around the modules would be > > > given > > > > as well. > > > > > > > > > > > > > > > > > > > > On July 10, 2019 at 18:15:21, Mike Thomsen (mikerthom...@gmail.com) > > > wrote: > > > > > > > > I agree. It's very well thought out. One change to consider is > > splitting > > > > the extensions further into two separate repos. One that would serve > > as a > > > > standard library of sorts for other component developers and another > > that > > > > would include everything else. Things like the Record API would go > into > > > the > > > > former so that we could have a more conservative release schedule > going > > > > forward with those components. > > > > > > > > On Wed, Jul 10, 2019 at 4:17 PM Andy LoPresto <alopre...@apache.org> > > > > wrote: > > > > > > > > > Thanks Kevin, this looks really promising. > > > > > > > > > > Updating the link here as I think the page may have moved: > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > > > > > < > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring > > > > > > > > > > > > > > > > Andy LoPresto > > > > > alopre...@apache.org > > > > > alopresto.apa...@gmail.com > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > > > > > > > On Jul 10, 2019, at 12:08 PM, Kevin Doran <kdo...@apache.org> > > wrote: > > > > > > > > > > > > Hi NiFi Dev Community, > > > > > > > > > > > > Jeff Storck, Bryan Bende, and I have been collaborating back and > > > forth > > > > > > on a proposal for how to restructure the NiFi source code into > > > smaller > > > > > > Maven projects and repositories based on the discussion that took > > > > > > place awhile back on this thread. I'm reviving this older thread > in > > > > > > order to share that proposal with the community and generate > > farther > > > > > > discussion about at solidifying a destination and a plan for how > to > > > > > > get there. > > > > > > > > > > > > Specifically, the proposal we've started working on has three > > parts: > > > > > > > > > > > > 1. Goals (more or less a summary of the earlier discussion that > > took > > > > > > place on this thread) > > > > > > 2. Proposed end state of the new Maven project and repository > > > structure > > > > > > 3. Proposed approach for how to get from where we are today to > the > > > > > > desired end state > > > > > > > > > > > > The proposal is on the Apache NiFi Wiki [1], so that we can all > > > > > > collaborate on it or leave comments there. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring > > > > > > > > > > > > Thanks, > > > > > > Kevin, Jeff, and Bryan > > > > > > > > > > > > On Thu, May 30, 2019 at 1:31 PM Kevin Doran <kdo...@apache.org> > > > wrote: > > > > > >> > > > > > >> I am also in favor of splitting the nifi maven project up into > > > smaller > > > > > >> projects with independent release cycles in order to decouple > > > > > >> development at well defined boundaries/interfaces and also to > > > > > >> facilitate code reuse. > > > > > >> > > > > > >> In anticipation of eventually working towards a NiFi 2.0 that > > > > > >> introduces bigger changes for developers and users, I've started > > > work > > > > > >> on a nifi-commons project in which I've extracted out some of > the > > > code > > > > > >> that originally got ported from NiFi -> NiFi Registry, and now > > > exists > > > > > >> as similar code in both projects, into a standalone modular > > library. > > > > > >> That premilinary work is here on my personal github account for > > now > > > > > >> [1]. > > > > > >> > > > > > >> So far, it only contains some security code in a submodule, and > > is a > > > > > >> WIP (more work coming when I have time), but the idea is > > > nifi-commons > > > > > >> could have several libraries/modules and would be released > > > > > >> periodically to use across nifi and registry. If we are talking > > > about > > > > > >> spliting the nifi project into framework and extensions, then > > > > > >> nifi-commons might be a good home for code that needs to be > shared > > > > > >> across those two sub projects as well, such as the nifi-api bits > > Joe > > > > > >> mentioned. > > > > > >> > > > > > >> As part of this larger effort, I would be happy to help get a > > > > > >> nifi-commons repository started in Apache where we can move > shared > > > > > >> code such as nifi-api to prepare for splitting nifi-framework > and > > > > > >> nifi-extensions. It also occurs to me that if nifi-framework and > > > > > >> nifi-extensions are being released independently, nifi-assembly > > > should > > > > > >> probably just become a project that pulls in and assembles the > > > latest > > > > > >> releases of framework and extensions. > > > > > >> > > > > > >> Overall, I think this would be beneficial for most of the work > > going > > > > > >> on in Apache NiFi, which would not have to cut across these > > > different > > > > > >> project and therefore would be easier to code, test, build, and > > > > > >> release. However, the level of difficulty will increase for > > changes > > > > > >> that will need to span multiple projects, though those are fewer > > in > > > > > >> number, so overall I think it would be a net win for the dev > > > > > >> community. > > > > > >> > > > > > >> [1] https://github.com/kevdoran/nifi-commons > > > > > >> > > > > > >> Thanks, > > > > > >> Kevin > > > > > >> > > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto < > > > alopre...@apache.org> > > > > > wrote: > > > > > >>> > > > > > >>> I am a strong +1 on the separation and reducing the build time. > > > With > > > > > that in mind, I think the process I brought up yesterday [1] of > > signing > > > > our > > > > > artifacts with GPG as part of the Maven build is paramount, because > > we > > > > > would now be consuming core code across multiple > > projects/repositories, > > > > so > > > > > there is even less guarantee the code is coming from “us”. > > > > > >>> > > > > > >>> [1] > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > < > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E > > > > > > > > > > > >>> > > > > > >>> Andy LoPresto > > > > > >>> alopre...@apache.org > > > > > >>> alopresto.apa...@gmail.com > > > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > > EF69 > > > > > >>> > > > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries <b...@jhu.edu> > > wrote: > > > > > >>>> > > > > > >>>> In regards to "We 'could' also split out the 'nifi-api'...", > > NiFi > > > > 2.0 > > > > > would > > > > > >>>> also be a good time to look at more clearly defining the > > > separation > > > > > between > > > > > >>>> the UI and the framework. Where nifi-api is the contract > between > > > the > > > > > >>>> extensions and the framework, the NiFi Rest api is the > contract > > > > > between the > > > > > >>>> UI and framework... These pieces could potentially be built / > > > > > deployed / > > > > > >>>> updated independently. > > > > > >>>> > > > > > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff <jtsw...@gmail.com> > > wrote: > > > > > >>>> > > > > > >>>>> In the same category of challenges that Peter pointed out, it > > > might > > > > > be > > > > > >>>>> difficult for Travis to build the "framework" and > "extensions" > > > > > projects if > > > > > >>>>> there are changes in a PR that affect both projects. > > > > > >>>>> > > > > > >>>>> Is there a good way in Travis to have the workspace/maven > repo > > > > shared > > > > > >>>>> between projects in a single build? > > > > > >>>>> > > > > > >>>>> It's probably always in the direction of the extensions > project > > > > > needing > > > > > >>>>> something new to be added to the framework project rather > than > > > the > > > > > other > > > > > >>>>> way around, but it'll be tricky to get that working right in > > > Travis > > > > > if it's > > > > > >>>>> not possible to set up the Travis build to know it needs to > > > deploy > > > > > the > > > > > >>>>> framework project artifacts into a maven repo that the > > extension > > > > > project > > > > > >>>>> will use. > > > > > >>>>> > > > > > >>>>> One way might be to make sure that changes to the framework > > > project > > > > > must be > > > > > >>>>> in master before the extensions project can make use of them, > > but > > > > > that > > > > > >>>>> would require a "default master" build for the framework > > project > > > > > which > > > > > >>>>> builds master after each commit, and deploys the build > > artifacts > > > to > > > > a > > > > > >>>>> persistent maven repo that the extension project builds can > > > access. > > > > > It > > > > > >>>>> also makes project-spanning change-sets take longer to review > > and > > > > > get fully > > > > > >>>>> committed to master. > > > > > >>>>> > > > > > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) < > > > > > pwi...@micron.com> > > > > > >>>>> wrote: > > > > > >>>>> > > > > > >>>>>> One more "not awesome" would be that core changes that > affect > > > > > extensions > > > > > >>>>>> will be a little harder to test. If I make a core change > that > > > > > changes the > > > > > >>>>>> signature of an interface/etc... I'll need to do some extra > > work > > > > to > > > > > make > > > > > >>>>>> sure I don't break extensions that use it. > > > > > >>>>>> > > > > > >>>>>> Still worth it, just one more thing to mention. > > > > > >>>>>> > > > > > >>>>>> -----Original Message----- > > > > > >>>>>> From: Joe Witt <joew...@apache.org> > > > > > >>>>>> Sent: Thursday, May 30, 2019 9:19 AM > > > > > >>>>>> To: dev@nifi.apache.org > > > > > >>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and > > extension > > > > > repos and > > > > > >>>>>> releases > > > > > >>>>>> > > > > > >>>>>> Team, > > > > > >>>>>> > > > > > >>>>>> We've discussed this a bit over the years in various forms > but > > > it > > > > > again > > > > > >>>>>> seems time to progress this topic and enough has changed I > > think > > > > to > > > > > >>>>> warrant > > > > > >>>>>> it. > > > > > >>>>>> > > > > > >>>>>> Tensions: > > > > > >>>>>> 1) Our build times take too long. In travis-ci for instance > it > > > > > takes 40 > > > > > >>>>>> minutes when it works. > > > > > >>>>>> 2) The number of builds we do has increased. We do us/jp/fr > > > builds > > > > > on > > > > > >>>>>> open and oracle JDKs. That is 6 builds. > > > > > >>>>>> 3) We want to add Java 11 support such that one could build > > > with 8 > > > > > or 11 > > > > > >>>>>> and the above still apply. The becomes 6 builds. > > > > > >>>>>> 4) With the progress in NiFi registry we can now load > > artifacts > > > > > there and > > > > > >>>>>> could pull them into NiFi. And this integration will only > get > > > > > better. > > > > > >>>>>> 5) The NiFi build is too huge and cannot grow any longer or > > else > > > > we > > > > > >>>>> cannot > > > > > >>>>>> upload convenience binaries. > > > > > >>>>>> > > > > > >>>>>> We cannot solve all the things just yet but we can make > > > progress. > > > > I > > > > > >>>>>> suggest we split apart the NiFi 'framework/application' in > its > > > own > > > > > >>>>> release > > > > > >>>>>> cycle and code repository from the 'nifi extensions' into > its > > > own > > > > > >>>>>> repository and release cycle. The NiFi release would still > > pull > > > in > > > > > a > > > > > >>>>>> specific set of extension bundles so to our end users at > this > > > time > > > > > there > > > > > >>>>> is > > > > > >>>>>> no change. In the future we could also just stop including > the > > > > > extensions > > > > > >>>>>> in nifi the application and they could be sourced at runtime > > as > > > > > needed > > > > > >>>>> from > > > > > >>>>>> the registry (call that a NiFi 2.x thing). > > > > > >>>>>> > > > > > >>>>>> Why does this help? > > > > > >>>>>> - Builds would only take as long as just extensions take or > > just > > > > > core/app > > > > > >>>>>> takes. This reduces time for each change cycle and reduces > > load > > > on > > > > > >>>>>> travis-ci which runs the same tests over and over and over > for > > > > each > > > > > pull > > > > > >>>>>> request/push regardless of whether it was an extension or > > core. > > > > > >>>>>> > > > > > >>>>>> - It moves us toward the direction we're heading anyway > > whereby > > > > > >>>>> extensions > > > > > >>>>>> can have their own lifecycle from the framework/app itself. > > > > > >>>>>> > > > > > >>>>>> How is this not awesome: > > > > > >>>>>> - Doesn't yet solve for the large builds problem. I think > > we'll > > > > get > > > > > >>>>> there > > > > > >>>>>> with a NiFi 2.x release which fully leverages nifi-registry > > for > > > > > retrieval > > > > > >>>>>> of all extensions. > > > > > >>>>>> - Adds another 'thing we need to do a release cycle for'. > This > > > is > > > > > >>>>>> generally unpleasant but it is paid for once a release cycle > > and > > > > it > > > > > does > > > > > >>>>>> allow us to release independently for new cool > > extensions/fixes > > > > > apart > > > > > >>>>> from > > > > > >>>>>> the framework itself. > > > > > >>>>>> > > > > > >>>>>> Would be great to hear others thoughts if they too feel it > is > > > time > > > > > to > > > > > >>>>> make > > > > > >>>>>> this happen. > > > > > >>>>>> > > > > > >>>>>> Thanks > > > > > >>>>>> Joe > > > > > >>>>>> > > > > > >>>>> > > > > > >>> > > > > > > > > > > > > > > > > > > > >
