Another approach would be to have a time-based build, say no more frequently than once/day. You would lose granularity to a specific commit, but reduce compute cycles required by reducing the number of builds being executed.
Just a thought. On Mon, Apr 19, 2021 at 2:34 PM David Handermann <exceptionfact...@gmail.com> wrote: > Chris,, > > Thanks for the description of the modular approach. Here is one GitHub > action that supports conditional execution based on Git changes: > > https://github.com/dorny/paths-filter > > It would take some effort to implement an approach that covers all the > bases, but using the Maven also-make and project-list command line options > should support building both a module and its dependencies. The list of > changes files can be passed to another action that could determine one or > more modules to build. > > Regards, > David Handermann > > On Mon, Apr 19, 2021 at 12:56 PM Chris Sampson > <chris.samp...@naimuri.com.invalid> wrote: > > > Our approach is the use a `git diff` between source and destination > > branches when a branch is merged (and the same for on dev branch builds). > > > > Each component within the repo then checks for whether any files within > its > > directories were changed (bearing in mind a dev branch may consist of > > multiple commits over time, so don't just look at changes in individual > > commits). > > > > I'm not sure if the mechanics for the existing nifi repo though with its > > use of: > > > > * GutHub Actions (can a git diff be performed at the start and then > > following actions use the output?) > > > > * Maven with sub-modules (can they be optionally built based on such > > conditions? The GitHub Action could create files depending upon what's > been > > changed in order to activate profiles maybe, would that work?) > > > > This is broadly how I've implemented a multi-component build previously, > > but not using Maven sub-modules (each component is typically its own > docker > > image and built separately from others). > > > > > > Cheers, > > > > Chris Sampson > > > > On Mon, 19 Apr 2021, 18:35 Joe Witt, <joe.w...@gmail.com> wrote: > > > > > Chris, > > > > > > Yeah that would be very helpful. But do you have any idea how that > > > might be achieved in this environment? > > > > > > Thanks > > > > > > On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson > > > <chris.samp...@naimuri.com.invalid> wrote: > > > > > > > > Could an approach of building only the updated parts of the repo help > > to > > > > reduce build times? > > > > > > > > For example, changes to the classes under the AWS bundle (and only > that > > > > bundle) would only need those classes to be built and tested. > > > > > > > > Where such an approach gets a bit more complex is interdependence > > between > > > > parts of the repo. For example, if nifi-api is updated, should all > > > classes > > > > be built and tested? > > > > > > > > As part of a release, the entire repo could then be built and tested. > > > > > > > > This approach might help if the majority of repo changes are to > > > individual > > > > NARs rather than wider ranging. I'm also assuming this would be > > possible > > > > via GitHub Actions (I have no experience of them, but have > implemented > > > this > > > > kind of approach in a Drone.io mono-repo previously). > > > > > > > > > > > > Cheers, > > > > > > > > Chris Sampson > > > > > > > > On Mon, 19 Apr 2021, 17:16 David Handermann, < > > exceptionfact...@gmail.com > > > > > > > > wrote: > > > > > > > > > This background is very helpful to keep in mind when evaluating new > > and > > > > > updated unit tests. There are definitely some expensive tests that > > > could > > > > > be streamlined, but introducing a separate version lifecycle for > > > framework > > > > > and extensions seems like it is becoming more necessary. Moving > to a > > > Java > > > > > 11 baseline would also reduce the need to build on multiple > versions, > > > but > > > > > based on other discussions, it sounds like that is not currently > > > scheduled. > > > > > > > > > > I have noticed that Windows builds can run for a longer period of > > > time, is > > > > > there a reason that the Windows workflow definition does not have > the > > > same > > > > > timeout setting as the Linux and macOS builds? Introducing one > would > > > at > > > > > least kill off problematic Windows builds. > > > > > > > > > > Regards, > > > > > David Handermann > > > > > > > > > > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt <joe.w...@gmail.com> > > wrote: > > > > > > > > > > > Thanks for bringing this up. The most clear next step I can > > envision > > > > > > at this point is that we break up our core framework from our > > > > > > extensions. Not obvious how best to break this up but we need > to. > > > > > > The build times are insane. > > > > > > > > > > > > Joe > > > > > > > > > > > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler < > > ottobackwa...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > As you can probably imagine, it takes a lot of resources in > order > > > to CI > > > > > > for all the Apache projects. Periodically this becomes an issue, > > as > > > the > > > > > > donated resources from cloud CI providers ( Travis and now GitHub > > > > > Actions ) > > > > > > end up queuing and delaying builds across Apache projects because > > of > > > > > larger > > > > > > projects and their requirements. > > > > > > > > > > > > > > The discussions center around a few common themes: > > > > > > > > > > > > > > - the CI requirements of large complex projects dominate the > > Apache > > > > > Queue > > > > > > > - how those projects can supplement their builds in a way > > > acceptable to > > > > > > ASF INFRA > > > > > > > - how those projects can have per project metering such that a > > > project > > > > > > can pay for hours over the ’norm’ > > > > > > > - how to optimize projects > > > > > > > > > > > > > > This issue is currently being discussed again on the @builds > > apache > > > > > > list. I’m sending this over to the Nifi Dev community because > Nifi > > > > > itself > > > > > > has been mentioned as one of the top users of GitHub action > > > resources by > > > > > > some measure. Many of the other projects are actively taking > > > measures > > > > > to > > > > > > decrease or optimize their usage, and we should probably think > > about > > > how > > > > > we > > > > > > can do so as well. > > > > > > > > > > > > > > Here is the *current* thread: > > > > > > > > > > > > > > > > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3e > > > > > > < > > > > > > > > > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3E > > > > > > > > > > > > > > > > > > > > > Here is the message where 13 projects are listed > > > > > > > > > > > > > > > > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8...@apache.org%3e > > > > > > < > > > > > > > > > > > > > > > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8...@apache.org%3E > > > > > > > > > > > > > > > > > > > > > There are many other threads regarding GitHub Action limits and > > > > > > resources if you start scrolling back through the archives. > > > > > > > > > > > > > > I’m posting this to hopefully kick off some discussions. > > > > > > > > > > > > > > > > >
