Apache Project issues with available GitHub Action resources across projects

2021-04-19 Thread Otto Fowler
As you can probably imagine, it takes a lot of resources in order to CI for all the Apache projects. Periodically this becomes an issue, as the donated resources from cloud CI providers ( Travis and now GitHub Actions ) end up queuing and delaying builds across Apache projects because of larger

Re: Apache Project issues with available GitHub Action resources across projects

2021-04-19 Thread Joe Witt
Thanks for bringing this up. The most clear next step I can envision at this point is that we break up our core framework from our extensions. Not obvious how best to break this up but we need to. The build times are insane. Joe On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler wrote: > > As you can

NIFI-8161 PR 4773 migration from SimpleDateFormat to DateTimeFormatter in NiFi expressions

2021-04-19 Thread Arek Burdach
Hi, Some time ago I've prepared a PR: https://github.com/apache/nifi/pull/4773 that changes formatter used for formatting/printing date from/to string from SimpleDateFormat to DateTimeFormatter. I did it because I've made some benchmarks and figure out that it is quite important bottle neck i

Re: Apache Project issues with available GitHub Action resources across projects

2021-04-19 Thread David Handermann
This background is very helpful to keep in mind when evaluating new and updated unit tests. There are definitely some expensive tests that could be streamlined, but introducing a separate version lifecycle for framework and extensions seems like it is becoming more necessary. Moving to a Java 11

Re: Apache Project issues with available GitHub Action resources across projects

2021-04-19 Thread Chris Sampson
Could an approach of building only the updated parts of the repo help to reduce build times? For example, changes to the classes under the AWS bundle (and only that bundle) would only need those classes to be built and tested. Where such an approach gets a bit more complex is interdependence betw

Re: Apache Project issues with available GitHub Action resources across projects

2021-04-19 Thread Joe Witt
Chris, Yeah that would be very helpful. But do you have any idea how that might be achieved in this environment? Thanks On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson wrote: > > Could an approach of building only the updated parts of the repo help to > reduce build times? > > For example, chan

Re: Apache Project issues with available GitHub Action resources across projects

2021-04-19 Thread Chris Sampson
Our approach is the use a `git diff` between source and destination branches when a branch is merged (and the same for on dev branch builds). Each component within the repo then checks for whether any files within its directories were changed (bearing in mind a dev branch may consist of multiple c

Re: Apache Project issues with available GitHub Action resources across projects

2021-04-19 Thread David Handermann
Chris,, Thanks for the description of the modular approach. Here is one GitHub action that supports conditional execution based on Git changes: https://github.com/dorny/paths-filter It would take some effort to implement an approach that covers all the bases, but using the Maven also-make and p