Hi, I just wanted to let you know that with Otavio, we finally could make the Github's runner uses the maven daemon (with 2 threads) as you may have seen. It seems to help a little bit since the validation of a PR now takes between 45-48 minutes while it used to take more than an hour. I know that it is not revolutionary, but I'm afraid that with 2 cores on the runner, it will be hard to do much better.
Would it make sense to do the same on the Jenkins' runner? Regards, Nicolas ________________________________ From: Nicolas Filotto <nfilo...@talend.com> Sent: Thursday, February 10, 2022 14:10 To: dev@camel.apache.org <dev@camel.apache.org> Subject: Re: Parallel Build in Camel !-------------------------------------------------------------------| This Message Is From an External Sender This message came from outside your organization. Exercise caution when opening attachments or clicking any links. |-------------------------------------------------------------------! Hi Guillaume, That's awsome, no I didn't know that option, thanks for sharing, it will definitively help. Actually even the maven daemon as it is now (wondeful job by the way) could help. Indeed, on my machine even with 2 threads, the command "mvnd clean install -Pfastinstall -Psourcecheck -T 2" takes 32 minutes while the equivalent with maven takes 48 minutes. I know that we cannot predict the results on the build pipeline based on this but at least it sounds like it could be interesting to try it, don't you agree? Regards, Nicolas ________________________________ From: Guillaume Nodet <gno...@apache.org> Sent: Thursday, February 10, 2022 11:24 To: dev@camel.apache.org <dev@camel.apache.org> Subject: Re: Parallel Build in Camel !-------------------------------------------------------------------| This Message Is From an External Sender This message came from outside your organization. Exercise caution when opening attachments or clicking any links. |-------------------------------------------------------------------! Fwiw, i did a lot of work on the build a while ago, but my main goal was to optimize subsequent `mvnd -DskipTests` run. Ideall, in such cases, no jars should be changed at all, but that's not the case completely, especially for maven plugins (try running `ls -ltr **/*.jar` after a build). There are pending changes in maven to fix that and I plan to push some changes in camel (mostly plugins updates) once those are released. In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime` option to gather statistics on the mojo execution time. In addition, I've been working a few weeks ago on the maven-build-cache-extension [1], which is a powerful extension that provides a full per-module build cache. The cache can be configured locally and remotely (where the output of a module will be downloaded from the remote cache instead of being built locally). It currently requires some changes in maven-core that will only be released in maven 3.9.0. However, I can try to set up an experimental mvnd branch that would embed this extension if that's of any interest. Guillaume [1] https://urldefense.com/v3/__https://github.com/apache/maven-build-cache-extension__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehHmiEG7g$ Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <angusyo...@gmail.com> a écrit : > Thanks for looking into this. > > On a related note, I think one area where we could investigate potential > improvements to the compilation is by looking at the performance of our > maven plugins and code generators. > > For example the camel-endpointdsl module takes a long time to generate > sources and build (that one takes almost 2 minutes to run on GH). The > camel-componentdsl one takes about 1 minute. I suspect that if we could > trace why they are so slow, there may be a route for optimizing the build > time (and if we are lucky, that could happen across the whole build). > > A secondary branch of investigation could be looking at whether we can > adjust our build so that we can have it running with "process-classes" > target, to avoid the costly process of compressing and installing the jars. > > Kind regards > > On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nfilo...@talend.com> > wrote: > > > Hi again, > > > > Just to let you know that it appears that there are only 2 cores on the > > target server which is obviously not enough to have an impact on the > build > > time. > > > > Regards, > > Nicolas > > ________________________________ > > From: Nicolas Filotto <nfilo...@talend.com> > > Sent: Wednesday, February 9, 2022 11:33 > > To: dev@camel.apache.org <dev@camel.apache.org> > > Subject: Re: Parallel Build in Camel > > > > !-------------------------------------------------------------------| > > This Message Is From an External Sender > > This message came from outside your organization. > > Exercise caution when opening attachments or clicking any > > links. > > |-------------------------------------------------------------------! > > > > Hi, > > > > Thank you very much for your feedbacks. Here is the related PR > > > https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$ > > . > > > > Regards, > > Nicolas > > > > ________________________________ > > From: Otavio Rodolfo Piske <angusyo...@gmail.com> > > Sent: Wednesday, February 9, 2022 10:26 > > To: dev@camel.apache.org <dev@camel.apache.org> > > Subject: Re: Parallel Build in Camel > > > > !-------------------------------------------------------------------| > > This Message Is From an External Sender > > This message came from outside your organization. > > Exercise caution when opening attachments or clicking any > > links. > > |-------------------------------------------------------------------! > > > > Hi, > > > > I think it would be OK to give it a try. Please, feel free to open a PR > > with this suggestion. > > > > Kind regards > > > > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nfilo...@talend.com> > > wrote: > > > > > Hi, > > > > > > If the parallel build is something that can be used in the build > > pipeline, > > > why not simply starting by adding the -T option of maven to the > commands > > > launched by the build ( > > > > > > https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$ > > ) > > > as short/middle term solution before eventually considering the maven > > > daemon? What do you think of it? > > > > > > Regards, > > > Nicolas > > > > > > ________________________________ > > > From: Otavio Rodolfo Piske <angusyo...@gmail.com> > > > Sent: Wednesday, February 9, 2022 10:03 > > > To: dev@camel.apache.org <dev@camel.apache.org> > > > Subject: Re: Parallel Build in Camel > > > > > > !-------------------------------------------------------------------| > > > This Message Is From an External Sender > > > This message came from outside your organization. > > > Exercise caution when opening attachments or clicking any > > > links. > > > |-------------------------------------------------------------------! > > > > > > Hi, > > > > > > Yeah, that helps a bit, although it's still quite a lot. One day I'd > like > > > to take a closer look at some of the plugins we have on the build to > see > > if > > > there's some easy speed up we could do. > > > > > > As for the Github, I think we depend on 2 things: > > > 1. I believe we need a Github action that contains and runs maven > daemon > > > 2. I believe this action needs to be approved somehow by the ASF. I am > > not > > > sure about this, though ... maybe others in the community know more > about > > > it. > > > > > > Kind regards > > > > > > > > > > > > As a recipient of an email from Talend, your contact personal data will > > be > > > on our systems. Please see our privacy notice (updated August 2020) at > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> > > > > > > > > > > > > > -- > > Otavio R. Piske > > > > > https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$ > > > > As a recipient of an email from Talend, your contact personal data will > be > > on our systems. Please see our privacy notice (updated August 2020) at > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> > > > > > > > > As a recipient of an email from Talend, your contact personal data will > be > > on our systems. Please see our privacy notice (updated August 2020) at > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> > > > > > > > > -- > Otavio R. Piske > https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehQC12Kv4$ > -- ------------------------ Guillaume Nodet As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice (updated August 2020) at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice. <https://www.talend.com/privacy/>
