Great work, thanks for contributing! For now I'd keep it as it is, so we can experiment with the parallel runner on GH. We have a few more things happening on the Jenkins pipeline and it would be better to keep it as stable as possible for now.
Kind regards On Wed, Feb 16, 2022 at 5:43 PM Nicolas Filotto <nfilo...@talend.com> wrote: > Hi, > > I just wanted to let you know that with Otavio, we finally could make the > Github's runner uses the maven daemon (with 2 threads) as you may have > seen. It seems to help a little bit since the validation of a PR now takes > between 45-48 minutes while it used to take more than an hour. I know that > it is not revolutionary, but I'm afraid that with 2 cores on the runner, it > will be hard to do much better. > > Would it make sense to do the same on the Jenkins' runner? > > Regards, > Nicolas > > ________________________________ > From: Nicolas Filotto <nfilo...@talend.com> > Sent: Thursday, February 10, 2022 14:10 > To: dev@camel.apache.org <dev@camel.apache.org> > Subject: Re: Parallel Build in Camel > > !-------------------------------------------------------------------| > This Message Is From an External Sender > This message came from outside your organization. > Exercise caution when opening attachments or clicking any > links. > |-------------------------------------------------------------------! > > Hi Guillaume, > > That's awsome, no I didn't know that option, thanks for sharing, it will > definitively help. > > Actually even the maven daemon as it is now (wondeful job by the way) > could help. > Indeed, on my machine even with 2 threads, the command "mvnd clean install > -Pfastinstall -Psourcecheck -T 2" takes 32 minutes while the equivalent > with maven takes 48 minutes. > > I know that we cannot predict the results on the build pipeline based on > this but at least it sounds like it could be interesting to try it, don't > you agree? > > Regards, > Nicolas > > ________________________________ > From: Guillaume Nodet <gno...@apache.org> > Sent: Thursday, February 10, 2022 11:24 > To: dev@camel.apache.org <dev@camel.apache.org> > Subject: Re: Parallel Build in Camel > > !-------------------------------------------------------------------| > This Message Is From an External Sender > This message came from outside your organization. > Exercise caution when opening attachments or clicking any > links. > |-------------------------------------------------------------------! > > Fwiw, i did a lot of work on the build a while ago, but my main goal was to > optimize subsequent `mvnd -DskipTests` run. > Ideall, in such cases, no jars should be changed at all, but that's not the > case completely, especially for maven plugins (try running `ls -ltr > **/*.jar` after a build). There are pending changes in maven to fix that > and I plan to push some changes in camel (mostly plugins updates) once > those are released. > > In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime` > option to gather statistics on the mojo execution time. > > In addition, I've been working a few weeks ago on the > maven-build-cache-extension [1], which is a powerful extension that > provides a full per-module build cache. The cache can be configured > locally and remotely (where the output of a module will be downloaded from > the remote cache instead of being built locally). It currently requires > some changes in maven-core that will only be released in maven 3.9.0. > However, I can try to set up an experimental mvnd branch that would embed > this extension if that's of any interest. > > Guillaume > > [1] > https://urldefense.com/v3/__https://github.com/apache/maven-build-cache-extension__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehHmiEG7g$ > > Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <angusyo...@gmail.com> > a écrit : > > > Thanks for looking into this. > > > > On a related note, I think one area where we could investigate potential > > improvements to the compilation is by looking at the performance of our > > maven plugins and code generators. > > > > For example the camel-endpointdsl module takes a long time to generate > > sources and build (that one takes almost 2 minutes to run on GH). The > > camel-componentdsl one takes about 1 minute. I suspect that if we could > > trace why they are so slow, there may be a route for optimizing the build > > time (and if we are lucky, that could happen across the whole build). > > > > A secondary branch of investigation could be looking at whether we can > > adjust our build so that we can have it running with "process-classes" > > target, to avoid the costly process of compressing and installing the > jars. > > > > Kind regards > > > > On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nfilo...@talend.com> > > wrote: > > > > > Hi again, > > > > > > Just to let you know that it appears that there are only 2 cores on the > > > target server which is obviously not enough to have an impact on the > > build > > > time. > > > > > > Regards, > > > Nicolas > > > ________________________________ > > > From: Nicolas Filotto <nfilo...@talend.com> > > > Sent: Wednesday, February 9, 2022 11:33 > > > To: dev@camel.apache.org <dev@camel.apache.org> > > > Subject: Re: Parallel Build in Camel > > > > > > !-------------------------------------------------------------------| > > > This Message Is From an External Sender > > > This message came from outside your organization. > > > Exercise caution when opening attachments or clicking any > > > links. > > > |-------------------------------------------------------------------! > > > > > > Hi, > > > > > > Thank you very much for your feedbacks. Here is the related PR > > > > > > https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$ > > > . > > > > > > Regards, > > > Nicolas > > > > > > ________________________________ > > > From: Otavio Rodolfo Piske <angusyo...@gmail.com> > > > Sent: Wednesday, February 9, 2022 10:26 > > > To: dev@camel.apache.org <dev@camel.apache.org> > > > Subject: Re: Parallel Build in Camel > > > > > > !-------------------------------------------------------------------| > > > This Message Is From an External Sender > > > This message came from outside your organization. > > > Exercise caution when opening attachments or clicking any > > > links. > > > |-------------------------------------------------------------------! > > > > > > Hi, > > > > > > I think it would be OK to give it a try. Please, feel free to open a PR > > > with this suggestion. > > > > > > Kind regards > > > > > > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nfilo...@talend.com> > > > wrote: > > > > > > > Hi, > > > > > > > > If the parallel build is something that can be used in the build > > > pipeline, > > > > why not simply starting by adding the -T option of maven to the > > commands > > > > launched by the build ( > > > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$ > > > ) > > > > as short/middle term solution before eventually considering the maven > > > > daemon? What do you think of it? > > > > > > > > Regards, > > > > Nicolas > > > > > > > > ________________________________ > > > > From: Otavio Rodolfo Piske <angusyo...@gmail.com> > > > > Sent: Wednesday, February 9, 2022 10:03 > > > > To: dev@camel.apache.org <dev@camel.apache.org> > > > > Subject: Re: Parallel Build in Camel > > > > > > > > !-------------------------------------------------------------------| > > > > This Message Is From an External Sender > > > > This message came from outside your organization. > > > > Exercise caution when opening attachments or clicking any > > > > links. > > > > |-------------------------------------------------------------------! > > > > > > > > Hi, > > > > > > > > Yeah, that helps a bit, although it's still quite a lot. One day I'd > > like > > > > to take a closer look at some of the plugins we have on the build to > > see > > > if > > > > there's some easy speed up we could do. > > > > > > > > As for the Github, I think we depend on 2 things: > > > > 1. I believe we need a Github action that contains and runs maven > > daemon > > > > 2. I believe this action needs to be approved somehow by the ASF. I > am > > > not > > > > sure about this, though ... maybe others in the community know more > > about > > > > it. > > > > > > > > Kind regards > > > > > > > > > > > > > > > > As a recipient of an email from Talend, your contact personal data > will > > > be > > > > on our systems. Please see our privacy notice (updated August 2020) > at > > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> > > > > > > > > > > > > > > > > > > -- > > > Otavio R. Piske > > > > > > > > > https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$ > > > > > > As a recipient of an email from Talend, your contact personal data will > > be > > > on our systems. Please see our privacy notice (updated August 2020) at > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> > > > > > > > > > > > > As a recipient of an email from Talend, your contact personal data will > > be > > > on our systems. Please see our privacy notice (updated August 2020) at > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> > > > > > > > > > > > > > -- > > Otavio R. Piske > > > https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehQC12Kv4$ > > > > > -- > ------------------------ > Guillaume Nodet > > As a recipient of an email from Talend, your contact personal data will be > on our systems. Please see our privacy notice (updated August 2020) at > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> > > > > As a recipient of an email from Talend, your contact personal data will be > on our systems. Please see our privacy notice. < > https://www.talend.com/privacy/> > > > -- Otavio R. Piske http://orpiske.net
