Re: Possibly speeding up tests with Gradle - anyone interested?
I dropped the ball in following up with the gradle contact that Nick met at ApacheCon. Konstantin, if you'd like to coordinate a meeting with him (and any other devs!), please let me know. I just sent a follow up message to him for potentially trying to set something up next week. Best, Tim On Wed, Nov 9, 2022 at 4:08 PM Konstantin Gribov wrote: > > Any follow up on using build cache and/or experiment with migration to > gradle? I have a little bit of expertise with gradle and wanted to join in > with the experiment. > Compilation avoidance, dependency resolution engine and ease of writing > ad-hoc plugins are very strong points with gradle. I migrated a bunch of > projects at work to gradle and improved things quite a lot. > > Configuration phase could be quite long, I'm impatiently waiting when > configuration cache lands. It also avoids dependency resolution step on > each gradle run. > > Nicholas, could you elaborate on what you mean with "New versions of gradle > result in full redownload all the deps"? Never seen such behavior and it > sounds like a bug or some elaborate repositories' misconfiguration. I only > got extremely slow resolution with the spring dependency plugin and > misconfigured reverse proxy before Nexus. > > As for build cache it brings some good on the ci but not much on local > development in cases where we tried to use it. IIRC Build Cache server > itself is free if we want to use it and burden Infra with running it. But > likely it doesn't support clustering/HA which could be an issue for the ASF > scale. > > We certainly could use build scans for free publishing them from CI builds, > it would help with analyzing dependency graph and build performance. > > -- > Best regards, > Konstantin Gribov > > пт, 14 окт. 2022 г., 17:13 Nicholas DiPiazza : > > > Is there a branch started on the Gradle thing? I have some cycles and can > > use them to upgrade to Gradle on Tika, if desired. > > > > On Thu, Oct 6, 2022 at 8:50 AM Nick Burch wrote: > > > > > On Thu, 6 Oct 2022, Tim Allison wrote: > > > > Happy to chat. Please put them in touch. > > > > > > Excellent, thanks Tim! > > > > > > Other than your past talks, have we got any info (eg on the wiki?) about > > > how to run the regression corpus? > > > > > > > I've been really impressed with what the POI team has done migrating > > > > from ant to gradle. On Tika, I don't think we have any special needs > > > > that would require deep gradle knowledge, but given the number of > > > > modules now, it will be non-trivial. Also, I take Nick D's concerns > > > > seriously. > > > > > > We don't have to swap from Maven - they have a plugin that integrates it > > > > > > Nick > > > > >
Re: Possibly speeding up tests with Gradle - anyone interested?
Any follow up on using build cache and/or experiment with migration to gradle? I have a little bit of expertise with gradle and wanted to join in with the experiment. Compilation avoidance, dependency resolution engine and ease of writing ad-hoc plugins are very strong points with gradle. I migrated a bunch of projects at work to gradle and improved things quite a lot. Configuration phase could be quite long, I'm impatiently waiting when configuration cache lands. It also avoids dependency resolution step on each gradle run. Nicholas, could you elaborate on what you mean with "New versions of gradle result in full redownload all the deps"? Never seen such behavior and it sounds like a bug or some elaborate repositories' misconfiguration. I only got extremely slow resolution with the spring dependency plugin and misconfigured reverse proxy before Nexus. As for build cache it brings some good on the ci but not much on local development in cases where we tried to use it. IIRC Build Cache server itself is free if we want to use it and burden Infra with running it. But likely it doesn't support clustering/HA which could be an issue for the ASF scale. We certainly could use build scans for free publishing them from CI builds, it would help with analyzing dependency graph and build performance. -- Best regards, Konstantin Gribov пт, 14 окт. 2022 г., 17:13 Nicholas DiPiazza : > Is there a branch started on the Gradle thing? I have some cycles and can > use them to upgrade to Gradle on Tika, if desired. > > On Thu, Oct 6, 2022 at 8:50 AM Nick Burch wrote: > > > On Thu, 6 Oct 2022, Tim Allison wrote: > > > Happy to chat. Please put them in touch. > > > > Excellent, thanks Tim! > > > > Other than your past talks, have we got any info (eg on the wiki?) about > > how to run the regression corpus? > > > > > I've been really impressed with what the POI team has done migrating > > > from ant to gradle. On Tika, I don't think we have any special needs > > > that would require deep gradle knowledge, but given the number of > > > modules now, it will be non-trivial. Also, I take Nick D's concerns > > > seriously. > > > > We don't have to swap from Maven - they have a plugin that integrates it > > > > Nick > > >
Re: Possibly speeding up tests with Gradle - anyone interested?
Is there a branch started on the Gradle thing? I have some cycles and can use them to upgrade to Gradle on Tika, if desired. On Thu, Oct 6, 2022 at 8:50 AM Nick Burch wrote: > On Thu, 6 Oct 2022, Tim Allison wrote: > > Happy to chat. Please put them in touch. > > Excellent, thanks Tim! > > Other than your past talks, have we got any info (eg on the wiki?) about > how to run the regression corpus? > > > I've been really impressed with what the POI team has done migrating > > from ant to gradle. On Tika, I don't think we have any special needs > > that would require deep gradle knowledge, but given the number of > > modules now, it will be non-trivial. Also, I take Nick D's concerns > > seriously. > > We don't have to swap from Maven - they have a plugin that integrates it > > Nick >
Re: Possibly speeding up tests with Gradle - anyone interested?
On Thu, 6 Oct 2022, Tim Allison wrote: Happy to chat. Please put them in touch. Excellent, thanks Tim! Other than your past talks, have we got any info (eg on the wiki?) about how to run the regression corpus? I've been really impressed with what the POI team has done migrating from ant to gradle. On Tika, I don't think we have any special needs that would require deep gradle knowledge, but given the number of modules now, it will be non-trivial. Also, I take Nick D's concerns seriously. We don't have to swap from Maven - they have a plugin that integrates it Nick
Re: Possibly speeding up tests with Gradle - anyone interested?
On Wed, 5 Oct 2022, Nicholas DiPiazza wrote: Are they offering the Gradle Build Cache stuff free for apache projects? There's an announcement at ApacheCon in about an hour... I think the Infra team are still working out the details on how it'll all work. However, there's an additional offer to Tika for us to get some help on our tests, especially the regression run. (I think it's open to other ASF projects with "interesting" tests but we're the first ones to ask!) Nick
Re: Possibly speeding up tests with Gradle - anyone interested?
On Wed, 5 Oct 2022, Oleg Tikhonov wrote: Honestly I am trying to port our project to gradle. But it goes not well. It is good idea. Is some folk can help, we can do it together. Apparently Gradle Enterprise works with both Gradle and Maven! So we don't even have to change our build - https://docs.gradle.com/enterprise/maven-extension/ Nick
Re: Possibly speeding up tests with Gradle - anyone interested?
Happy to chat. Please put them in touch. I've been really impressed with what the POI team has done migrating from ant to gradle. On Tika, I don't think we have any special needs that would require deep gradle knowledge, but given the number of modules now, it will be non-trivial. Also, I take Nick D's concerns seriously. If there's something cheap and easy to do with our current build, great. If we're looking at a full migration, have at it on a dev branch! Perhaps we can put that in a Tika 3.x that requires Java 11...early next year? Cheers, Tim On Wed, Oct 5, 2022 at 3:05 PM Nick Burch wrote: > > Hi All > > At ApacheCon this week, a Bob and myself ended up chatting with the folks > from Gradle, who are keen to help ASF projects, and are discussing with > the Infra team. > > The easier bit - they think they might be able to help speed up our maven > build, especially the running of tests. Anyone have some time to give that > a try? Will pass details along to anyone with the volunteer cycles > > The interesting bit - we told them about the regression corpus, and they > got very excited as it sounds completely different to most of their normal > "my build is slow" type problems. The size of it, and the fact that it > isn't a simple pass/fail, seemed to catch their interest. Anyone (though > probably only Tim...) intersted in talking them through how it works, and > maybe getting one of their team access to the VM? > > Cheers > Nick
Re: Possibly speeding up tests with Gradle - anyone interested?
Yeah i'd be interested in the details. Are they offering the Gradle Build Cache stuff free for apache projects? In my experience, Gradle actually can slow things down quite a bit in general day-to-day development. But you get tradeoff of having a much more functional build process (tasks written in Kotlin). Example on one of my builds, a simple *./gradlew tasks* Took 18 seconds most of that time sat around derping in the Configuring step. It can also leak Gradle daemons like a Siv. Example from my same project: > Starting a Gradle Daemon, 4 busy and 18 incompatible Daemons could not be reused, use --status for details They sold us Gradle Build Cache but we quickly abandoned it because it kept causing errors and licensing problems. New versions of gradle result in full redownload all the deps. Very interested to hear how they aim to improve speed. On Wed, Oct 5, 2022 at 2:05 PM Nick Burch wrote: > Hi All > > At ApacheCon this week, a Bob and myself ended up chatting with the folks > from Gradle, who are keen to help ASF projects, and are discussing with > the Infra team. > > The easier bit - they think they might be able to help speed up our maven > build, especially the running of tests. Anyone have some time to give that > a try? Will pass details along to anyone with the volunteer cycles > > The interesting bit - we told them about the regression corpus, and they > got very excited as it sounds completely different to most of their normal > "my build is slow" type problems. The size of it, and the fact that it > isn't a simple pass/fail, seemed to catch their interest. Anyone (though > probably only Tim...) intersted in talking them through how it works, and > maybe getting one of their team access to the VM? > > Cheers > Nick >
Re: Possibly speeding up tests with Gradle - anyone interested?
Hi Nick, Honestly I am trying to port our project to gradle. But it goes not well. It is good idea. Is some folk can help, we can do it together. +1 Cheers, Oleg On Wed, Oct 5, 2022, 22:05 Nick Burch wrote: > Hi All > > At ApacheCon this week, a Bob and myself ended up chatting with the folks > from Gradle, who are keen to help ASF projects, and are discussing with > the Infra team. > > The easier bit - they think they might be able to help speed up our maven > build, especially the running of tests. Anyone have some time to give that > a try? Will pass details along to anyone with the volunteer cycles > > The interesting bit - we told them about the regression corpus, and they > got very excited as it sounds completely different to most of their normal > "my build is slow" type problems. The size of it, and the fact that it > isn't a simple pass/fail, seemed to catch their interest. Anyone (though > probably only Tim...) intersted in talking them through how it works, and > maybe getting one of their team access to the VM? > > Cheers > Nick >
Possibly speeding up tests with Gradle - anyone interested?
Hi All At ApacheCon this week, a Bob and myself ended up chatting with the folks from Gradle, who are keen to help ASF projects, and are discussing with the Infra team. The easier bit - they think they might be able to help speed up our maven build, especially the running of tests. Anyone have some time to give that a try? Will pass details along to anyone with the volunteer cycles The interesting bit - we told them about the regression corpus, and they got very excited as it sounds completely different to most of their normal "my build is slow" type problems. The size of it, and the fact that it isn't a simple pass/fail, seemed to catch their interest. Anyone (though probably only Tim...) intersted in talking them through how it works, and maybe getting one of their team access to the VM? Cheers Nick