Hi all, Although validating cross platform builds at pre-commit would be the most effective approach, I understand the huge disadvantage caused by the slowdown. The best way to tackle this would be to enable parallel builds for the different platforms. I had given it a shot about a year ago[1], it didn't go well and ran into all sorts of random errors. I think we should make the parallel builds run on different agents as opposed to starting the builds parallelly on the same agent (which is what I had done earlier).
So, I'll settle down to integrating the full suite of platform builds into the nightly builds. Could anyone please point me to the Jenkins job for this? [1] = https://github.com/apache/hadoop/pull/3166 Thanks, --Gautham On Fri, 6 May 2022 at 21:04, Ayush Saxena <ayush...@gmail.com> wrote: > From functional point of view it does makes sense to validate all the > platforms as part of the builds, but the Pre commits builds taking time is > now no longer a small things, In past one year or may be two, we have > already increased it more than twice as compared to what it was before, If > someone has a change in HDFS, which includes both hdfs-client & > hadoop-hdfs, it takes more than 5 hours, which long back was around 2 hours. > With the current state, I don't think we should just go and add these > extra overheads. Having them as part of the nightly builds does makes sense > for now. > > In future if we feel there is a strong need for this and we start to see > very frequent failures in some other platforms and we are left with no > other option but to integrate it in our pre-commit jobs, we can explore > having these build phases running in parallel, along with trying other > phases also to run in parallel like compilation/javadoc builds of JDK-8 & > JDK-11 can run in parallel and may be explore other opportunities as well > to compensate for this time. > > For now lets just integrate it our nightly builds only and circle back > again here in future if the need be. > > -Ayush > > On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang <weic...@apache.org> wrote: > >> Running builds for all platforms for each and every PR seems too >> excessive. >> >> How about doing all platform builds in the nightly jobs? >> >> On Fri, May 6, 2022 at 8:02 AM Steve Loughran <ste...@cloudera.com.invalid >> > >> wrote: >> >> > I'm not enthusiastic here as it not only makes the builds slower, it >> > reduces the #of builds we can through a day >> > >> > one thing I am wondering is could we remove java8 support on some >> branches? >> > >> > make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and >> this >> > summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11 >> > only. >> > that would cut buld and test time for those trunk PRs in half...after >> which >> > the preospect of building on more than one platform becomes more viable. >> > >> > On Thu, 5 May 2022 at 15:34, Gautham Banasandra <gaur...@apache.org> >> > wrote: >> > >> > > Hi Hadoop devs, >> > > >> > > Last week, there was a Hadoop build failure on Debian 10 caused by >> > > https://github.com/apache/hadoop/pull/3988. In >> dev-support/jenkins.sh, >> > > there's the capability to build and test Hadoop across the supported >> > > platforms. Currently, we're limiting this only for those PRs having >> only >> > > C/C++ changes[1], since C/C++ changes are more likely to cause >> > > cross-platform build issues and bypassing the full platform build for >> non >> > > C/C++ PRs would save a great deal of CI time. However, the build >> failure >> > > caused by PR #3988 motivates me to enable the capability to build and >> > > test Hadoop for all the supported platforms for ALL the PRs. >> > > >> > > While this may cause longer CI run duration for each PR, it would >> > > immensely minimize the risk of breaking Hadoop across platforms and >> > > saves us a lot of debugging time. Kindly post your opinion regarding >> this >> > > and I'll move to enable this capability for all PRs if the response is >> > > sufficiently positive. >> > > >> > > [1] = >> > > >> > > >> > >> https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 >> > > >> > > >> > > Thanks, >> > > --Gautham >> > > >> > >> >
