>From functional point of view it does makes sense to validate all the platforms as part of the builds, but the Pre commits builds taking time is now no longer a small things, In past one year or may be two, we have already increased it more than twice as compared to what it was before, If someone has a change in HDFS, which includes both hdfs-client & hadoop-hdfs, it takes more than 5 hours, which long back was around 2 hours. With the current state, I don't think we should just go and add these extra overheads. Having them as part of the nightly builds does makes sense for now.
In future if we feel there is a strong need for this and we start to see very frequent failures in some other platforms and we are left with no other option but to integrate it in our pre-commit jobs, we can explore having these build phases running in parallel, along with trying other phases also to run in parallel like compilation/javadoc builds of JDK-8 & JDK-11 can run in parallel and may be explore other opportunities as well to compensate for this time. For now lets just integrate it our nightly builds only and circle back again here in future if the need be. -Ayush On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang <weic...@apache.org> wrote: > Running builds for all platforms for each and every PR seems too excessive. > > How about doing all platform builds in the nightly jobs? > > On Fri, May 6, 2022 at 8:02 AM Steve Loughran <ste...@cloudera.com.invalid > > > wrote: > > > I'm not enthusiastic here as it not only makes the builds slower, it > > reduces the #of builds we can through a day > > > > one thing I am wondering is could we remove java8 support on some > branches? > > > > make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and > this > > summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11 > > only. > > that would cut buld and test time for those trunk PRs in half...after > which > > the preospect of building on more than one platform becomes more viable. > > > > On Thu, 5 May 2022 at 15:34, Gautham Banasandra <gaur...@apache.org> > > wrote: > > > > > Hi Hadoop devs, > > > > > > Last week, there was a Hadoop build failure on Debian 10 caused by > > > https://github.com/apache/hadoop/pull/3988. In dev-support/jenkins.sh, > > > there's the capability to build and test Hadoop across the supported > > > platforms. Currently, we're limiting this only for those PRs having > only > > > C/C++ changes[1], since C/C++ changes are more likely to cause > > > cross-platform build issues and bypassing the full platform build for > non > > > C/C++ PRs would save a great deal of CI time. However, the build > failure > > > caused by PR #3988 motivates me to enable the capability to build and > > > test Hadoop for all the supported platforms for ALL the PRs. > > > > > > While this may cause longer CI run duration for each PR, it would > > > immensely minimize the risk of breaking Hadoop across platforms and > > > saves us a lot of debugging time. Kindly post your opinion regarding > this > > > and I'll move to enable this capability for all PRs if the response is > > > sufficiently positive. > > > > > > [1] = > > > > > > > > > https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 > > > > > > > > > Thanks, > > > --Gautham > > > > > >
