how about for trunk we -wait for it- declare it java11 only? do that and we cut out a lot of the CI build.
we would have to mandate a test run through branch-3.3 before any cherrypick there On Sat, 7 May 2022 at 11:19, Ayush Saxena <ayush...@gmail.com> wrote: > Three for trunk: > Java-8 > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/ > > Java-11 > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/ > > ARM > > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/ > > -Ayush > > > > > On 07-May-2022, at 11:49 AM, Gautham Banasandra <gaur...@apache.org> > wrote: > > > > Hi all, > > > > Although validating cross platform builds at pre-commit > > would be the most effective approach, I understand the > > huge disadvantage caused by the slowdown. The best > > way to tackle this would be to enable parallel builds for > > the different platforms. I had given it a shot about a year > > ago[1], it didn't go well and ran into all sorts of random > > errors. I think we should make the parallel builds run on > > different agents as opposed to starting the builds parallelly > > on the same agent (which is what I had done earlier). > > > > So, I'll settle down to integrating the full suite of platform > > builds into the nightly builds. Could anyone please point > > me to the Jenkins job for this? > > > > [1] = https://github.com/apache/hadoop/pull/3166 > > > > Thanks, > > --Gautham > > > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena <ayush...@gmail.com> wrote: > >> > >> From functional point of view it does makes sense to validate all the > >> platforms as part of the builds, but the Pre commits builds taking time > is > >> now no longer a small things, In past one year or may be two, we have > >> already increased it more than twice as compared to what it was before, > If > >> someone has a change in HDFS, which includes both hdfs-client & > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 > hours. > >> With the current state, I don't think we should just go and add these > >> extra overheads. Having them as part of the nightly builds does makes > sense > >> for now. > >> > >> In future if we feel there is a strong need for this and we start to see > >> very frequent failures in some other platforms and we are left with no > >> other option but to integrate it in our pre-commit jobs, we can explore > >> having these build phases running in parallel, along with trying other > >> phases also to run in parallel like compilation/javadoc builds of JDK-8 > & > >> JDK-11 can run in parallel and may be explore other opportunities as > well > >> to compensate for this time. > >> > >> For now lets just integrate it our nightly builds only and circle back > >> again here in future if the need be. > >> > >> -Ayush > >> > >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang <weic...@apache.org> > wrote: > >>> > >>> Running builds for all platforms for each and every PR seems too > >>> excessive. > >>> > >>> How about doing all platform builds in the nightly jobs? > >>> > >>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran > <ste...@cloudera.com.invalid > >>>> > >>> wrote: > >>> > >>>> I'm not enthusiastic here as it not only makes the builds slower, it > >>>> reduces the #of builds we can through a day > >>>> > >>>> one thing I am wondering is could we remove java8 support on some > >>> branches? > >>>> > >>>> make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and > >>> this > >>>> summers branch-3.3 release (which I'd rebadge 3.4) would ship as java > 11 > >>>> only. > >>>> that would cut buld and test time for those trunk PRs in half...after > >>> which > >>>> the preospect of building on more than one platform becomes more > viable. > >>>> > >>>> On Thu, 5 May 2022 at 15:34, Gautham Banasandra <gaur...@apache.org> > >>>> wrote: > >>>> > >>>>> Hi Hadoop devs, > >>>>> > >>>>> Last week, there was a Hadoop build failure on Debian 10 caused by > >>>>> https://github.com/apache/hadoop/pull/3988. In > >>> dev-support/jenkins.sh, > >>>>> there's the capability to build and test Hadoop across the supported > >>>>> platforms. Currently, we're limiting this only for those PRs having > >>> only > >>>>> C/C++ changes[1], since C/C++ changes are more likely to cause > >>>>> cross-platform build issues and bypassing the full platform build for > >>> non > >>>>> C/C++ PRs would save a great deal of CI time. However, the build > >>> failure > >>>>> caused by PR #3988 motivates me to enable the capability to build and > >>>>> test Hadoop for all the supported platforms for ALL the PRs. > >>>>> > >>>>> While this may cause longer CI run duration for each PR, it would > >>>>> immensely minimize the risk of breaking Hadoop across platforms and > >>>>> saves us a lot of debugging time. Kindly post your opinion regarding > >>> this > >>>>> and I'll move to enable this capability for all PRs if the response > is > >>>>> sufficiently positive. > >>>>> > >>>>> [1] = > >>>>> > >>>>> > >>>> > >>> > https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 > >>>>> > >>>>> > >>>>> Thanks, > >>>>> --Gautham > >>>>> > >>>> > >>> > >> >
