+1 on Java 11 for branch-3.3 I'd also like to start addressing the test instability by addressing collisions between tests. Currently the tests are working in the same shared resources either directly through the system properties like test.build.data, indirectly through utility classes like GenericTestUtils, or modifying shared resources within target/test-classes. I've been working towards cleaning up flaky tests, and I plan on submitting some patches today. ________________________________ From: Steve Loughran <ste...@cloudera.com.INVALID> Sent: Monday, May 9, 2022 7:50 AM To: Ayush Saxena <ayush...@gmail.com> Cc: Gautham Banasandra <gaur...@apache.org>; Hadoop Common <common-dev@hadoop.apache.org>; Hdfs-dev <hdfs-...@hadoop.apache.org>; yarn-dev <yarn-...@hadoop.apache.org>; d...@hadoop.apache.org <d...@hadoop.apache.org> Subject: Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
how about for trunk we -wait for it- declare it java11 only? do that and we cut out a lot of the CI build. we would have to mandate a test run through branch-3.3 before any cherrypick there On Sat, 7 May 2022 at 11:19, Ayush Saxena <ayush...@gmail.com> wrote: > Three for trunk: > Java-8 > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/ > > Java-11 > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/ > > ARM > > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/ > > -Ayush > > > > > On 07-May-2022, at 11:49 AM, Gautham Banasandra <gaur...@apache.org> > wrote: > > > > Hi all, > > > > Although validating cross platform builds at pre-commit > > would be the most effective approach, I understand the > > huge disadvantage caused by the slowdown. The best > > way to tackle this would be to enable parallel builds for > > the different platforms. I had given it a shot about a year > > ago[1], it didn't go well and ran into all sorts of random > > errors. I think we should make the parallel builds run on > > different agents as opposed to starting the builds parallelly > > on the same agent (which is what I had done earlier). > > > > So, I'll settle down to integrating the full suite of platform > > builds into the nightly builds. Could anyone please point > > me to the Jenkins job for this? > > > > [1] = https://github.com/apache/hadoop/pull/3166 > > > > Thanks, > > --Gautham > > > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena <ayush...@gmail.com> wrote: > >> > >> From functional point of view it does makes sense to validate all the > >> platforms as part of the builds, but the Pre commits builds taking time > is > >> now no longer a small things, In past one year or may be two, we have > >> already increased it more than twice as compared to what it was before, > If > >> someone has a change in HDFS, which includes both hdfs-client & > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 > hours. > >> With the current state, I don't think we should just go and add these > >> extra overheads. Having them as part of the nightly builds does makes > sense > >> for now. > >> > >> In future if we feel there is a strong need for this and we start to see > >> very frequent failures in some other platforms and we are left with no > >> other option but to integrate it in our pre-commit jobs, we can explore > >> having these build phases running in parallel, along with trying other > >> phases also to run in parallel like compilation/javadoc builds of JDK-8 > & > >> JDK-11 can run in parallel and may be explore other opportunities as > well > >> to compensate for this time. > >> > >> For now lets just integrate it our nightly builds only and circle back > >> again here in future if the need be. > >> > >> -Ayush > >> > >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang <weic...@apache.org> > wrote: > >>> > >>> Running builds for all platforms for each and every PR seems too > >>> excessive. > >>> > >>> How about doing all platform builds in the nightly jobs? > >>> > >>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran > <ste...@cloudera.com.invalid > >>>> > >>> wrote: > >>> > >>>> I'm not enthusiastic here as it not only makes the builds slower, it > >>>> reduces the #of builds we can through a day > >>>> > >>>> one thing I am wondering is could we remove java8 support on some > >>> branches? > >>>> > >>>> make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and > >>> this > >>>> summers branch-3.3 release (which I'd rebadge 3.4) would ship as java > 11 > >>>> only. > >>>> that would cut buld and test time for those trunk PRs in half...after > >>> which > >>>> the preospect of building on more than one platform becomes more > viable. > >>>> > >>>> On Thu, 5 May 2022 at 15:34, Gautham Banasandra <gaur...@apache.org> > >>>> wrote: > >>>> > >>>>> Hi Hadoop devs, > >>>>> > >>>>> Last week, there was a Hadoop build failure on Debian 10 caused by > >>>>> https://github.com/apache/hadoop/pull/3988. In > >>> dev-support/jenkins.sh, > >>>>> there's the capability to build and test Hadoop across the supported > >>>>> platforms. Currently, we're limiting this only for those PRs having > >>> only > >>>>> C/C++ changes[1], since C/C++ changes are more likely to cause > >>>>> cross-platform build issues and bypassing the full platform build for > >>> non > >>>>> C/C++ PRs would save a great deal of CI time. However, the build > >>> failure > >>>>> caused by PR #3988 motivates me to enable the capability to build and > >>>>> test Hadoop for all the supported platforms for ALL the PRs. > >>>>> > >>>>> While this may cause longer CI run duration for each PR, it would > >>>>> immensely minimize the risk of breaking Hadoop across platforms and > >>>>> saves us a lot of debugging time. Kindly post your opinion regarding > >>> this > >>>>> and I'll move to enable this capability for all PRs if the response > is > >>>>> sufficiently positive. > >>>>> > >>>>> [1] = > >>>>> > >>>>> > >>>> > >>> > https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 > >>>>> > >>>>> > >>>>> Thanks, > >>>>> --Gautham > >>>>> > >>>> > >>> > >> >
