Have raised HADOOP-18241 There is a WIP PR linked to it as well.... https://issues.apache.org/jira/browse/HADOOP-18241
On Wed, 11 May 2022 at 01:28, Ayush Saxena <ayush...@gmail.com> wrote: > +1 for moving to Java 11 for trunk. > If nobody objects by EOW will create a jira for that > > -Ayush > > On 10-May-2022, at 11:18 PM, Gautham Banasandra <gaur...@apache.org> > wrote: > > > +1 for moving to Java 11. > > Thanks, > --Gautham > > On Mon, 9 May 2022 at 19:50, Steve Vaughan <em...@stevevaughan.me> wrote: > >> +1 on Java 11 for branch-3.3 >> >> I'd also like to start addressing the test instability by addressing >> collisions between tests. Currently the tests are working in the same >> shared resources either directly through the system properties like >> test.build.data, indirectly through utility classes like GenericTestUtils, >> or modifying shared resources within target/test-classes. I've been >> working towards cleaning up flaky tests, and I plan on submitting some >> patches today. >> ------------------------------ >> *From:* Steve Loughran <ste...@cloudera.com.INVALID> >> *Sent:* Monday, May 9, 2022 7:50 AM >> *To:* Ayush Saxena <ayush...@gmail.com> >> *Cc:* Gautham Banasandra <gaur...@apache.org>; Hadoop Common < >> common-...@hadoop.apache.org>; Hdfs-dev <hdfs-dev@hadoop.apache.org>; >> yarn-dev <yarn-...@hadoop.apache.org>; d...@hadoop.apache.org < >> d...@hadoop.apache.org> >> *Subject:* Re: [DISCUSS] Enabling all platform builds in CI for all >> Hadoop PRs >> >> how about for trunk we -wait for it- declare it java11 only? >> >> do that and we cut out a lot of the CI build. >> >> we would have to mandate a test run through branch-3.3 before any >> cherrypick there >> >> On Sat, 7 May 2022 at 11:19, Ayush Saxena <ayush...@gmail.com> wrote: >> >> > Three for trunk: >> > Java-8 >> > >> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/ >> > >> > Java-11 >> > >> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/ >> > >> > ARM >> > >> > >> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/ >> > >> > -Ayush >> > >> > > >> > > On 07-May-2022, at 11:49 AM, Gautham Banasandra <gaur...@apache.org> >> > wrote: >> > > >> > > Hi all, >> > > >> > > Although validating cross platform builds at pre-commit >> > > would be the most effective approach, I understand the >> > > huge disadvantage caused by the slowdown. The best >> > > way to tackle this would be to enable parallel builds for >> > > the different platforms. I had given it a shot about a year >> > > ago[1], it didn't go well and ran into all sorts of random >> > > errors. I think we should make the parallel builds run on >> > > different agents as opposed to starting the builds parallelly >> > > on the same agent (which is what I had done earlier). >> > > >> > > So, I'll settle down to integrating the full suite of platform >> > > builds into the nightly builds. Could anyone please point >> > > me to the Jenkins job for this? >> > > >> > > [1] = https://github.com/apache/hadoop/pull/3166 >> > > >> > > Thanks, >> > > --Gautham >> > > >> > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena <ayush...@gmail.com> >> wrote: >> > >> >> > >> From functional point of view it does makes sense to validate all the >> > >> platforms as part of the builds, but the Pre commits builds taking >> time >> > is >> > >> now no longer a small things, In past one year or may be two, we have >> > >> already increased it more than twice as compared to what it was >> before, >> > If >> > >> someone has a change in HDFS, which includes both hdfs-client & >> > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 >> > hours. >> > >> With the current state, I don't think we should just go and add these >> > >> extra overheads. Having them as part of the nightly builds does makes >> > sense >> > >> for now. >> > >> >> > >> In future if we feel there is a strong need for this and we start to >> see >> > >> very frequent failures in some other platforms and we are left with >> no >> > >> other option but to integrate it in our pre-commit jobs, we can >> explore >> > >> having these build phases running in parallel, along with trying >> other >> > >> phases also to run in parallel like compilation/javadoc builds of >> JDK-8 >> > & >> > >> JDK-11 can run in parallel and may be explore other opportunities as >> > well >> > >> to compensate for this time. >> > >> >> > >> For now lets just integrate it our nightly builds only and circle >> back >> > >> again here in future if the need be. >> > >> >> > >> -Ayush >> > >> >> > >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang <weic...@apache.org> >> > wrote: >> > >>> >> > >>> Running builds for all platforms for each and every PR seems too >> > >>> excessive. >> > >>> >> > >>> How about doing all platform builds in the nightly jobs? >> > >>> >> > >>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran >> > <ste...@cloudera.com.invalid >> > >>>> >> > >>> wrote: >> > >>> >> > >>>> I'm not enthusiastic here as it not only makes the builds slower, >> it >> > >>>> reduces the #of builds we can through a day >> > >>>> >> > >>>> one thing I am wondering is could we remove java8 support on some >> > >>> branches? >> > >>>> >> > >>>> make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, >> and >> > >>> this >> > >>>> summers branch-3.3 release (which I'd rebadge 3.4) would ship as >> java >> > 11 >> > >>>> only. >> > >>>> that would cut buld and test time for those trunk PRs in >> half...after >> > >>> which >> > >>>> the preospect of building on more than one platform becomes more >> > viable. >> > >>>> >> > >>>> On Thu, 5 May 2022 at 15:34, Gautham Banasandra < >> gaur...@apache.org> >> > >>>> wrote: >> > >>>> >> > >>>>> Hi Hadoop devs, >> > >>>>> >> > >>>>> Last week, there was a Hadoop build failure on Debian 10 caused by >> > >>>>> https://github.com/apache/hadoop/pull/3988. In >> > >>> dev-support/jenkins.sh, >> > >>>>> there's the capability to build and test Hadoop across the >> supported >> > >>>>> platforms. Currently, we're limiting this only for those PRs >> having >> > >>> only >> > >>>>> C/C++ changes[1], since C/C++ changes are more likely to cause >> > >>>>> cross-platform build issues and bypassing the full platform build >> for >> > >>> non >> > >>>>> C/C++ PRs would save a great deal of CI time. However, the build >> > >>> failure >> > >>>>> caused by PR #3988 motivates me to enable the capability to build >> and >> > >>>>> test Hadoop for all the supported platforms for ALL the PRs. >> > >>>>> >> > >>>>> While this may cause longer CI run duration for each PR, it would >> > >>>>> immensely minimize the risk of breaking Hadoop across platforms >> and >> > >>>>> saves us a lot of debugging time. Kindly post your opinion >> regarding >> > >>> this >> > >>>>> and I'll move to enable this capability for all PRs if the >> response >> > is >> > >>>>> sufficiently positive. >> > >>>>> >> > >>>>> [1] = >> > >>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 >> > >>>>> >> > >>>>> >> > >>>>> Thanks, >> > >>>>> --Gautham >> > >>>>> >> > >>>> >> > >>> >> > >> >> > >> >
