+1 for moving to Java 11.

Thanks,
--Gautham

On Mon, 9 May 2022 at 19:50, Steve Vaughan <em...@stevevaughan.me> wrote:

> +1 on Java 11 for branch-3.3
>
> I'd also like to start addressing the test instability by addressing
> collisions between tests.  Currently the tests are working in the same
> shared resources either directly through the system properties like
> test.build.data, indirectly through utility classes like GenericTestUtils,
> or modifying shared resources within target/test-classes.  I've been
> working towards cleaning up flaky tests, and I plan on submitting some
> patches today.
> ------------------------------
> *From:* Steve Loughran <ste...@cloudera.com.INVALID>
> *Sent:* Monday, May 9, 2022 7:50 AM
> *To:* Ayush Saxena <ayush...@gmail.com>
> *Cc:* Gautham Banasandra <gaur...@apache.org>; Hadoop Common <
> common-...@hadoop.apache.org>; Hdfs-dev <hdfs-dev@hadoop.apache.org>;
> yarn-dev <yarn-...@hadoop.apache.org>; d...@hadoop.apache.org <
> d...@hadoop.apache.org>
> *Subject:* Re: [DISCUSS] Enabling all platform builds in CI for all
> Hadoop PRs
>
> how about for trunk we -wait for it- declare it java11 only?
>
> do that and we cut out a lot of the CI build.
>
> we would have to mandate a test run through branch-3.3 before any
> cherrypick there
>
> On Sat, 7 May 2022 at 11:19, Ayush Saxena <ayush...@gmail.com> wrote:
>
> > Three for trunk:
> > Java-8
> >
> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/
> >
> > Java-11
> >
> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/
> >
> > ARM
> >
> > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/
> >
> > -Ayush
> >
> > >
> > > On 07-May-2022, at 11:49 AM, Gautham Banasandra <gaur...@apache.org>
> > wrote:
> > >
> > > Hi all,
> > >
> > > Although validating cross platform builds at pre-commit
> > > would be the most effective approach, I understand the
> > > huge disadvantage caused by the slowdown. The best
> > > way to tackle this would be to enable parallel builds for
> > > the different platforms. I had given it a shot about a year
> > > ago[1], it didn't go well and ran into all sorts of random
> > > errors. I think we should make the parallel builds run on
> > > different agents as opposed to starting the builds parallelly
> > > on the same agent (which is what I had done earlier).
> > >
> > > So, I'll settle down to integrating the full suite of platform
> > > builds into the nightly builds. Could anyone please point
> > > me to the Jenkins job for this?
> > >
> > > [1] = https://github.com/apache/hadoop/pull/3166
> > >
> > > Thanks,
> > > --Gautham
> > >
> > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena <ayush...@gmail.com> wrote:
> > >>
> > >> From functional point of view it does makes sense to validate all the
> > >> platforms as part of the builds, but the Pre commits builds taking
> time
> > is
> > >> now no longer a small things, In past one year or may be two, we have
> > >> already increased it more than twice as compared to what it was
> before,
> > If
> > >> someone has a change in HDFS, which includes both hdfs-client &
> > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2
> > hours.
> > >> With the current state, I don't think we should just go and add these
> > >> extra overheads. Having them as part of the nightly builds does makes
> > sense
> > >> for now.
> > >>
> > >> In future if we feel there is a strong need for this and we start to
> see
> > >> very frequent failures in some other platforms and we are left with no
> > >> other option but to integrate it in our pre-commit jobs, we can
> explore
> > >> having these build phases running in parallel, along with trying other
> > >> phases also to run in parallel like compilation/javadoc builds of
> JDK-8
> > &
> > >> JDK-11 can run in parallel and may be explore other opportunities as
> > well
> > >> to compensate for this time.
> > >>
> > >> For now lets just integrate it our nightly builds only and circle back
> > >> again here in future if the need be.
> > >>
> > >> -Ayush
> > >>
> > >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang <weic...@apache.org>
> > wrote:
> > >>>
> > >>> Running builds for all platforms for each and every PR seems too
> > >>> excessive.
> > >>>
> > >>> How about doing all platform builds in the nightly jobs?
> > >>>
> > >>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran
> > <ste...@cloudera.com.invalid
> > >>>>
> > >>> wrote:
> > >>>
> > >>>> I'm not enthusiastic here as it not only makes the builds slower, it
> > >>>> reduces the #of builds we can through a day
> > >>>>
> > >>>> one thing I am wondering is could we remove java8 support on some
> > >>> branches?
> > >>>>
> > >>>> make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build,
> and
> > >>> this
> > >>>> summers branch-3.3 release (which I'd rebadge 3.4) would ship as
> java
> > 11
> > >>>> only.
> > >>>> that would cut buld and test time for those trunk PRs in
> half...after
> > >>> which
> > >>>> the preospect of building on more than one platform becomes more
> > viable.
> > >>>>
> > >>>> On Thu, 5 May 2022 at 15:34, Gautham Banasandra <gaur...@apache.org
> >
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Hadoop devs,
> > >>>>>
> > >>>>> Last week, there was a Hadoop build failure on Debian 10 caused by
> > >>>>> https://github.com/apache/hadoop/pull/3988. In
> > >>> dev-support/jenkins.sh,
> > >>>>> there's the capability to build and test Hadoop across the
> supported
> > >>>>> platforms. Currently, we're limiting this only for those PRs having
> > >>> only
> > >>>>> C/C++ changes[1], since C/C++ changes are more likely to cause
> > >>>>> cross-platform build issues and bypassing the full platform build
> for
> > >>> non
> > >>>>> C/C++ PRs would save a great deal of CI time. However, the build
> > >>> failure
> > >>>>> caused by PR #3988 motivates me to enable the capability to build
> and
> > >>>>> test Hadoop for all the supported platforms for ALL the PRs.
> > >>>>>
> > >>>>> While this may cause longer CI run duration for each PR, it would
> > >>>>> immensely minimize the risk of breaking Hadoop across platforms and
> > >>>>> saves us a lot of debugging time. Kindly post your opinion
> regarding
> > >>> this
> > >>>>> and I'll move to enable this capability for all PRs if the response
> > is
> > >>>>> sufficiently positive.
> > >>>>>
> > >>>>> [1] =
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103
> > >>>>>
> > >>>>>
> > >>>>> Thanks,
> > >>>>> --Gautham
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>

Reply via email to