Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
Have raised HADOOP-18241 There is a WIP PR linked to it as well https://issues.apache.org/jira/browse/HADOOP-18241 On Wed, 11 May 2022 at 01:28, Ayush Saxena wrote: > +1 for moving to Java 11 for trunk. > If nobody objects by EOW will create a jira for that > > -Ayush > > On 10-May-2022, at 11:18 PM, Gautham Banasandra > wrote: > > > +1 for moving to Java 11. > > Thanks, > --Gautham > > On Mon, 9 May 2022 at 19:50, Steve Vaughan wrote: > >> +1 on Java 11 for branch-3.3 >> >> I'd also like to start addressing the test instability by addressing >> collisions between tests. Currently the tests are working in the same >> shared resources either directly through the system properties like >> test.build.data, indirectly through utility classes like GenericTestUtils, >> or modifying shared resources within target/test-classes. I've been >> working towards cleaning up flaky tests, and I plan on submitting some >> patches today. >> -- >> *From:* Steve Loughran >> *Sent:* Monday, May 9, 2022 7:50 AM >> *To:* Ayush Saxena >> *Cc:* Gautham Banasandra ; Hadoop Common < >> common-...@hadoop.apache.org>; Hdfs-dev ; >> yarn-dev ; d...@hadoop.apache.org < >> d...@hadoop.apache.org> >> *Subject:* Re: [DISCUSS] Enabling all platform builds in CI for all >> Hadoop PRs >> >> how about for trunk we -wait for it- declare it java11 only? >> >> do that and we cut out a lot of the CI build. >> >> we would have to mandate a test run through branch-3.3 before any >> cherrypick there >> >> On Sat, 7 May 2022 at 11:19, Ayush Saxena wrote: >> >> > Three for trunk: >> > Java-8 >> > >> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/ >> > >> > Java-11 >> > >> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/ >> > >> > ARM >> > >> > >> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/ >> > >> > -Ayush >> > >> > > >> > > On 07-May-2022, at 11:49 AM, Gautham Banasandra >> > wrote: >> > > >> > > Hi all, >> > > >> > > Although validating cross platform builds at pre-commit >> > > would be the most effective approach, I understand the >> > > huge disadvantage caused by the slowdown. The best >> > > way to tackle this would be to enable parallel builds for >> > > the different platforms. I had given it a shot about a year >> > > ago[1], it didn't go well and ran into all sorts of random >> > > errors. I think we should make the parallel builds run on >> > > different agents as opposed to starting the builds parallelly >> > > on the same agent (which is what I had done earlier). >> > > >> > > So, I'll settle down to integrating the full suite of platform >> > > builds into the nightly builds. Could anyone please point >> > > me to the Jenkins job for this? >> > > >> > > [1] = https://github.com/apache/hadoop/pull/3166 >> > > >> > > Thanks, >> > > --Gautham >> > > >> > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena >> wrote: >> > >> >> > >> From functional point of view it does makes sense to validate all the >> > >> platforms as part of the builds, but the Pre commits builds taking >> time >> > is >> > >> now no longer a small things, In past one year or may be two, we have >> > >> already increased it more than twice as compared to what it was >> before, >> > If >> > >> someone has a change in HDFS, which includes both hdfs-client & >> > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 >> > hours. >> > >> With the current state, I don't think we should just go and add these >> > >> extra overheads. Having them as part of the nightly builds does makes >> > sense >> > >> for now. >> > >> >> > >> In future if we feel there is a strong need for this and we start to >> see >> > >> very frequent failures in some other platforms and we are left with >> no >> > >> other option but to integrate it in our pre-commit jobs, we can >> explore >> > >> having these build phases running in parallel, along with trying >> othe
Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
+1 for moving to Java 11 for trunk. If nobody objects by EOW will create a jira for that -Ayush > On 10-May-2022, at 11:18 PM, Gautham Banasandra wrote: > > > +1 for moving to Java 11. > > Thanks, > --Gautham > >> On Mon, 9 May 2022 at 19:50, Steve Vaughan wrote: >> +1 on Java 11 for branch-3.3 >> >> I'd also like to start addressing the test instability by addressing >> collisions between tests. Currently the tests are working in the same >> shared resources either directly through the system properties like >> test.build.data, indirectly through utility classes like GenericTestUtils, >> or modifying shared resources within target/test-classes. I've been working >> towards cleaning up flaky tests, and I plan on submitting some patches today. >> From: Steve Loughran >> Sent: Monday, May 9, 2022 7:50 AM >> To: Ayush Saxena >> Cc: Gautham Banasandra ; Hadoop Common >> ; Hdfs-dev ; >> yarn-dev ; d...@hadoop.apache.org >> >> Subject: Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs >> >> how about for trunk we -wait for it- declare it java11 only? >> >> do that and we cut out a lot of the CI build. >> >> we would have to mandate a test run through branch-3.3 before any >> cherrypick there >> >> On Sat, 7 May 2022 at 11:19, Ayush Saxena wrote: >> >> > Three for trunk: >> > Java-8 >> > >> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/ >> > >> > Java-11 >> > >> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/ >> > >> > ARM >> > >> > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/ >> > >> > -Ayush >> > >> > > >> > > On 07-May-2022, at 11:49 AM, Gautham Banasandra >> > wrote: >> > > >> > > Hi all, >> > > >> > > Although validating cross platform builds at pre-commit >> > > would be the most effective approach, I understand the >> > > huge disadvantage caused by the slowdown. The best >> > > way to tackle this would be to enable parallel builds for >> > > the different platforms. I had given it a shot about a year >> > > ago[1], it didn't go well and ran into all sorts of random >> > > errors. I think we should make the parallel builds run on >> > > different agents as opposed to starting the builds parallelly >> > > on the same agent (which is what I had done earlier). >> > > >> > > So, I'll settle down to integrating the full suite of platform >> > > builds into the nightly builds. Could anyone please point >> > > me to the Jenkins job for this? >> > > >> > > [1] = https://github.com/apache/hadoop/pull/3166 >> > > >> > > Thanks, >> > > --Gautham >> > > >> > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena wrote: >> > >> >> > >> From functional point of view it does makes sense to validate all the >> > >> platforms as part of the builds, but the Pre commits builds taking time >> > is >> > >> now no longer a small things, In past one year or may be two, we have >> > >> already increased it more than twice as compared to what it was before, >> > If >> > >> someone has a change in HDFS, which includes both hdfs-client & >> > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 >> > hours. >> > >> With the current state, I don't think we should just go and add these >> > >> extra overheads. Having them as part of the nightly builds does makes >> > sense >> > >> for now. >> > >> >> > >> In future if we feel there is a strong need for this and we start to see >> > >> very frequent failures in some other platforms and we are left with no >> > >> other option but to integrate it in our pre-commit jobs, we can explore >> > >> having these build phases running in parallel, along with trying other >> > >> phases also to run in parallel like compilation/javadoc builds of JDK-8 >> > & >> > >> JDK-11 can run in parallel and may be explore other opportunities as >> > well >> > >> to compensate for this time. >> > >> >> > >> For now lets just integrate it our nightly builds only and circle
Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
+1 for moving to Java 11. Thanks, --Gautham On Mon, 9 May 2022 at 19:50, Steve Vaughan wrote: > +1 on Java 11 for branch-3.3 > > I'd also like to start addressing the test instability by addressing > collisions between tests. Currently the tests are working in the same > shared resources either directly through the system properties like > test.build.data, indirectly through utility classes like GenericTestUtils, > or modifying shared resources within target/test-classes. I've been > working towards cleaning up flaky tests, and I plan on submitting some > patches today. > -- > *From:* Steve Loughran > *Sent:* Monday, May 9, 2022 7:50 AM > *To:* Ayush Saxena > *Cc:* Gautham Banasandra ; Hadoop Common < > common-...@hadoop.apache.org>; Hdfs-dev ; > yarn-dev ; d...@hadoop.apache.org < > d...@hadoop.apache.org> > *Subject:* Re: [DISCUSS] Enabling all platform builds in CI for all > Hadoop PRs > > how about for trunk we -wait for it- declare it java11 only? > > do that and we cut out a lot of the CI build. > > we would have to mandate a test run through branch-3.3 before any > cherrypick there > > On Sat, 7 May 2022 at 11:19, Ayush Saxena wrote: > > > Three for trunk: > > Java-8 > > > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/ > > > > Java-11 > > > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/ > > > > ARM > > > > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/ > > > > -Ayush > > > > > > > > On 07-May-2022, at 11:49 AM, Gautham Banasandra > > wrote: > > > > > > Hi all, > > > > > > Although validating cross platform builds at pre-commit > > > would be the most effective approach, I understand the > > > huge disadvantage caused by the slowdown. The best > > > way to tackle this would be to enable parallel builds for > > > the different platforms. I had given it a shot about a year > > > ago[1], it didn't go well and ran into all sorts of random > > > errors. I think we should make the parallel builds run on > > > different agents as opposed to starting the builds parallelly > > > on the same agent (which is what I had done earlier). > > > > > > So, I'll settle down to integrating the full suite of platform > > > builds into the nightly builds. Could anyone please point > > > me to the Jenkins job for this? > > > > > > [1] = https://github.com/apache/hadoop/pull/3166 > > > > > > Thanks, > > > --Gautham > > > > > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena wrote: > > >> > > >> From functional point of view it does makes sense to validate all the > > >> platforms as part of the builds, but the Pre commits builds taking > time > > is > > >> now no longer a small things, In past one year or may be two, we have > > >> already increased it more than twice as compared to what it was > before, > > If > > >> someone has a change in HDFS, which includes both hdfs-client & > > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 > > hours. > > >> With the current state, I don't think we should just go and add these > > >> extra overheads. Having them as part of the nightly builds does makes > > sense > > >> for now. > > >> > > >> In future if we feel there is a strong need for this and we start to > see > > >> very frequent failures in some other platforms and we are left with no > > >> other option but to integrate it in our pre-commit jobs, we can > explore > > >> having these build phases running in parallel, along with trying other > > >> phases also to run in parallel like compilation/javadoc builds of > JDK-8 > > & > > >> JDK-11 can run in parallel and may be explore other opportunities as > > well > > >> to compensate for this time. > > >> > > >> For now lets just integrate it our nightly builds only and circle back > > >> again here in future if the need be. > > >> > > >> -Ayush > > >> > > >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang > > wrote: > > >>> > > >>> Running builds for all platforms for each and every PR seems too > > >>> excessive. > > >>> > > >>> How about doing all platform builds in t
Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
how about for trunk we -wait for it- declare it java11 only? do that and we cut out a lot of the CI build. we would have to mandate a test run through branch-3.3 before any cherrypick there On Sat, 7 May 2022 at 11:19, Ayush Saxena wrote: > Three for trunk: > Java-8 > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/ > > Java-11 > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/ > > ARM > > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/ > > -Ayush > > > > > On 07-May-2022, at 11:49 AM, Gautham Banasandra > wrote: > > > > Hi all, > > > > Although validating cross platform builds at pre-commit > > would be the most effective approach, I understand the > > huge disadvantage caused by the slowdown. The best > > way to tackle this would be to enable parallel builds for > > the different platforms. I had given it a shot about a year > > ago[1], it didn't go well and ran into all sorts of random > > errors. I think we should make the parallel builds run on > > different agents as opposed to starting the builds parallelly > > on the same agent (which is what I had done earlier). > > > > So, I'll settle down to integrating the full suite of platform > > builds into the nightly builds. Could anyone please point > > me to the Jenkins job for this? > > > > [1] = https://github.com/apache/hadoop/pull/3166 > > > > Thanks, > > --Gautham > > > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena wrote: > >> > >> From functional point of view it does makes sense to validate all the > >> platforms as part of the builds, but the Pre commits builds taking time > is > >> now no longer a small things, In past one year or may be two, we have > >> already increased it more than twice as compared to what it was before, > If > >> someone has a change in HDFS, which includes both hdfs-client & > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 > hours. > >> With the current state, I don't think we should just go and add these > >> extra overheads. Having them as part of the nightly builds does makes > sense > >> for now. > >> > >> In future if we feel there is a strong need for this and we start to see > >> very frequent failures in some other platforms and we are left with no > >> other option but to integrate it in our pre-commit jobs, we can explore > >> having these build phases running in parallel, along with trying other > >> phases also to run in parallel like compilation/javadoc builds of JDK-8 > & > >> JDK-11 can run in parallel and may be explore other opportunities as > well > >> to compensate for this time. > >> > >> For now lets just integrate it our nightly builds only and circle back > >> again here in future if the need be. > >> > >> -Ayush > >> > >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang > wrote: > >>> > >>> Running builds for all platforms for each and every PR seems too > >>> excessive. > >>> > >>> How about doing all platform builds in the nightly jobs? > >>> > >>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran > > >>> wrote: > >>> > I'm not enthusiastic here as it not only makes the builds slower, it > reduces the #of builds we can through a day > > one thing I am wondering is could we remove java8 support on some > >>> branches? > > make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and > >>> this > summers branch-3.3 release (which I'd rebadge 3.4) would ship as java > 11 > only. > that would cut buld and test time for those trunk PRs in half...after > >>> which > the preospect of building on more than one platform becomes more > viable. > > On Thu, 5 May 2022 at 15:34, Gautham Banasandra > wrote: > > > Hi Hadoop devs, > > > > Last week, there was a Hadoop build failure on Debian 10 caused by > > https://github.com/apache/hadoop/pull/3988. In > >>> dev-support/jenkins.sh, > > there's the capability to build and test Hadoop across the supported > > platforms. Currently, we're limiting this only for those PRs having > >>> only > > C/C++ changes[1], since C/C++ changes are more likely to cause > > cross-platform build issues and bypassing the full platform build for > >>> non > > C/C++ PRs would save a great deal of CI time. However, the build > >>> failure > > caused by PR #3988 motivates me to enable the capability to build and > > test Hadoop for all the supported platforms for ALL the PRs. > > > > While this may cause longer CI run duration for each PR, it would > > immensely minimize the risk of breaking Hadoop across platforms and > > saves us a lot of debugging time. Kindly post your opinion regarding > >>> this > > and I'll move to enable this capability for all PRs if the response > is > > sufficiently positive. > > > > [1] = > > > > > > >>> > https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev
Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
Three for trunk: Java-8 https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/ Java-11 https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/ ARM https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/ -Ayush > > On 07-May-2022, at 11:49 AM, Gautham Banasandra wrote: > > Hi all, > > Although validating cross platform builds at pre-commit > would be the most effective approach, I understand the > huge disadvantage caused by the slowdown. The best > way to tackle this would be to enable parallel builds for > the different platforms. I had given it a shot about a year > ago[1], it didn't go well and ran into all sorts of random > errors. I think we should make the parallel builds run on > different agents as opposed to starting the builds parallelly > on the same agent (which is what I had done earlier). > > So, I'll settle down to integrating the full suite of platform > builds into the nightly builds. Could anyone please point > me to the Jenkins job for this? > > [1] = https://github.com/apache/hadoop/pull/3166 > > Thanks, > --Gautham > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena wrote: >> >> From functional point of view it does makes sense to validate all the >> platforms as part of the builds, but the Pre commits builds taking time is >> now no longer a small things, In past one year or may be two, we have >> already increased it more than twice as compared to what it was before, If >> someone has a change in HDFS, which includes both hdfs-client & >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 hours. >> With the current state, I don't think we should just go and add these >> extra overheads. Having them as part of the nightly builds does makes sense >> for now. >> >> In future if we feel there is a strong need for this and we start to see >> very frequent failures in some other platforms and we are left with no >> other option but to integrate it in our pre-commit jobs, we can explore >> having these build phases running in parallel, along with trying other >> phases also to run in parallel like compilation/javadoc builds of JDK-8 & >> JDK-11 can run in parallel and may be explore other opportunities as well >> to compensate for this time. >> >> For now lets just integrate it our nightly builds only and circle back >> again here in future if the need be. >> >> -Ayush >> >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang wrote: >>> >>> Running builds for all platforms for each and every PR seems too >>> excessive. >>> >>> How about doing all platform builds in the nightly jobs? >>> >>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran >>> >>> wrote: >>> I'm not enthusiastic here as it not only makes the builds slower, it reduces the #of builds we can through a day one thing I am wondering is could we remove java8 support on some >>> branches? make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and >>> this summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11 only. that would cut buld and test time for those trunk PRs in half...after >>> which the preospect of building on more than one platform becomes more viable. On Thu, 5 May 2022 at 15:34, Gautham Banasandra wrote: > Hi Hadoop devs, > > Last week, there was a Hadoop build failure on Debian 10 caused by > https://github.com/apache/hadoop/pull/3988. In >>> dev-support/jenkins.sh, > there's the capability to build and test Hadoop across the supported > platforms. Currently, we're limiting this only for those PRs having >>> only > C/C++ changes[1], since C/C++ changes are more likely to cause > cross-platform build issues and bypassing the full platform build for >>> non > C/C++ PRs would save a great deal of CI time. However, the build >>> failure > caused by PR #3988 motivates me to enable the capability to build and > test Hadoop for all the supported platforms for ALL the PRs. > > While this may cause longer CI run duration for each PR, it would > immensely minimize the risk of breaking Hadoop across platforms and > saves us a lot of debugging time. Kindly post your opinion regarding >>> this > and I'll move to enable this capability for all PRs if the response is > sufficiently positive. > > [1] = > > >>> https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 > > > Thanks, > --Gautham > >>> >>
Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
Hi all, Although validating cross platform builds at pre-commit would be the most effective approach, I understand the huge disadvantage caused by the slowdown. The best way to tackle this would be to enable parallel builds for the different platforms. I had given it a shot about a year ago[1], it didn't go well and ran into all sorts of random errors. I think we should make the parallel builds run on different agents as opposed to starting the builds parallelly on the same agent (which is what I had done earlier). So, I'll settle down to integrating the full suite of platform builds into the nightly builds. Could anyone please point me to the Jenkins job for this? [1] = https://github.com/apache/hadoop/pull/3166 Thanks, --Gautham On Fri, 6 May 2022 at 21:04, Ayush Saxena wrote: > From functional point of view it does makes sense to validate all the > platforms as part of the builds, but the Pre commits builds taking time is > now no longer a small things, In past one year or may be two, we have > already increased it more than twice as compared to what it was before, If > someone has a change in HDFS, which includes both hdfs-client & > hadoop-hdfs, it takes more than 5 hours, which long back was around 2 hours. > With the current state, I don't think we should just go and add these > extra overheads. Having them as part of the nightly builds does makes sense > for now. > > In future if we feel there is a strong need for this and we start to see > very frequent failures in some other platforms and we are left with no > other option but to integrate it in our pre-commit jobs, we can explore > having these build phases running in parallel, along with trying other > phases also to run in parallel like compilation/javadoc builds of JDK-8 & > JDK-11 can run in parallel and may be explore other opportunities as well > to compensate for this time. > > For now lets just integrate it our nightly builds only and circle back > again here in future if the need be. > > -Ayush > > On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang wrote: > >> Running builds for all platforms for each and every PR seems too >> excessive. >> >> How about doing all platform builds in the nightly jobs? >> >> On Fri, May 6, 2022 at 8:02 AM Steve Loughran > > >> wrote: >> >> > I'm not enthusiastic here as it not only makes the builds slower, it >> > reduces the #of builds we can through a day >> > >> > one thing I am wondering is could we remove java8 support on some >> branches? >> > >> > make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and >> this >> > summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11 >> > only. >> > that would cut buld and test time for those trunk PRs in half...after >> which >> > the preospect of building on more than one platform becomes more viable. >> > >> > On Thu, 5 May 2022 at 15:34, Gautham Banasandra >> > wrote: >> > >> > > Hi Hadoop devs, >> > > >> > > Last week, there was a Hadoop build failure on Debian 10 caused by >> > > https://github.com/apache/hadoop/pull/3988. In >> dev-support/jenkins.sh, >> > > there's the capability to build and test Hadoop across the supported >> > > platforms. Currently, we're limiting this only for those PRs having >> only >> > > C/C++ changes[1], since C/C++ changes are more likely to cause >> > > cross-platform build issues and bypassing the full platform build for >> non >> > > C/C++ PRs would save a great deal of CI time. However, the build >> failure >> > > caused by PR #3988 motivates me to enable the capability to build and >> > > test Hadoop for all the supported platforms for ALL the PRs. >> > > >> > > While this may cause longer CI run duration for each PR, it would >> > > immensely minimize the risk of breaking Hadoop across platforms and >> > > saves us a lot of debugging time. Kindly post your opinion regarding >> this >> > > and I'll move to enable this capability for all PRs if the response is >> > > sufficiently positive. >> > > >> > > [1] = >> > > >> > > >> > >> https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 >> > > >> > > >> > > Thanks, >> > > --Gautham >> > > >> > >> >
Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
>From functional point of view it does makes sense to validate all the platforms as part of the builds, but the Pre commits builds taking time is now no longer a small things, In past one year or may be two, we have already increased it more than twice as compared to what it was before, If someone has a change in HDFS, which includes both hdfs-client & hadoop-hdfs, it takes more than 5 hours, which long back was around 2 hours. With the current state, I don't think we should just go and add these extra overheads. Having them as part of the nightly builds does makes sense for now. In future if we feel there is a strong need for this and we start to see very frequent failures in some other platforms and we are left with no other option but to integrate it in our pre-commit jobs, we can explore having these build phases running in parallel, along with trying other phases also to run in parallel like compilation/javadoc builds of JDK-8 & JDK-11 can run in parallel and may be explore other opportunities as well to compensate for this time. For now lets just integrate it our nightly builds only and circle back again here in future if the need be. -Ayush On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang wrote: > Running builds for all platforms for each and every PR seems too excessive. > > How about doing all platform builds in the nightly jobs? > > On Fri, May 6, 2022 at 8:02 AM Steve Loughran > > wrote: > > > I'm not enthusiastic here as it not only makes the builds slower, it > > reduces the #of builds we can through a day > > > > one thing I am wondering is could we remove java8 support on some > branches? > > > > make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and > this > > summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11 > > only. > > that would cut buld and test time for those trunk PRs in half...after > which > > the preospect of building on more than one platform becomes more viable. > > > > On Thu, 5 May 2022 at 15:34, Gautham Banasandra > > wrote: > > > > > Hi Hadoop devs, > > > > > > Last week, there was a Hadoop build failure on Debian 10 caused by > > > https://github.com/apache/hadoop/pull/3988. In dev-support/jenkins.sh, > > > there's the capability to build and test Hadoop across the supported > > > platforms. Currently, we're limiting this only for those PRs having > only > > > C/C++ changes[1], since C/C++ changes are more likely to cause > > > cross-platform build issues and bypassing the full platform build for > non > > > C/C++ PRs would save a great deal of CI time. However, the build > failure > > > caused by PR #3988 motivates me to enable the capability to build and > > > test Hadoop for all the supported platforms for ALL the PRs. > > > > > > While this may cause longer CI run duration for each PR, it would > > > immensely minimize the risk of breaking Hadoop across platforms and > > > saves us a lot of debugging time. Kindly post your opinion regarding > this > > > and I'll move to enable this capability for all PRs if the response is > > > sufficiently positive. > > > > > > [1] = > > > > > > > > > https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 > > > > > > > > > Thanks, > > > --Gautham > > > > > >
Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
Running builds for all platforms for each and every PR seems too excessive. How about doing all platform builds in the nightly jobs? On Fri, May 6, 2022 at 8:02 AM Steve Loughran wrote: > I'm not enthusiastic here as it not only makes the builds slower, it > reduces the #of builds we can through a day > > one thing I am wondering is could we remove java8 support on some branches? > > make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and this > summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11 > only. > that would cut buld and test time for those trunk PRs in half...after which > the preospect of building on more than one platform becomes more viable. > > On Thu, 5 May 2022 at 15:34, Gautham Banasandra > wrote: > > > Hi Hadoop devs, > > > > Last week, there was a Hadoop build failure on Debian 10 caused by > > https://github.com/apache/hadoop/pull/3988. In dev-support/jenkins.sh, > > there's the capability to build and test Hadoop across the supported > > platforms. Currently, we're limiting this only for those PRs having only > > C/C++ changes[1], since C/C++ changes are more likely to cause > > cross-platform build issues and bypassing the full platform build for non > > C/C++ PRs would save a great deal of CI time. However, the build failure > > caused by PR #3988 motivates me to enable the capability to build and > > test Hadoop for all the supported platforms for ALL the PRs. > > > > While this may cause longer CI run duration for each PR, it would > > immensely minimize the risk of breaking Hadoop across platforms and > > saves us a lot of debugging time. Kindly post your opinion regarding this > > and I'll move to enable this capability for all PRs if the response is > > sufficiently positive. > > > > [1] = > > > > > https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 > > > > > > Thanks, > > --Gautham > > >
Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
I'm not enthusiastic here as it not only makes the builds slower, it reduces the #of builds we can through a day one thing I am wondering is could we remove java8 support on some branches? make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and this summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11 only. that would cut buld and test time for those trunk PRs in half...after which the preospect of building on more than one platform becomes more viable. On Thu, 5 May 2022 at 15:34, Gautham Banasandra wrote: > Hi Hadoop devs, > > Last week, there was a Hadoop build failure on Debian 10 caused by > https://github.com/apache/hadoop/pull/3988. In dev-support/jenkins.sh, > there's the capability to build and test Hadoop across the supported > platforms. Currently, we're limiting this only for those PRs having only > C/C++ changes[1], since C/C++ changes are more likely to cause > cross-platform build issues and bypassing the full platform build for non > C/C++ PRs would save a great deal of CI time. However, the build failure > caused by PR #3988 motivates me to enable the capability to build and > test Hadoop for all the supported platforms for ALL the PRs. > > While this may cause longer CI run duration for each PR, it would > immensely minimize the risk of breaking Hadoop across platforms and > saves us a lot of debugging time. Kindly post your opinion regarding this > and I'll move to enable this capability for all PRs if the response is > sufficiently positive. > > [1] = > > https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 > > > Thanks, > --Gautham >
[DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
Hi Hadoop devs, Last week, there was a Hadoop build failure on Debian 10 caused by https://github.com/apache/hadoop/pull/3988. In dev-support/jenkins.sh, there's the capability to build and test Hadoop across the supported platforms. Currently, we're limiting this only for those PRs having only C/C++ changes[1], since C/C++ changes are more likely to cause cross-platform build issues and bypassing the full platform build for non C/C++ PRs would save a great deal of CI time. However, the build failure caused by PR #3988 motivates me to enable the capability to build and test Hadoop for all the supported platforms for ALL the PRs. While this may cause longer CI run duration for each PR, it would immensely minimize the risk of breaking Hadoop across platforms and saves us a lot of debugging time. Kindly post your opinion regarding this and I'll move to enable this capability for all PRs if the response is sufficiently positive. [1] = https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103 Thanks, --Gautham
