Re: ASF Jenkins Builds
Thanks! Gavin McDonald 于2018年11月4日周日 下午12:50写道： > H12 to H19 have been upgraded with 4TB disks and are now re-puppetised and > back online. > > Except H18 which is being re-looked into, working on it. > > HTH > > Gav... > > > On Fri, Nov 2, 2018 at 11:20 AM Chris Thistlethwaite > wrote: > >> Yes, we're missing a bunch of H nodes as they being upgraded. Those >> nodes are all donated, so infra has to wait for them to be handed over. >> While having those nodes back will be a huge help, having hourly builds >> and a nightly run every 6 hours is a bit excessive. As you can see, if >> there is any hiccup or change in the Jenkins infrastructure, these >> frequent builds pile up and have to be killed/paused until everything >> catches up. >> >> Thanks again for the quick response on rescheduling those builds! >> >> -Chris T. >> >> On Fri, 2018-11-02 at 07:14 +0800, 张铎(Duo Zhang) wrote: >> > Flakey is only part of all the UTs so usually it should be finished >> > soon and should not queue lots of jobs. >> > >> > And recently I found that half of the machines under the Hadoop label >> > are gone, and filed an infra issue. The guy claimed that the machines >> > are being reinstalled and will be back soon. Maybe this is the root >> > cause that why the resources are not enough? >> > >> > Thanks. >> > >> > Chris Thistlethwaite 于2018年11月2日 周五00:41写道： >> > > Awesome! Thanks for the quick response. >> > > >> > > -Chris T. >> > > >> > > On Thu, 2018-11-01 at 09:38 -0700, Stack wrote: >> > > > Thank you for the notice. I'm making the change now. You should >> > > > notice difference in an hour or so. >> > > > https://issues.apache.org/jira/browse/HBASE-21424 is our issue to >> > > > address. >> > > > Thanks, >> > > > S >> > > > >> > > > On Thu, Nov 1, 2018 at 9:09 AM Chris Thistlethwaite < >> > > > chr...@apache.org> wrote: >> > > > > Greetings! >> > > > > >> > > > > During the Jenkins outage yesterday I noticed a ton of builds >> > > from >> > > > > HBase-Flaky-Tests >> > > > > >> > > >> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Flaky-Tests/ >> > > > > in >> > > > > the queue. Turns out this runs a bunch of pipeline builds every >> > > > > hour >> > > > > which clogs up Jenkins, both for you and other projects. For >> > > > > example, >> > > > > branch-2.0 is currently queuing 3 builds, waiting on the 4th to >> > > > > finish, >> > > > > and it's also behind the HBase Nightly. >> > > > > >> > > > > That brings me to HBase Nightly >> > > > > >> > > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/ >> > > > > it >> > > > > runs every 6 hours, which is a bit excessive for a nightly >> > > build >> > > > > which >> > > > > by definition should be once a day. Especially as it gets >> > > > > dangerously >> > > > > close to running into itself as builds currently around 4-5 >> > > hours >> > > > > of >> > > > > build time. >> > > > > >> > > > > I suggest something more like Flaky-Tests every 6 hours and the >> > > > > Nightly >> > > > > once a day. If you agree to these changes, feel free to update >> > > > > Jenkins. >> > > > > Otherwise, I'll update the jobs in the next few days if there >> > > is no >> > > > > response. >> > > > > >> > > > > Please add t...@infra.apache.org and/or my address to any >> > > replies >> > > > > as >> > > > > we're not subbed to your dev list. >> > > > > >> > > > > Thank you, >> > > > > Chris T. >> > > > > >> > > > > >> > > >> >> > > -- > Gav... >
Re: ASF Jenkins Builds
H12 to H19 have been upgraded with 4TB disks and are now re-puppetised and back online. Except H18 which is being re-looked into, working on it. HTH Gav... On Fri, Nov 2, 2018 at 11:20 AM Chris Thistlethwaite wrote: > Yes, we're missing a bunch of H nodes as they being upgraded. Those > nodes are all donated, so infra has to wait for them to be handed over. > While having those nodes back will be a huge help, having hourly builds > and a nightly run every 6 hours is a bit excessive. As you can see, if > there is any hiccup or change in the Jenkins infrastructure, these > frequent builds pile up and have to be killed/paused until everything > catches up. > > Thanks again for the quick response on rescheduling those builds! > > -Chris T. > > On Fri, 2018-11-02 at 07:14 +0800, 张铎(Duo Zhang) wrote: > > Flakey is only part of all the UTs so usually it should be finished > > soon and should not queue lots of jobs. > > > > And recently I found that half of the machines under the Hadoop label > > are gone, and filed an infra issue. The guy claimed that the machines > > are being reinstalled and will be back soon. Maybe this is the root > > cause that why the resources are not enough? > > > > Thanks. > > > > Chris Thistlethwaite 于2018年11月2日 周五00:41写道： > > > Awesome! Thanks for the quick response. > > > > > > -Chris T. > > > > > > On Thu, 2018-11-01 at 09:38 -0700, Stack wrote: > > > > Thank you for the notice. I'm making the change now. You should > > > > notice difference in an hour or so. > > > > https://issues.apache.org/jira/browse/HBASE-21424 is our issue to > > > > address. > > > > Thanks, > > > > S > > > > > > > > On Thu, Nov 1, 2018 at 9:09 AM Chris Thistlethwaite < > > > > chr...@apache.org> wrote: > > > > > Greetings! > > > > > > > > > > During the Jenkins outage yesterday I noticed a ton of builds > > > from > > > > > HBase-Flaky-Tests > > > > > > > > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Flaky-Tests/ > > > > > in > > > > > the queue. Turns out this runs a bunch of pipeline builds every > > > > > hour > > > > > which clogs up Jenkins, both for you and other projects. For > > > > > example, > > > > > branch-2.0 is currently queuing 3 builds, waiting on the 4th to > > > > > finish, > > > > > and it's also behind the HBase Nightly. > > > > > > > > > > That brings me to HBase Nightly > > > > > > > > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/ > > > > > it > > > > > runs every 6 hours, which is a bit excessive for a nightly > > > build > > > > > which > > > > > by definition should be once a day. Especially as it gets > > > > > dangerously > > > > > close to running into itself as builds currently around 4-5 > > > hours > > > > > of > > > > > build time. > > > > > > > > > > I suggest something more like Flaky-Tests every 6 hours and the > > > > > Nightly > > > > > once a day. If you agree to these changes, feel free to update > > > > > Jenkins. > > > > > Otherwise, I'll update the jobs in the next few days if there > > > is no > > > > > response. > > > > > > > > > > Please add t...@infra.apache.org and/or my address to any > > > replies > > > > > as > > > > > we're not subbed to your dev list. > > > > > > > > > > Thank you, > > > > > Chris T. > > > > > > > > > > > > > > > -- Gav...
Re: ASF Jenkins Builds
Yes, we're missing a bunch of H nodes as they being upgraded. Those nodes are all donated, so infra has to wait for them to be handed over. While having those nodes back will be a huge help, having hourly builds and a nightly run every 6 hours is a bit excessive. As you can see, if there is any hiccup or change in the Jenkins infrastructure, these frequent builds pile up and have to be killed/paused until everything catches up. Thanks again for the quick response on rescheduling those builds! -Chris T. On Fri, 2018-11-02 at 07:14 +0800, 张铎(Duo Zhang) wrote: > Flakey is only part of all the UTs so usually it should be finished > soon and should not queue lots of jobs. > > And recently I found that half of the machines under the Hadoop label > are gone, and filed an infra issue. The guy claimed that the machines > are being reinstalled and will be back soon. Maybe this is the root > cause that why the resources are not enough? > > Thanks. > > Chris Thistlethwaite 于2018年11月2日 周五00:41写道： > > Awesome! Thanks for the quick response. > > > > -Chris T. > > > > On Thu, 2018-11-01 at 09:38 -0700, Stack wrote: > > > Thank you for the notice. I'm making the change now. You should > > > notice difference in an hour or so. > > > https://issues.apache.org/jira/browse/HBASE-21424 is our issue to > > > address. > > > Thanks, > > > S > > > > > > On Thu, Nov 1, 2018 at 9:09 AM Chris Thistlethwaite < > > > chr...@apache.org> wrote: > > > > Greetings! > > > > > > > > During the Jenkins outage yesterday I noticed a ton of builds > > from > > > > HBase-Flaky-Tests > > > > > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Flaky-Tests/ > > > > in > > > > the queue. Turns out this runs a bunch of pipeline builds every > > > > hour > > > > which clogs up Jenkins, both for you and other projects. For > > > > example, > > > > branch-2.0 is currently queuing 3 builds, waiting on the 4th to > > > > finish, > > > > and it's also behind the HBase Nightly. > > > > > > > > That brings me to HBase Nightly > > > > > > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/ > > > > it > > > > runs every 6 hours, which is a bit excessive for a nightly > > build > > > > which > > > > by definition should be once a day. Especially as it gets > > > > dangerously > > > > close to running into itself as builds currently around 4-5 > > hours > > > > of > > > > build time. > > > > > > > > I suggest something more like Flaky-Tests every 6 hours and the > > > > Nightly > > > > once a day. If you agree to these changes, feel free to update > > > > Jenkins. > > > > Otherwise, I'll update the jobs in the next few days if there > > is no > > > > response. > > > > > > > > Please add t...@infra.apache.org and/or my address to any > > replies > > > > as > > > > we're not subbed to your dev list. > > > > > > > > Thank you, > > > > Chris T. > > > > > > > > > >
Re: ASF Jenkins Builds
Flakey is only part of all the UTs so usually it should be finished soon and should not queue lots of jobs. And recently I found that half of the machines under the Hadoop label are gone, and filed an infra issue. The guy claimed that the machines are being reinstalled and will be back soon. Maybe this is the root cause that why the resources are not enough? Thanks. Chris Thistlethwaite 于2018年11月2日 周五00:41写道： > Awesome! Thanks for the quick response. > > -Chris T. > > On Thu, 2018-11-01 at 09:38 -0700, Stack wrote: > > Thank you for the notice. I'm making the change now. You should > > notice difference in an hour or so. > > https://issues.apache.org/jira/browse/HBASE-21424 is our issue to > > address. > > Thanks, > > S > > > > On Thu, Nov 1, 2018 at 9:09 AM Chris Thistlethwaite < > > chr...@apache.org> wrote: > > > Greetings! > > > > > > During the Jenkins outage yesterday I noticed a ton of builds from > > > HBase-Flaky-Tests > > > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Flaky-Tests/ > > > in > > > the queue. Turns out this runs a bunch of pipeline builds every > > > hour > > > which clogs up Jenkins, both for you and other projects. For > > > example, > > > branch-2.0 is currently queuing 3 builds, waiting on the 4th to > > > finish, > > > and it's also behind the HBase Nightly. > > > > > > That brings me to HBase Nightly > > > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/ > > > it > > > runs every 6 hours, which is a bit excessive for a nightly build > > > which > > > by definition should be once a day. Especially as it gets > > > dangerously > > > close to running into itself as builds currently around 4-5 hours > > > of > > > build time. > > > > > > I suggest something more like Flaky-Tests every 6 hours and the > > > Nightly > > > once a day. If you agree to these changes, feel free to update > > > Jenkins. > > > Otherwise, I'll update the jobs in the next few days if there is no > > > response. > > > > > > Please add t...@infra.apache.org and/or my address to any replies > > > as > > > we're not subbed to your dev list. > > > > > > Thank you, > > > Chris T. > > > > > > > >
Re: ASF Jenkins Builds
Awesome! Thanks for the quick response. -Chris T. On Thu, 2018-11-01 at 09:38 -0700, Stack wrote: > Thank you for the notice. I'm making the change now. You should > notice difference in an hour or so. > https://issues.apache.org/jira/browse/HBASE-21424 is our issue to > address. > Thanks, > S > > On Thu, Nov 1, 2018 at 9:09 AM Chris Thistlethwaite < > chr...@apache.org> wrote: > > Greetings! > > > > During the Jenkins outage yesterday I noticed a ton of builds from > > HBase-Flaky-Tests > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Flaky-Tests/ > > in > > the queue. Turns out this runs a bunch of pipeline builds every > > hour > > which clogs up Jenkins, both for you and other projects. For > > example, > > branch-2.0 is currently queuing 3 builds, waiting on the 4th to > > finish, > > and it's also behind the HBase Nightly. > > > > That brings me to HBase Nightly > > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/ > > it > > runs every 6 hours, which is a bit excessive for a nightly build > > which > > by definition should be once a day. Especially as it gets > > dangerously > > close to running into itself as builds currently around 4-5 hours > > of > > build time. > > > > I suggest something more like Flaky-Tests every 6 hours and the > > Nightly > > once a day. If you agree to these changes, feel free to update > > Jenkins. > > Otherwise, I'll update the jobs in the next few days if there is no > > response. > > > > Please add t...@infra.apache.org and/or my address to any replies > > as > > we're not subbed to your dev list. > > > > Thank you, > > Chris T. > > > >
Re: ASF Jenkins Builds
Thank you for the notice. I'm making the change now. You should notice difference in an hour or so. https://issues.apache.org/jira/browse/HBASE-21424 is our issue to address. Thanks, S On Thu, Nov 1, 2018 at 9:09 AM Chris Thistlethwaite wrote: > Greetings! > > During the Jenkins outage yesterday I noticed a ton of builds from > HBase-Flaky-Tests > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Flaky-Tests/ in > the queue. Turns out this runs a bunch of pipeline builds every hour > which clogs up Jenkins, both for you and other projects. For example, > branch-2.0 is currently queuing 3 builds, waiting on the 4th to finish, > and it's also behind the HBase Nightly. > > That brings me to HBase Nightly > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/ it > runs every 6 hours, which is a bit excessive for a nightly build which > by definition should be once a day. Especially as it gets dangerously > close to running into itself as builds currently around 4-5 hours of > build time. > > I suggest something more like Flaky-Tests every 6 hours and the Nightly > once a day. If you agree to these changes, feel free to update Jenkins. > Otherwise, I'll update the jobs in the next few days if there is no > response. > > Please add t...@infra.apache.org and/or my address to any replies as > we're not subbed to your dev list. > > Thank you, > Chris T. > > >
