FYI. Still not fixed. Others experience this as well: https://github.com/travis-ci/worker/issues/604
On Tue, Jul 23, 2019 at 11:34 AM Jarek Potiuk <[email protected]> wrote: > No good news yet. We are getting randomly assigned 1CPU /3.5GB mem > instances still. Infrastructure is on it. > > On Tue, Jul 23, 2019 at 10:49 AM Jarek Potiuk <[email protected]> > wrote: > >> It looks like we are back to the original specs. I am runnning tests and >> re-enable everything if I see it works. >> >> J. >> >> On Tue, Jul 23, 2019 at 10:34 AM Jarek Potiuk <[email protected]> >> wrote: >> >>> From INFRA: "I have confirmed that our builds appear to be running with >>> 3.75GB memory and 1 core currently. This does not match Travis' standard >>> specs (7.5GB and 2 cores), and I have raised a ticket with their support. I >>> will respond when we hear back from Travis." >>> >>> >>> On Tue, Jul 23, 2019 at 10:26 AM Jarek Potiuk <[email protected]> >>> wrote: >>> >>>> It's definitely confirmed that the problem is on Travis CI side: >>>> >>>> I re-run the commit before the new CI was introduced (I cherry-picked a >>>> small doc fix related to recent sphinx dependency update) and it fails in >>>> exactly the same way (memory and cpu problems): >>>> https://travis-ci.org/apache/airflow/builds/562450592. >>>> >>>> For now I cannot do much but wait for the INFRA's response (and work on >>>> GitLab CI replacement of Travis). >>>> >>>> I recommend to bring some pop-corn. It's going to be an interesting one >>>> to watch. >>>> >>>> J. >>>> >>>> On Tue, Jul 23, 2019 at 9:43 AM Jarek Potiuk <[email protected]> >>>> wrote: >>>> >>>>> It's now pretty consistent and happens pretty much every time using >>>>> the old build system - for example here: >>>>> https://travis-ci.org/apache/airflow/builds/562435992. >>>>> >>>>> I will cancel all PRs and disable automated PR build on Travis until >>>>> we solve the problem - as it is pointless - new PRs will simply queue and >>>>> fail constantly. >>>>> >>>>> I opened critical infrastructure ticket: >>>>> https://issues.apache.org/jira/browse/INFRA-18787 and I am running >>>>> some additional tests - I run the builds from commit before the new CI so >>>>> that I see if another change since then could cause it. >>>>> >>>>> J. >>>>> >>>>> >>>>> On Tue, Jul 23, 2019 at 8:55 AM Jarek Potiuk <[email protected]> >>>>> wrote: >>>>> >>>>>> Update2: I can confirm that the same memory/resource related issues >>>>>> happen in my Travis CI forks with reverted changes :( >>>>>> https://travis-ci.org/potiuk/airflow/builds/562430507 . I will >>>>>> escalate it to Travis/APACHE infrastructure >>>>>> >>>>>> On Tue, Jul 23, 2019 at 8:35 AM Jarek Potiuk < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Update: it looks like it's Travis's problem: I reverted the CI >>>>>>> changes and we have the same CPU problem in the old build: >>>>>>> https://travis-ci.org/potiuk/airflow/jobs/562430517 . >>>>>>> >>>>>>> On Tue, Jul 23, 2019 at 8:32 AM Jarek Potiuk < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hello everyone, >>>>>>>> >>>>>>>> We've started to experience some random failures on Travis relaated >>>>>>>> to lack of resources: those are either Out of Memory errors or lack of >>>>>>>> CPUS >>>>>>>> to run Kubernetes builds. >>>>>>>> >>>>>>>> I tried to rerun those, thinking it was an intermittent error. It >>>>>>>> started happening yesterday and I have not seen it before so I rather >>>>>>>> doubt >>>>>>>> it is related to the latest changes. >>>>>>>> >>>>>>>> But I do not want to risk everyone being blocked so I am testing >>>>>>>> now on my own fork if reverting the latest CI changes help. I will let >>>>>>>> you >>>>>>>> know and will revert in case I found old CI works in a stable way. >>>>>>>> >>>>>>>> In the meantime - I will cancel all outstanding builds that are >>>>>>>> blocking our queue and will test it both old CI and new CI in our fork >>>>>>>> :( >>>>>>>> (Travis queue limit is not helping). >>>>>>>> >>>>>>>> Can you please hold on with rebasing/pushing new PRs until I check >>>>>>>> it. >>>>>>>> >>>>>>>> Example failures: >>>>>>>> >>>>>>>> >>>>>>>> - OSError: [Errno 12] Cannot allocate memory ( >>>>>>>> https://travis-ci.org/apache/airflow/jobs/562395978) >>>>>>>> - [ERROR NumCPU]: the number of available CPUs 1 is less than >>>>>>>> the required 2 ( >>>>>>>> https://travis-ci.org/apache/airflow/jobs/562395978) >>>>>>>> >>>>>>>> >>>>>>>> J. >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Jarek Potiuk >>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>>>> >>>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Jarek Potiuk >>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>>> >>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Jarek Potiuk >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>> >>>>>> M: +48 660 796 129 <+48660796129> >>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> Jarek Potiuk >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>> >>>>> M: +48 660 796 129 <+48660796129> >>>>> [image: Polidea] <https://www.polidea.com/> >>>>> >>>>> >>>> >>>> -- >>>> >>>> Jarek Potiuk >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>> >>>> M: +48 660 796 129 <+48660796129> >>>> [image: Polidea] <https://www.polidea.com/> >>>> >>>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
