Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Mark Williams]

Adi Roiban graciously reviewed the PR and I deployed the new master.cfg to 
buildbot.twistedmatrix.com.  It looks like the macOS builder is now working 
(more) reliably.

Please note that the required macOS build is now known as 
"buildbot/osx10.11-py2.7-coverage", where before it was 
"buildbot/osx10.10-py2.7-coverage".  This name change reflects the updated 
operating system.

On Wed, Apr 4, 2018, at 5:28 PM, Mark Williams wrote:
> On Mon, Apr 2, 2018, at 4:13 AM, Adi Roiban wrote:
> > 
> > FYI buildbot/osx10.10-py2.7 is broken again
> > 
> 
> The Build master's configuration was updated outside of twisted-infra/
> braid.  I've checked it in and brought the repo more in line with 
> reality:
> 
> https://github.com/twisted-infra/braid/commit/55402aaa0a8da3b8d6ee737764287866171084f4
> 
> I also have PR that should fix the intermittent failures:
> 
> https://github.com/twisted-infra/braid/pull/250
> 
> If you're interested, please review that PR.
> 
> > -
> > 
> > Circle-CI is pretty fast, but we are also burning the time pretty fast.
> > We have used 90% of the time allocated between 19 March of 19 April.
> > 
> > Regards,
> > -- 
> > Adi Roiban
> > _
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
> --
>   Mark Williams
>   m...@enotuniq.org
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


-- 
  Mark Williams
  m...@enotuniq.org

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Mark Williams]

On Mon, Apr 2, 2018, at 4:13 AM, Adi Roiban wrote:
> 
> FYI buildbot/osx10.10-py2.7 is broken again
> 

The Build master's configuration was updated outside of twisted-infra/braid.  
I've checked it in and brought the repo more in line with reality:

https://github.com/twisted-infra/braid/commit/55402aaa0a8da3b8d6ee737764287866171084f4

I also have PR that should fix the intermittent failures:

https://github.com/twisted-infra/braid/pull/250

If you're interested, please review that PR.

> -
> 
> Circle-CI is pretty fast, but we are also burning the time pretty fast.
> We have used 90% of the time allocated between 19 March of 19 April.
> 
> Regards,
> -- 
> Adi Roiban
> _
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

--
  Mark Williams
  m...@enotuniq.org

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[jonathan vanasco]

> On Apr 2, 2018, at 7:13 AM, Adi Roiban  wrote:
> 
> FYI buildbot/osx10.10-py2.7 is broken again


FWIW, `pip` on OSX (perhaps others?) swallows the 403 error about TLS brownouts 
on pypi for the 1.0/1.1 ending next week. No error is raised, and you just get 
an empty match for installable versions of anything. (it needs to be run as 
`-` to see/know there is an actual error).  If that is the issue, the 9.0.3 
version of pip is compatible with the new TLS and upgrading to it will fix 
things.

This broke my CI system a few weeks ago, and has done the same for everyone i 
know with a CI running on macs.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Craig Rodrigues]

On Sun, Mar 25, 2018 at 5:01 AM, Adi Roiban  wrote:

> Hi,
>
>
> On 25 March 2018 at 05:14, Craig Rodrigues  wrote:
> > Hi,
> >
> > Can someone check this builder:
> >
> > https://buildbot.twistedmatrix.com/builders/osx10.10-py2.7
> >
> > It is failing randomly in weird ways during builds.  For example,
> > in some cases it fails to install tox.  In other cases, the tests
> terminate
> > in
> > some random spot.
>
> I will try and take a look at it.
>
> If someone knows more about the slave running this build, please add
> the info on wiki at:
>
> https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves
>


Where is this MacOS builder?  Is it in Rackspace, some cloud provider, or
other?

--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Adi Roiban]

On 27 March 2018 at 23:54, Glyph  wrote:

>
>
>
> On Mon, Mar 26, 2018, at 10:39 PM, Glyph wrote:
>
>
>
> On Mar 26, 2018, at 3:32 AM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
> On Mon, Mar 26, 2018 at 2:58 AM, Glyph  wrote:
>
> Alternately...
>
> The reason we're doing this at all is that Travis's macOS situation was
> pretty dismal when we last tried it.  But then, the graphs on
> https://www.traviscistatus.com at the time matched that experience:
> routinely there were build queues in the hundreds.  Right now the "Backlog
> macOS Builds for Open Source Projects" shows a completely flat line, zero
> for the last 24 hours.
>
> It might be worth trying out Travis again.
>
>
> FWIW, I moved another project's macOS CI from Travis to CircleCI about a
> month ago because the Travis macOS job queue was so long.  Counting queue
> time, CircleCI finishes those jobs around 6x faster than TravisCI.
>
> Maybe things have improved on Travis but they're still pretty bad.
>
>
> That's a bummer, but thanks for providing the data point.
>
> -g
>
>
>
> I guess I just needed to look during the PST workday...
>
>
FYI buildbot/osx10.10-py2.7 is broken again


-

Circle-CI is pretty fast, but we are also burning the time pretty fast.
We have used 90% of the time allocated between 19 March of 19 April.

Regards,
-- 
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Glyph]

On Mon, Mar 26, 2018, at 10:39 PM, Glyph wrote:
> 
> 
>> On Mar 26, 2018, at 3:32 AM, Jean-Paul Calderone
>>  wrote:>> 
>> On Mon, Mar 26, 2018 at 2:58 AM, Glyph
>>  wrote:>>> Alternately...
>>> 
>>> The reason we're doing this at all is that Travis's macOS situation
>>> was pretty dismal when we last tried it.  But then, the graphs on
>>> https://www.traviscistatus.com[1] at the time matched that
>>> experience: routinely there were build queues in the hundreds.
>>> Right now the "Backlog macOS Builds for Open Source Projects" shows
>>> a completely flat line, zero for the last 24 hours.>>> 
>>> It might be worth trying out Travis again.
>> 
>> FWIW, I moved another project's macOS CI from Travis to CircleCI
>> about a month ago because the Travis macOS job queue was so long.
>> Counting queue time, CircleCI finishes those jobs around 6x faster
>> than TravisCI.>> 
>> Maybe things have improved on Travis but they're still pretty bad.
> 
> That's a bummer, but thanks for providing the data point.
> 
> -g


I guess I just needed to look during the PST workday... 
 


Links:

  1. https://www.traviscistatus.com/
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Glyph]

> On Mar 26, 2018, at 3:32 AM, Jean-Paul Calderone  
> wrote:
> 
> On Mon, Mar 26, 2018 at 2:58 AM, Glyph  > wrote:
> Alternately...
> 
> The reason we're doing this at all is that Travis's macOS situation was 
> pretty dismal when we last tried it.  But then, the graphs on 
> https://www.traviscistatus.com  at the time 
> matched that experience: routinely there were build queues in the hundreds.  
> Right now the "Backlog macOS Builds for Open Source Projects" shows a 
> completely flat line, zero for the last 24 hours.
> 
> It might be worth trying out Travis again.
> 
> FWIW, I moved another project's macOS CI from Travis to CircleCI about a 
> month ago because the Travis macOS job queue was so long.  Counting queue 
> time, CircleCI finishes those jobs around 6x faster than TravisCI.
> 
> Maybe things have improved on Travis but they're still pretty bad.

That's a bummer, but thanks for providing the data point.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Jean-Paul Calderone]

On Mon, Mar 26, 2018 at 2:58 AM, Glyph  wrote:

> Alternately...
>
> The reason we're doing this at all is that Travis's macOS situation was
> pretty dismal when we last tried it.  But then, the graphs on
> https://www.traviscistatus.com at the time matched that experience:
> routinely there were build queues in the hundreds.  Right now the "Backlog
> macOS Builds for Open Source Projects" shows a completely flat line, zero
> for the last 24 hours.
>
> It might be worth trying out Travis again.
>

FWIW, I moved another project's macOS CI from Travis to CircleCI about a
month ago because the Travis macOS job queue was so long.  Counting queue
time, CircleCI finishes those jobs around 6x faster than TravisCI.

Maybe things have improved on Travis but they're still pretty bad.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Glyph]

> On Mar 25, 2018, at 5:01 AM, Adi Roiban  wrote:
> 
> Hi,
> 
> 
> On 25 March 2018 at 05:14, Craig Rodrigues  wrote:
>> Hi,
>> 
>> Can someone check this builder:
>> 
>> https://buildbot.twistedmatrix.com/builders/osx10.10-py2.7
>> 
>> It is failing randomly in weird ways during builds.  For example,
>> in some cases it fails to install tox.  In other cases, the tests terminate
>> in
>> some random spot.
> 
> I will try and take a look at it.

Adi, myself, and Mark looked at it by turns and eventually I think we got it 
into a mostly working state.

Builds are now passing.  However, if you attempt to view the logs of a build in 
progress, it appears to fail with a purple exception.  This actually looks like 
it's a bug in the build master which I'm a little confused about why it isn't 
affecting other builders.

> If someone knows more about the slave running this build, please add
> the info on wiki at:
> 
> https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves 
> 

Looks like you beat me to the punch: 
https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves/d3 


Is there other information you need?

>> This builder is marked as "Required", so it is a bit annoying.
> 
> My understanding from the previous discussion about the supported
> platforms is that any build is required.
> The osx is there as I reminder to run the buildbot tests and not to
> merge a PR with only Travis and Appveyor.
> 
> I have reconfigured GitHub to require  ubuntu16.04-py2.7-coverage as
> this is backed by 3 slaves.

Primarily it's required because it's the only reasonable way we have to verify 
that things work on macOS, which is by far our most popular developer operating 
system.  I've reverted this configuration change, since it's more reasonable to 
give the reviewers some discretion about the fact that Travis is representative 
of Linux than that a random assortment of non-macOS buildbots are 
representative of macOS.

> We have the circle-ci macOS and for the next  2 days, we can use that :)
> We are already used 55% of the macOS time and Circle-CI was enalbled
> in trunk in the last 2 days...so I don't think that we can use
> Circle-CI for macOS for the long term.
> I will write them to see if we can get an exception or a discount.

I would definitely like it if we could move to something hosted :).

> Otherwise, I plan to rent a macOS  so that it can act as a second
> slave for osx10.10-py2.7
> So far, the support from macincloud was not helpful or friendly, but I
> guess that we can just try their service for 1 month and see how it
> goes.

Bummer.  This seems like the sort of thing that we should have the Conservancy 
pay for, rather than you personally.  Better yet, of course, to get a 
commercially interested macOS user to supply some hardware or mac cloud boxes 
or something.

Alternately...

The reason we're doing this at all is that Travis's macOS situation was pretty 
dismal when we last tried it.  But then, the graphs on 
https://www.traviscistatus.com  at the time 
matched that experience: routinely there were build queues in the hundreds.  
Right now the "Backlog macOS Builds for Open Source Projects" shows a 
completely flat line, zero for the last 24 hours.

It might be worth trying out Travis again.

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Adi Roiban]

Hi,


On 25 March 2018 at 05:14, Craig Rodrigues  wrote:
> Hi,
>
> Can someone check this builder:
>
> https://buildbot.twistedmatrix.com/builders/osx10.10-py2.7
>
> It is failing randomly in weird ways during builds.  For example,
> in some cases it fails to install tox.  In other cases, the tests terminate
> in
> some random spot.

I will try and take a look at it.

If someone knows more about the slave running this build, please add
the info on wiki at:

https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves


> This builder is marked as "Required", so it is a bit annoying.
>


My understanding from the previous discussion about the supported
platforms is that any build is required.
The osx is there as I reminder to run the buildbot tests and not to
merge a PR with only Travis and Appveyor.

I have reconfigured GitHub to require  ubuntu16.04-py2.7-coverage as
this is backed by 3 slaves.



-

We have the circle-ci macOS and for the next  2 days, we can use that :)
We are already used 55% of the macOS time and Circle-CI was enalbled
in trunk in the last 2 days...so I don't think that we can use
Circle-CI for macOS for the long term.
I will write them to see if we can get an exception or a discount.

Otherwise, I plan to rent a macOS  so that it can act as a second
slave for osx10.10-py2.7
So far, the support from macincloud was not helpful or friendly, but I
guess that we can just try their service for 1 month and see how it
goes.

-- 
Adi Roiban

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Problem with osx10.10-py2.7 builder

[Craig Rodrigues]

Hi,

Can someone check this builder:

https://buildbot.twistedmatrix.com/builders/osx10.10-py2.7

It is failing randomly in weird ways during builds.  For example,
in some cases it fails to install tox.  In other cases, the tests terminate
in
some random spot.

This builder is marked as "Required", so it is a bit annoying.

--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python