Re: [Twisted-Python] Enable pre-commit.ci for twisted/twisted

[Jean-Paul Calderone]

On Thu, Jul 15, 2021 at 10:26 AM Adi Roiban  wrote:

> Hi,
>
> As part of this PR https://github.com/twisted/twisted/pull/1619 Thomas
> has requested to enable the https://pre-commit.ci  service for
> twisted/twisted repository
>
> The pre-commit.ci is kind of an auto-pilot for our pre-commit
> configuration.
>
> We already do post-commit checks via the pre-commit framework and I think
> that some of the core developers have pre-commit enabled as a git hook.
>
> The convenience if pre-commit.ci is that if you don't have the git-hook
> enable and you commit dirty code, pre-commit.ci will fix it for you.
>
> pre-commit.ci will also automatically create PR to update the pre-commit
> dependencies.
>
> This all comes at the cost of allowing the external pre-commit.ci server
> to write to twisted/twisted repository.
>
> pre-commit.ci was already enabled for twisted/ldaptor
>
> pre-commit.ci is currently enabled for twisted/twisted for testing while
> a final decision is made.
>
> I am +0 on this change due to security reasons...but I do think that it
> will reduce a bit of the frustration for first time contributors.
>

Can you elaborate on this a little bit?  First time contributors won't be
able to push a branch to twisted/twisted.  Does giving pre-commit.ci write
access to twisted/twisted also give it write access to forks of
twisted/twisted?  Or maybe only to branches in forks with PRs targeting
twisted/twisted?


> If there are no strong comments against enabling pre-commit.ci, in a
> week, I plan to approve the PR and have the pre-commit.ci enabled for the
> long term.
>

Thanks for raising this question with the mailing list and for supplying
details about the motivation.  Given that there is no justification for the
change written on the ticket or in the PR itself, this is really helpful.

Thanks,
Jean-Paul


> We can always rewoke to pre-commit.ci write access if needed.
>
>
Cheers
> --
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

[Jean-Paul Calderone]

On Wed, Mar 3, 2021 at 3:08 AM Glyph  wrote:

>
> On Mar 2, 2021, at 6:10 PM, Jean-Paul Calderone 
> wrote:
>
> Policy aside, this change doesn't seem like much of an improvement to me.
> If I were to guess, I would guess the change was made to satisfy some check
> Mypy is now being asked to make about Twisted.  If that's the case, it
> seems unfortunate that real-world software is suffering so that a synthetic
> goal can be achieved.  I do recognize there is a perception that practical
> value can come from attending to the errors Mypy reports.  It would
> probably benefit everyone if more care were taken to consider the
> real-world consequences of changes that are made to satisfy the
> non-real-world goalposts set by tools like Mypy.
>
>
> I do think that Mypy was likely highlighting a real issue here, and it
> should have been fixed.  We had documented interfaces already, and we were
> failing to adhere to them properly, and Mypy was complaining about that.
>
> For easy reference, the change that made these fixes is here
> https://github.com/twisted/twisted/pull/1298
>
> This genre of fix is definitely something we should unwind over time,
> although it does make it easier to start mypy-clean rather than have
> hundreds of exclusion lines that need to be manually adjusted, so on
> balance I agree with this tradeoff at the point it was made.
>

Broadly, I agree.  But not with this part.  It seems like there is clearly
a trade-off that is better for everyone.  The trade-off represented by
#1298:

   - Breaks application code without providing any new functionality or
   fixing any bugs
   - Captures a long list of TODOs as an arbitrarily complex collection of
   sources spread across the entire codebase
   - All the work of addressing the problems still remains to be done

 Contrast this with any basic ratchet-style approach (for example, capture
the list of mypy errors, then allow any PR that either removes some or
doesn't add any new ones):

   - At the outset, no application code breaks because nothing has actually
   changed
   - As work towards *fixing* the TODOs progresses, if it *does* break
   application code then at least applications have a chance at some benefit
   - As the work towards fixing the TODOs progresses, maintainers can
   easily look up what issues remain by consulting the list of errors that
   feeds the ratchet-style job.

The exact mechanism for the ratchet isn't the point here.  Maybe Mypy plays
more nicely with inline comments telling it to ignore something, I don't
know.  The point is:

   - A centralized list of stuff left to do is better than a mess smeared
   across the whole codebase
   - If we're going to risk breaking applications we should at least try to
   offer some value to them
   - We shouldn't make applications deal with every change twice when we
   could just disturb them once

Jean-Paul


> Specifically what I mean by "this genre of fix" is that you can always
> make mypy happy by transforming code like this:
>
> class Thing:
> pass
>
> Thing().stuff()
>
>
> into code like this:
>
> class Thing:
> @property
> def stuff(self):
> raise AttributeError("no such thing as `stuff`")
>
> Thing().stuff()
>
>
> The runtime behavior is the same, but now Mypy can't tell you about this
> class of bug any more.
>
> So, at some point, we should slowly unwind all `NotImplementedError`
> methods and find ways to either implement them for real or make their
> required use raise at import time.
>
> Finally: let's not develop an aversion to new tooling and change because
> it might create problems; experience over the last few years has shown me
> that Mypy can catch *tons* of real bugs and it's well worth getting the
> codebase to type check.  If we want to prevent breakages like this in the
> future, the answer is not to stop trying to get linters and typecheckers to
> run cleanly with arbitrary changes, but to figure out some kind of *continuous
> integration *solution that's actually continuous with our downstream
> dependencies
>
> If dependencies could start testing against Twisted trunk in some
> capacity, we could get notified close to when unintentionally breaking
> changes occur, and dependencies can let us know well before the release
> happens, and we can either revert or they can fix things if the error is on
> their end.  If initially, say, crossbar and matrix would like to work with
> us to set up some kind of repeatable pattern we can suggest to others, that
> would be great.
>
> -g
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

[Jean-Paul Calderone]

On Tue, Mar 2, 2021 at 3:09 PM Tobias Oberstein 
wrote:

> Hi all,
>
> rgd Twisted v21.2.0, thanks a lot to all involved in this release! the
> release, mypy, more py3, .. +1! really appreciated.
>
> now, fwiw, just wanted to let you know (sorry, too late) about a
> regression with
>
> https://github.com/crossbario/crossbar
>
> that actually originates from
>
> https://github.com/crossbario/autobahn-python/issues/1470
>
> =>
>
> with Twisted v20.3.0 calling
> twisted.internet.protocol.ProcessProtocol.getPeer raises an AttributeError
>
> with Twisted v21.2.0 calling the same method raises an
> NotImplementedError
>
> =>
>
> this fix broke our workaround for the bug now fixed;)
>
> https://github.com/crossbario/autobahn-python/pull/1469/files
>
> and released in:
>
> https://pypi.org/project/autobahn/21.3.1/
> https://pypi.org/project/crossbar/21.3.1/
>
> Cheers,
> /Tobias
>
>
It seems like this change does not actually violate Twisted's compatibility
policy.  This is not to say introducing such regressions is desirable, but
when constructing such work-arounds it might be worth paying
closer-than-average attention to what the policy actually is.

The policy is currently documented at
https://twistedmatrix.com/documents/current/core/development/policy/compatibility-policy.html#new-attributes
(and that anchor takes you to the section that I think allows the change).

Policy aside, this change doesn't seem like much of an improvement to me.
If I were to guess, I would guess the change was made to satisfy some check
Mypy is now being asked to make about Twisted.  If that's the case, it
seems unfortunate that real-world software is suffering so that a synthetic
goal can be achieved.  I do recognize there is a perception that practical
value can come from attending to the errors Mypy reports.  It would
probably benefit everyone if more care were taken to consider the
real-world consequences of changes that are made to satisfy the
non-real-world goalposts set by tools like Mypy.

Jean-Paul



> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Thread Cancelled

[Jean-Paul Calderone]

On Mon, Jan 25, 2021 at 5:14 PM Robert DiFalco 
wrote:

> So I have a simple question from all this. Is there a twisted idiom I can
> use so that deferred returned from a Klein route are not canceled if the
> client resets the connection? It's fine if it's before I've gotten the
> request body, but once I've gotten the request body I want all deferreds to
> succeed. I only wan't the streaming of the final result to fail.
>

As far as I know, you can't make Klein not cancel that Deferred.  However,
you can return a different Deferred.

def ignore_cancellation(d):

ignore_d = Deferred()

d.chainDeferred(ignore_d)

return ignore_d


Wherever you're returning a Deferred to Klein now, if you return
ignore_cancellation(that_deferred) instead then either:

   - it will run to completion and deliver its result to ignore_d which
   will deliver it to Klein; or
   - Klein will cancel it and the cancellation will not propagate to the
   original Deferred so that operation can complete (when it does, it will
   deliver the result to the cancelled Deferred which will drop it on the
   floor)

The documentation for Deferred cancellation could be a little bit clearer
on how cancellation works for Deferreds that are related in various ways,
including this way.  I'm not sure how you would discover this behavior
except for reading the implementation or doing experiments.  The way that
might make sense to think about it, though, is that results only propagate
in one direction down the chain - from d to ignore_d - they never flow the
other way.  So anything that happens to ignore_d cannot affect d.

Jean-Paul



> On Sun, Jan 24, 2021 at 7:40 PM Robert DiFalco 
> wrote:
>
>> Well, I've dealt with this issue with other languages. Not sure how to
>> deal with it in Klein/Twisted. This operation is idempotent so I suppose
>> what I'd like to happen is have the whole chain of deferred's succeed but
>> then just not be able to write the response to the socket -- but not
>> interrupt the chain. Unfortunately, what I really need is my operations to
>> be atomic -- introduce a two phase commit or some such. But that's too big
>> of a job on this legacy code base for now. Right now I'd be content if I
>> could save the database record and then send the SQS message to AWS and not
>> have that SQS send interrupted if the client closed the socket.. Is there a
>> simple way to achieve that? If I got the request body I'm good, I don't
>> care if I can't write the response.
>>
>> One other thing that would be nice is to know why a deferred was
>> canceled. If the client close a connection I might like to ignore the
>> cancel, but I think what is happening is that Twisted is pretty smart. So
>> it either knows I'm going to make a write to SQS using Boto that is
>> inbound, so it is somehow able to cancel that operation -- perhaps more
>> likely the client closed the connection and the connection canceled defers
>> after I called deferToThread but before a thread was available to run on.
>> Either way, I'd like them all to run, and just fail to write the final
>> response from the end of the chain.
>>
>> Sorry, too many words.
>>
>>
>>
>>
>> On Sun, Jan 24, 2021 at 3:22 PM Glyph  wrote:
>>
>>> If you're dealing with lots of clients on the public internet, sometimes
>>> this is just gonna happen, for a variety of reasons; it's normal.  We would
>>> welcome better error reporting for this scenario so it doesn't require the
>>> kind of debugging you just did :-).
>>>
>>> -g
>>>
>>> On Jan 24, 2021, at 2:58 PM, Robert DiFalco 
>>> wrote:
>>>
>>> That makes sense, thank you. A timeout seems unlikely but maybe the
>>> client is closing the connection due to a network issue. This is an
>>> extremely rare occurrence.
>>>
>>> On Sun, Jan 24, 2021 at 2:41 PM Glyph  wrote:
>>>
 While a socket is open and receiving data, recv() will either give you
 a non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA
 EAGAIN) if no bytes are ready.  A result of zero bytes (the empty string)
 means "end of file" - the other end has closed the socket.

 So what's happening here is your client is timing out or otherwise
 canceling its request by closing the socket, and this is the correct,
 intentional response to that scenario.

 -g

 On Jan 24, 2021, at 11:57 AM, Robert DiFalco 
 wrote:

 You're absolutely right, I meant "cancel the deferred". I don't grok
 server sockets very well so maybe someone can help. But apparently, klein
 does a .doRead from our server socket (getting the request from the
 client?). This returns a "why" of "connection done" so that closes the
 connection before we have written our response to the client, and that
 cancels the deferred SQS write.


 https://github.com/racker/python-twisted-core/blob/master/twisted/internet/selectreactor.py#L148-L155

 The method above is "doRead". Which calls this:

 

Re: [Twisted-Python] reactor for Linux io_uring

[Jean-Paul Calderone]

On Tue, Jan 5, 2021 at 6:49 AM Barry Scott 
wrote:

> On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
> > In investigating async file I/O I came across this. In a nutshell it's
> > the new epoll()
>
> > It's marginally more efficient although this is only apparent at very
> > high loads.
>
> > What's more interesting is that io_uring accepts files as
> > well as network/pipe handles: avoiding the need for threads.
>
> What threads? Why do you call out file FDs different from socket FDs?
>
> The point of io_uring is to avoid transitions between user and kernel
> right?
> Nothing to do with thread.
>
> In current twisted you can run complex network code without threads
> already.
>
> > Here's a good intro: https://unixism.net/loti/index.html
>
> Also there is full coverage of io_uring on lwn.net.
> Its a fast evolving kernel API.
>
> > If people think an IoUringReactor is worthwhile I'll open a ticket and
> > make a start.
>
> I'm guessing that you will need to write a Python extension to get at
> io_uring or use ctypes. Is that what you where expecting?
>
> > However it will need a reviewer... :-)
>
> Yes this is going to be complex code that few people have any experience
> with.
>

Just so there's a counter-perspective out there, I would like to suggest
that reactors are neither magical nor particular complex in the grand
scheme of software and that if you start with the assumption that a piece
of code is going to be complex and hard to review then you will almost
certainly create a piece of software that is complex and hard to review.

My recommendation would be to instead start with the assumption that it's
just a bit more mundane code binding a relatively small and straightforward
C API related to copying bytes from one location to another.  Yea, maybe
there will be some hassles getting it to compile smoothly in all the
desired environments or maybe there will be a few tricky areas for memory
management or some other kind of low-level, localized bookkeeping - but
those kinds of issues don't have to make for a complex piece of code.
Starting from this assumption, try to produce an implementation that is
straightforward, well-factored, and easily reviewed.  What do you have to
lose?

Jean-Paul


>
> Barry
>
> > Ian
> >
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
>
>
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Plan/Goal for GitHub Sponsors

[Jean-Paul Calderone]

On Sun, Jan 3, 2021 at 10:00 PM Ian Haywood  wrote:

> I think having a project coordinator as   a *first* priority isn't a good
> idea (not saying it isn't down the track)
>
> Currently there's little to coordinate as the core team aren't active in
> planning or review. can the team be expanded ? (I'm not angling for a role
> myself)
>

Since Glyph has asked that this discussion move to a different forum, I
won't respond to this counter-proposal here.

Jean-Paul


> Ian
> On 4/01/2021 11:59 am, Adi Roiban wrote:
>
>
>
> On Mon, 4 Jan 2021 at 00:54, Glyph  wrote:
>
>> It’s complicated and I’m not a lawyer, so maybe it is indeed not a
>> problem. But in brief it’s like trademark protection, kind of, in that it
>> becomes SFC’s problem to be aware that you’ve said these things and tell
>> you not to say them. The twisted project (which is a bit of an amorphous
>> concept to begin with) has authorized SFC to be its fiscal sponsor, the SFC
>> has gone through  the rigamarole with the IRS to ensure this is an
>> exempt-able public benefit activity, and now someone is making claims about
>> the project hiring, which they’re on the hook for. How does the IRS know
>> your status of affiliation with the project or the conservancy for sure?
>> Somebody has to investigate it, investigating means asking a bunch of
>> questions and sucking up the SFC’s time and energy, even if no enforcement
>> action is ever formally taken.
>>
>> In short: talk to the SFC first about the project’s status, get an actual
>> official recommendation and not my random opinion about what may or may not
>> be a problem, before doing anything related to fundraising. I can’t say
>> anything authoritative about what is allowed, because as far as I
>> understand it, *nothing* is allowed without untangling the PLC/approval
>> process first. :-)
>>
>> (Except to raise money for the already-authorized expenses related to the
>> continued hosting of twistedmatrix.com, of course.)
>>
>
> Thanks for the info and sorry for the trouble :)
>
> I will try to send a (private) message to SFC these days to untangle the
> PLC team and I will add you to CC.
>
> Until we solve the PLC issue, I would consider this discussion blocked.
>
> Cheers
> --
> Adi Roiban
>
> ___
> Twisted-Python mailing 
> listTwisted-Python@twistedmatrix.comhttps://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Plan/Goal for GitHub Sponsors

[Jean-Paul Calderone]

On Wed, Dec 23, 2020 at 4:41 PM Adi Roiban  wrote:

> Hi,
>
> I started a separate discussion to confirm the goal for a future
> fundraising.
>
> Top priority - Pay someone to help with the review queue
> Nice to have - Move server/services to Azure VM -  We have at least $100
> monthly allowance for Azure not sure if we still have the huge $2000
> allowance on rackspace.
> Nice to have - Migrate Trac wiki to GitHub Wiki
> Nice to have - Migrate Trac Ticket to GitHub Issues
>

I have a suggestion for another priority to be inserted above that top
priority - pay someone to look after the day-to-day logistics of the
project.  For example: to triage tickets in the issue tracker (identify
duplicates to avoid redundant effort, classify issues by feature request vs
defect vs regression, etc), keep track of the release process so releases
are completed in a timely manner, identify and eliminate friction in the
development process, and identify big-picture directions / priorities /
roadmap items and track and coordinate efforts to achieve them.  I don't
think someone could be hired to set the Twisted roadmap but someone could
be hired to solicit this information from core developers and the wider
community and organize it into a coherent plan.

Ideally this person could also look after fundraising efforts to ensure
that there are funds to continue to support their other activities.

Helping with the review queue is great but it's a purely reactive
activity.  This is fine so far as it goes but it leaves the project without
a coherent direction, which in turn makes less productive use of the
resources available.  The project should continue to operate reactively to
address issues raised by the community but to really stay relevant, the
core Twisted team itself also needs to identify coherent future goals and
work to achieve them.

Messing with CI configuration, wikis, issue tracking, etc, may also all be
beneficial but they're not useful goals in themselves - they should all be
in support of a goal like reducing operational overhead to allow resources
to be directed elsewhere or removing roadblocks that stand in the way of
other contributors having an impact.

Jean-Paul


>
> That is all..but first we should solve the release queue.
> Cheers
> --
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Upcoming Twisted Release

[Jean-Paul Calderone]

On Mon, Dec 28, 2020 at 6:13 AM Adi Roiban  wrote:

>
>>> I plan to update the release documentation to make it clear that all
>>> release blocker tichet
>>> should have an owner and there are plans to fix the ticket in a maximum
>>> of 2 weeks.
>>>
>>> Otherwise we risk to block the release forever... and if we delay
>>> forever people will start using "trunk"
>>> and if everybody is using trunk, what is the point of a release :) ?
>>>
>>
>> Part of the point is that when someone runs `pip install ...` they get a
>> *working* version of Twisted, to the best of the project's ability to
>> provide one.
>>
>> Fortunately many regressions aren't that difficult to resolve.  At worst,
>> find the merge that introduced them and revert it.  This works best when
>> regressions are found in a timely manner, of course.  Of course it's also
>> nice if the problem can be fixed without backing out whatever (presumably
>> desirable) set of changes it came along with.
>>
>> Part of the release managers job is to motivate this kind of work to
>> happen.  A standing policy to revert the cause of a regression can also
>> serve as good motivation to get the other kind of fix in, too.
>>
>> It's better if these known regressions don't linger for months, though.
>> It looks like the Buildbot PR had a failing CI run in October.  I'd suggest
>> that not waiting until December is a good way to avoid having these kinds
>> of situations turn into a larger problem.
>>
>> Jean-Paul
>>
>>
> Thanks Kyle and Jean-Paul for your feedback.
>
> I guess there are no comments against removing a ticket from the
> release-blocking list if the ticket is not active for 1 or 2 weeks.
>
>
Commenting against this was the main reason for my earlier reply.  I've
left my quoted reply above.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Upcoming Twisted Release

[Jean-Paul Calderone]

On Sun, Dec 27, 2020 at 6:59 PM Adi Roiban  wrote:

> Hi Craig,
>
> On Sun, 27 Dec 2020 at 20:10, Craig Rodrigues 
> wrote:
>
>>
>>
>> On Sat, Dec 26, 2020 at 3:50 PM Adi Roiban  wrote:
>>
>>> Hi,
>>>
>>> I plan to act as a release manager for the next release and follow the
>>> plan documented at
>>>
>>>
>>> https://docs.twistedmatrix.com/en/latest/core/development/policy/release-process.html
>>>
>>>
>> I was previously working on releasing Twisted.  I was running into
>> various roadblocks, but was moving forward,
>> and got permission from Glyph to move forward with this.
>> Has this changed?
>>
>
> If you want to do the release, I am more than happy to not have to do the
> release myself :)
>
>
>> Unfortunately, Amber did not respond to any e-mails that I sent to her
>> and Glyph, so I tried to move forward the
>> best that I could.
>>
>
> Is there anything still blocking you ?
> Can I help?
>
>
>>>
>>> So no other tickets are in the blocker queue:
>>> https://twistedmatrix.com/trac/report/26
>>>
>>> --
>>>
>>> Do you know any other release blocker issues?
>>>
>>>
>> I filed this:
>> https://twistedmatrix.com/trac/ticket/10070
>>
>> which I found with Pierre Tardy's help by running buildbot's test suite
>> against Twisted trunk.
>> This looks like a problem on the Twisted side, and should be fixed before
>> a Twisted release is pushed out.
>>
>
> OK. I closed https://twistedmatrix.com/trac/ticket/10069 as I think it's
> a duplicate.
>
> Do you or Pierre plan to fix that ticket?
>
> I think that you should block the release only if someone is committed to
> fixing the release blocker.
>
> I plan to update the release documentation to make it clear that all
> release blocker tichet
> should have an owner and there are plans to fix the ticket in a maximum of
> 2 weeks.
>
> Otherwise we risk to block the release forever... and if we delay forever
> people will start using "trunk"
> and if everybody is using trunk, what is the point of a release :) ?
>

Part of the point is that when someone runs `pip install ...` they get a
*working* version of Twisted, to the best of the project's ability to
provide one.

Fortunately many regressions aren't that difficult to resolve.  At worst,
find the merge that introduced them and revert it.  This works best when
regressions are found in a timely manner, of course.  Of course it's also
nice if the problem can be fixed without backing out whatever (presumably
desirable) set of changes it came along with.

Part of the release managers job is to motivate this kind of work to
happen.  A standing policy to revert the cause of a regression can also
serve as good motivation to get the other kind of fix in, too.

It's better if these known regressions don't linger for months, though.  It
looks like the Buildbot PR had a failing CI run in October.  I'd suggest
that not waiting until December is a good way to avoid having these kinds
of situations turn into a larger problem.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Current twisted dns client just doesn't work

[Jean-Paul Calderone]

On Fri, Dec 11, 2020 at 10:31 AM spam tam  wrote:

> Yes. I would like to replace ANY with A and  requests.
> I created pull request: https://github.com/twisted/twisted/pull/1488
> It's an easy solution. I just request for A records and if A doesn't
> provide IP I create  request.
>

Thanks for your work on this so far.  Are you interested in finishing up
the PR (at least go down the rest of the checklist)?  If so, wonderful.  If
not, it would be good to know and maybe someone else can pick up the task
from here.

Jean-Paul


>
> On Fri, Dec 11, 2020 at 6:03 PM Barry Scott 
> wrote:
>
>> On Friday, 11 December 2020 14:23:49 GMT spam tam wrote:
>> > Dis you read the whole my email?
>> > Did you read this:
>> > https://blog.cloudflare.com/rfc8482-saying-goodbye-to-any/
>> >
>> > ANY is not supported by internet. Sometime works sometime not
>>
>> Oh I missed that. That is very interesting.
>> Is that what you are trying to fix in twisted? The use of ANY?
>>
>> Barry
>>
>> >
>> > Пт, 11 дек. 2020 г. в 12:26, Barry Scott :
>> >
>> > > On Thursday, 10 December 2020 23:29:33 GMT spam tam wrote:
>> > > > I would like to find problems with ANY. But I think that there is no
>> > > > problem.
>> > > > DNS servers don't provide standard response for ANY request. My
>> local
>> > > > machine doesn't provide correct response for request:
>> > >
>> > > So you need to fix your network infra not twisted right?
>> > >
>> > > Barry
>> > >
>> > >
>> > > >
>> > > > $ dig amazon.in any
>> > > >
>> > > > ; <<>> DiG 9.16.1-Ubuntu <<>> amazon.in any
>> > > > ;; global options: +cmd
>> > > > ;; connection timed out; no servers could be reached
>> > > >
>> > > > My VPS server provide such response:
>> > > >
>> > > > ; <<>> DiG 9.10.3-P4-Ubuntu <<>> amazon.in any
>> > > > ;; global options: +cmd
>> > > > ;; Got answer:
>> > > > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54130
>> > > > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
>> > > >
>> > > > ;; OPT PSEUDOSECTION:
>> > > > ; EDNS: version: 0, flags:; udp: 512
>> > > > ;; QUESTION SECTION:
>> > > > ;amazon.in. IN  ANY
>> > > >
>> > > > ;; ANSWER SECTION:
>> > > > amazon.in.  3599IN  HINFO   "RFC8482" ""
>> > > >
>> > > > ;; Query time: 40 msec
>> > > > ;; SERVER: 8.8.8.8#53(8.8.8.8)
>> > > > ;; WHEN: Thu Dec 10 22:10:39 UTC 2020
>> > > > ;; MSG SIZE  rcvd: 59
>> > > >
>> > > > It sometimes provides another response. But the problem is that
>> behaviour
>> > > > with ANY is not stable.
>> > > > The present and the future of ANY are hazy. Read more here:
>> > > > https://blog.cloudflare.com/rfc8482-saying-goodbye-to-any/
>> > > >
>> > > > If you don't see the problem please provide your opinion. I would
>> like to
>> > > > find solution with ANY but it seems it is impossible.
>> > > >
>> > > > So what do you think?
>> > > >
>> > > > On Thu, Dec 10, 2020 at 8:30 PM Barry Scott <
>> barry.sc...@forcepoint.com>
>> > > > wrote:
>> > > >
>> > > > > Answers to but your emails in line.
>> > > > >
>> > > > > I've added the twisted list back in so others can comment.
>> > > > >
>> > > > > On Wednesday, 9 December 2020 21:17:51 GMT spam tam wrote:
>> > > > > > As additional information you can check how operation system
>> works
>> > > with
>> > > > > dns.
>> > > > > > You can run
>> > > > > >
>> > > > > > *sudo tcpdump -i lo -v port 53*
>> > > > > > as UDP local sniffer.
>> > > > >
>> > > > > Agreed great tool to debug this stuff with.
>> > > > > And use wireshark to decode the output.
>> > > > >
>> > > > > > And run
>> > > > > >
>> > > > > > *dig google.com *
>> > > > > >
>> > > > > > And you will see that it makes A request. Not ANY
>> > > > >
>> > > > > That is the dig default to use A. Use this to do a any query.
>> > > > >
>> > > > >dig google.com any
>> > > > >
>> > > > > I checked the man page to see if anything extra can be printed
>> but it
>> > > > > looks like
>> > > > > the default is to print everything dig knows how to print. The
>> options
>> > > only
>> > > > > remove output it seems.
>> > > > >
>> > > > > > On Wed, Dec 9, 2020 at 11:42 PM spam tam 
>> > > wrote:
>> > > > > >
>> > > > > > > Yes. You are correct. My local dns just is not stable. But try
>> > > this:
>> > > > > > >
>> > > > > > > python3.8 dns_lookup6.py amazon.in
>> > > > > > >
>> > > > > > > It provides response:
>> > > > > > >
>> > > > > > > --- dnsLookupDone ([> > > ttl=3599s
>> > > > > > > auth=False>], [], [])
>> > > > > > > --- dnsLookupDone RR > > > ttl=3599s
>> > > > > > > auth=False> type 13 payload > ttl=3599>
>> > > > > > >
>> > > > > > > There are not CNAME, A,  or NS records. This means that
>> current
>> > > > > > > twisted.names.common.extractRecord will not provide IP.
>> > > > > > > What do you think about this? May be we should replace dns ANY
>> > > request
>> > > > > > > with A and ?
>> > > > >
>> > > > > Now that is interesting because dig returns 54 lines of output
>> from:

Re: [Twisted-Python] Unjellying and circular references?

[Jean-Paul Calderone]

On Fri, Dec 11, 2020 at 8:19 AM Jasper Phillips 
wrote:

> I'm using perspective broker to transfer objects in a networked game,
> which I'm having trouble unjellying -- the remote versions wind up with
> dangling twisted.persisted.crefutil._Dereference instances, so don't match
> the original objects.
>

Hi!

I'd strongly suggest you switch to a different protocol.  HTTP is quite
capable.  It has some annoying corners but you can avoid them if you
control the client and the server.  If you really want a Twisted-originated
protocol, AMP is a much better choice than PB.  It offers a little bit more
functionality than HTTP without going to the extreme complexity of PB (I am
not going to try to explain those complexities here, I'm just offering my
personal conclusion based on 20 years of experience developing and/or using
Twisted).

This is not to say that PB is the wrong choice for all applications ... but
probably almost all.


> I'm seeing this for objects that have circular references to each other.
> I've refactored things to mostly avoid circular references and sidestep
> this, but have one remaining case where I find circular references mean
> clearer code that I'm reluctant to refactor.
>
> Is there some trick I'm missing to avoid _Dereferences?
>

No, it's supposed to Just Work™ so you've found a bug in some part of the
implementation.  If you can produce a minimal reproducing example then it
may be worth a bug report.  But after that, I'd suggest investigating HTTP
or AMP as a replacement for PB.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] ClientFactory design

[Jean-Paul Calderone]

On Mon, Nov 30, 2020 at 6:48 AM Peter Westlake 
wrote:

> I'm studying an SSH client implementation using Conch, and was surprised
> to see that the ClientFactory subclass had a connect() method which calls
> connectTCP. This isn't really a problem - the code works - but it seems
> like an odd place to do the connection. Am I right to be surprised?
>

Putting transport setup on the factory unnecessarily couples protocol and
transport logic.  More flexibility is provided by keeping these concerns
independent.


>
> Peter.
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Retry: Conch Issues

[Jean-Paul Calderone]

On Thu, Sep 24, 2020 at 8:54 PM Robert DiFalco 
wrote:

> Why exactly is makeDirectory not firing any callbacks or errbacks and what
> can I add at the Factory level to make it do so?
>

class MySSHClientFactory(SSHClientFactory):
> def clientConnectionLost(self, connector, reason):
> self.op.addErrback(reason)
>
> def makeDirectory(self, path, attrs):
> self.op = Deferred()
> def _cbSuccess(result):
> self.op.callback(result)
>
> return self.sftpClient.makeDirectory(path, 
> attrs).addCallback(_cbSuccess)d
>
>
This code is wrong.  It takes what *may* be a perfectly good Deferred from
makeDirectory and breaks the error chain off of it.  The correct way to do
this would be more like `makeDirectory(...).chainDeferred(self.op)`.
However, `self.op` and `_cbSuccess` are doing nothing useful here so you
may as well just delete all that and return the `makeDirectory(...)`
Deferred.

If *that* Deferred doesn't fire its errback chain when the connection is
closed, that's a bug in Conch.  If you can put together a minimal example
and file a ticket, that would be wonderful and perhaps someone can get it
fixed for you.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

[Jean-Paul Calderone]

On Fri, Sep 11, 2020 at 1:34 PM  wrote:

> Hey guys,
>
>
>
> Last year I hit a condition discussed in this ticket:
> https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a
> twisted.internet.tcp.Port.
>
>
>
> I ignored it at the time since it was just on Linux, and my main platform
> was Windows.  Now I’m coming back to it.  I’ll add context on the problem
> below, but first I want to ask a high-level, design-type question with
> multiprocessing and Twisted:
>
>
>
> Referencing Jean-Paul’s comment at the end of ticket 4759, I read you
> shouldn’t fork a process (multiprocessing module) that already has a
> Twisted reactor.  Understood.  But what about a parent process (not doing
> anything Twisted) forking child processes, where each child process starts
> their own Twisted reactor?  Is that intended to work from the Twisted
> perspective?
>

To answer the asked question, I don't think there is rigorous (or even
casual) testing of very much of Twisted in the context of "some Twisted
code has been loaded into memory and then the process forked".  So while it
seems like a reasonable thing, I wouldn't say there's currently much effort
being put towards making it a *supported* usage of Twisted.  Of course this
can change at almost any moment if someone decides to commit the effort.

To dig a bit further into the specific problem, even if you only *import* the
reactor in the parent process and then fork a child and try to start the
reactor in the child, I strongly suspect epollreactor will break.  This is
because the epoll object is created by reactor instantiation (as opposed to
being delayed until the reactor is run).  epoll objects have a lot of weird
behavior.  See the *Questions and Answers* section of the epoll(7) man page.

I don't know if this is the cause of your particular expression of these
symptoms (it certainly doesn't apply to the *original* bug report which is
on FreeBSD where there is no epoll) but it's at least *a possible* cause.

This could probably be fixed in Twisted by only creating the epoll object
when run is called.  There's nothing particularly difficult about that
change but it does involve touching a lot of the book-keeping logic since
that all assumes it can register file descriptors before the reactor is
started (think reactor.listenTCP(...); reactor.run()).

I'm not sure but it may also be the case that only delaying creation of the
*waker* until the reactor starts would also fix this.  This is because as
long as the epoll object remains empty a lot of the weird behavior is
avoided and the waker is probably the only thing that actually gets added
to it if you're just *importing* the reactor but not *running* it before
forking.

Alternatively, your application *should* be able to fix it by studiously
avoiding the import of twisted.internet.reactor (directly or transitively,
of course).  You could add some kind of assertion about the state of
sys.modules immediately before your forking code to develop some confidence
you've managed this.

And if this is really an epoll problem then switching to poll or select
reactor would also presumably get rid of the issue.

Jean-Paul


>
>
>
> Context:
>
> I only hit this problem on Linux, not Windows.
>
>
>
> The software project (github.com/opencontentplatform/ocp) has been a lot
> of fun, especially with walking the tight rope in using multi-processing,
> multi-threading, and Twisted reactors.  The main controller process kicks
> off about 10 child processes, each doing different types of work.  In
> general though, the child processes individually start a Twisted reactor,
> connect to Kafka, connect to a database, use a shared REST API, and some
> listen for connecting clients to accomplish work.
>
>
>
> I test on Linux about once a year, so too many changes to rollback and
> figure out that way.  It was working on Linux 2 years ago, but last year’s
> testing and current testing, receive the doWrite error.  It continues
> running fine on Windows.  I’ve gone back about 2 years of versions with
> Python3, Twisted, and dependent libs… on both Windows and Linux.  Every
> version change yields the same result - continues to work on Windows and
> continues to hit the error on Linux.  So something I added has caused Linux
> to throw that error.
>
>
>
> I’m not explicitly sharing much between the main process and the sub
> processes.  They are spun up with (1) a shared multiprocessing.Event() - to
> have the main process shut the children down, (2) their own unique
> multiprocessing.Event() - to have the child processes notify back to the
> parent, and (3) a deep copy of a dictionary (containing a bunch of settings
> that remain constant).  The main process uses twisted.logger, but for
> testing I strip that out to remove any twisted imports in the main
> process.  So I’m not importing anything Twisted in the main process, and I
> don’t believe I’m explicitly sharing something I shouldn’t.  Seems like
> something is implicitly being 

Re: [Twisted-Python] Management of PyPI maintainers (as related to qt5reactor)

[Jean-Paul Calderone]

On Thu, Jul 30, 2020 at 3:58 PM Kyle Altendorf  wrote:

> On 2020-07-30 14:10, Jean-Paul Calderone wrote:
>
> >
> > So, if I've missed something, maybe you can help clarify.
>
> qt5reactor isn't particularly active and and my hope in it moving into
> the Twisted organization is that if all 'active' maintainers are lost
> and someone else volunteers later, an organizational maintainer could
> choose to give the new volunteer the necessary authority.  It may well
> be that this is a silly reason to make the move but I haven't been
> corrected about it yet.  :]
>
> I didn't originate the 1password suggestion but if a Twisted PyPI
> account were created, as Adi mentioned, and the credentials stored in
> 1password then that would associate control with the Twisted
> organization rather than individual developers.  The presently 'active'
> individual developers would presumably retain their PyPI maintainership
> rights as well.
>
> Any more clear now?
>
>
Yep, thanks for clarifying.


> Cheers,
> -kyle
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Management of PyPI maintainers (as related to qt5reactor)

[Jean-Paul Calderone]

On Thu, Jul 30, 2020 at 10:34 AM Kyle Altendorf  wrote:

> Following up on:
> https://github.com/twisted/qt5reactor/issues/50#issuecomment-658432478
>
> qt5reactor has recently been moved into the Twisted organization on
> GitHub.  The intent is that being in an org will make it less likely
> that existing maintainers disappear and the project is stranded with
> nobody having the authority to pass it on to any new maintainers.  If we
> happen to get more people interested in maintenance that's a bonus, but
> it is not the reason for the move.
>
> The question is, how should the Twisted organization manage PyPI access
> for its projects?  Glyph mentioned there is a 1password account that
> could be relevant.  I have not used 1password personally so I don't know
> any details about how it would fit in here.  Twisted itself has six
> maintainers listed on PyPI: exarkun, glyph, hawkowl, itamarst, jml, and
> markrwilliams.
>
> Any opinions?  1Password vs.
> just-add-a-couple-maintainers-to-the-qt5reactor-pypi vs. ...?
>
>
Can you clarify this a bit?  PyPI has perfectly serviceable support for
multiple maintainers per project.  What benefits come from sharing some
kind of credentials (and what credentials) via a tool like 1Password?

It seems like folks who should be qt5reactor PyPI maintainers can have
their personal PyPI accounts configured as maintainers on PyPI and then the
problem's solved.

So, if I've missed something, maybe you can help clarify.

Thanks,
Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] mypy integrated with CI for twisted

[Jean-Paul Calderone]

On Wed, Jun 24, 2020 at 12:44 AM Glyph  wrote:

>
>
> On Jun 23, 2020, at 5:34 AM, Adi Roiban  wrote:
>
> Hi Craig,
>
> On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues 
> wrote:
>
>> I have merged some more fixes for mypy to Twisted trunk branch.
>>
>> In trunk, you can run mypy with:
>>
>> *tox -e mypy*
>>
>> Currently this results in *171* errors, which is way down from >1000
>> errors
>> a month ago.
>>
>> In addition, if you look at any new PR's there is a *Mypy Ubuntu* job
>> running on Azure pipeline, which runs mypy.  Right now errors from this
>> job
>> are ignored and does not block the PR.  However, if we can get the mypy
>> errors down to zero, we can make mypy status a blocker for the PR.
>>
>>
> Thanks for working on this.
>
> Looking forward to have a real green mypy build.
>
> A general question: Why Twisted used Azure Devops and not GitHub actions?
>
>
> Azure Pipelines gave us substantially more parallel capacity than is
> available via Github Actions, which means we can make build statuses appear
> much sooner.  Plus they support more platforms.
>

Does Twisted have a special deal with Azure Pipelines?  Or is the use of
past-tense in this sentence intentional? :)  Or are the docs for the
respective platforms wrong/misleading?

https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent-jobs?view=azure-devops
says
free-tier public projects get 10 parallel jobs.

https://help.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits
says
free tier projects get 20 parallel jobs.

(Of course this says nothing about the number of supported platforms.)

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twistedchecker now uses pylint >= 2.4.4, can we use type annotations in Twisted now?

[Jean-Paul Calderone]

On Sun, Jun 7, 2020 at 7:22 PM Craig Rodrigues 
wrote:

> I have merged a few PR's to trunk which eliminate hundreds of errors
> encountered with:
>
>
> *tox -e mypy*
> I think we can take several passes with more PR's to whack away all these
> mypy errors,
> and turn on mypy as part of the default CI for Twisted.
>
> I have seen a few errors like:
>
> src/twisted/words/protocols/jabber/sasl_mechanisms.py:47:1: error:
> 'Anonymous' is missing following 'ISASLMechanism' interface members:
> getResponse.  [misc]
> class Anonymous(object):
> src/twisted/words/protocols/jabber/sasl_mechanisms.py:61:1: error: 'Plain'
> is missing following 'ISASLMechanism' interface members: getResponse.
>  [misc]
> class Plain(object):
> src/twisted/internet/_dumbwin32proc.py:110:1: error: 'Process' is missing
> following 'twisted.internet.interfaces.ITransport' interface members:
> getHost, getPeer.  [misc]
> class Process(_pollingfile._PollingTimer, BaseProcess):
> src/twisted/internet/process.py:959:1: error: 'PTYProcess' is missing
> following 'twisted.internet.interfaces.ITransport' interface members:
> getHost, getPeer.  [misc]
> class PTYProcess(abstract.FileDescriptor, _BaseProcess):
> src/twisted/internet/process.py:959:1: error: 'PTYProcess' is missing
> following 'IProcessTransport' interface members: closeChildFD,
> writeToChild.  [misc]
> class PTYProcess(abstract.FileDescriptor, _BaseProcess):
> src/twisted/internet/base.py:504:1: error: 'ReactorBase' is missing
> following 'IReactorCore' interface members: run.  [misc]
> class ReactorBase(PluggableResolverMixin)
>
>
> For a class to properly implement a Zope interface, is it mandatory that
> it  implement every method in that interface?
>

Yes.


>
> If we modify the classes with mypy errors to properly implement these
> methods (even with no-ops) is that the correct
> way to go?
>

Who does this serve?  I would say no, this is not correct.  If a type
declares it implements an interface and it cannot provide useful
implementations of every method/attribute, then it made a mistake in its
declaration or the interface has the wrong methods/attributes.

Jean-Paul

>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

[Jean-Paul Calderone]

On Thu, May 21, 2020 at 4:46 PM Sereysethy TOUCH 
wrote:

> I am on MacOS, it is a development phase, but deployment will be on Linux.
>
> It is hard to run dtrace/dtruss on MacOS due to the "system integrity
> protection" on MacOS. I got some output but very limited information.
>
> So if there is a workaround you might suggest?
>

I would definitely recommend nailing down the exact cause of the problem.
I think that system integrity protection can be bypassed, perhaps by
creating a virtualenv?  I am not particularly familiar with macOS but I'm
sure there's some good information somewhere out there.

If the problem is the blocking network operations performed by
`PolicyClient` then you'll probably have to move even the instantiation of
that class to a separate thread.  This is probably good anyway, since it's
not necessarily the case that it is safe to create the object in one thread
and then use it in another.  Better to create and use it all in the same
thread.

Jean-Paul


>
> TS
>
> On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
>> On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH <
>> touch.sereyse...@gmail.com> wrote:
>>
>>> Hi Jean-Paul,
>>>
>>> I found the error after working on a short, self contained, correcte
>>> example. It is not about twisted and normal thread in general, but
>>> something else.
>>>
>>> After more tests, I come to realise that whenever I instantiate this
>>> class PolicyClient from this library
>>> https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in
>>> my protocol or just anywhere, twisted app either freezes or terminates, and
>>> this *only* happens when the app runs as a daemon, created using a
>>> service (.tac file).
>>>
>>> I hope you can take a look at the class PolicyClient, why it causes
>>> problem to reactor main loop. Please point me to where the idea might come
>>> from.
>>>
>>> I can give you the example but not sure if it is enough, as the client
>>> needs to connect to a server.
>>>
>>
>> It looks like PolicyClient does a blocking HTTP request in __init__.
>> Since PolicyClient is instantiated in the reactor thread, this will block
>> the reactor.  Perhaps this call is hanging for some reason when the process
>> has daemonized?  This would be unusual but not unheard of.  For example,
>> there are macOS environments where a process does not have access to the
>> network if it is not associated with a windowing session.
>>
>> What platform do you observe the problem on, and what more can you learn
>> about what the process does on its way to hanging (eg can you run strace on
>> it)?
>>
>> Jean-Paul
>>
>>
>>>
>>> Thank you,
>>> TS
>>>
>>>
>>> On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
>>> exar...@twistedmatrix.com> wrote:
>>>
>>>> On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
>>>> touch.sereyse...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am developing a twisted app, and it runs as a service using twistd
>>>>> -y to start the app.
>>>>> I am having a problem of running a thread. I know it is not
>>>>> recommended to use thread, but the library that I use, the object created
>>>>> is running in a thread.
>>>>>
>>>>> Here is the problem:
>>>>>
>>>>> 1) if I start reactor by running reactor.run() directly, thread is
>>>>> running fine
>>>>> 2) if I run it as a service using twisted, the thread is not running,
>>>>> it runs but it seems to be blocked, because I tried to write something to
>>>>> file using time.sleep(), but file is empty.
>>>>>
>>>>> Is there something that I miss? How can I debug this?
>>>>>
>>>>
>>>> A good next step would be to create at Short, Self Contained, Correct
>>>> (Compilable), Example <http://sscce.org/> and share it.
>>>>
>>>> Jean-Paul
>>>>
>>>>
>>>>>
>>>>> Thank you,
>>>>>
>>>>> TS
>>>>> ___
>>>>> Twisted-Python mailing list
>>>>> Twisted-Python@twistedmatrix.com
>>>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>>>
>>>> ___
>>>> Twisted-Python mailing list
>>>> Twisted-Python@twistedmatrix.com
>>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>>
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

[Jean-Paul Calderone]

On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH 
wrote:

> Hi Jean-Paul,
>
> I found the error after working on a short, self contained, correcte
> example. It is not about twisted and normal thread in general, but
> something else.
>
> After more tests, I come to realise that whenever I instantiate this class
> PolicyClient from this library
> https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in
> my protocol or just anywhere, twisted app either freezes or terminates, and
> this *only* happens when the app runs as a daemon, created using a
> service (.tac file).
>
> I hope you can take a look at the class PolicyClient, why it causes
> problem to reactor main loop. Please point me to where the idea might come
> from.
>
> I can give you the example but not sure if it is enough, as the client
> needs to connect to a server.
>

It looks like PolicyClient does a blocking HTTP request in __init__.  Since
PolicyClient is instantiated in the reactor thread, this will block the
reactor.  Perhaps this call is hanging for some reason when the process has
daemonized?  This would be unusual but not unheard of.  For example, there
are macOS environments where a process does not have access to the network
if it is not associated with a windowing session.

What platform do you observe the problem on, and what more can you learn
about what the process does on its way to hanging (eg can you run strace on
it)?

Jean-Paul


>
> Thank you,
> TS
>
>
> On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
>> On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
>> touch.sereyse...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I am developing a twisted app, and it runs as a service using twistd -y
>>> to start the app.
>>> I am having a problem of running a thread. I know it is not recommended
>>> to use thread, but the library that I use, the object created is running in
>>> a thread.
>>>
>>> Here is the problem:
>>>
>>> 1) if I start reactor by running reactor.run() directly, thread is
>>> running fine
>>> 2) if I run it as a service using twisted, the thread is not running, it
>>> runs but it seems to be blocked, because I tried to write something to file
>>> using time.sleep(), but file is empty.
>>>
>>> Is there something that I miss? How can I debug this?
>>>
>>
>> A good next step would be to create at Short, Self Contained, Correct
>> (Compilable), Example <http://sscce.org/> and share it.
>>
>> Jean-Paul
>>
>>
>>>
>>> Thank you,
>>>
>>> TS
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

[Jean-Paul Calderone]

On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
touch.sereyse...@gmail.com> wrote:

> Hello,
>
> I am developing a twisted app, and it runs as a service using twistd -y to
> start the app.
> I am having a problem of running a thread. I know it is not recommended to
> use thread, but the library that I use, the object created is running in a
> thread.
>
> Here is the problem:
>
> 1) if I start reactor by running reactor.run() directly, thread is running
> fine
> 2) if I run it as a service using twisted, the thread is not running, it
> runs but it seems to be blocked, because I tried to write something to file
> using time.sleep(), but file is empty.
>
> Is there something that I miss? How can I debug this?
>

A good next step would be to create at Short, Self Contained, Correct
(Compilable), Example  and share it.

Jean-Paul


>
> Thank you,
>
> TS
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Jean-Paul Calderone]

On Tue, May 19, 2020 at 8:33 AM Barry Scott 
wrote:

> On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
>
> >
> > If looking improve process management in Twisted, I think that the
> > lowest-hanging fruit is to watch children with pidfd, rather than reaping
> > all children on SIGCHLD. This O(n) reaping --- one waitpid() call per
> > running process --- starts to consume substantial CPU. IIRC it pegged a
> > core at 20,000 processes or so.
>
> Why not waitpid for all children (pid=0 or pid=-1) in a single call?
>
> Barry
>

Because then you might reap a child process someone launched with a
different API - eg the Python subprocess module.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] unified filesystem API (was: SMB server component for twisted)

[Jean-Paul Calderone]

Is anyone going to mention twisted.vfs?

On Tue, May 19, 2020 at 8:33 AM Barry Scott 
wrote:

> On Saturday, 16 May 2020 02:14:57 BST Ian Haywood wrote:
> > On 16/05/2020 10:55 am, Ian Haywood wrote:
> > > On 15/05/2020 10:52 am, Glyph wrote:
> > >> y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega  > >> 
> > >>
> > >>>   I think it's great to get an SMB implementation in the Twisted
> > >>> org, but why would we even consider adding something like this to
> > >>> the main Twisted project?
> > >
> > > The advantage of twisted itself is cross-protocol abstractions such as
> > > cred. Of course you don't have to be in the repo to use them, but in
> > > practice developers need the discipline of a single project to
> > > maintain consistency, otherwise the human tendency to reinvent wheels
> > > is too strong
> >
> > Apropos we have two APIs for exporting filesystems which are broadly
> > similar: SFTP in conch and FTP itself, SMB is about to be a third,
> > ideally we should have one or have them descend from one another. It's
> > slightly harder than it sounds as SMB is a file-access, as opposed to
> > file-transfer, protocol and supports some extra features such as locking.
>
> I do not see a common class tree works for things that are so different
> as SMB and FTP.
>
> Barry
>
>
>
> >
> > Ian
> >
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how will twisted handle generator-based coroutines deprecation?

[Jean-Paul Calderone]

On Wed, May 6, 2020 at 11:44 AM Vladimir Chudyk 
wrote:

> Looking at the docs I found that Generators-based coroutines, something
> that twisted's inline callbacks depend a lot, will be deprecated soon by
> the time python 3.10 is released.
>
>
> https://docs.python.org/3/library/asyncio-task.html#generator-based-coroutines
>
>
> * NoteSupport for generator-based coroutines is deprecated and is
> scheduled for removal in Python 3.10.*
>
> This is a problem since any future python release will come without the
> generator-based coroutines that twisted needs, so any higher version of
> python 3.10 will, for the most part, completely broke a lot of twisted code
> that already exists.
> How will then twisted handle this deprecation?
>

This seems to be a deprecation of the asyncio library's implementation of
generator-based coroutines.  It doesn't seem like it should affect any code
written using `inlineCallbacks`, only code that uses `asyncio.coroutine`.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] The Python 2.7 Plan

[Jean-Paul Calderone]

On Wed, Dec 11, 2019 at 4:06 PM Wilfredo Sánchez Vega 
wrote:

>   With Python 2.7 support dropping in a matter of weeks, I'd like to pick
> up from hawkowl's thread back in March about Python 2.7 support for Twisted.
>

Despite the widely circulated January 1 2020 date, it seems that python-dev
nevertheless plans to make a Python 2.7 release in April 2020 -
https://www.python.org/dev/peps/pep-0373/

This is not to say the questions here don't need answering or that there
isn't some urgency at this point, but it seems like the January 1 date is
essentially an arbitrary deadline without a clear connection to the support
status of CPython 2.7.

Or does anyone know a different way to reconcile these apparently
conflicting dates?

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Continued towncrier support?

[Jean-Paul Calderone]

On Thu, Aug 8, 2019 at 4:42 PM Sebastian Berg 
wrote:

> Hi all,
>
> I am not sure that this is the right place, but if I understand
> correctly towncrier grew out of twisted. For NumPy (and likely more of
> the scientific python ecosystem), we are looking into alternatives to
> create release notes right now.
>
> Towncrier seems to be a very nice option, but currently has some
> limitations which do not fit to how our release notes are formatted. I
> have started on a PR (and saw old stalled ones that did similar
> things), which give towncrier enough flexibility to be used by us
> nicely.
>

I like towncrier too!  And there seems to be plenty of community interest.
It would be sad to see this interest turn into a bunch of incompatible
forks.

To this end, I just filed https://github.com/hawkowl/towncrier/issues/159

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] self.channel.writeHeaders(version, code, reason, headers)\nbuiltins.AttributeError: \'NoneType\' object has no attribute \'writeHeaders\'\

[Jean-Paul Calderone]

On Mon, Aug 5, 2019 at 2:15 PM Waqar Khan  wrote:

> False alarm. Seems like there are some sneaky conditions when I get the
> error message.
> Like before, I use to get notifyFinish error everytime. Now, it seems that
> 6/10 times  things are "clean" but then 4/10 times there are notifyFinish
> errors.
> Wondering on your suggestion. How do I ensure whether the notifyFinish
> error deferred has been fired or not.
> Could I have like a vanilla "HelloWorld" example?
> Thanks
>
>
You can find an example of this here -
https://twistedmatrix.com/documents/current/web/howto/web-in-60/interrupted.html

Jean-Paul


> On Mon, Aug 5, 2019 at 2:24 AM Waqar Khan  wrote:
>
>> Hi Glyph,
>> I am not sure I understand.
>>
>> Is there a method/variable in request which keeps a track whether
>> notifyFinish has been fired..
>> So, I can do something like.
>>
>> if not request.hasFiredNotifyFinish:
>>request.finish()
>> ??
>>
>>
>> I have sort of able to get around this issue.. though I can't put a
>> finger on what actually worked.
>> Here is what I did.. First change to 19.7rc01 version.. and just fix the
>> channel issue.
>>
>> Next  self.fetch_response(request).. This is an async def.. So what I did
>> was..
>>
>> async def fetch_response(request):
>> future = {}
>> try:
>>future = await some other async def.. which returns future
>> except asyncio.CancelledError as e:
>>print("Error..", e)
>> return future
>>
>> I basically ended up doing this everywhere where there is async/await.
>> Next, I added this:
>> d.addErrback(lambda failure: failure.trap(asyncio.CancelledError))
>>
>>
>> So.. now.. I don't see the notifyFinish error anymore. And I am bit
>> terrified not to touch anything.. :-D
>>
>> But, I want to try out your suggestion as that seems like a more solid
>> way to handle the issue. But I am not sure I quite understand, how do I
>> figure out whether notifyFinish has been called before calling finish?
>>
>>
>> On Mon, Aug 5, 2019 at 1:56 AM Glyph  wrote:
>>
>>>
>>>
>>> On Aug 4, 2019, at 9:04 PM, Waqar Khan  wrote:
>>>
>>> Ah yes.. That is true..
>>>
>>> If I comment out request.finish() (Here is the doc which I tried to
>>> followed:
>>> https://twistedmatrix.com/documents/13.0.0/web/howto/web-in-60/interrupted.html
>>> )
>>> Then actually.. when I try to test out the code... (via curl or like
>>> doing requests.get .. to the URI)..
>>> it is just stuck..
>>>
>>>
>>> Yep!  This makes sense; if you don't call `.finish()` *at all* then
>>> your code will never tell the client that it's done.
>>>
>>> But if you call `.finish()` *after notifyFinish() has fired* then you
>>> get that error.
>>>
>>> If you change your code to only call `.finish()` if the deferred
>>> returned by notifyFinish() has not fired or failed yet, do you get the
>>> error?  It's possible that you still do, in which case, there's a bug in
>>> Twisted that needs to be fixed.
>>>
>>> I do want to convey my thanks for the help. Really appreciate it.
>>>
>>>
>>> Thanks for using Twisted :)
>>> -g
>>>
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Getting nice logging outputs in py.test

[Jean-Paul Calderone]

On Fri, Jul 5, 2019 at 1:55 AM Chris Withers  wrote:

> On 04/07/2019 21:43, Jean-Paul Calderone wrote:
> > How are you running the reactor in your pytest suites? I've not
> found a
> > sane way to do this and so would recommend sticking with trial,
> crufty
> > and old though it feels...
> >
> >
> > What feels crufty and old about trial?  And why is "old" a negative?
>
> "old" is negative here because things have moved on and trial hasn't.
>
> pytest feels like it's become the python standard now, so there's that.
> It's rich ecosystem of plugins is another one.
> The reliance on subclassing TestCase is nother.
> No fixtures.
>

Just for anyone who's interested, if you use testtools TestCase then you
get fixtures and a load of other nice features.  This is what I almost
always do for new test development now.  Interoperability between different
libraries is one of the few nice things that comes from xUnit so trial can
load and run these tests.

Poor support in pycharm for trial versus first class support for pytest.
>
> That's just off the top of my head, use pytest for a year or two and
> coming back to trial may well feel the same.
>

Don't assume I haven't used pytest just because I don't immediately and
unquestioningly express complete agreement with you. :)

Anyhow, I am less interested in trying to convince you you're wrong than I
am in learning from your experience - hence the request to expand on your
assessment.

Jean-Paul


>
> cheers,
>
> Chris
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Getting nice logging outputs in py.test

[Jean-Paul Calderone]

On Thu, Jul 4, 2019 at 1:50 PM Chris Withers  wrote:

> On 04/07/2019 14:39, Thomas Westfeld wrote:
> > Dear all,
> >
> > I am having an issue with logging of my Twisted application. I am using
> the new twisted.logger framework. The output produced by twist or twistd
> when runnning my plugin is nice.
> >
> > However when running a test suite using py.test I only get this as an
> output to stdout:
>

Are you letting pytest eat your stdout?  That's the default behavior, isn't
it?


>
> How are you running the reactor in your pytest suites? I've not found a
> sane way to do this and so would recommend sticking with trial, crufty
> and old though it feels...
>
>
What feels crufty and old about trial?  And why is "old" a negative?

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Azure Pipelines

[Jean-Paul Calderone]

On Fri, Jun 21, 2019 at 12:05 PM Glyph  wrote:

>
>
> On Jun 21, 2019, at 6:39 AM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
> Hello,
>
> I submitted ticket:9655 <https://twistedmatrix.com/trac/ticket/9655> /
> PR#1155 <https://github.com/twisted/twisted/pull/1155> for review a bit
> over a week ago.  It is suffering from a red mark on the CI report due to
> Azure Pipelines complaining there was some "communication" problem "with
> the server".  I imagine this is a spurious failure due to transient network
> conditions on the CI system.  There doesn't appear to be a retry button to
> find out, though.
>
>
> You need to be signed in to see it, but there is a "retry" button under
> the vertical-ellipsis button in the top right, next to "all logs".
>
> I clicked it for you (annoyingly I had to do this twice, there was another
> transient CI-system-related error on a different pipeline, maybe they've
> been having an outage?) and the build is now green.
>

Annoying, indeed. :/


>
> I can't see any Azure Pipeline configuration in the repository, I don't
> see much information on the ticket that (maybe?) introduced Azure
> Pipeline usage <https://twistedmatrix.com/trac/ticket/9454>.
>
>
> Once you're logged in, I think you can also see the build configuration,
> under "edit pipeline" in the same menu the "retry" item is in.  Not sure if
> there are some permissions to manage here for either the retry or edit
> buttons.  If there are, and you can't see either of them, let's get them
> set up for anyone who wants them; but I think Github org membership should
> be sufficient, since from what I can tell in their control panel that's how
> I had permission.
>
> I can't sign in to Azure or whatever the system driving this is called.
>
>
> The system is broadly called "Microsoft", I think :-), but you do have to
> log in to this specific corner of it (Azure DevOps?).
>
> Anyway - if you click on the Azure build status "Details" link on Github,
> you'll get to this page:
>
> https://twistedmatrix.visualstudio.com/twisted/_build/results?buildId=958
>
> On that page, there's a "Sign in" link in the top right.  When you click
> it, you'll see a login.microsoftonline.com page with a "sign in with
> github" link at the bottom.  Use that and you should be all set.
>

When I follow that link and grant permission on the GitHub side, I get
taken back to a microsoft.com URL that eventually fails with an infinite
redirect error. :/  So I am not able to look at the build page while signed
in.

Presumably there is something about my browser profile that triggers this
but, of course, I have no idea what.

Jean-Paul


>
> What is to become of this PR?
>
>
> Hopefully it'll get landed now ;).
>

Thanks!
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Azure Pipelines

[Jean-Paul Calderone]

Hello,

I submitted ticket:9655  /
PR#1155  for review a bit
over a week ago.  It is suffering from a red mark on the CI report due to
Azure Pipelines complaining there was some "communication" problem "with
the server".  I imagine this is a spurious failure due to transient network
conditions on the CI system.  There doesn't appear to be a retry button to
find out, though.

I can't see any Azure Pipeline configuration in the repository, I don't see
much information on the ticket that (maybe?) introduced Azure Pipeline usage
.  I can't sign in to Azure or
whatever the system driving this is called.

What is to become of this PR?

Thanks,
Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Backward incompatible fixes to backward incompatible changes

[Jean-Paul Calderone]

On Thu, Apr 25, 2019 at 12:52 PM Jean-Paul Calderone <
exar...@twistedmatrix.com> wrote:

> Hello all,
>
> I'd like to discuss the project policy/process for dealing with incorrect
> backward incompatibilities that end up released.
>

Considering the lack of interest in the topic, I'm going to suppose that
there is no policy and the decision is made case-by-case.  In the case I
referenced, I'm going to approve the change in behavior as "compatible" by
on the grounds that it restores the original behavior.

Jean-Paul


> As a case study, we could look at
> https://twistedmatrix.com/trac/ticket/9410
> <https://twistedmatrix.com/trac/ticket/9410#comment:17> /
> https://github.com/twisted/twisted/pull/1106
>
> In Twisted 16.3.0 the behavior of Request.write was changed so that it
> raises an AttributeError if called after the connection has closed.  Prior
> to that release, it silently did nothing in that case.
>
> Now, three years later, we have a PR which proposed restoring the behavior
> of silently ignoring the write to a closed connection.
>
> The original change was incompatible.  The new change is incompatible.
> What should win out?
>
> Jean-Paul
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Backward incompatible fixes to backward incompatible changes

[Jean-Paul Calderone]

Hello all,

I'd like to discuss the project policy/process for dealing with incorrect
backward incompatibilities that end up released.

As a case study, we could look at https://twistedmatrix.com/trac/ticket/9410
 /
https://github.com/twisted/twisted/pull/1106

In Twisted 16.3.0 the behavior of Request.write was changed so that it
raises an AttributeError if called after the connection has closed.  Prior
to that release, it silently did nothing in that case.

Now, three years later, we have a PR which proposed restoring the behavior
of silently ignoring the write to a closed connection.

The original change was incompatible.  The new change is incompatible.
What should win out?

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to get extReceived call from SSHCommandClientEndpoint

[Jean-Paul Calderone]

On Fri, Apr 5, 2019 at 2:08 PM Sereysethy TOUCH 
wrote:

> Hi Sean,
>
> Yes it is a method of SSHChannel, but when I read the source code, the
> _CommandChannel only implements dataReceived which calls protocol's method
> but it does not implement extReceived.
>
> When you suggest to call it like that 
> (self.transport.conn.channels[0].extReceived()),
> how do I know when extended data is received?
>
> For now I have subclassed _CommandChannel and also SSHCommandClientEndpoint
> in order to add extReceived.
>

An interesting question might be ... what do you want to happen to such
data?  Do you want it mixed in with the stdout stream?

For an IProtocol-based API, there aren't a lot of other options.  Maybe it
would be useful to have a similar API that works in terms of
IProcessProtocol instead of IProtocol?

There is a ProcessEndpoint that takes a flag that controls what it does
with stderr - currently supporting only two options, log it or drop it.

What did you hook extReceived up to in your subclass?

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] why can't a callback be called with a deferred?

[Jean-Paul Calderone]

On Wed, Feb 27, 2019 at 10:07 AM Scott, Barry 
wrote:

>
> The code I have posted is the good version so it works and the assert
> does not fire. My goal is to show what I assume is the correct way to code
> a function that uses internal Deferred(), not give an example that breaks.
>
> This should cause the the assert as the code must wait for the thread to
> return a result.
>
> def doWork():
> d = Deferred()
> d2 = deferToThread(doWorkHelper)
> d.callback(d2)
> return d
>

Ah.  I see.  Indeed, this code is wrong and triggers the assert.  For what
it's worth, there is a simpler solution than the example you gave.  It uses
the "chainDeferred" method:

def doWork():
d = Deferred()
d2 = deferToThread(doWorkHelper)
d2.chainDeferred(d)
return d

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] why can't a callback be called with a deferred?

[Jean-Paul Calderone]

On Wed, Feb 27, 2019 at 9:34 AM Scott, Barry 
wrote:

> On Tuesday, 26 February 2019 06:34:28 GMT Glyph wrote:
> > > On Feb 25, 2019, at 3:32 AM, Scott, Barry 
> > > wrote:>
> > > On Tuesday, 19 February 2019 11:00:57 GMT Chris Withers wrote:
> > >> Hi All,
> > >>
> > >> There's this assert:
> > >>
> > >>
> https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/defer.
> > >> py# L459
> > >>
> > >> ...and I'd like to understand why it's there.
> > >
> > > We hit this assert when porting from very old twisted to current
> twisted.
> > > In all cases the problem was with our code that used deferreds in a
> poor,
> > > not well understood way. After refactoring we are a lot happier with
> the
> > > resulting code as it easier to maintain now.
> >
> > Thanks for the feedback, Barry!
> >
> > It would still be great to figure out, if we can, how we might make the
> > error message a bit more legible to folks with less knowledge of
> Twisted's
> > internals.
>
> Let suppose that I need work done by doWork function.
> It returns a deferred for me to hang call backs and error backs on.
>
> d = doWork()
> d.addCallback(handleWorkDone)
>
> In my handleWorkDone I expect to get the result of doWork completing.
>
> The assert fires if instead of a result value is returned a Deferred is
> returned. This I consider a bug in the doWork() implementation.
>

This doesn't sound right.  Can you provide an example implementation of
doWork that provokes this behavior?  Here's an implementation that seems
like it matches your description and which does not provoke the behavior:

def doWork():
d = Deferred()
d.callback("result")
return d

d = doWork()
d.addCallback(handleWorkDone)

This doesn't trigger the assert.  This calls handleWorkDone with "result".
If you simplify the code so the Deferred interaction remains the same but
all the extraneous code is removed, it looks like this:

d = Deferred()
d.callback("result")
d.addCallback(handleWorkDone)

which *must* work or Deferred is completely useless.

Jean-Paul



>
> What must happen in doWork is that it must arrange that
> any Deferred it used internally has an addCallback used to
> cause the d returned to the user to complete. Leaking the
> any internal Deferred() objects must not happen to the user
> of doWork.
>
> def doWork():
> d = Deferred()
>
> def completeWork(result, d):
> d.callback(result)
>
> inner_d = doAsyncWork()
> inner_d.addCallback(completeWork, d)
>
> return d
>
> The error message would need to say something like:
> "Cannot return a Deferred as a result. Did you forgot to addCallback to
> the
> deferred?"
>
> Maybe add something to docs based on the above and refer to it in the
> message?
>
> Barry
>
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] asyncio.reactor.install()

[Jean-Paul Calderone]

On Tue, Dec 18, 2018 at 7:43 PM Jeff Grimmett  wrote:

> This pattern works:
>
>
>
> *from twisted.internet import wxreactorreactor = wxreactor.install()*
>

The fact that a reactor is returned by install is a bit of a mistake and is
not really part of the interface of *install.*

For whatever reason, asyncioreactor.install doesn't bother to return the
reactor it creates.

You should do this instead:

from twisted.internet import wxreactor

wxreactor.install()

from twisted.internet import reactor


Or don't do reactor installation yourself at all, use twistd or twist or
some other similar tool.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Thread Consumption Problem in Daemon?

[Jean-Paul Calderone]

On Mon, Nov 19, 2018 at 8:26 AM Maarten ter Huurne 
wrote:

> On maandag 19 november 2018 12:40:20 CET Darren Govoni wrote:
> > Hi,
> >   I am using twisted to run my Flask app via WSGI like so.
> >
> > twistd --pidfile $PORT/pidfile -l $PORT/logfile  -n web --port
> > tcp:$PORT --wsgi my.app
> >
> > Naturally, I have functions representing routes that enter and exit
> > just fine.
> >
> > However, I notice the twisted daemon process is :"gathering threads".
> > Eventually system runs out of them.
> >
> >  Here's a full status for one twisted server. 504 threads???
>
> I have a server running inside twistd which uses exactly 1 thread after
> running for a few weeks, so the problem may not be in twistd itself.
>
> I'm using a reverse-proxy HTTP setup though, not WSGI. Maybe the problem
> is specific to WSGI, Flask or your application?
>


Twisted's WSGI support definitely uses threads (as this is essentially a
requirement of WSGI).  It uses the reactor thread pool (if you launch it
from the CLI with twistd) which used to be limited to 10 threads.  I don't
know if the same limit is in place these days.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "disconnecting properly" in tests still hangs on macOS

[Jean-Paul Calderone]

On Wed, Nov 14, 2018 at 3:50 AM Chris Withers  wrote:

> Right, so, I've been trying to get the technique in
> https://jml.io/pages/how-to-disconnect-in-twisted-really.html to work
> for me.
>
> No hating please, most of my testing in the past has involved hitting a
> relational database, so there's already a TCP connection flying around,
> one more won't make any difference.
>
> jml's example, exactly as-is on that page, hangs around 30-40% of the
> time when running on my macOS laptop. From changing the teardown to look
> like this:
>
>  def tearDown(self):
>  ds = defer.maybeDeferred(self.serverPort.stopListening)
>  dc = defer.maybeDeferred(self.clientConnection.disconnect)
>  print()
>
>  ds.addCallback(lambda _: print('serverPort.stopListening'))
>  dc.addCallback(lambda _:
> print('self.clientConnection.disconnect'))
>  self.clientDisconnected.addCallback(lambda _:
> print('self.clientDisconnected'))
>  self.serverDisconnected.addCallback(lambda _:
> print('self.serverDisconnected'))
>  self.serverDisconnected.addErrback(lambda _:
> print('self.serverDisconnected:', _))
>  return defer.gatherResults([ds, dc, self.clientDisconnected,
> self.serverDisconnected])
>
> ...it appears that it's the serverDisconnected deferred that's failing
> to fire. I can't reproduce this on Linux as of yet, so I'm guessing this
> is a difference between the SelectReactor used on macOS and the
> EPollReactor used on Linux.
>
> What's the best way to go about debugging a non-firing deferred like this?
>

Track it backwards towards the earliest/lowest possible source of the
event.  You've already started this: you noticed that the result of
`gatherResults` doesn't fire and you investigate somehow and learned that
this is because `serverDisconnected` didn't fire.  Now keep going.  Why
didn't `serverDisconnected` fire?  It looks like it should be fired by
`ServerProtocol.connectionLost`.  Does that method ever get called?  If so,
you've now identified the break in the chain and you just have to figure
out why `connectionLost` doesn't manage to make `serverDisconnected` fire.
If not, keep going.  What code calls `ServerProtocol.connectionLost` - does
that run, etc.  Repeat until you find the break.


>
> Anyone know what this might be?
>

I know that macOS often delivers events in a different order compared to
Linux.  This has sometimes caused problems, particularly in the test
suite.  I can't quite see how that would explain this behavior though so
maybe it's something else.  It might be useful to share your version of
Python, Twisted, macOS, and the reactor you're using, in case anyone wants
to try to replicate.

Jean-Paul


>
> cheers,
>
> Chris
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "blocking" on a response to a client

[Jean-Paul Calderone]

On Tue, Nov 6, 2018 at 6:42 AM Chris Withers  wrote:

> On 05/11/2018 18:32, meejah wrote:
> > Chris Withers  writes:
> >
> >> So, separate question: I have a client that's making a connection to a
> >> websocket from a server. The client needs to send an "auth" message
> >> once the websocket is up, and the client code shouldn't progress until
> >> it receives an affirmative response. How do I do this in a
> >> non-blocking way when using twisted? (inside an inlineDeferred method
> >> in this case!)
> >
> > There's nothing really special about an @inlineCallbacks method -- it
> > just returns a Deferred so you can handle it like any other
> > Deferred-returning method.
>
> Thanks, but the special bit I was asking about is having my
> application's logical flow pause until the websocket is successfully up
> and a successful response has been received, all without blocking the
> reactor.
>
> How do I do that?
>
>
Put the rest of your code in a callback on a Deferred that fires when the
event you're interested in happens.  When using inlineCallbacks, that's
like:

yield the_event_youre_interested_in
# ... more code

At least, this is the extremely-localized solution to the problem.  There
are many possible ways to structure such a thing.  It's easy to describe
this one because you said you're using inlineCallbacks and didn't say
anything else about how your code is organized.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Issue with TLS failure modes

[Jean-Paul Calderone]

On Wed, Sep 5, 2018 at 9:40 AM Robert DiFalco 
wrote:

> Is the timeout the only way to handle this kind of TLS handshake error?
>

It might be easier to answer your original question if you include a Short,
Self Contained, Correct (Compilable), Example .

Jean-Paul


> Sent from my iPhone
>
> > On Aug 30, 2018, at 10:14 AM, Robert DiFalco 
> wrote:
> >
> > I'm having an issue properly handling TLS failure modes. For example
> consider the EchoServer and EchoClient code. If I use a TLS client with a
> TCP4 server, I do not get a handshake exception until I abort the
> connection. But I don't want to abort the connection unless I get a
> handshake error.
> >
> > What I'd like to do is to check the handshake status in my protocol
> before my client sends bytes to the server. I'd like my send message to be
> able to raise the : [('SSL routines',
> 'ssl23_read', 'ssl handshake failure'). But for some reason it seems to get
> lost until I abort the connection. Does this sound familiar to anyone?
> >
> > What I've done for now is setup a Timeout mixin so that after my
> client.send, if I do not get an ACK back (which my particular protocol
> does) within two minutes, I just abort the connection. This then calls
> connectionLost with the correct SSL.Error. But if it's in the error queue
> (and found during the course of abortConnection then isn't there a way to
> find it sooner? Like before my timeout and before I call send on the client
> protocol?
> >
> > Thanks!
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Using twistd with -c option causes permission error

[Jean-Paul Calderone]

On Tue, Aug 28, 2018 at 5:54 AM Richard Shea  wrote:

>
> With Apache the process starts as root, reads the key and then makes the
> apache process run as a different, less powerful, user  but I can't see how
> you can do the equivalent for twistd ? Am I overlooking something ?
>
>
twistd has `--uid` and `--gid` options for switching to another user after
`privilegedStartService` runs.  However, I'm not sure how much help this
will be since there is a strong desire to replace twistd with twist and
twist does not have these features.  Perhaps someone who has been working
on twist can explain the preferred workflow using that tool.


>
> Thanks
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] [ANN] txkube 0.3.0

[Jean-Paul Calderone]

Hello all,

I'm pleased to announce a new release of txkube, a Twisted-based library
for interacting with Kubernetes using the HTTP API.  The big news for this
release is support for Python 3.6.  Also included is support for multiple
configuration files in the KUBECONFIG environment variable which allows for
better configuration management practices.

Here is an example of txkube usage, taken from the README:

   from __future__ import print_function
   from twisted.internet.task import react

   from txkube import network_kubernetes_from_context

   @react
   def main(reactor):
   k8s = network_kubernetes_from_context(reactor, u"minikube")
   d = k8s.versioned_client()
   d.addCallback(
   lambda client: client.list(client.model.v1.Namespace)
   )
   d.addCallback(print)
   return d

You can download txkube from PyPI <https://pypi.python.org/pypi>
You can contribute to its development on GitHub
<https://github.com/LeastAuthority/txkube>.

Thanks to Least Authority TFA GmbH <https://leastauthority.com/> for
sponsoring this development and to Craig Rodrigues for his efforts on
Python 3 porting work.

Jean-Paul Calderone
<https://as.ynchrono.us/>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twistedmatrix.com: scheduled maintenance July 18th

[Jean-Paul Calderone]

On Fri, Jul 13, 2018 at 4:27 AM Evilham  wrote:

> Am 13.07.2018 um 08:35 schrieb Glyph:
> >> DNS could be pointed at an S3-backed website to serve a maintenance
> >> message, at least.  Not optimal, but better than ECONNREFUSED I would
> >> think.  I'd be happy to set this up and foot the bill - if someone
> >> provides the content (I can make web pages but no one really wants
> that).
> >
> > Please do so!
> >
> > Although keep in mind that it is the DNS primary as well, so you'll have
> > to update it with enough of a buffer that the secondary has cached the
> > address by the time it goes offline.
>
> I could also set this up on a Data Center in Switzerland; without
> spending a penny :).
> Only TLS is kind of a thing though, I'd probably generate LE certs to
> keep visitors happy and promise to delete those after the maintenance
> time-window.
> It'd make sense if that's kind of a no-go ^^.
>
> If it does sound like sth to do, just let me know in advance and I'll
> set up the thing + be available to trigger LE cert generation as soon as
> DNS is set up.
> (In that case, I noticed twistedmatrix.com doesn't have HSTS headers, so
> note to self: do not set those up :-D)
>
>
> As for content, I have a placeholder I've used in the past, which I've
> adapted to Twisted. Feel free to use/modify/not use at will.
>
> Preview: https://evilham.com/twisted_placeholder/
> Source: https://evilham.com/twisted_placeholder.tar.gz
>
>
Thanks.  That content is way better than I would have done.  I've put it up
at
http://2a28c83f-1613-4f82-a99c-cf7ca21326f8.s3-website-us-east-1.amazonaws.com/
just to see.

The TLS issue will be the same on AWS as you mentioned.  If the maintenance
site is to have HTTPS (which seems clearly to be the case) and we host on
AWS then I'll also need a certificate.  AWS CertManager would probably be
the way to go for this.

I probably already have access to all the things I need to get a new
certificate issued.  I'm happy to go ahead and complete the AWS setup if
that sounds good to everyone else.

Whichever way we go, thanks for the placeholder content and the offer to
host it, Evilham.

Jean-Paul

>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twistedmatrix.com: scheduled maintenance July 18th

[Jean-Paul Calderone]

On Thu, Jul 12, 2018 at 10:29 AM Adi Roiban  wrote:

> On Thu, 12 Jul 2018 at 12:57, Jean-Paul Calderone
>  wrote:
> >
> > On Wed, Jul 11, 2018 at 11:14 PM Mark Williams  wrote:
> >>
> >> Note that this is the minimum maintenance necessary to keep
> twistedmatrix.com running.  We need money to realize improvements to our
> infrastructure, like newer hardware for the existing server and additional
> hosts that could continue to serve twistedmatrix.com during a maintenance
> window.
> >
> >
> > DNS could be pointed at an S3-backed website to serve a maintenance
> message, at least.  Not optimal, but better than ECONNREFUSED I would
> think.  I'd be happy to set this up and foot the bill - if someone provides
> the content (I can make web pages but no one really wants that).
> >
> > Jean-Paul
> >
>
> Do we need fancy content?
>
> I was thinking to just use a plain text message explaining what is
> going on there.
>
> 
>
> There is also the option to host in Azure. Twisted has $120 each month
> for Azure services... so there will be no extra payment.
>
> Let me know if you want/need access to the Twisted Azure subscription.
>

I've not used Azure before.  Cheaper for me to pay the AWS bill than learn
a different cloud. :)


>
> Azure Blob does not have a "default page"...so you need to set up a
> CDN to serve a static page out of a container.
>
> There is one easy way to create a site from a GitHub repo with Azure
> App Service:
> https://github.com/adiroiban/azure-web-app-klein
>
> --
> Adi Roiban
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twistedmatrix.com: scheduled maintenance July 18th

[Jean-Paul Calderone]

On Wed, Jul 11, 2018 at 11:14 PM Mark Williams  wrote:

> Hello,
>
> The server that hosts twistedmatrix.com will be down for maintenance on
> Wednesday July 18th.  Our hosting company, Tummy.com, will be increasing
> the size of the /boot partition to accommodate the larger kernels typical
> of modern Linux distributions.  This requires a full reload of all
> partitions' data.
>
> The downtime is expected to be 24 hours, or through Thursday, July 19th.
> During the downtime no twistedmatrix.com sites will be available.
>
> Because Trac will be down, no pull requests should be merged between July
> 18th and 19th.
>
> Twisted's documentation will be inaccessible during that time as well.
>
> Note that this is the minimum maintenance necessary to keep
> twistedmatrix.com running.  We need money to realize improvements to our
> infrastructure, like newer hardware for the existing server and additional
> hosts that could continue to serve twistedmatrix.com during a maintenance
> window.
>

DNS could be pointed at an S3-backed website to serve a maintenance
message, at least.  Not optimal, but better than ECONNREFUSED I would
think.  I'd be happy to set this up and foot the bill - if someone provides
the content (I can make web pages but no one really wants that).

Jean-Paul


>
> Tax-deductible donations can be made to Twisted via the Software Freedom
> Conservancy.  More information is available here:
> https://twistedmatrix.com/trac/wiki/TwistedSoftwareFoundation#HowtoDonate
>
> --
>   Mark Williams
>   m...@enotuniq.org
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] subunitreporter 18.7.5

[Jean-Paul Calderone]

Hello all,

I'm happy to announce a new project, subunitreporter
, which provides a reporter
plugin for Trial which emits subunit v2 output.

Twisted includes a subunit v1 reporter for Trial.  However, the v1 output
format must generally be converted to the v2 format before much can be done
with it.  Additionally, the reporter included with Twisted does not include
timing information.  The output generated by subunitreporter does include
this information.

One useful thing to do with a subunit v2 stream is convert it to junitxml
and let CircleCI ingest it.  This is precisely the reason I've written this
reporter.  You can see this in action on some recent Tahoe-LAFS builds.  In
all cases, builds now include a summary of the number of tests run.

Additionally:
On success , slowest
test are reported.
On failure , the
failing tests and their failure details are summarized.
Further analysis and reporting
 is
also done.

I find this to be a nice development convenience when using CircleCI.
Other build systems are likely to be able to use either the subunit v2
output or (like CircleCI) the equivalent junitxml which can trivially be
produced from the subunit v2 output.

My thanks to Least Authority TFA GmbH  for
sponsoring these specific Tahoe-LAFS infrastructure improvements as well as
the subunitreporter project.

Enjoy,
Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Strategy for finding source of unhandled error in Deferred

[Jean-Paul Calderone]

On Mon, Mar 26, 2018 at 10:47 AM, Barry Scott <barry.sc...@forcepoint.com>
wrote:

> On Sunday, 18 March 2018 15:23:58 BST Jean-Paul Calderone wrote:
> >
> > Try this:
> >
> > from twisted.internet.defer import setDebugging
> > setDebugging(True)
> >
> > This gives you creation and callback stacks in the logs.
>
> If you need to track down who put the value into the deferred
> you need to patch the defer.py code to log that info.
>
> There are 2 places that a value enters the Deferred.
> The first is the callback() call. The other is in
>
> defer.py:653 Deffered._runCallbacks()
>
> 653:current.result = callback(current.result,
> *args,
> **kw)
>

No, you don't:

exarkun@baryon:/tmp$ cat deferreddebug.py
from twisted.internet.defer import fail, setDebugging
setDebugging(True)
def foo():
return fail(Exception("boo"))
foo()
exarkun@baryon:/tmp$ python deferreddebug.py
Unhandled error in Deferred:
(debug:  C: Deferred was created:
 C:  File "deferreddebug.py", line 5, in 
 C:foo()
 C:  File "deferreddebug.py", line 4, in foo
 C:return fail(Exception("boo"))
 C:  File
"/home/exarkun/.local/lib/python2.7/site-packages/twisted/internet/defer.py",
line 106, in fail
 C:d = Deferred()
 I: First Invoker was:
 I:  File "deferreddebug.py", line 5, in 
 I:foo()
 I:  File "deferreddebug.py", line 4, in foo
 I:return fail(Exception("boo"))
 I:  File
"/home/exarkun/.local/lib/python2.7/site-packages/twisted/internet/defer.py",
line 107, in fail
 I:d.errback(result)
)
exarkun@baryon:/tmp$


Although it is true that you only get this behavior for the "Unhandled
error in Deferred" case.  If you explicitly log a Failure from a Deferred,
even with Deferred debugging on, you will not get a callstack for the
creator or invoker.  You'll just get whatever callstack the Failure has
(and the Failure is independent of the Deferred).

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

[Jean-Paul Calderone]

On Mon, Mar 26, 2018 at 2:58 AM, Glyph  wrote:

> Alternately...
>
> The reason we're doing this at all is that Travis's macOS situation was
> pretty dismal when we last tried it.  But then, the graphs on
> https://www.traviscistatus.com at the time matched that experience:
> routinely there were build queues in the hundreds.  Right now the "Backlog
> macOS Builds for Open Source Projects" shows a completely flat line, zero
> for the last 24 hours.
>
> It might be worth trying out Travis again.
>

FWIW, I moved another project's macOS CI from Travis to CircleCI about a
month ago because the Travis macOS job queue was so long.  Counting queue
time, CircleCI finishes those jobs around 6x faster than TravisCI.

Maybe things have improved on Travis but they're still pretty bad.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Strategy for finding source of unhandled error in Deferred

[Jean-Paul Calderone]

On Sun, Mar 18, 2018 at 11:21 AM, Jean-Paul Calderone <
exar...@twistedmatrix.com> wrote:

> On Sun, Mar 18, 2018 at 11:04 AM, Alex Railean <ralie...@gmail.com> wrote:
>
>> Hello,
>>
>> I have inherited a misbehaving system and am trying to troubleshoot it.
>> The code
>> is not documented and I cannot produce a minimal working example that
>> illustrates
>> the problem yet, as I don't know which parts are involved and how they
>> relate.
>>
> This produces the following entries in the log:
>> Unhandled error in Deferred:
>> CRITICAL _legacy   twcore Unhandled error in Deferred:
>>
>> Traceback (most recent call last):
>> Failure: __builtin__.type: 
>> CRITICAL _legacy   twcore
>>
>>
>>
>> How can I get more clues about what is going on? I was expecting a line
>> number,
>> or a complete call-stack.
>>
>
Try this:

from twisted.internet.defer import setDebugging
setDebugging(True)

This gives you creation and callback stacks in the logs.

Jean-Paul


>
>>
>> What are the recommended troubleshooting steps for finding the Deferred
>> in which
>> the error occurred?  There are are around 30 of them throughout the code,
>> is
>> there a heuristic I can use to narrow down the list of suspects?
>>
>>
>>
>>
>>
>>
>>
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Strategy for finding source of unhandled error in Deferred

[Jean-Paul Calderone]

On Sun, Mar 18, 2018 at 11:04 AM, Alex Railean  wrote:

> Hello,
>
> I have inherited a misbehaving system and am trying to troubleshoot it.
> The code
> is not documented and I cannot produce a minimal working example that
> illustrates
> the problem yet, as I don't know which parts are involved and how they
> relate.
>
This produces the following entries in the log:
> Unhandled error in Deferred:
> CRITICAL _legacy   twcore Unhandled error in Deferred:
>
> Traceback (most recent call last):
> Failure: __builtin__.type: 
> CRITICAL _legacy   twcore
>
>
>
> How can I get more clues about what is going on? I was expecting a line
> number,
> or a complete call-stack.
>
>
> What are the recommended troubleshooting steps for finding the Deferred in
> which
> the error occurred?  There are are around 30 of them throughout the code,
> is
> there a heuristic I can use to narrow down the list of suspects?
>
>
>
>
>
>
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] List of required builds before a merge

[Jean-Paul Calderone]

On Tue, Mar 13, 2018 at 9:36 AM, Adi Roiban  wrote:

> We can try Circle-CI.
> They don't offer any free versions. Even for open source.
> When I did the initial work for Twisted with Travis and Appveyor I
> contacted Circle-CI to see if we can get a discount.
>
> They offered OSX - Seed plan which comes with 500 minutes/month - free.
>

They advertise this on their site now:
https://circleci.com/pricing/#faq-section-os-x

So I would expect Twisted could get it.

Your point about the minutes limit is a good one though.  What is 500
minutes to Twisted?  Around 30 builds?  It seems likely Twisted would run
out most months.  Certainly, maintaining separate coverage and non-coverage
builds would totally blow this limit.  And one of the big advantages of
hosted CI over buildbot - letting third-party PRs trigger builds - would
seem to be out of the question.


> I stopped as the people on IRC told me that Circle-CI is not better than
> Travis.
>

Presumably at some point this was true.  These days, in my experience,
CircleCI definitely seems competitive.  It has near feature parity (and
more features in some areas), a less crufty configuration system, and seems
better resourced (but this may just be better resource management - eg, the
500 minute macOS limit for free open source users).

Of course, there's also the fact that the macOS Buildbot slaves are
*not* offline
right now.  I know they've gone offline but as far as I know, their
maintainer is fairly responsive to reports of this.  So, is macOS really
part of this problem?

The immediate problem is all of the Rackspace-hosted slaves that are gone,
right?

So maybe what's more interesting is that CircleCI offers a way to replace
those - with either VM or container (including custom-image) builders.

Jean-Paul


>
> I am happy to try again with Circle-CI
>
> --
>
> We might go over 500 minutes. I suggest running the tests in stages.
>
> Run twistedchecker/pyflakes/newsfragment/Ubuntu tests first.
> Only when they all pass we should trigger Windows and OSX tests.
>
> I am also running the tests on stage... For example, Debian/RHEL/SUSE
> pass 99.99% if Ubuntu pass... so those tests are executed only later
> in the stage.
>
> --
>
>
> I don't have much time to contribute to Twisted infrastructure, and I
> would like to spend the available time doing reviews and helping
> people contribute to Twisted.
>
> If there is a better plan, I am happy to go with that.
>
> Thanks for your time :)
> Adi
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Waiting for a contended resource

[Jean-Paul Calderone]

On Mon, Mar 12, 2018 at 3:52 PM, Ilya Skriblovsky  wrote:

> Hi, Richard,
>
> I've used class like this to cache the result of Expensive Calculation:
>
> class DeferredCache:
> pending = None
> result = None
> failure = None
>
> def __init__(self, expensive_func):
> self.expensive_func = expensive_func
>
> def __call__(self):
> if self.pending is None:
> def on_ready(result):
> self.result = result
> def on_fail(failure):
> self.failure = failure
>
> self.pending = defer.maybeDeferred(self.
> expensive_func).addCallbacks(on_ready, on_fail)
>
> return self.pending.addCallback(self._return_result)
>
>
This seems like basically a correct answer to me.  However, I suggest one
small change.

You probably want to create and return a new Deferred for each result.  If
you don't, then your internal `pending` Deferred is now reachable by
application code.

As written, an application might (very, very reasonably):

d = getResource()
d.addCallback(long_async_operation)

Now `pending` has `long_async_operation` as a callback on its chain.  This
will prevent anyone else from getting a result until `long_async_operation`
is done.

You can fix this by:

result = Deferred()
self.pending.addCallback(self._return_result).chainDeferred(result)
return result

Now the application can only reach `result`.  Nothing they do to `result`
will make much difference to `pending` because `chainDeferred` only puts
`callback` (and `errback`) onto `pending`'s callback chain.  `callback` and
`errback` don't wait on anything.

You have to be a little careful with `chainDeferred` because it doesn't
have the recursion-avoidance logic that implicit chaining has.  However,
that doesn't matter in this particular case because the chain depth is
fixed at two (`pending` and `result`).  The problems only arise if you
extend the chain out in this direction without bound.

Jean-Paul



> def _return_result(self, _):
> return self.failure or self.result
>
> Using it you can get rid of DeferredLocks:
>
> deferred_cache = DeferredCache(do_expensive_calculation)
>
> def getResource():
> return deferred_cache()
>
> It will start `expensive_func` on the first call. The second and
> consequtive calls will return deferreds that resolves with the result when
> expensive_func is done. If you call it when result is already here, it will
> return alread-fired deferred.
>
> Of course, it will require some more work if you need to pass arguments to
> `expensive_func` and memoize results per arguments values.
>
> -- ilya
>
> пн, 12 мар. 2018 г. в 22:38, L. Daniel Burr :
>
>> Hi Richard,
>>
>> On March 12, 2018 at 1:49:41 PM, Richard van der Hoff (rich...@matrix.org)
>> wrote:
>>
>> Hi folks,
>>
>> I thought I'd poll the list on the best way to approach a problem in
>> Twisted.
>>
>> The background is that we have a number of resources which can be
>> requested by a REST client, and which are calculated on demand. The
>> calculation is moderately expensive (can take multiple seconds), so the
>> results of the calculation are cached so multiple lookups of the same
>> resource are more efficient.
>>
>> The problem comes in trying to handle multiple clients requesting the
>> same resource at once. Obviously if 200 clients all request the same
>> resource at the same time, we don't want to fire off 200 calculation
>> requests.
>>
>> The approach we adopted was, effectively, to maintain a lock for each
>> resource:
>>
>> > lock = defer.DeferredLock()
>> > cached_result = None
>> >
>> > @defer.inlineCallbacks
>> > def getResource():
>> > yield lock.acquire()
>> > try:
>> > if cached_result is None:
>> > cached_result = yield do_expensive_calculation()
>> > defer.returnValue(cached_result)
>> > finally:
>> > lock.release()
>>
>> (Of course one can optimise the above to avoid getting the lock if we
>> already have the cached result - I've omitted that for simplicity.)
>>
>> That's all very well, but it falls down when we get more than about 200
>> requests for the same resource: once the calculation completes, we can
>> suddenly serve all the requests, and the Deferreds returned by
>> DeferredLock end up chaining together in a way that overflows the stack.
>>
>> I reported this as http://twistedmatrix.com/trac/ticket/9304 and, at the
>> time, worked around it by adding a call to reactor.callLater(0) into our
>> implementation. However, Jean-Paul's comments on that bug implied that
>> we were approaching the problem in completely the wrong way, and instead
>> we should be avoiding queuing up work like this in the first place.
>>
>>
>> You mention using callLater to solve this problem, so I’m guessing that
>> instead of using a lock you are re-scheduling the call to getResource if
>> there is no cached_result value.  I’ve used this solution plenty of times
>> across 

Re: [Twisted-Python] List of required builds before a merge

[Jean-Paul Calderone]

On Mon, Mar 12, 2018 at 7:59 AM, Adi Roiban  wrote:

> Hi,
>
> It is not clear to me what builders need to pass before we can merge
> something.
>
> I expect that all supported "platforms" need to pass, but it is not
> clear what are the currently supported platforms.
>
> We have this info in the wiki but it does not help.
> https://twistedmatrix.com/trac/wiki/ReviewProcess#Authors:
> Howtomergethechangetotrunk
>
> In GitHub I can see Travis / Appveyor and OSX from Buildot as "Required"
>
> Is that all?
>
> --
>
> If I check the "supported" group in Buildbot, I see many more builders.
> The problem is that a significant number of slaves are down and those
> builders are not available.
>
> 
>
> Is Fedora still supported and required?
>
> ---
>
> I suggest to use GitHub "Required" marker to document what platforms
> are supported.
>
> We don't have time to maintain the infrastructure, so I suggest to
> drop support for anything that is not supported by Travis and
> Appveyor.
>

It would help to have a list of what coverage this would remove.  What
platforms are only covered by Travis and Appveyor?   What tests are only
run there?  What platforms are only covered by Buildbot?  What tests are
only run there?

Without this information, it's not really possible to make an informed
decision.  No user cares about whether we drop buildbot.  Some user might
care if we, for example, drop HTTP support.


>
> I know that this might be disruptive.
> I think that we need it in order to raise awareness that supporting a
> platform is not easy.
> If someone (including me) cares about a platform they should find a
> way to help to project supporting that platform.
>
>
Note that some people cared about some platforms and they found a way to
help in donating a buildslave.  Do the operators of the offline slaves
*know* that the slaves are offline?  Maybe all that's missing is some
notification to the operators when their slave goes away.  If that's all,
jumping straight to "throw away all of buildbot" seems like an overreaction.

Jean-Paul



> What do you think?
> --
> Adi Roiban
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Unhandled exceptions and observability

[Jean-Paul Calderone]

On Wed, Dec 27, 2017 at 11:18 PM, Svein Seldal  wrote:

> Hi
>
>
> I'm not sure how to write this email, but please let me try. I'd like to
> address something that I see as a limitation in Twisted. It might be that
> my use case is odd or that I'm outside the scope of Twisted, but non the
> less, I'd hope this could be a relevant topic.
>
> Problem:
>
> Unhandled exceptions can leave the application in a half-working state,
> and the in-app observability for them is difficult to obtain. Instead of
> terminating the whole application, the rest of the app can still keep
> running, and can be completely unaware of the failure.
>
> This applies to unhandled errbacks in Deferred and principally to any
> other reactor callbacks. E.g. it can occur in Deferreds being used
> internally in Twisted, where direct access to the object isn't available to
> the caller.
>
> As a user of Twisted, I would like to have the option to catch or fail my
> application completely when these unhandled exceptions occur, as would be
> expected in a sequential program.
>
>
I'm not sure I agree with the problem statement or your idea (below) for
solving it.  However, it's straightforward to implement your idea with
current and all recent versions of Twisted.  This program will exit rapidly:


from twisted.internet import reactor
> from twisted.python.log import addObserver
> def stop_on_errors(event):
> if event['isError']:
> reactor.stop()
> addObserver(stop_on_errors)
> def fail():
> 1/0
> reactor.callLater(0, fail)
> reactor.run()


Someone else can probably demonstrate how to do the same thing using
twisted.logger instead.


>
> Background:
>
> I have a larger application using many simultaneous TCP, UDP and UNIX
> connections. As with Twisted, the app is grouped in functions, where most
> of the heavy lifting are done in black-box-ish modules. There is of course,
> no guarantee for everything to work smoothly and if something fails, the
> entire application stops as a clear indication of the failure. However,
> there have been some occasions where this application is found to be
> half-dead, due to a failure occurring in a reactor-based callback that can
> only be seen by reading the logs. The main application is unfortunately
> unaware of its own failure.
>

The black boxes should probably not be so black that they hide whether they
are working or broken from the calling code.  What if they are broken in a
way that doesn't raise an exception?  What if they are broken in a way that
doesn't signal whatever ad hoc channel you invent or discover for
determining if they are broken?  The only real solution is for error
signaling to be a guaranteed part of the interface to the black box.


>
> AFAIK Twisted has no direct mechanism for handling errors that might occur
> when user code is called from the reactor. Or even worse, the caller does
> not know about the occurred failure unless the caller has direct access to
> the failing object. I believe this is more dangerous to reliability than
> the plain failing applications is, due to lower observability.
>

Correct.  No *direct* mechanism.  Various indirect mechanisms exist,
though, such as the logging example given above.


>
> Lets say the following code is used in a running application:
>
>from twisted.internet.task import LoopingCall
>class Foo:
>  def __init__(self):
>self.loop = LoopingCall(self.cb)
>self.loop.start(2, False)
>  def cb(self):
>self.count += 1
>
># Main app does this:
>try:
>  foo = Foo()
>except:
>  print "Won't happen"
>  raise
>
> The code will fail due to the programmical error in cb, but the calling
> application won't fail and thinks everything is fine. The methodology in
> debugging errors like this is by looking through the logs.
>

Foo is broken.  It uses the global reactor.  It creates side-effects in
__init__.  It creates a Deferred (LoopingCall.start) without attaching
callbacks or errbacks.


>
>
> The 0-solution:
>
> Everywhere a function is being called from the reactor, the user is
> responsible to handling all exceptions. As is the current case.
>
> However, this is not completely straight forward. try-expect are great to
> catch expected errors, but it's easy to forget and ignore the unexpected
> ones. Like in the example above. The safeguard for this would be something
> like:
>
>def cb(self):
>  try:
> self.count += 1
>  except:
> print "Whoops. Unexpected"
> signal_main_app()
>
> And in a large application, there are many entrypoints (e.g. methods in a
> protcol handler), so the code becomes very cluttered. Plus it puts the
> responsibility for the user to implement the signal_main_app() framework.
>

You say "cluttered".  Someone else might say "has correct error handling
code".  You also don't need to modify every piece of application code this
way.  You can compose this error handling into just about 

[Twisted-Python] [ANN] txaws 0.5.0

[Jean-Paul Calderone]

Hello all,

I'm pleased to announce the release of txAWS 0.5.0.  txAWS is a library for
interacting with Amazon Web Services (AWS) using Twisted.

You can download the release from PyPI .

Since the last release, the following enhancements have been made:

Features
> 
> - txaws.s3.client.S3Client.get_bucket now accepts a ``prefix`` parameter
> for
>   selecting a subset of S3 objects. (#78)
> - txaws.ec2.client.EC2Client now has a ``get_console_output`` method
> binding
>   the ``GetConsoleOutput`` API. (#82)


Thanks to everyone who contributed and to Least Authority TFA GmbH
 for sponsoring my work on this release.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Accepting/merging patches for unsupported platforms without CI

[Jean-Paul Calderone]

On Sun, Nov 26, 2017 at 5:30 PM, Glyph  wrote:

>
>
> Quite often—as I believe is the case for the patch you’re referring to
> here—patches are adjusting behaviors which would be difficult to cleanly
> integration-test on the platform in question anyway, and the appropriate
> thing to do is to make some mock match the observed behavior of the
> platform in question.


Just because there's enough misunderstanding about how to do this kind of
thing and the words that describe it, I wanted to point out that this isn't
really a "mock" - and it's certainly not something you would do with a
library with "mock" in its name.

For those who don't know (not Glyph), a mock is an object that somehow
records an exact sequence of operations (like "my foo method was called
with bar" or "my baz attribute was accessed").  These tend to result in
very fragile tests because they operate at such a specific level.  Quite
often entirely legitimate implementation changes invalidate the
expectations of some mock and force corresponding test suite maintenance.

A better approach is to make sure the platform-specific interface being
tested has been reduced to its smallest possible scope and then create a
verified fake which implements that interface.  The verification is
performed by a test suite which can run against both implementations (the
fake and the real - with the real version only being tested in the
necessary environment).  All other related tests can use the verified fake
instead of the real implementation and will run on all platforms.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] COMMERCIAL:Re: Clarification or pausing a consumer

[Jean-Paul Calderone]

On Mon, Oct 23, 2017 at 8:10 AM, Barry Scott <barry.sc...@forcepoint.com>
wrote:

> On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
> > > On Oct 13, 2017, at 5:47 PM, Jean-Paul Calderone
> > > <exar...@twistedmatrix.com> wrote:
> > >
> > > On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <
> barry.sc...@forcepoint.com
> > > <mailto:barry.sc...@forcepoint.com>> wrote: I have a app that needs to
> > > send data as an http response. But the data is produced slowly. More
> > > slowly then the time taken to transmit it.
> > >
> > > The consumer keeps calling resumeProducing, but there is no data to
> send
> > > often.
> > >
> > > This sounds like a bug in the consumer.  There are two cases.
> > >
> > > If the producer is "streaming" or "push" (two words, one meaning) then
> the
> > > consumer should call resumeProducing once and let data come.  If it
> wants
> > > to slow down the data, it can call pauseProducing.  There's nothing
> else
> > > it should be doing.
> > This is the good kind of producer.
> >
> > > If the producer is "not streaming" or "pull" (again, one meaning) then
> the
> > > consumer should call resumeProducing and wait for a write call.  Then
> it
> > > should call resumeProducing again (if it wants more data).  It should
> not
> > > call resumeProducing again while waiting for a write call.
> > This is the bad kind.  It is basically pointless.  It was a design error
> to
> > include it and it should be removed from Twisted eventually.
>
> Oh. This is surprising to here what is the rational behind this "good" vs.
> "bad"?
>

What makes it "bad" is that it creates extra work for every consumer
implementation.  The extra work is implementing largely the same thing
every time so it's also redundant work.  The logic for knowing when to call
resumeProducing again is always exactly the same regardless of the
consumer.  It doesn't belong inside every consumer.


>
> My use case is that I'm a proxy and I can only produce what has been
> received
> from the other side.
>
>
This sounds like it should be a push producer to me.


> A hard CPU loop calling resumeProducing is not appropiate for this use
> case.
>
>
This is appropriate behavior for neither kind of producer.  As I wrote in
my first reply, a consumer with a pull ("not streaming", "bad") producer
should call resumeProducing once and then wait until write is called on
it.  After write is called on it, it may call resumeProducing again if it
wants.

Jean-Paul


> I had to work around what looked like a bug that the streaming parameter is
> ignored and call unregisterProducer/registedProduced to get back usable
> behaviour.
>
> Barry
>
> > > What is the correct way to pause the consumer so that it does not keep
> > > calling resumeProducing? unregisterProducer?
> > >
> > > Probably just fix the consumer implementation to not be broken.  Or
> switch
> > > to tubes which has a simpler model and (in principle) fewer places to
> > > make mistakes.
> > My guess is not that the consumer is broken, but rather, you registered
> your
> > producer as the bad kind of producer ("pull") and it's just doing as it
> was
> > told, which is to keep calling resumeProducing over and over again every
> > time it wants data.
> >
> > Let us know how it works out!
> >
> > > Then when the next block of data is available what is the correct way
> to
> > > resume the consumer? registerProducer?
> > >
> > > Related to that why does HTTPChannel.registerProducer convert a
> > > IPullProducer into a IPushProducer using _PullToPush?
> > >
> > > Probably as an attempt to simplify the implementation - to make it so
> the
> > > consumer can pretend there's just one kind of producer instead of two
> > > (which would be nice for everyone).
> > >
> > > Jean-Paul
> > >
> > > ___
> > > Twisted-Python mailing list
> > > Twisted-Python@twistedmatrix.com
> > > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Clarification or pausing a consumer

[Jean-Paul Calderone]

On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott 
wrote:

> I have a app that needs to send data as an http response. But the data is
> produced slowly. More slowly then the time taken to transmit it.
>
> The consumer keeps calling resumeProducing, but there is no data to send
> often.
>

This sounds like a bug in the consumer.  There are two cases.

If the producer is "streaming" or "push" (two words, one meaning) then the
consumer should call resumeProducing once and let data come.  If it wants
to slow down the data, it can call pauseProducing.  There's nothing else it
should be doing.

If the producer is "not streaming" or "pull" (again, one meaning) then the
consumer should call resumeProducing and wait for a write call.  Then it
should call resumeProducing again (if it wants more data).  It should not
call resumeProducing again while waiting for a write call.


> What is the correct way to pause the consumer so that it does not keep
> calling resumeProducing? unregisterProducer?
>

Probably just fix the consumer implementation to not be broken.  Or switch
to tubes which has a simpler model and (in principle) fewer places to make
mistakes.


>
> Then when the next block of data is available what is the correct way to
> resume the consumer? registerProducer?
>
> Related to that why does HTTPChannel.registerProducer convert a
> IPullProducer
> into a IPushProducer using _PullToPush?
>

Probably as an attempt to simplify the implementation - to make it so the
consumer can pretend there's just one kind of producer instead of two
(which would be nice for everyone).

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTPS reverse proxy using twistd, twist, or twisted.web?

[Jean-Paul Calderone]

On Tue, Oct 10, 2017 at 3:17 PM, Craig Rodrigues 
wrote:

> Hi,
>
> Can someone give me a short snippet for how to set
> up a HTTPS reverse proxy using twistd, twist, or twisted.web ?
>
>
This requires CONNECT support in the Twisted Web server.  Has anyone
implemented this yet?

Jean-Paul



> I'd like to add an example of this to the documentation examples,
> since it is a pretty common thing to do these days.
>
> --
> Craig
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to correctly run Sqlite with Twisted?

[Jean-Paul Calderone]

On Tue, Sep 19, 2017 at 4:59 PM, Goffi  wrote:

> Hello,
>
> I'm using Sqlite3 module through Twisted's enterpirse.adbapi, I create the
> ConnectionPool instance like this:
>
>self.dbpool = ConnectionPool("sqlite3", db_filename,
> check_same_thread=False)
>
> You can see the code at https://repos.goffi.org/sat/
> file/tip/src/memory/sqlite.py
>
> Sometime, the writing is failing with following exception:
>
>   Failure instance: Traceback: :
> database is
> locked
>
> So I wonder if the database is correctly used, did anybody experienced
> something similar with Twisted and Sqlite ?
>
>
"Correctly" has so many shades.  Here's another possible direction for
exploration:

https://github.com/twisted/axiom

Jean-Paul
http://as.ynchrono.us/
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Forcing Twisted / Klein to use HTTPS

[Jean-Paul Calderone]

On Wed, Aug 30, 2017 at 11:36 AM, Chris Norman  wrote:

> Hi all,
>
> I'm trying to force Klein to use HTTPS, and so far I have a custom error
> handler which redirects the user to the HTTPs version of the page via a
> check_secure function which takes the request object and raises the right
> error if request.isSecure() is False.
>
>
> Is there a better global way to enforce HTTPS with any part of Klein or
> Twisted?
>
>
The way I do this is to make one Site that listens on HTTP and does nothing
except redirect to HTTPS.  The real site only listens on HTTPS.


>
> Cheers,
>
>
> Chris
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 17.5 vs txkube

[Jean-Paul Calderone]

I hope that won't discourage anyone else from taking a look.

On Wed, Aug 30, 2017 at 8:09 AM, Amber Hawkie Brown <
hawk...@atleastfornow.net> wrote:

>
> On 30 Aug 2017, at 22:05, Jean-Paul Calderone <exar...@twistedmatrix.com>
> wrote:
>
> Hello,
>
> Sadly the txkube test suite is still pretty busted against Twisted 17.5.
> It was broken for a long time with Twisted 17.1 as well.  I'm not sure if
> these are exactly the same problems or not but they look somewhat related.
>
> https://travis-ci.org/LeastAuthority/txkube/builds/269920984
>
> If anyone feels like getting to the bottom of this, I'd appreciate it.
>
> Jean-Paul
>
>
> I took a stab at some fixes a while ago in https://github.com/
> LeastAuthority/txkube/pull/135 -- some using private APIs, if I remember,
> sadly -- but I don't have the time at the moment to get it in a mergable
> state. But it might be a good start for others.
>
> - Amber
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted 17.5 vs txkube

[Jean-Paul Calderone]

Hello,

Sadly the txkube test suite is still pretty busted against Twisted 17.5.
It was broken for a long time with Twisted 17.1 as well.  I'm not sure if
these are exactly the same problems or not but they look somewhat related.

https://travis-ci.org/LeastAuthority/txkube/builds/269920984

If anyone feels like getting to the bottom of this, I'd appreciate it.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Is there a nice way to write transport-agnostic protocols and servers/clients?

[Jean-Paul Calderone]

On Thu, Aug 17, 2017 at 8:52 AM, Kevin Conway 
wrote:

> > What UDP endpoint?  There isn't one.  When I say "endpoint" I'm talking
> about the Twisted API collection that centers around
> *twisted.internet.endpoints*.  Is that what you're talking about?
>
> Oh... Yes, that is true. I think I was reading UNIX and saying UDP in my
> head. I'm really sorry for the confusion I'm causing. On the bright side, I
> now understand the problem. Sorry, folks.
>

Aha!  Yes, re-reading what you wrote and substituting UNIX for UDP,
everything makes much more sense.  Thanks for clarifying. :)

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Is there a nice way to write transport-agnostic protocols and servers/clients?

[Jean-Paul Calderone]

On Thu, Aug 17, 2017 at 1:35 AM, Kevin Conway 
wrote:

> > There is no datagram endpoints API as yet, though (see #4471), only
> stream endpoints.
>
> I've read through the ticket. What I still don't understand is the
> significance of this of this? Other than some irrelevant callbacks, what
> prevents anyone from using the existing UDP endpoint? It's not clear from
> this thread why the specific transport abstraction is critically
> important to the data protocol.
>

What UDP endpoint?  There isn't one.  When I say "endpoint" I'm talking
about the Twisted API collection that centers around
*twisted.internet.endpoints*.  Is that what you're talking about?


>
> If you are building a Protocol then you use the 'dataReceived' callback
> to parse the data. Can someone elaborate on why it matters if the bytes get
> here via a datagram or TCP stream?
>

Often, protocols which have a TCP variation and a UDP variation use
different framing depending on which one they're running over.  Therefore,
you cannot have a transport-neutral  parser.  Parsing depends on the
transport.

Also, in Twisted, if you use UDP, you must implement a *DatagramProtocol* and
the *datagramReceived* method.  If you use TCP, you must implement a
*Protocol* and the *dataReceived* method.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Is there a nice way to write transport-agnostic protocols and servers/clients?

[Jean-Paul Calderone]

On Wed, Aug 16, 2017 at 9:59 AM, Tristan Seligmann 
wrote:

> On Wed, 16 Aug 2017 at 15:28 Kevin Conway 
> wrote:
>
>> Maybe I misunderstand the issue, but I believe the Endpoints API solves
>> for this. https://twistedmatrix.com/documents/16.4.1/core/howto/
>> endpoints.html
>>
>> This abstracts the transport and allows you to focus on only interpreting
>> the bytes transmitted over that transport by defining a Protocol and a
>> Factory that plug into strports. This should allow you to have the business
>> logic defined in only one location while listening/requesting over
>> different transports.
>>
> There is no datagram endpoints API as yet, though (see #4471), only stream
> endpoints.
>
>
Also, when there are datagram endpoints, they won't be compatible with
stream endpoints.  Endpoints don't abstract the transport, they abstract
*creating* the transport.

Twisted provides a uniform abstraction over stream-based transports like
TCP and gives you *ITransport* to implement an *IProtocol* against.  It
also provides a uniform abstraction over datagram-based transports like UDP
and gives you *IUDPTransport* (oops; should have been *IDatagramTransport* and
some other things about it should probably be changed too) to implement a
*DatagramProtocol* (oops; no interface for this one, just subclass: see,
this should change) against.  But it doesn't provide a uniform abstraction
over both stream-based and datagram-based transports.

Implementing such a thing may be possible and even a good idea but no one
has yet done it for Twisted (as far as I know).

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Is there a nice way to write transport-agnostic protocols and servers/clients?

[Jean-Paul Calderone]

On Wed, Aug 16, 2017 at 7:53 AM, Jarosław Fedewicz <
jaroslaw.fedew...@gmail.com> wrote:

> This makes a perfect sense. (I ended up implementing it in this sorta way,
> but the other way around: it was a primarily-datagram protocol that could
> use TCP as a transport, provided there is appropriate framing.)
>
> However, while there is really a need for a shim to make use of
> polytransport protocols (DNS, syslog, SIP, NFS are a few), it looks like
> every Twisted user should bring their own. This bothers me, but just a
> little bit.
>

I don't think anyone working on Twisted has yet looked closely at these
various protocols to try to understand if there is a unifying abstraction
that would make a stream/datagram adapter possible.  There are differences
between the protocols that you mention that need to be accounted for and so
far everyone who has worked on implementations for Twisted has implemented
a custom solution for a particular protocol.  If you want to work on
figuring out what the abstraction might be, that seems useful.  I'd also
recommend you take a look at tubes  as a
potentially useful direction for solving this problem.


>
> Also, your reply does not address the little discrepancies that exist in
> Twisted between two kinds of datagram protocols (namely, Unix datagram
> sockets and UDP): seems like UDP APIs progressed while Unix datagram socket
> support was "left behind". I don't know if this is deliberate or not.
>

There's not as much interest in UNIX datagrams as there is in UDP.  There's
no particular reason why the UNIX datagram support shouldn't be brought up
to be on par with the UDP support.


>
> One thing that is also notably missing from the puzzle is that there is no
> support for datagram endpoints. With stream sockets, I can now use strings
> and clientFromString()/serverFromString() and be blissfully ignorant if
> it's TCP or a Unix socket. I cannot do the same with datagram sockets, I
> need to be actively aware if what I'm dealing with is a UDP socket or a
> Unix datagram socket.
>
> I'd gladly implement something to fill the gap, but I fear my contribution
> would be rejected because, say, it doesn't conform to the general direction
> Twisted is heading, whatever that may be. Or because datagram protocols are
> somehow inferior as a class. Or because Unix datagram sockets in particular
> should die and burn in hell.
>
>
Likewise, there's not as much interest in datagram transports as stream
transports.  There's no particular reason not to introduce datagram
endpoints  - someone just needs to do the work.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Chunked decoding missing from HTTPClient?

[Jean-Paul Calderone]

On Tue, Aug 15, 2017 at 8:43 AM, Barry Scott 
wrote:

> I see that HTTPChannel knows about chunking and indeed works well.
>
> However shouldn't HTTPClient also support chunking?
> Or Am I missing something to get chunking working for the response from a
> server?
>
>
Take a look at
http://twistedmatrix.com/documents/current/web/howto/client.html which
documents the newer HTTP client API in Twisted.  You may also want to have
a look at https://treq.readthedocs.io/en/latest/ which provides another
layer on top of that which may be even easier to use for your purposes.

HTTPClient is old, implements HTTP/1.0, and will probably be deprecated and
removed before too much longer.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] 17.5.0 _newtls.pu:207 loseConnection signature problem

[Jean-Paul Calderone]

On Fri, Aug 11, 2017 at 6:46 AM, Barry Scott 
wrote:

> I porting some code to latest twisted and I;m seeing the old code expects
> to
> call loseConenction with a reason.
>

This code makes the mistake of treating a peculiarity of a particular
implementation of an interface as the interface itself.


>
> But loseConnection in class ConnectionMixin does not have the parameter.
>

Nor does the definition of loseConnection on the interface:

http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.ITransport.html#loseConnection


>
> Is this an oversight or deliberate?
>

Code that goes beyond the guarantees of the interface is limiting itself to
working with a particular implementation.  Sometimes it is less than clear
whether some behavior is intended to be guaranteed by the interface or not
- but in this case, it's pretty clear.  loseConnection accepts no
arguments.  Code that passes an argument may work with a specific
implementation but there's no guarantee it will work with other
implementations.  And "other implementations" includes "future versions of
a specific implementation".

So, it's deliberate.  If you want to make the application code in question
more portable across implementations, it should stop passing an argument.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [ANN] txAWS 0.4.0

[Jean-Paul Calderone]

On Thu, Jul 13, 2017 at 3:58 AM, Glyph Lefkowitz <gl...@twistedmatrix.com>
wrote:

>
> On Jul 11, 2017, at 3:41 AM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
> Do you see particular similarities between boto/boto3 and txAWS?  Do you
> think there is something to be gained by explicitly mirroring something
> done in those libraries?
>
>
> The major thing to be gained by mimicking boto3, specifically (*not* boto
> 2) is its usage of botocore, which is a bunch of structured, static data
> describing the parameters used by various AWS APIs.  This could potentially
> increase the coverage of APIs that txAWS supports.
>
> (boto2 had quite a bit of high-level convenience functionality that I wish
> boto3 had kept though; I don't think the "nothing but botocore wrappers"
> approach is necessarily something that should be copied)
>
>
Ah.  Yea.  I would like it if someone would do some work on that.
https://github.com/twisted/txaws/issues/13

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [ANN] txAWS 0.4.0

[Jean-Paul Calderone]

On Tue, Jul 4, 2017 at 4:08 PM, Glyph <gl...@twistedmatrix.com> wrote:

>
> > On Jul 4, 2017, at 4:30 AM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
> >
> > Hello all,
> >
> > I'm pleased to announce the release of txAWS 0.4.0.  txAWS is a library
> for interacting with Amazon Web Services (AWS) using Twisted.
>
> Woohoo!
>
> This reminds me; I have a question about txAWS: if I have an EC2 instance
> or ECS instance with a machine role, will txAWS pick up the ambient
> credentials from ... I guess it's the metadata server that provides them?
>
>
Nope, it totally won't.  There's been some progress recently in improving
the way credentials are supplied to txAWS.  I hope there's some more soon
to add these kinds of capabilities.

Jean-Paul



> -glyph
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] [ANN] txAWS 0.4.0

[Jean-Paul Calderone]

Hello all,

I'm pleased to announce the release of txAWS 0.4.0.  txAWS is a library for
interacting with Amazon Web Services (AWS) using Twisted.

You can download the release from PyPI <https://pypi.python.org/pypi/txAWS>.

txAWS development uses GitHub for issues and source control
<https://github.com/twisted/txaws>.

Since the last release, the following enhancements have been made:

Bugfixes


- txaws now correctly signs requests with paths that require urlencoding.
(#20)


Features


- The Route53 client now recognizes all of the basic resource record types.
  (#50)
- txaws now supports reading the AWS_SHARED_CREDENTIALS_FILE and environment
  variable. (#52)
- txAWS now raises a CredentialsNotFoundError when it cannot locate
  credentials. Catching the previously-raised ValueError is now deprecated.
  (#53)
- txaws.credentials.AWSCredentials now supports loading different sets of
  credentials via AWS_PROFILE. (#54)

Thanks to everyone who contributed and to Least Authority Enterprises
<https://leastauthority.com/> for sponsoring my work on this release.

Jean-Paul Calderone
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Cannot access raise ticket in Issue Tracker

[Jean-Paul Calderone]

On Mon, Jun 19, 2017 at 4:33 PM, David Worenklein 
wrote:

> Hi,
>
>
>
> I’m trying to post a documentation bug (ReverseProxyResource says the path
> parameter type is str, but actually it needs to be bytes)
>
>
>
> I’m trying to access http://twistedmatrix.com/trac/newticket but I get an
> Error: Forbidden
>
>
>
> When I try to log in at http://twistedmatrix.com/trac/login I get an
> Error: Not Found
>
>
Where did you get this login link?  When I visit the front page logged out,
the login link goes to https://twistedmatrix.com/trac/github/login

Jean-Paul


>
>
> Thanks in advance,
>
> David
>
>
>
>
>
>
> This communication may contain privileged and/or confidential information.
> If you are not the intended recipient, you are hereby notified that you
> have received this message in error and that any review, dissemination,
> distribution or copying of this message is STRICTLY PROHIBITED. Any views
> or opinions presented are solely those of the author and do not necessarily
> represent those of Shelter Growth Capital Partners LLC, SG Capital Partners
> LLC or their affiliates. If you have received this communication in error,
> please notify the sender immediately by email, and delete the original
> message. We do not waive confidentiality if you have received this
> communication in error. This is not an offer to sell, or a solicitation of
> an offer to buy any financial product or an official confirmation of any
> transaction.
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Custom log "files" with twisted.logger

[Jean-Paul Calderone]

Hello,

I would like to be able to send events from the new logging system
(twisted.logger) to a remote log collection system.  If I'm using twist then
it seems options for configuring this part of logging are limited.  The
choices appear to be logging to stdout, stdout, or a file on the filesystem.

If I write my own command-line entrypoint then I can probably do whatever I
want by writing a custom observer and hooking it up appropriately.  I'd
rather not have to do that, though.

I could also go back to twistd and use --logger.  But since interest in
twistd appears to be on the decline, in favor of twist, I'd rather avoid
that as well.

It seems like the idea would be something like --logger for twist, perhaps
replacing the --log-file option.  If there were a way to pass some
arguments that would be pretty nice as well.

twist --logger myapp.something:x=y,a=b
  twist --logger twisted.logger.FileLogObserver.fromPath:path=/foo/bar

(Actual syntax not that important to me; consistency with endpoint syntax
would have some advantages; something that deals with the necessities of
quoting and nesting would be better, of course.   Also, I know everyone
hates Twisted's plugins but they would make for a nicer UI than FQPNs here.)

The first thing I'm likely to do with this is bridge twisted.logger events
to Eliot.  I think this will provide the most useful logs (at least *some* of
the logs from Twisted will have a meaningful Eliot context, though
certainly nowhere near all).  Also I already have a system for configuring
Eliot log destinations (which supports passing along arguments, so I can
already tell Eliot where my remote log collector is with a command-line
argument).

What are the chances of getting support for this scenario into twist itself?

Thanks,
Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] [ANN] txkube 0.2.0

[Jean-Paul Calderone]

Hello all,

I'm pleased to announce a new release of txkube, a Twisted-based
library for interacting with Kubernetes using the HTTP API.  The big
news for this release is compatibility with Kubernetes 1.6 and an
easier path to compatibility with future Kubernetes releases.  This
release also improves txkube's ability to authenticate against
different Kubernetes configurations.


Here is an example of txkube usage, taken from the README:

   from __future__ import print_function
   from twisted.internet.task import react

   from txkube import network_kubernetes_from_context

   @react
   def main(reactor):
   k8s = network_kubernetes_from_context(reactor, u"minikube")
   d = k8s.versioned_client()
   d.addCallback(
   lambda client: client.list(client.model.v1.Namespace)
   )
   d.addCallback(print)
   return d

You can download txkube from PyPI .
You can contribute to its development on GitHub
.

Thanks to Least Authority Enterprises 
for sponsoring this development.

Jean-Paul Calderonehttp://as.ynchrono.us/
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 17.5.0rc2 RC Announcement

[Jean-Paul Calderone]

On Mon, May 29, 2017 at 3:27 PM, Glyph <gl...@twistedmatrix.com> wrote:

>
> On May 26, 2017, at 4:38 AM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
> Some of the txkube problems have been resolved but either not entirely or
> some new ones have been introduced.  Attached is a log of the failures from
> the unit test suite.  I haven't investigated these new errors in depth yet
> so I won't say whether these are txkube's fault or Twisted's fault... but
> at least *something* is at fault.
>
>
> I do think we need to block the final release until we at least know where
> the responsibility lies for these tracebacks :-\.  These all still pass on
> 16.6.0 I presume?
>

They do, yea.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 17.5.0rc2 RC Announcement

[Jean-Paul Calderone]

On Fri, May 26, 2017 at 2:35 AM, Amber Brown 
wrote:

> Hi everyone,
>
> Fresh from PyCon US, comes a 17.5 prerelease. Fancy things in this release:
>
> - twisted.python.url has been spun out into the new 'hyperlink' package;
> importing twisted.python.url is now a compatibility alias
> - Support for OpenSSL 1.1.0.
> - Fixes around the reactor DNS resolver changes in 17.1, solving all known
> regressions
> - Deferred.asFuture and Deferred.fromFuture, to allow you to map asyncio
> Futures to Twisted Deferreds and vice versa, for use the with Python 3+
> asyncioreactor in Twisted
> - Support for TLS 1.3 ciphersuites, in advance of a released OpenSSL to
> enable the protocol
> - Further Python 3 support in twisted.web, initial support in
> twisted.mail.smtp
>
> Please see the NEWS.rst file and download the tarball at
> https://twistedmatrix.com/Releases/rc/17.5.0rc2/ . (rc1 fell to metadata
> mishaps!)
>
> Let me know if you notice anything out of the ordinary. If there's no
> problems, it'll release in the next week or so.
>
>
Thanks to all who contributed to this (and previous) efforts.

Some of the txkube problems have been resolved but either not entirely or
some new ones have been introduced.  Attached is a log of the failures from
the unit test suite.  I haven't investigated these new errors in depth yet
so I won't say whether these are txkube's fault or Twisted's fault... but
at least *something* is at fault.

Jean-Paul
[ERROR]
Traceback (most recent call last):
Failure: testtools.testresult.real._StringException: Traceback (most recent 
call last):
  File 
"/tmp/txkube-twisted-17.5.0rc2/local/lib/python2.7/site-packages/txkube/test/test_authentication.py",
 line 124, in test_bearer_token_authorization
request_bytes = self._authorized_request(token=token, headers=None)
  File 
"/tmp/txkube-twisted-17.5.0rc2/local/lib/python2.7/site-packages/txkube/test/test_authentication.py",
 line 108, in _authorized_request
[(host, port, factory, _, _)] = reactor.tcpClients
ValueError: need more than 0 values to unpack


txkube.test.test_authentication.AuthenticateWithServiceAccountTests.test_bearer_token_authorization
===
[ERROR]
Traceback (most recent call last):
Failure: testtools.testresult.real._StringException: Traceback (most recent 
call last):
  File 
"/tmp/txkube-twisted-17.5.0rc2/local/lib/python2.7/site-packages/txkube/test/test_authentication.py",
 line 140, in test_other_headers_preserved
request_bytes = self._authorized_request(token=token, headers=headers)
  File 
"/tmp/txkube-twisted-17.5.0rc2/local/lib/python2.7/site-packages/txkube/test/test_authentication.py",
 line 108, in _authorized_request
[(host, port, factory, _, _)] = reactor.tcpClients
ValueError: need more than 0 values to unpack


txkube.test.test_authentication.AuthenticateWithServiceAccountTests.test_other_headers_preserved
===
[ERROR]
Traceback (most recent call last):
Failure: testtools.testresult.real._StringException: eliot-log: {{{
2ad4f697-40a8-4bbe-bcf1-e54383e849ac
└── network-client:delete@1/started
├── kind: Namespace
├── name: e
├── namespace: None
├── timestamp: 1495798510.55902
├── network-client:request@2,1/started
│   ├── method: DELETE
│   ├── timestamp: 1495798510.559532
│   ├── url: https://kubernetes.example.invalid./api/v1/namespaces/e
│   └── network-client:request@2,2/succeeded
│   └── timestamp: 1495798510.564767
└── network-client:delete@3/failed
├── exception: txkube._exception.KubernetesError
├── reason: 
└── timestamp: 1495798510.565738

ffd04195-0b06-44db-9dc3-bc6f99ae5326
└── poll:start@1/started
├── obj
│   ├── apiVersion: v1
│   ├── kind: Namespace
│   └── metadata
│   ├── annotations
│   ├── finalizers: []
│   ├── labels
│   │   ├── c: x
│   │   └── v8: k
│   ├── name: e
│   └── ownerReferences: []
├── timestamp: 1495798510.566214
├── 
│   ├── poll_iteration: None
│   └── timestamp: 1495798510.566402
├── network-client:get@3,1/started
│   ├── kind: Namespace
│   ├── name: e
│   ├── namespace: None
│   ├── timestamp: 1495798510.566587
│   ├── network-client:request@3,2,1/started
│   │   ├── method: GET
│   │   ├── timestamp: 1495798510.566854
│   │   ├── url: https://kubernetes.example.invalid./api/v1/namespaces/e
│   │   └── network-client:request@3,2,2/succeeded
│   │   └── timestamp: 1495798510.570049
│   └── network-client:get@3,3/failed
│   ├── exception: txkube._exception.KubernetesError
│   ├── reason: 
│   └── timestamp: 1495798510.571157
└── poll:start@4/succeeded
└── timestamp: 1495798510.571529

3fd46547-0772-441e-9896-6a52f1f8500b
└── 

Re: [Twisted-Python] Welcome to the "Twisted-Python" mailing list

[Jean-Paul Calderone]

On Wed, Apr 19, 2017 at 4:11 PM, Adam Wozniak 
wrote:

> This is a problem in twisted.  Buldbot uses twisted.
>
> To make buildbot git polling work for me, I had to make the change below.
>
> Based on zturner comment #17 in
> http://trac.buildbot.net/ticket/2936



I left a comment on that ticket forwarding the straightforward fix that can
be applied to BuildBot to address this problem.

It's certainly true that there are shortcomings in the spawnProcess API so
I won't try to argue that this *isn't* a problem in Twisted.  However, some
of those very problems make it difficult to *fix* any of the problems.  It
seems more likely that spawnProcess will eventually be replaced by a better
API (if the problems are ever fixed at all, that is).  Meanwhile, since the
fix in BuildBot is quite simple, that seems like a good direction to pursue.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [ANN] kubetop 17.4.17.1

[Jean-Paul Calderone]

On Mon, Apr 17, 2017 at 9:44 PM, Glyph Lefkowitz <gl...@twistedmatrix.com>
wrote:

>
> On Apr 17, 2017, at 10:24 AM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
> Hello all,
>
> I'm pleased to announce the initial release of kubetop, a command-line
> tool in the style of top(1) for displaying current usage and activity of a
> Kubernetes cluster.
>
>
> Personally I'm not using Kubernetes much, but this looks SUPER handy.  And
> it seems to install cleanly with `pipsi install kubetop` :-).  Thanks for
> the announcement!
>
> I did see this in my install output though:
>
>   Running setup.py bdist_wheel for attr ... done
>   Stored in directory: /Users/glyph/Library/Caches/pip/wheels/39/a7/44/
> cce905c824a2266d3e5747b2de72c8d73f9c8e9f0a71f066cd
>
> Is this actually supposed to depend on https://pypi.python.org/pypi/attr ?
> It seems likely to me that this was a typo somewhere in the dependency
> chain for https://pypi.python.org/pypi/attrs .
>

Oops!  Thanks for pointing that out.  It is indeed a mistake.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] [ANN] kubetop 17.4.17.1

[Jean-Paul Calderone]

Hello all,

I'm pleased to announce the initial release of kubetop, a command-line tool
in the style of top(1) for displaying current usage and activity of a
Kubernetes cluster.

kubetop 17.4.17.1 will format information about your Kubernetes cluster
like this:

kubetop - 13:02:57
Node 0 CPU%   9.80 MEM% 57.97 (   2 GiB/   4 GiB)  POD%  7.27 (  8/110) Ready
Node 1 CPU%  21.20 MEM% 59.36 (   2 GiB/   4 GiB)  POD%  3.64 (  4/110) Ready
Node 2 CPU%  99.90 MEM% 58.11 (   2 GiB/   4 GiB)  POD%  7.27 (  8/110) Ready
Pods:   20 total0 running0 terminating0 pending
 POD   (CONTAINER)%CPU MEM   %MEM
s4-infrastructure-3073578190-2k2vw75.5  782.05 MiB  20.76
  (subscription-converger)72.7  459.11 MiB
 (grid-router) 2.7   98.07 MiB
 (web) 0.1   67.61 MiB
(subscription-manager) 0.0   91.62 MiB
   (foolscap-log-gatherer) 0.0   21.98 MiB
   (flapp) 0.0   21.46 MiB
  (wormhole-relay) 0.0   22.19 MiB


kubetop is meant as a handy tool to use at the spur of the moment.  It is
not intended to replace a real monitoring system.

kubetop's presentation is preliminary and a work in progress.
Contributions welcome.

You can find kubetop on PyPI <https://pypi.python.org/pypi/kubetop> and
GitHub <https://github.com/LeastAuthority/kubetop>.  Install it in the
usual way:

pip install kubetop


Thanks to Least Authority Enterprises <https://leastauthority.com/> for
sponsoring this development.

Jean-Paul Calderone
http://as.ynchrono.us/
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Beginner

[Jean-Paul Calderone]

On Wed, Apr 12, 2017 at 9:43 AM, Chris Norman <chris.norm...@googlemail.com>
wrote:

>
> On 12/04/2017 14:41, Jean-Paul Calderone wrote:
>
> On Wed, Apr 12, 2017 at 9:37 AM, Chris Norman <
> chris.norm...@googlemail.com> wrote:
>
>>
>>
>> On 12/04/2017 14:21, catch me wrote:
>>
>> Hey! I don't know I should ask this question or not but as a beginner I
>> have to ask.
>> I have just started learn python network programming and fortunately
>> found such an amazing Twisted Framework.
>>
>>
>> Welcome! Glad you found Twisted, it really is awesome!
>>
>>
>> As a beginner I could not be able to decide where to start I mean what
>> should I make first using twisted framework. I would be very thankful to
>> you for guiding me.
>>
>>
>> I personally like telnet-like servers. They are very easy to make with
>> the LineReceiver protocol (hint hint), and you can do a lot with them.
>>
>
> If you want *telnet* then you should probably use *twisted.conch.telnet*.
> It's a little bit more complicated than *LineReceiver* but it will
> actually speak telnet for you.  LineReceiver will get you some simple
> line-oriented interactions, though.
>
>
> Ah, than you. I could never figure out how to make the proper
> TelnetProtocol to do anything useful, and line-based is all I've ever
> really needed.
>

The module would certainly benefit from some prose documentation.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Beginner

[Jean-Paul Calderone]

On Wed, Apr 12, 2017 at 9:37 AM, Chris Norman 
wrote:

>
>
> On 12/04/2017 14:21, catch me wrote:
>
> Hey! I don't know I should ask this question or not but as a beginner I
> have to ask.
> I have just started learn python network programming and fortunately found
> such an amazing Twisted Framework.
>
>
> Welcome! Glad you found Twisted, it really is awesome!
>
>
> As a beginner I could not be able to decide where to start I mean what
> should I make first using twisted framework. I would be very thankful to
> you for guiding me.
>
>
> I personally like telnet-like servers. They are very easy to make with the
> LineReceiver protocol (hint hint), and you can do a lot with them.
>

If you want *telnet* then you should probably use *twisted.conch.telnet*.
It's a little bit more complicated than *LineReceiver* but it will actually
speak telnet for you.  LineReceiver will get you some simple line-oriented
interactions, though.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [ANN] txkube 0.1.0

[Jean-Paul Calderone]

On Tue, Apr 11, 2017 at 1:15 AM, Glyph Lefkowitz <gl...@twistedmatrix.com>
wrote:

>
> On Apr 10, 2017, at 11:11 AM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
> Hello all,
>
> I'm pleased to announce the first release of txkube, a Twisted-based
> library for interacting with Kubernetes using the HTTP API.
>
>
> ✨³✨
>
> This release supports several of the most commonly used basic Kubernetes
> objects, including Services, ConfigMaps, Deployments, ReplicaSets, and
> Pods.  While Kubernetes has many, many more object kinds, this collection
> of kinds already supports a very useful set of interactions.
>
>
> Thanks for the announcement!
>
> I do have one question about txkube, since this comes up periodically in
> every higher-level networking layer, and it's a place where I think Twisted
> has some advantages over other HTTP clients: is there a way to specify
> custom trust roots, or construct a custom Agent to pass in to txkube?
>
>
There's another constructor for IKubernetesService
<https://github.com/LeastAuthority/txkube/blob/b2e81400e40989696f82cabfd575d5283f7180e8/src/txkube/_interface.py>,
network_kubernetes
<https://github.com/LeastAuthority/txkube/blob/b2e81400e40989696f82cabfd575d5283f7180e8/src/txkube/_network.py#L41-L56>.
It takes an IAgent.  So one can do:

network_kubernetes(base_url=url, agent=Agent(contextFactory=...))

Does that approach fit with any emerging conventions for exposing this kind
of functionality?  I'd be happy to adapt it if there's a better pattern.

Also note that it's common for Kubernetes to be deployed with a self-signed
or other non-cartel certificate.  So even the API in the example,
network_kubernetes_from_context, doesn't trust the usual web-oriented
collection of CAs.  It only respects the certificate found in the local
Kubernetes configuration.  This means that there's another way to control
this - edit ~/.kube/config and put the desired CA certificate there.


> Here is an example of txkube usage, taken from the README:
>
>from __future__ import print_function
>from twisted.internet.task import react
>
>from txkube import v1, network_kubernetes_from_context
>
>@react
>
>
> btw, don't think I didn't notice this
> <https://github.com/twisted/twisted/pull/646#discussion_r96104930>...
>



Jean-Paul


>
>
>def main(reactor):
>k8s = network_kubernetes_from_context(reactor, u"minikube")
>client = k8s.client()
>d = client.list(v1.Namespace)
>d.addCallback(print)
>return d
>
> You can download txkube from PyPI <https://pypi.python.org/pypi>.
> You can contribute to its development on GitHub
> <https://github.com/LeastAuthority/txkube>.
>
> Thanks to Least Authority Enterprises <https://leastauthority.com/> for
> sponsoring this development.
>
>
> Thanks, LAE!
>
> -glyph
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] [ANN] txkube 0.1.0

[Jean-Paul Calderone]

Hello all,

I'm pleased to announce the first release of txkube, a Twisted-based
library for interacting with Kubernetes using the HTTP API.

This release supports several of the most commonly used basic Kubernetes
objects, including Services, ConfigMaps, Deployments, ReplicaSets, and
Pods.  While Kubernetes has many, many more object kinds, this collection
of kinds already supports a very useful set of interactions.

Here is an example of txkube usage, taken from the README:

   from __future__ import print_function
   from twisted.internet.task import react

   from txkube import v1, network_kubernetes_from_context

   @react
   def main(reactor):
   k8s = network_kubernetes_from_context(reactor, u"minikube")
   client = k8s.client()
   d = client.list(v1.Namespace)
   d.addCallback(print)
   return d

You can download txkube from PyPI <https://pypi.python.org/pypi>.
You can contribute to its development on GitHub
<https://github.com/LeastAuthority/txkube>.

Thanks to Least Authority Enterprises <https://leastauthority.com/> for
sponsoring this development.

Jean-Paul Calderone
http://as.ynchrono.us/
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] [ANN] txAWS 0.3.0

[Jean-Paul Calderone]

Hello all,

I'm pleased to announce the release of txAWS 0.3.0.  txAWS is a library for
interacting with
Amazon Web Services (AWS) using Twisted.

You can download the release from PyPI <https://pypi.python.org/pypi/txAWS>
 (https://pypi.python.org/pypi/txAWS).

txAWS development uses GitHub for issues and source control
<https://github.com/twisted/txaws> (https://github.com/twisted/txaws).

Since the last release, the following enhancements have been made:

   - Jean-Paul Calderone added basic Route53 support.
   - Mark Williams add Auth v4, now used by the S3 and Route53 support.
   - Francisco Souza added support for VPCs to some EC2 APIs.
   - Drew Smathers added multi-part S3 upload support.
   - Several people added other S3 client features such as pagination and
   streaming uploads.
   - *txaws.testing* now provides S3 and Route53 support.

Additionally, the following APIs have been deprecated:

   - The *client* attribute of *txaws.client.base.BaseQuery*
   - The *date* attribute of *txaws.s3.client.Query*

Thanks to everyone who contributed and to Least Authority Enterprises
<https://leastauthority.com/> (<https://leastauthority.com/>) for
sponsoring my work on this release.

Jean-Paul Calderone
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP Agent persistent connections not closed for some HTTPS sites

[Jean-Paul Calderone]

On Fri, Mar 10, 2017 at 3:26 AM, Cory Benfield <c...@lukasa.co.uk> wrote:

>
> On 9 Mar 2017, at 22:01, Jean-Paul Calderone <exar...@twistedmatrix.com>
> wrote:
>
> Run the server with TLSMemoryBIOProtocol and use the transport's
> pauseProducing method before you attempt the shutdown?
>
> Jean-Paul
>
>
> Yup, that will work. Similarly, having two TLSMemoryBIOProtocols
> communicating together and pausing one before calling loseConnection on the
> other should also work. Essentially, any situation you can conjure where
> the closing connection will manage not to see either a TLS CloseNotify or a
> TCP FIN or RST will do the job.
>

Indeed.  For that matter, you can have them interact in-memory (maybe
StringTransport is good enough as-is, maybe not, I'm not sure) and then
just stop shuttling the bytes in one direction.  (I/O-free tests for the
win)

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP Agent persistent connections not closed for some HTTPS sites

[Jean-Paul Calderone]

On Thu, Mar 9, 2017 at 3:40 PM, Adi Roiban  wrote:

> On 7 March 2017 at 09:31, Cory Benfield  wrote:
> >
> > On 6 Mar 2017, at 09:05, Adi Roiban  wrote:
> >
> > I am not sure about which code are we talking here.
> > The specific HTTP11ClientProtocol which will fix only the HTTP client
> > part or the generic TLSMemoryBIOProtocol code which might fix any TLS
> > connection?
> >
> >
> > I’m talking about the fact that calling TLSMemoryBIOProtocol.
> loseConnection
> > may not ever actually drop the underlying FD is the surprising part of
> the
> > code. We had to work around it in the server side by adding a timeout
> after
> > loseConnection is called to end up calling abortConnection.
> >
> > Cory
>
> In order to submit a patch, I am trying to write an automated tests for
> this.
> Were you able to reproduce this issue in a controlled environment.
>
> I am still not fully understanding why I get this behaviour with
> sharepoint.com sites.
>
> I am thinking at having a rigged TLS server which will keep the socket
> open but will not
> respond to the shutdown request but I still don't know how :)
>
>
Run the server with TLSMemoryBIOProtocol and use the transport's
pauseProducing method before you attempt the shutdown?

Jean-Paul


> --
> Adi Roiban
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] non-merge commits to trunk & "review" keyword

[Jean-Paul Calderone]

Hello,

GitHub apparently allows fast-forward merges to trunk.  Here's an example
of one:

  https://github.com/twisted/twisted/pull/730

This doesn't seem like a good thing.

   - The ticket is still open
   - There is no merge commit
   - There is no merge commit message
   - There are non-merge commits directly on trunk history (first parent)

  Anyone else have an opinion?

Also, on the same PR, it seems like folks have trouble managing the two
different ways to represent the review states: the "review" keyword in trac
and the accepted/etc status on the GitHub PR.  Maybe there shouldn't be two
different ways to represent this?

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Minor fixes in the “Customizing twistd logging” section

[Jean-Paul Calderone]

On Fri, Mar 3, 2017 at 2:57 AM, Роман Мещеряков  wrote:

> Hi everyone,
>
>
> *On the documentation web page
> https://twistedmatrix.com/documents/current/core/howto/application.html
> 
> in the “Customizing twistd logging” section there are 2 examples which as I
> can see are a bit outdated in terms of imported module names and some other
> minor details. I would like to fix that. Is this change is worth doing? If
> so, please guide how can I do it. My GitHub login, if it’s needed, is
> RomanMeR.*
>

Hi Роман,

Thanks for your interest in improving Twisted!  Have a look at
https://twistedmatrix.com/trac/wiki/TwistedDevelopment and then let us know
if you have further questions.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Test coverage requirements

[Jean-Paul Calderone]

On Mon, Feb 27, 2017 at 6:00 PM, Tristan Seligmann 
wrote:

> On Mon, 27 Feb 2017 at 21:54 Glyph Lefkowitz 
> wrote:
>
>> That said, it has been *improving* and if it keeps improving at the rate
>> it has been, I expect that we'd be able to put that coverage blocker back
>> in in another 3-4 months.  Perhaps something to talk about at PyCon.
>>
>
> I think at least one problem that we're suffering from here is our fault,
> rather than Codecov's: the coverage of the test suite is not stable due to
> non-determinism in the test suite. That is, the lines executed during a
> test run are not the same every time due to things like ordering / timing
> races / etc. This means that "changes" to coverage may show up for a
> particular PReven though nothing in that PR is actually responsible.
>
>
Changes to Twisted code which are only sometimes covered by the test suite
sound like they would violate a 100% coverage rule.  But I guess the
experience of looking at a codecov report is so bad/confusing that it's not
surprising authors/reviewers might fail to see what's going on and fix the
non-deterministic.

Particularly for code that requires coverage measurements on multiple
platforms (ie, you basically *can't* do it locally), it seems like it would
be easier (though, to be clear, *bad*) to just forget about it and hope
everything is covered...

A tool that pointed out coverage differences between multiple runs of the
same version of the code would be a useful thing to start pointing out
where these flaws in the Twisted test suite lie, right?  And then each area
could be given deterministic test coverage instead...
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] trac spam

[Jean-Paul Calderone]

On Mon, Feb 27, 2017 at 8:41 AM, Adi Roiban <a...@roiban.ro> wrote:

> On 27 February 2017 at 12:59, Jean-Paul Calderone
> <exar...@twistedmatrix.com> wrote:
> > Tickets 9058 and 9059 are spam.  I would have cleaned them up but it's
> not
> > clear from the UI how (actually I clicked a "delete" button in the UI but
> > the tickets are there so I guess "delete" doesn't mean what I think).
>
> I have removed the first one.
>
> After you click delete, you need to scroll and to confirm the action.
>
> I left the other one there as an exercise :)
>

Hm I thought I clicked the button at the bottom.  Maybe I was on the wrong
page?  Which admin page should I be on?

Jean-Paul



>
> > Jean-Paul
> >
> >
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
>
>
>
> --
> Adi Roiban
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] trac spam

[Jean-Paul Calderone]

Tickets 9058 and 9059 are spam.  I would have cleaned them up but it's not
clear from the UI how (actually I clicked a "delete" button in the UI but
the tickets are there so I guess "delete" doesn't mean what I think).

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Test coverage requirements

[Jean-Paul Calderone]

Hi,

I'm looking at some  recent
 trunk
 commits
 (also, others) that seem
to have non-trivial untested code at at ReviewProcess
.
I can't tell if the codecov reports are wrong or if the development process
documentation is wrong or if the commits just violate policy or (I guess)
some mix of the three.

Can anyone shed any light on this?

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Jean-Paul Calderone]

On Wed, Feb 22, 2017 at 7:08 AM, Phil Mayers 
wrote:

> On 22/02/17 11:12, Glyph Lefkowitz wrote:
>
> I'm tempted to launch into a diatribe about namespacing, containers, and
>> application isolation generally, but before I do - why is it that you
>> /want/ to use the system Python environment?  Had you just not
>> considered the option of virtual environments?
>>
>
> Awesome though it is, virtualenv can be very tedious if you need to
> install hundreds of megabytes of compiler and -devel packages. System
> packages are attractive precisely because you can avoid this.
>

I guess this is a big part of the motivation for Twisted itself to shed its
C extensions - to avoid 100% of these extra dependencies.  And presumably
also a big part of the motivation behind the development of support for
distributable binary wheels for Linux (which seems to have largely
succeeded at this point, though their use is not as widespread as one would
hope).


>
> I've had to do all sorts of tedious things with containers where I spin up
> a temporary container to build a bunch of .whl files, then actually install
> them in the final container - all to avoid bloating the container image
> with the build tools.
>
>
And I've been meaning to really dig in to the Nix toolchain for image
creation.  That seems like another thing that should be more widely used
(but it's hard to beat the "batteries included" tools Docker has, even if
those tools are awful).


> It's a real shame that binary wheels on Linux/PyPI aren't a thing. I
> understand the reasons why, but TBH it substantially reduces the value of
> the Python ecosystem for me.
>
>
Ah!  They are now, maybe you haven't seen them?

Check out https://github.com/pypa/manylinux and
https://github.com/pypa/python-manylinux-demo

Go is looking like a more and more attractive competitor to Python,
> frankly, having sidestepped this problem entirely.
>
>
A lot of people seem to be thinking that way.  *Personally*, with my
experience with Go (about five months solid at previous employer), I
wouldn't go anywhere near that stack.  There are plenty of *other *things
with appealing features that Python lacks which would seem to make a better
move. :)

But check out the story for binary wheels.  The state of things in Python
may not be *quite* as bad as you think.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to restart Twisted service in-process

[Jean-Paul Calderone]

On Sat, Feb 18, 2017 at 2:25 PM, Glyph Lefkowitz 
wrote:

>
> On Feb 2, 2017, at 1:17 AM, Роман Мещеряков 
> wrote:
>
> *Hello everyone! I wrote Twisted-based TCP server which is capable of
> running in several relatively different modes. When mode switch is needed,
> I would like the server to restart itself by some means, for it to read new
> mode from configuration file and create corresponding implementation. Also
> I wanted the restart to occur without re-running twistd process, because it
> seems to me more appropriate for Linux daemon.*
>
>
> Hi Roman,
>
> Sorry it took a while to get to this one :).
>
> If your solution is working for you, then it's fine.  You've correctly
> understood the API associated with IServiceCollection and your
> implementation (at least at first glance, I don't have time to test it
> exhaustively :)) is correct.
>
> However, there's a more philosophical question as well: if you really want
> to *re-start* from scratch, what exactly are you shutting down and
> starting up again?  Wouldn't you want to be able to load new code?  Simply
> stopping and starting objects won't re-initialize the process from the
> beginning, only from some indeterminate middle state where parts of the
> program are already set up; this doesn't have the usual desired effect of a
> "restart" where potentially unknown or buggy state is cleared away.
>
> You may want to consider simply calling execv https://docs.python.org/
> 2.7/library/os.html#os.execv with [sys.executable] + sys.argv rather than
> trying to do this in terms of Twisted's APIs.
>

Or better yet.  Don't be afraid to just exit.  Your process is being
managed by a supervisor with a rich feature set - like easily controlled
re-restart behavior, right (If not, what do you do when your daemon simply
*crashes*)?

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 17.1 Release Announcement

[Jean-Paul Calderone]

On Thu, Feb 16, 2017 at 12:47 AM, Glyph Lefkowitz <gl...@twistedmatrix.com>
wrote:

>
> On Feb 15, 2017, at 6:51 AM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
> It doesn't sound crazy to say "treq 16.12.0 test suite should be required
> to pass for Twisted trunk@HEAD (at least until around Twisted 17.12,
> probably?).
>
>
> That seems like a reasonable requirement, yes.
>
>
Here's *a* take on that: https://tm.tl/#9042 /
https://github.com/twisted/twisted/pull/710

I didn't pin treq to 16.12.0 but instead let it float on the most recent
release.  That seems safe as long as treq takes care not to issue a release
that contains changes that makes it incompatible with Twisted trunk@HEAD.
 treq CI tests against Twisted trunk@HEAD so I think this is low risk.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Trial & Hypothesis

[Jean-Paul Calderone]

Hey all,

A few folks had an IRC discussion about improving the trial & hypothesis
experience.  I took some notes and thought I'd share them here in case
anyone else is interested.

We're planning to pick up the discussion next Wednesday.  Feel free to join
in #twisted-dev on Freenode (we might set up a Google Hangout, details will
be on IRC if so).

Jean-Paul
* Goals
** Make Twisted / Hypothesis as convenient as Anything Else / Hypothesis.

*** trial doesn't discover "runTest" so Hypothesis stateful tests aren't 
discovery
 Make trial discover "runTest"
* This might break things
* Also TestLoader.getTestCaseNames is incompatible with test methods 
without a test_ prefix (so we need a new interface)

 Make Hypothesis name its test "test_something"
* Name could be an argument to a stateful TestCase factory

*** trial TestCases have extra helper methods not availabe to Hypothesis 
stateful tests
 Make base TestCase class an argument to a stateful TestCase factory

*** setUp / tearDown are called just once even though test body might be called 
many times; and
*** tests that return Deferred are simply incompatible with Hypothesis
 Hypothesis has an "executor" hook (eg new_style_executor) that might be a 
good place to hang reactor knowledge
 this could also be a place to hang better setUp / tearDown handling
* this would be incompatible with current operation which has separate 
setup_example/teardown_example hooks
* and there's still only a single TestCase instance having methods called 
on it repeatedly which could be surprising
** `trial -u` does this already though, so maybe less so for trial users
* but it might be better, anyway

*** Hypothesis does a lot of work within a single test method.
 A consequence is a ton of log data mixed together from runs with different 
values.
* It would be nice to be able to see minimal logging from just the most 
shrunk iteration.
 Maybe there could be a custom HypothesisSuite that knows how repeatedly 
instantiate a TestCase and run it more normally
* This replaces the loop inside a test_ method with a loop around something 
more like TestCase().run()
** might solve many problems at once
** confuses things like `trial -j`
** causes each test run to have a different number of tests, which is weird
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 17.1 Release Announcement

[Jean-Paul Calderone]

On Wed, Feb 15, 2017 at 9:11 AM, Tristan Seligmann <mithra...@mithrandi.net>
wrote:

> On Wed, 15 Feb 2017 at 15:23 Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
>>
>> I wonder about the details of how treq's failures went unnoticed.  Is
>> development sufficiently inactive that no one looked at CI between the
>> breakage and the release?  Wasn't that a period of several months?  Or does
>> it have a Twisted trunk@HEAD build that none of the contributors really
>> pay attention to?
>>
>
> The latter is the case; there are Twisted trunk builds, but they're
> flagged as "failures allowed" to avoid disrupting development with
> transient breakage. Unfortunately this means that basically nobody ever
> notices when they fail. For example, here's a failing build:
>
> https://travis-ci.org/twisted/treq/builds/194691849
>
> I haven't found a good solution to this problem yet in my own projects;
> I'm hoping someone else has some ideas?
>

Given an arbitrarily powerful CI system, you could do something.

Assume two projects, A and B, with a dependency between them (doesn't
matter which direction, I think; doesn't even have to be one-way though
obviously it should be).

When A changes, do a build of B.  Use the new A and the newest version of B
which had a successful build.  If this build succeeds, A and B are probably
still working well together.  If this build fails, there's a good chance A
just introduced a regression.

Apply the same logic with A and B reversed.

I expect you could implement this on BuildBot.  It might be more realistic
to do something slightly simpler - like pinning the version of B in the
A-changed builds (and vice versa).  This would miss regressions relating to
changes in B introduced after the pinned version - but maybe that's okay
(you could update pinned versions with each release of B or once every few
months or something).  I think you could implement this on something like
travis-ci without *much* difficulty.  Just duplicate (some of) the treq
(for example) travis-ci configuration in Twisted's travis-ci
configuration.  It doesn't sound crazy to say "treq 16.12.0 test suite
should be required to pass for Twisted trunk@HEAD (at least until around
Twisted 17.12, probably?).

And of course there's bound to be noise out of this system, too.  I'd guess
it would be little enough that the result is still valuable but hard to say
for sure without trying...

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 17.1 Release Announcement

[Jean-Paul Calderone]

On Wed, Feb 15, 2017 at 3:39 AM, Glyph Lefkowitz <gl...@twistedmatrix.com>
wrote:

>
> On Feb 11, 2017, at 5:54 PM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
> On Sat, Feb 11, 2017 at 5:03 AM, Amber Hawkie Brown <
> hawk...@atleastfornow.net> wrote:
>
>> On behalf of Twisted Matrix Laboratories, I am honoured to announce the
>> release of Twisted 17.1!
>>
>> The highlights of this release are:
>>
>> - twisted.web.client.Agent now supports IPv6! It's also now the primary
>> web client in Twisted, with twisted.web.client.getPage being deprecated in
>> favour of it and Treq.
>> - twisted.web.server has had many cleanups revolving around timing out
>> inactive clients.
>> - twisted.internet.ssl.CertificateOptions has had its `method` argument
>> deprecated, in favour of the new raiseMinimumTo, lowerMaximumSecurityTo,
>> and insecurelyLowerMinimumTo arguments, which take TLSVersion arguments.
>> This allows you to better give a range of versions of TLS you wish to
>> negotiate, rather than forcing yourself to any one version.
>> - twisted.internet.ssl.CertificateOptions will use OpenSSL's
>> MODE_RELEASE_BUFFERS, which will let it free unused memory that was held by
>> idle TLS connections.
>> - You can now call the new twist runner with `python -m twisted`.
>> - twisted.conch.ssh now has some ECDH key exchange support and supports
>> `hmac-sha2-384`.
>> - Better Unicode support in twisted.internet.reactor.spawnProcess,
>> especially on Windows on Python 3.6.
>> - More Python 3 porting in Conch, and more under-the-hood changes to
>> facilitate a Twisted-wide jump to new-style classes only on Python 2 in
>> 2018/2019. This release has also been tested on Python 3.6 on Linux.
>> - Lots of deprecated code removals, to make a sleeker, less confusing
>> Twisted.
>> - 60+ closed tickets.
>>
>> For more information, check the NEWS file (link provided below).
>>
>> You can find the downloads at <https://pypi.python.org/pypi/Twisted> (or
>> alternatively <http://twistedmatrix.com/trac/wiki/Downloads>). The NEWS
>> file is also available at <https://github.com/twisted/tw
>> isted/blob/twisted-17.1.0/NEWS>.
>>
>> Many thanks to everyone who had a part in this release - the supporters
>> of the Twisted Software Foundation, the developers who contributed code as
>> well as documentation, and all the people building great things with
>> Twisted!
>>
>>
> Thanks all!  Great to see the troubles of 16.7 mostly behind us.
>
>
> Yes, thanks!  Lots of good stuff in this release!
>
> I do hope there will be at 17.1.1 soon for <https://twistedmatrix.com/
> trac/ticket/9031> and particularly <https://twistedmatrix.com/
> trac/ticket/9032>.
>
>
> This is a little outside the normal process, but I added a milestone for
> these just so we have a thing to track when they're fixed and we should
> roll them up (and in case anything new appears):
>
> https://twistedmatrix.com/trac/milestone/17.1.1
>
> Something I should note about these regressions; they showed up as soon as
> Mimic upgraded its dependency: https://github.
> com/rackerlabs/mimic/pull/734
>
> This suggests to me that we really ought to have a more "active" phase of
> the release process where someone - let's say, the reviewer of the release
> ticket - goes out to check a select list of other projects' smoke tests to
> ensure that the prerelease is at least passing their CI.  For example, I
> would normally have tested Mimic myself, but I was out of commission with a
> nasty flu-like thing for a good chunk of the prerelease period, but since I
> happened to miss that window, well-known projects (heck, treq has its own
> tests for .testing, and those broke, and it's in the twisted org!) managed
> to break without getting noticed.
>

I wonder about the details of how treq's failures went unnoticed.  Is
development sufficiently inactive that no one looked at CI between the
breakage and the release?  Wasn't that a period of several months?  Or does
it have a Twisted trunk@HEAD build that none of the contributors really pay
attention to?

If the problem is attention, maybe some kind of alerting could be set up so
the folks who are actually working on Twisted have a direct feed of
information that might be related to that work creating regressions?  I
know there are problems with false positives from CI but the dial for
information flow seems to be set a bit too low right now.  Maybe turning it
up for a while and then re-evaluating makes sense.


> So, while it would be good if user projects reported breakages, we should
> be checking at