Re: [Twisted-Python] automat question

[Chris Withers]

On 15/03/2021 09:34, Glyph wrote:



Right, but as best I can tell, outputs in automat have to be defined 
as part of the state machine class, I need targets only available 
after class instantiation to be notified.


Outputs are just methods, so it sort of depends what you mean by "after 
class instantiation".  You can call whatever you want /in/ an output.


Right, but then I'd have to maintain my own list of subscribers and 
figure out a way to duplicate in that logic in each of the output 
methods defined on the machine.


Again... have outputs.  I think there's something you're leaving out 
about how you want to have some /generalized/ output, but without 
knowing a bit more it's hard to say how it could help more :).


I think it's what I said above. TBH, I'm probably going to end up 
writing "yet another state machine class" that does just what I need, 
but I think that's okay - not everything has to work for every 
situation :-)


It sounds like what you /might/ want here is soemthing for constructing 
arbitrary machines at runtime, 


The states and transitions are very rigid.

and automat is all about enforcing that 
your machine is static so that (for example) it can be visualized 
statically.  So yeah in that case, different library.


Take this example:

class LightSwitch(object):
_machine = MethodicalMachine()

@_machine.state(serialized="on")
def on_state(self):
"the switch is on"

@_machine.state(serialized="off", initial=True)
def off_state(self):
"the switch is off"

@_machine.input()
def flip(self):
"flip the switch"

on_state.upon(flip, enter=off_state, outputs=[])
off_state.upon(flip, enter=on_state, outputs=[])


What I'm looking to do is something along the lines of:

aSwitch = LightSwitch()
aSwitch.flip.addListener(myCallable)

myCallable might well be a deferred that something else is waiting on...

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] automat question

[Chris Withers]

On 09/03/2021 18:53, Glyph wrote:


On Mar 9, 2021, at 3:18 AM, Chris Withers <mailto:ch...@withers.org>> wrote:


I'm not sure we're quite on the same page: I'm not looking to inspect 
the state, but more be notified when certain edges are traversed.


The way you get notified when an edge is traversed is you add an output 
to that edge :).


Right, but as best I can tell, outputs in automat have to be defined as 
part of the state machine class, I need targets only available after 
class instantiation to be notified.


Again... have outputs.  I think there's something you're leaving out 
about how you want to have some /generalized/ output, but without 
knowing a bit more it's hard to say how it could help more :).


I think it's what I said above. TBH, I'm probably going to end up 
writing "yet another state machine class" that does just what I need, 
but I think that's okay - not everything has to work for every situation :-)


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] automat question

[Chris Withers]

I'm not sure we're quite on the same page: I'm not looking to inspect 
the state, but more be notified when certain edges are traversed.


That said: what's the best practice when you want to visualize the 
current state of a machine? In this case, the machine is for a unit of 
work in a scheduling app, and the UI has a list of these units where 
it'd make sense to show the current state.


The unit of work also forms the boundary API between the from end object 
model and the pool/scheduling back end. As such, each side can naturally 
call inputs methods on the machine, as I believe automat intends, but 
it's how to get notified by an output?


So, the front end may want to say "I no longer need this UoW done, 
please kill it", the back end then needs to run the code to kill 
whatever's running the UoW.
Likewise, when the backend finishes a UoW, it wants to call either 
machine.result(...) or machine.error(...) to say what happened, but how 
do I wire in the front end bit that needs to get called when this happens?


cheers,

Chris

On 07/03/2021 22:34, Glyph wrote:
Automat is designed to make this sort of thing intentionally annoying, 
as you have discovered:).


The idea is that if you want to know this sort of internal state, it’s 
for a specific reason. That's not a blanket "No" — see for example how 
automat deals with serialization — but each such interface should be 
minimal and thoughtfully designed. Otherwise a state machine library 
just becomes a bunch of complex infrastructure around making an 
arbitrary series of function calls, and loses all of its helpful 
formalisms.


So probably we do need to make a change to automat if you really need 
to do this, but first it's important to know what your use-case is. In 
a lot of cases the answer is "just make an output. Stop trying to make 
the application see into the guts of the framework." But without 
knowing it’s impossible to say!


-g

Hi,

Apologies if there's a better list for this, please let me know where...

I've grown to like Glyph's automat package a lot, but in a current
project, I have observers that need to know when a machine changes state.

What's the best way to let those observers know?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] automat question

[Chris Withers]

Hi,

Apologies if there's a better list for this, please let me know where...

I've grown to like Glyph's automat package a lot, but in a current 
project, I have observers that need to know when a machine changes state.


What's the best way to let those observers know?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] twisted vs asyncio vs tornado (and other event loops?)

[Chris Withers]

Hi All,

Just bumped into Glyph's post from last year:
https://twistedmatrix.com/pipermail/twisted-python/2019-April/032280.html

Doesn't look like there was much of a response, so wondered what had 
happened with the plans laid out?


Also interested in how people see crochet fitting into this world...

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

[chris]

[Closing the loop for future contextual searches]

 

This solved the problem.  Moving all Twisted reactor-related imports inside 
overloaded multiprocessing.Process.run() functions, allows a single controller 
process to manage many Twisted reactors running in child processes.  

 

Thanks again.

-Chris

 

 

From: Twisted-Python  On Behalf Of 
ch...@cmsconstruct.com
Sent: Saturday, September 12, 2020 12:07 PM
To: 'Twisted general discussion' 
Subject: Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

 

Hi Jean-Paul,

 

Thank you very much for the detailed answer.  And my appologies for not 
providing OS details; I’ve tested on CentOS and RedHat EL variants, not FreeBSD 
as the ticket discussed.  Looks like Red Hat (EL 7.6) is using epoll reactor, 
and the Windows side is using the select reactor.

 

Thanks for the direction on checking out sys.modules.  To avoid the reactor 
being loaded in the parent process, I can presumably move twisted imports 
within the multiprocessing child modules (from top, down into the run() 
functions).  I will see how far I need to go (e.g. if I can continue using 
Twisted’s JSON logging or if absolutely everything should be isolated until 
after child process startup).  But knowing I need to head that direction for 
epoll or other potential reactor conflicts - is very helpful.

 

Reminds me of the GI Joe cartoon in the early 1980’s that would end with, 
“knowing is half the battle.”

 

-Chris

 

 

From: Twisted-Python mailto:twisted-python-boun...@twistedmatrix.com> > On Behalf Of Jean-Paul 
Calderone
Sent: Friday, September 11, 2020 1:28 PM
To: Twisted general discussion mailto:twisted-python@twistedmatrix.com> >
Subject: Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

 

On Fri, Sep 11, 2020 at 1:34 PM mailto:ch...@cmsconstruct.com> > wrote:

Hey guys,

 

Last year I hit a condition discussed in this ticket: 
https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a 
twisted.internet.tcp.Port.  

 

I ignored it at the time since it was just on Linux, and my main platform was 
Windows.  Now I’m coming back to it.  I’ll add context on the problem below, 
but first I want to ask a high-level, design-type question with multiprocessing 
and Twisted:

 

Referencing Jean-Paul’s comment at the end of ticket 4759, I read you shouldn’t 
fork a process (multiprocessing module) that already has a Twisted reactor.  
Understood.  But what about a parent process (not doing anything Twisted) 
forking child processes, where each child process starts their own Twisted 
reactor?  Is that intended to work from the Twisted perspective?

 

To answer the asked question, I don't think there is rigorous (or even casual) 
testing of very much of Twisted in the context of "some Twisted code has been 
loaded into memory and then the process forked".  So while it seems like a 
reasonable thing, I wouldn't say there's currently much effort being put 
towards making it a supported usage of Twisted.  Of course this can change at 
almost any moment if someone decides to commit the effort.

 

To dig a bit further into the specific problem, even if you only import the 
reactor in the parent process and then fork a child and try to start the 
reactor in the child, I strongly suspect epollreactor will break.  This is 
because the epoll object is created by reactor instantiation (as opposed to 
being delayed until the reactor is run).  epoll objects have a lot of weird 
behavior.  See the Questions and Answers section of the epoll(7) man page.

 

I don't know if this is the cause of your particular expression of these 
symptoms (it certainly doesn't apply to the original bug report which is on 
FreeBSD where there is no epoll) but it's at least a possible cause.

 

This could probably be fixed in Twisted by only creating the epoll object when 
run is called.  There's nothing particularly difficult about that change but it 
does involve touching a lot of the book-keeping logic since that all assumes it 
can register file descriptors before the reactor is started (think 
reactor.listenTCP(...); reactor.run()).

 

I'm not sure but it may also be the case that only delaying creation of the 
waker until the reactor starts would also fix this.  This is because as long as 
the epoll object remains empty a lot of the weird behavior is avoided and the 
waker is probably the only thing that actually gets added to it if you're just 
importing the reactor but not running it before forking.

 

Alternatively, your application should be able to fix it by studiously avoiding 
the import of twisted.internet.reactor (directly or transitively, of course).  
You could add some kind of assertion about the state of sys.modules immediately 
before your forking code to develop some confidence you've managed this.

 

And if this is really an epoll problem then switching to poll or select reactor 
would also presumably get rid

Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

[chris]

Hi Jean-Paul,

 

Thank you very much for the detailed answer.  And my appologies for not 
providing OS details; I’ve tested on CentOS and RedHat EL variants, not FreeBSD 
as the ticket discussed.  Looks like Red Hat (EL 7.6) is using epoll reactor, 
and the Windows side is using the select reactor.

 

Thanks for the direction on checking out sys.modules.  To avoid the reactor 
being loaded in the parent process, I can presumably move twisted imports 
within the multiprocessing child modules (from top, down into the run() 
functions).  I will see how far I need to go (e.g. if I can continue using 
Twisted’s JSON logging or if absolutely everything should be isolated until 
after child process startup).  But knowing I need to head that direction for 
epoll or other potential reactor conflicts - is very helpful.

 

Reminds me of the GI Joe cartoon in the early 1980’s that would end with, 
“knowing is half the battle.”

 

-Chris

 

 

From: Twisted-Python  On Behalf Of 
Jean-Paul Calderone
Sent: Friday, September 11, 2020 1:28 PM
To: Twisted general discussion 
Subject: Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

 

On Fri, Sep 11, 2020 at 1:34 PM mailto:ch...@cmsconstruct.com> > wrote:

Hey guys,

 

Last year I hit a condition discussed in this ticket: 
https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a 
twisted.internet.tcp.Port.  

 

I ignored it at the time since it was just on Linux, and my main platform was 
Windows.  Now I’m coming back to it.  I’ll add context on the problem below, 
but first I want to ask a high-level, design-type question with multiprocessing 
and Twisted:

 

Referencing Jean-Paul’s comment at the end of ticket 4759, I read you shouldn’t 
fork a process (multiprocessing module) that already has a Twisted reactor.  
Understood.  But what about a parent process (not doing anything Twisted) 
forking child processes, where each child process starts their own Twisted 
reactor?  Is that intended to work from the Twisted perspective?

 

To answer the asked question, I don't think there is rigorous (or even casual) 
testing of very much of Twisted in the context of "some Twisted code has been 
loaded into memory and then the process forked".  So while it seems like a 
reasonable thing, I wouldn't say there's currently much effort being put 
towards making it a supported usage of Twisted.  Of course this can change at 
almost any moment if someone decides to commit the effort.

 

To dig a bit further into the specific problem, even if you only import the 
reactor in the parent process and then fork a child and try to start the 
reactor in the child, I strongly suspect epollreactor will break.  This is 
because the epoll object is created by reactor instantiation (as opposed to 
being delayed until the reactor is run).  epoll objects have a lot of weird 
behavior.  See the Questions and Answers section of the epoll(7) man page.

 

I don't know if this is the cause of your particular expression of these 
symptoms (it certainly doesn't apply to the original bug report which is on 
FreeBSD where there is no epoll) but it's at least a possible cause.

 

This could probably be fixed in Twisted by only creating the epoll object when 
run is called.  There's nothing particularly difficult about that change but it 
does involve touching a lot of the book-keeping logic since that all assumes it 
can register file descriptors before the reactor is started (think 
reactor.listenTCP(...); reactor.run()).

 

I'm not sure but it may also be the case that only delaying creation of the 
waker until the reactor starts would also fix this.  This is because as long as 
the epoll object remains empty a lot of the weird behavior is avoided and the 
waker is probably the only thing that actually gets added to it if you're just 
importing the reactor but not running it before forking.

 

Alternatively, your application should be able to fix it by studiously avoiding 
the import of twisted.internet.reactor (directly or transitively, of course).  
You could add some kind of assertion about the state of sys.modules immediately 
before your forking code to develop some confidence you've managed this.

 

And if this is really an epoll problem then switching to poll or select reactor 
would also presumably get rid of the issue.

 

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] doWrite on twisted.internet.tcp.Port

[chris]

Hey guys,

 

Last year I hit a condition discussed in this ticket:
https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a
twisted.internet.tcp.Port.  

 

I ignored it at the time since it was just on Linux, and my main platform
was Windows.  Now I'm coming back to it.  I'll add context on the problem
below, but first I want to ask a high-level, design-type question with
multiprocessing and Twisted:

 

Referencing Jean-Paul's comment at the end of ticket 4759, I read you
shouldn't fork a process (multiprocessing module) that already has a Twisted
reactor.  Understood.  But what about a parent process (not doing anything
Twisted) forking child processes, where each child process starts their own
Twisted reactor?  Is that intended to work from the Twisted perspective?

 

 

Context:

I only hit this problem on Linux, not Windows.

 

The software project (github.com/opencontentplatform/ocp) has been a lot of
fun, especially with walking the tight rope in using multi-processing,
multi-threading, and Twisted reactors.  The main controller process kicks
off about 10 child processes, each doing different types of work.  In
general though, the child processes individually start a Twisted reactor,
connect to Kafka, connect to a database, use a shared REST API, and some
listen for connecting clients to accomplish work.

 

I test on Linux about once a year, so too many changes to rollback and
figure out that way.  It was working on Linux 2 years ago, but last year's
testing and current testing, receive the doWrite error.  It continues
running fine on Windows.  I've gone back about 2 years of versions with
Python3, Twisted, and dependent libs. on both Windows and Linux.  Every
version change yields the same result - continues to work on Windows and
continues to hit the error on Linux.  So something I added has caused Linux
to throw that error.  

 

I'm not explicitly sharing much between the main process and the sub
processes.  They are spun up with (1) a shared multiprocessing.Event() - to
have the main process shut the children down, (2) their own unique
multiprocessing.Event() - to have the child processes notify back to the
parent, and (3) a deep copy of a dictionary (containing a bunch of settings
that remain constant).  The main process uses twisted.logger, but for
testing I strip that out to remove any twisted imports in the main process.
So I'm not importing anything Twisted in the main process, and I don't
believe I'm explicitly sharing something I shouldn't.  Seems like something
is implicitly being exposed/shared across Linux child processes, that aren't
on Windows.

 

The tracebacks come through on Linux (sometimes randomly), on the console of
the parent controller process.  No need to paste here, since it's the same
as the ticket shows.  I can't reliably reproduce the problem, but I know if
I stop/start client connections
(ServerFactory/twisted.internet.interfaces.IReactorTCP and
twisted.internet.protocol.ReconnectingClientFactory) then it will eventually
happen.  I need to devote time at whittling down the code and attempting to
create a reliable test case. if even possible.

 

The error is slightly different when running HTTP vs HTTPS, but the story is
the same.  It cripples whatever child process that hits it, from doing much
of anything thereafter.  Not much luck with troubleshooting.  The tracebacks
do not include a calling function from my code, to tell me where to start.
And it happens across different child process types, so not the same one
each time.  When I throw debuggers on the child processes, the problem seems
to mask itself.  Well, at least I didn't hit the problem over the last 3
days using pudb and stepping through code at breakpoints.

 

I'm absolutely open to suggestions for troubleshooting, but first wanted to
take a HUGE step back and ask a design question regarding Twisted and
multiprocessing.

 

Thanks!

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] unified filesystem API

[Chris Withers]

On 16/05/2020 07:42, Glyph wrote:



On May 15, 2020, at 11:30 PM, Chris Withers <mailto:ch...@withers.org>> wrote:


On 16/05/2020 06:55, Glyph wrote:
This does point out one of my secret hopes for SMB: that a file 
server's maintainers will care enough about file throughput that 
we'll finally get a centralized, official way of doing async file I/O 
that we can share with SFTP, FTP, and HTTP :).


I got excited by the title, but this all seems to be able giving 
non-filesystems a file-system like API.


What are you referring to as non-filesystems?


Ah, sorry, misread the above.

If I've missed something, please let me know, but I couldn't find an 
async interface for writing to actual file systems.


In Twisted? There isn't one, really, which is exactly the issue.  There 
are various operating system interfaces for this but none of them are great.


What's the asyncio spin on this? We server like uvicorn are all writing 
logs, are they too just punting on the blocking nature of writing to 
filesystems?


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] unified filesystem API

[Chris Withers]

On 16/05/2020 06:55, Glyph wrote:

This does point out one of my secret hopes for SMB: that a file server's 
maintainers will care enough about file throughput that we'll finally get a 
centralized, official way of doing async file I/O that we can share with SFTP, 
FTP, and HTTP :).


I got excited by the title, but this all seems to be able giving 
non-filesystems a file-system like API.


If I've missed something, please let me know, but I couldn't find an 
async interface for writing to actual file systems. It feels like 
Twisted has just punted on them as "fast enough" to not need async 
interfaces, but that can really not be the case for networked 
filesystems (NFS, GPFS, etc) where the blocking time for reading or 
writing can be seconds (or minutes if it's having a bad day!).


What's the Twisted solution for these kinds of things? Defer all file IO 
into a thread?


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Getting nice logging outputs in py.test

[Chris Withers]

On 04/07/2019 21:43, Jean-Paul Calderone wrote:

How are you running the reactor in your pytest suites? I've not found a
sane way to do this and so would recommend sticking with trial, crufty
and old though it feels...


What feels crufty and old about trial?  And why is "old" a negative?


"old" is negative here because things have moved on and trial hasn't.

pytest feels like it's become the python standard now, so there's that.
It's rich ecosystem of plugins is another one.
The reliance on subclassing TestCase is nother.
No fixtures.
Poor support in pycharm for trial versus first class support for pytest.

That's just off the top of my head, use pytest for a year or two and 
coming back to trial may well feel the same.


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Getting nice logging outputs in py.test

[Chris Withers]

On 04/07/2019 14:39, Thomas Westfeld wrote:

Dear all,

I am having an issue with logging of my Twisted application. I am using the new 
twisted.logger framework. The output produced by twist or twistd when runnning 
my plugin is nice.

However when running a test suite using py.test I only get this as an output to 
stdout:


How are you running the reactor in your pytest suites? I've not found a 
sane way to do this and so would recommend sticking with trial, crufty 
and old though it feels...


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Binary wheels for Twisted on Windows?

[Chris Satterthwaite]

Hi Griatch,

 

Our stack-based solution for customers hosting our platform requires the same.  
We could compile and distribute, but then we’d have to support dependencies… 
and I’ve never been a fan of embedded dependencies.  We do that for software 
installation (embedding python and libs into a binary via PyInstaller), but for 
production run-time it’s just the modules – hence similar dependencies on 
developer tools being installed.

 

The Microsoft link has changed over the years.  I’ve pasted what we have in our 
documentation for those tools below, which looks like it’s validated on Windows 
10 Pro, Win server 2012 R2, and Win server 2016.  Also, on the Server side, I 
believe you also need to be mindful of the .NET versioning… in case there’s 
another application dependency.

 

===

Get the developer tools from Microsoft:

 

1.  Go to the following URL:  https://www.visualstudio.com/vs/community/
2.  Download and run the vs_community file 
3.  Select "Desktop development with C++" 
4.  Reboot when finished 

 

-Chris

 

 

From: Twisted-Python  On Behalf Of 
Griatch Art
Sent: Tuesday, May 28, 2019 6:04 AM
To: twisted-python@twistedmatrix.com
Subject: [Twisted-Python] Binary wheels for Twisted on Windows?

 

Hi, 

 

I'm investigating installing the Evennia MU* server on Windows. We use Twisted 
and will be requiring Python3.7 in our next release. I need to make easy-to-use 
install instructions since a lot of Windows users use our library. 

 

I tested with a Windows7 64bit VM and installed everything from scratch to 
emulate what a non-dev Windows user would see. I don't have Windows10 so can't 
compare to the install experience there (but Windows7 64bit is still relevant, 
having something like 24% of the active Windows user-base according to Steam).

 

Using pip to install Evennia, at the Twisted requirement install step I run 
into an error telling me that I need "Microsoft Visual C++ build tools" from 
the URL https://visualstudio.microsoft.com/downloads. The first issue is that 
there does not appear to be any build-tools package named like that on that 
page or sub-page (at least not what I could find after digging around). I tried 
to install a few similarly-named packages, like "Visual Studio Build Tools", 
but  had no luck getting past the Twisted install point. So that 
recommendation-string should likely be updated. 

 

The thing is though, while I could probably personally figure out how to set it 
up eventually, our Windows users are likely the least tech-savvy of our users. 
Requiring them to set up a compiler environment (despite us telling them that 
Python code does not need compilation) a bit too much. 

It seems Twisted has distributed binary Windows wheels in the past, would it be 
possible to get them again? Or should I recommend some other, specific install 
procedure for our Windows users?

 

Cheers, 

Griatch, Evennia dev

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] finding out what's blocking the reactor

[Chris Withers]

On 14/05/2019 16:39, Scott, Barry wrote:

On Monday, 13 May 2019 15:26:01 BST Chris Withers wrote:

Hi All,

On some busy instances of the app I'm working with/on, we see websocket
connections being dropped due to not ping/ponging in time, I'm fairly
sure this is as a result of some user-written callbacks blocking the
reactor for long periods of time (seconds...).

What's the best way to debug a twisted application to find out what's
blocking the reactor?


Is the process CPU bound or IO bound?


Yes ;-)


If CPU bound then cprofile or the tool Glyph mentioned would help.

This advice is from a Linux perspective.

If its IO bound the you may be waiting on a blocking operation.
You could use strace on the process to see what its doing.


Good call on strace, will do so next time one of these happens.


As its hung for seconds you could connect gdb to the process and see a python
backtrace using the py-bt command. You are likely to catch the offending code.


I'm failing miserably to get gdb to find debug symbols for python. I've 
installed the system python's debug symbols rpm, I've installed gdb .py 
that ships in the python source tree (so at least I *have* a py-bt 
command now!) - what am I likely to be missing?



You could adding logging to your code to show the time of callback entry and
exit.


Right, but if the callback yields, that will be falsely shown as 
blocking, no?


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] finding out what's blocking the reactor

[Chris Withers]

On 14/05/2019 22:14, Werner Thie wrote:

Hi all

with periods of seemingly no activity the gc comes to mind.


Hmm, gc blocking the process is an interesting thought, what evidence 
would you see with gdb, etc, to show that this was the problem?


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] finding out what's blocking the reactor

[Chris Withers]

Hi All,

On some busy instances of the app I'm working with/on, we see websocket 
connections being dropped due to not ping/ponging in time, I'm fairly 
sure this is as a result of some user-written callbacks blocking the 
reactor for long periods of time (seconds...).


What's the best way to debug a twisted application to find out what's 
blocking the reactor?


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] ws(s):// urls and host/port duplication in twisted/autobahn code

[Chris Withers]

On 02/04/2019 09:07, Tobias Oberstein wrote:
Right, but correct me if I'm wrong, my understanding is that URL 
stands for Univeral Resource Locator and URLs can encode all of the 
things you 


WebSocket needs a _HTTP_ URL


Not sure I follow, I specify ws:// and things work. What am I missing?
(I know the initial protocol is http and that's "upgraded" into 
websocket, but the url is still ws:// or wss://, right?)



fwiw, WAMP is a protocol with a clear-cut, rigorous and decoupled layering:


Do browsers speak WAMP? Do firewalls get upset with it?

describe. They certainly contain the host and port, so shouldn't there 
be a graceful way to specify a URL once and have everything that needs 
that info or a subset of it get it from there?


a (fully qualified) HTTP URL of course encodes a host and port, but that 
only covers transports that have those notions


Not sure what you mean by fully qualified?

(This isn't necessarily aimed at AutoBahn, seems to be something 
common in Twisted...)


things look only "simple" when leaving out details;)


There's never a reasonable excuse for duplicating the same information 
within a few lines of code.


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Withers]

On 02/04/2019 08:25, Tobias Oberstein wrote:
https://github.com/crossbario/autobahn-python/blob/master/examples/twisted/wamp/basic/client_using_clientservice.py 


Any way to get rid of the ugly duplication between the url in line 69 
and the endpoint on line 72?


I seem to remember that ends up happening with or without ClientService.


I am aware, but no, it's designed like this, because WebSocket requires 
a HTTP URL (with potentially path components, query parameters etc) in 
the opening handshake while the _transport_ does not necessarily have that.


Autobahn allows you to run WebSocket over basically everything that is a 
bidirectional byte pipe: TCP of course, but also Unix domain socket, 
pipes, Tor onion services (via https://github.com/meejah/txtorcon) and 
even serial ports!


Right, but correct me if I'm wrong, my understanding is that URL stands 
for Univeral Resource Locator and URLs can encode all of the things you 
describe. They certainly contain the host and port, so shouldn't there 
be a graceful way to specify a URL once and have everything that needs 
that info or a subset of it get it from there?


(This isn't necessarily aimed at AutoBahn, seems to be something common 
in Twisted...)


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Withers]

On 26/03/2019 09:53, Tobias Oberstein wrote:
The Autobahn guys still show ReconnectingClientFactory in their docs, 


Where did you find that? That would be a doc bug, but in the _docs_, 
there is no reference to ReconnectingClientFactory


(cpy372_3) oberstet@intel-nuci7:~$ find 
~/scm/crossbario/autobahn-python/docs/ -type f -exec grep -Hi 
"ReconnectingClientFactory" {} \;

(cpy372_3) oberstet@intel-nuci7:~$

We do have some example code using ReconnectingClientFactory though:

(cpy372_3) oberstet@intel-nuci7:~$ find 
~/scm/crossbario/autobahn-python/examples/ -type f -exec grep -Hi 
"ReconnectingClientFactory" {} \; | wc -l

8


Yeah, this one: 
https://github.com/crossbario/autobahn-python/tree/master/examples/twisted/websocket/reconnecting


Would be good to get that converted...


 1. ReconnectingClientFactory is destined for deprecation, eventually.
  You should just adopt the "new" thing now so that if we get more
    energy to cycle the APIs and delete the old stuff, you'll have less
    hassle to deal with.  ("New" is in quotes here since it's been
    around for well over 3 years at this point; Autobahn should update
    too, not just you.)


Autobahn will automatically use

twisted.application.internet.ClientService

for auto-reconnect when on Twisted 16.1.0+

https://github.com/crossbario/autobahn-python/blob/master/autobahn/twisted/wamp.py#L349 


I'm not using WAMP, just raw websocket.


 2. ClientService works with endpoints, which means you can use it with
    /any/ kind of transport, like SSH transports, subprocesses, etc.
  Most practically, it works with HostnameEndpoint which is a much
    better way to get TLS than connectSSL; 


What's the simplest way to connect a ClientService to a websocket url 
(ie: ws://host/endpoint or wss://host/endpoint)



 1. if you want to shut down a ReconnectingClientFactory:
 1. you have to call stopTrying, then uh...
 2. find the last protocol it built with buildProtocol, then


This hasn't proved to hard.


 3. grab its transport (hope it's saving that transport as
    '.transport', because it doesn't actually have to)
 4. call loseConnection


Why? Sending a WebSocket protocol-level close from the client seems to 
work just fine?



 5. It's more testable because it just takes its clock and reactor as
    constructor parameters, rather than requiring post-hoc
    poorly-documented attribute patching to become testable.


I've abstracted all that away into https://github.com/cjw296/carly.
See 
https://github.com/cjw296/carly/blob/master/tests/test_autobahn_websocket_client.py 
for example.

I do still have writing docs for carly on my list of things to do.

It would be great if Twisted ever had a real testing reactor, but I 
think we all know that's never going to happen ;-)


Besides, it's actually useful to test with real networking...

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Satterthwaite]

Hi Chris,

The print and sleep statements are just my efforts at converting something big 
down to something small for a test case.  In the actual client I'm using 
twisted.logger for FilteringLogObserver's and Logger's.  The looping call for 
system health was also a shorted version; I wasn't sure if looping calls were 
the culprit at the start.  I appologise for the fluff and if I introduced 
confusion.  I just figured that posting multiple classes and thousands of lines 
would get less response than an attempted short test sample.


> when you say "When the client hits a problem, it calls 
> transport.loseConnection()", where is that call to transport.loseConnection?

The transport.loseConnection call is on line 82 of the test client code; it's 
inside the cleanup() function.


> Now, as to what your problem is, I suspect it's this call to stopTrying:
> https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L87

I may be missing this from the Twisted side, but I understood that stopFactory 
was only called when the reactor was cleaning up.  So I added stopTrying in 
there for the ReconnectingClientFactory portion, amongst many other lines (that 
I removed for the example) for stopping external dependencies.  All that seemed 
to work fine.


When I called loseConnection with just a single established connection, 
stopFactory was called and ReconnectingClientFactory was not restarted or 
reconnected.  Seemed like ReconnectingClientFactory was not expecting a new 
connection attempt after the loseConnection call.  Additionally, it seemed like 
the factory expected to clean up and stop the reactor whenever loseConnection 
was called with that single connection.  And of course in most use-cases, that 
makes sense.  But my goal was to hang on to the factory after loseConnection, 
and continue work after external dependencies came back online.


> when you noticed some Twisted source code that works off factory.numPorts, 
> where is that code?

To troubleshoot, I used __dict__ to start investigating variable values with 
the factory and client.  And I used that to start searching through the twisted 
code in site-packages.  I noticed at least one function which conditionally 
worked off numPorts in order to shut things down 
(twisted.internet.protocol.AbstractDatagramProtocol.doStop).  Here's the code:
assert self.numPorts > 0
self.numPorts = self.numPorts - 1
self.transport = None
if not self.numPorts:
if self.noisy:
log.msg("Stopping protocol %s" % self)
self.stopProtocol()

And so the work around I implemented was to conditionally increase this number 
while I controlled the disconnect/reconnect.  Seemed to work fine in practice.


Thanks again for the responses.  Given all the benefits Glyph mentioned of 
ClientService - I've added that migration into my roadmap.

-Chris


-Original Message-
From: Twisted-Python  On Behalf Of 
Chris Withers
Sent: Tuesday, March 26, 2019 2:28 AM
To: twisted-python@twistedmatrix.com
Subject: Re: [Twisted-Python] stop/start client connections with loseConnection 
in ReconnectingClientFactory

On 22/03/2019 23:26, Chris Satterthwaite wrote:
> Hi Chris,
> 
> The files I attached (with the full classes) made it through to my email, but 
> I wondered if they would they go through to everyone.
> 
> Here's a gist with the same scripts:
> https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519
> f
> 
> And yep, as you mentioned, a 'sleep' is definitely blocking.  That's not in 
> the production version; I just dropped it in here for the test script to 
> simulate something.

Observations:

- Your super call at
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L46,
I'd expect that to be super(ServiceClientFactory, self).__init__(), but your 
spelling may be a python 3 only thing that works?

- Those sleeps are going to cause you more problems than they solve.

- What does this seek to achieve? 
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L88-L89

- Why not use twisted logging instead of print? By setting it to debug, you'll 
get lots of into about what twisted is doing, and by using logging you won't 
need to do all that manual traceback printing.

- when you say "When the client hits a problem, it calls 
transport.loseConnection()", where is that call to transport.loseConnection?

- when you noticed some Twisted source code that works off factory.numPorts, 
where is that code? Can you provide a link? This doesn't sound right...

Now, as to what your problem is, I suspect it's this call to stopTrying:
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L87

The factory is stopped and started again by ReconnectingClientFactory, so you 
don't w

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Withers]

On 22/03/2019 23:26, Chris Satterthwaite wrote:

Hi Chris,

The files I attached (with the full classes) made it through to my email, but I 
wondered if they would they go through to everyone.

Here's a gist with the same scripts:
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f

And yep, as you mentioned, a 'sleep' is definitely blocking.  That's not in the 
production version; I just dropped it in here for the test script to simulate 
something.


Observations:

- Your super call at 
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L46, 
I'd expect that to be super(ServiceClientFactory, self).__init__(), but 
your spelling may be a python 3 only thing that works?


- Those sleeps are going to cause you more problems than they solve.

- What does this seek to achieve? 
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L88-L89


- Why not use twisted logging instead of print? By setting it to debug, 
you'll get lots of into about what twisted is doing, and by using 
logging you won't need to do all that manual traceback printing.


- when you say "When the client hits a problem, it calls 
transport.loseConnection()", where is that call to transport.loseConnection?


- when you noticed some Twisted source code that works off 
factory.numPorts, where is that code? Can you provide a link? This 
doesn't sound right...


Now, as to what your problem is, I suspect it's this call to stopTrying:
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file-testclient-py-L87

The factory is stopped and started again by ReconnectingClientFactory, 
so you don't want that there as it means you stop trying every time 
there's a disconnection.


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Withers]

On 24/03/2019 04:30, Glyph wrote:
I'd further note that ClientService is /generally/ the new, good way to 
do things and ReconnectingClientFactory is the old, bad way.  Our hope 
is to eventually deprecate ReconnectingClientFactory and most of the 
APIs that it uses, but this is a big project that we have not been able 
to make much progress on in the last, ahem, decade or so.


What's the big advantage(s) of ClientService over 
ReconnectingClientFactory?


The Autobahn guys still show ReconnectingClientFactory in their docs, 
and I remember looking at this before and ending up going with 
ReconnectingClientFactory because it works and didn't look like it'd 
need much effort to integrated into an existing code base.


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Satterthwaite]

Hi Sean,

 

Thanks for the suggestion; I’ll take a look.

 

-Chris

 

 

From: Twisted-Python  On Behalf Of 
Sean DiZazzo
Sent: Friday, March 22, 2019 3:09 PM
To: Twisted general discussion 
Subject: Re: [Twisted-Python] stop/start client connections with loseConnection 
in ReconnectingClientFactory

 

You may want to look at twisted.application.internet.ClientService 
<https://twistedmatrix.com/documents/18.7.0/api/twisted.application.internet.ClientService.html>
 .  It uses the new endpoints instead of the `connectTCP()` stuff.  Not sure if 
it applies in your situation, but it has all of the retry logic built in, so 
that may make it easier to work with.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Satterthwaite]

Hi Chris,

The files I attached (with the full classes) made it through to my email, but I 
wondered if they would they go through to everyone.

Here's a gist with the same scripts:
https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f

And yep, as you mentioned, a 'sleep' is definitely blocking.  That's not in the 
production version; I just dropped it in here for the test script to simulate 
something.

Thanks!
-Chris

-Original Message-
From: Twisted-Python  On Behalf Of 
Chris Withers
Sent: Friday, March 22, 2019 1:54 PM
To: twisted-python@twistedmatrix.com
Subject: Re: [Twisted-Python] stop/start client connections with loseConnection 
in ReconnectingClientFactory

On 22/03/2019 17:08, Chris Satterthwaite wrote:
>  def clientConnectionLost(self, connector, reason):

Without the rest of your class, it's difficult to see some of the potential 
problems... Could you put it on a gist somewhere?

> 
>  print('  factory clientConnectionLost: 
> reason: {}'.format(reason))
> 
>  # if self.disconnectedOnPurpose:
> 
>  # ## Hack to keep reactor 
> alive
> 
>  # print('  factory
> clientConnectionLost: increasing numPorts')
> 
>  # self.numPorts += 1
> 
>  # self.numPortsChanged = 
> True
> 
>  # 
> self.disconnectedOnPurpose = False
> 
>  print('  ... simulate client going 
> idle before attempting restart...')
> 
>  time.sleep(5)

This will block the reactor, so don't think you should be doing it...

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Withers]

On 22/03/2019 17:08, Chris Satterthwaite wrote:

     def clientConnectionLost(self, connector, reason):


Without the rest of your class, it's difficult to see some of the 
potential problems... Could you put it on a gist somewhere?




     print('  factory clientConnectionLost: 
reason: {}'.format(reason))


     # if self.disconnectedOnPurpose:

     #     ## Hack to keep reactor alive

     #     print('  factory 
clientConnectionLost: increasing numPorts')


     #     self.numPorts += 1

     #     self.numPortsChanged = True

     # 
self.disconnectedOnPurpose = False


     print('  ... simulate client going idle 
before attempting restart...')


     time.sleep(5)


This will block the reactor, so don't think you should be doing it...

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory

[Chris Satterthwaite]

Hello community,

 

First of all - thanks for an awesome platform!  I'm brand new to this
community, but have been using Twisted a couple years.

 

Reason for posting:

I've hit a condition with ReconnectingClientFactory that I'm not sure is per
design.  I have a work around right now, but need your perspective.  Seems
like there should be a better/right way to do this.

 

Attempted design:

I'd like to have long running TCP clients (forever until stopped), with a
long running TCP server.  When a long running client hits a problem with a
dependency (database is down, kafka bus unavailable, external API not
responding, etc), I want the client to go offline for a while and then come
back online. an automated, self-recovery type action.  Since it's not ok to
start/stop/restart the Twisted Reactor, I am letting the client finish
whatever it can do, disconnect from the service, destruct the dependencies,
wait for a period of time, and then attempt a clean re-initialization of
those dependencies along with reconnecting to the Twisted Server.

 

Problem case:

I'm using the ReconnectingClientFactory in my client.  When the client hits
a problem, it calls transport.loseConnection().  But whenever the client
calls this, after the disconnect - it does not reconnect; stopFactory is
called and everything exits. 

 

Work around:

I noticed some Twisted source code that works off factory.numPorts.  If
numPorts is 1 and the client loses the connection, it goes to 0 and calls
the cleanup.  So I conditionally increase this number right before
intentionally disconnecting, and then reset that after reconnecting.  This
solves the problem, but it's a hack.  

 

I'll attach the test scripts to this post (if attachments are allowed), but
the main code is with these functions in the factory:

 

def clientConnectionLost(self, connector, reason):

print('  factory clientConnectionLost:
reason: {}'.format(reason))

# if self.disconnectedOnPurpose:

# ## Hack to keep reactor alive

# print('  factory
clientConnectionLost: increasing numPorts')

# self.numPorts += 1

# self.numPortsChanged = True

# self.disconnectedOnPurpose =
False

print('  ... simulate client going idle
before attempting restart...')

time.sleep(5)

 
ReconnectingClientFactory.clientConnectionLost(self, connector, reason)

print('  factory clientConnectionLost:
end.\n')

 

def clientConnectionMade(self):

print('  factory clientConnectionMade:
starting numPorts: {}'.format(self.numPorts))

# if self.numPortsChanged :

# ## Resetting from hacked value

# print('  factory
clientConnectionMade: decreasing numPorts')

# self.numPorts -= 1

# self.numPortsChanged = False

print('  factory clientConnectionMade:
finished numPorts: {}'.format(self.numPorts))

 

def cleanup(self):

print('factory cleanup: calling
loseConnection')

if self.connectedClient is not None:

 
self.connectedClient.transport.loseConnection()

self.disconnectedOnPurpose =
True

 

With the above lines commented out, once the cleanup call does
transport.loseConnection(), the factory stops at the end of
clientConnectionLost. 

 

 

Sample scripts/logs:

I've tried to create short test scripts and corresponding logs (with the
client failing, and then with it restarting when I use the workaround).
I've cut out several thousand lines to get down to something simple for the
example test scripts, but I know the client is still a little long.  Again,
I'm not sure if attachments work on the mailing list, but I'll attempt to
attach the client/server scripts with the corresponding pass/fail logs.

 

Thanks!

 

-Chris

 

import os, sys, traceback
import json, time, datetime, psutil
from twisted.internet.protocol import ReconnectingClientFactory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor, task, defer, threads
from contextlib import suppress

class CustomLineReceiverProtocol(LineReceiver):
delimiter = b':==:'

class ServiceClientProtocol(CustomLineReceiverProtocol):
def connectionMade(self):
print('protocol connectionMade')
self.factory.connectedClient = self
self.factory.clientConnectionMade

Re: [Twisted-Python] ANN: Eliot 1.7, the causal logging library, now with inlineCallbacks support

[Chris Withers]

Hey Stranger, long time no speak :-)

On 21/03/2019 15:27, Itamar Turner-Trauring wrote:
Python and Twisted's built-in |logging| output a stream of factoids: 
they’re interesting, but you can’t really tell what’s going on.


  * Why is your application slow?
  * What caused this code path to be chosen?
  * Why did this error happen?

Standard logging can’t answer these questions.

But with a better model you could understand what and why things 
happened in your application. You could pinpoint performance 
bottlenecks, you could understand what happened when, who called what.


That is what Eliot does. |Eliot| is a Python logging system that outputs 
causal chains of *actions*: actions can spawn other actions, and 
eventually they either *succeed or fail*. The resulting logs tell you 
the story of what your software did: what happened, and what caused it.


Have you looked at tools like https://www.jaegertracing.io/?
Feels like quite a similar approach to Eliot, and it's be fantastic if 
Eliot could provide contexts to Jaeger for Twisted applications!


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] giving a reason when adding a timeout

[Chris Withers]

Unless I'm missing something, Deferred.addTimeout is really unhelpful in
terms on providing context about *what* timed out.
TimeoutError(, 'Deferred') just isn't that useful.

How come addTimeout doesn't let you specify a textual reason, or
otherwise provide some context about the timeout?

Am I missing something obvious here?


I don't know what kind of context do you need :)


foo.addTimeout(myTimeout, reactor, message='never heard back')


You can add your own errback and add your extra error handling there.


What's the must succinct way to get the above by adding an errback?


There is also onTimeoutCancel argument.


Pretty clunky to write a whole function for either this or the errback 
just to change the useless 'Deferred' string in the current 
implementation with something useful.



If there is anything missing from the API, feel free to send a pull
request in GitHub.


What's the likely turn around time from me submitting a PR to when it 
ends up in a released version of Twisted that I can use?


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] giving a reason when adding a timeout

[Chris Withers]

Hi All,

Unless I'm missing something, Deferred.addTimeout is really unhelpful in 
terms on providing context about *what* timed out.

TimeoutError(, 'Deferred') just isn't that useful.

How come addTimeout doesn't let you specify a textual reason, or 
otherwise provide some context about the timeout?


Am I missing something obvious here?

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] a farewell to buildbot

[Chris Withers]

  
  
What do you make of CircleCI?

On 03/03/2019 06:45, Amber Brown wrote:


  
  I have been switching matrix.org to Buildkite's open
source offering (where you bring your own builders). It might be
an option for things like codespeed, which we still would want
to be consistent.


Other than that, I was taking a look at Azure
  Pipelines last year. I think we could get a decent amount of
  hosted, easy to maintain build infrastructure through
  Microsoft, which includes the potential to also ditch running
  our own Windows for testing.


- Amber
  
  
  
On Sat., 2 Mar. 2019, 22:33
  Glyph, 
  wrote:


  Thanks
to our long-suffering contributors Adi Roiban and Kyle
Altendorf, we now have macOS builds running on circleCI! 
Supposedly we have sufficient resources to actually run all
our builds, too, without running out of CI juice before the
end of each month :-).


As such, I've changed our repository configuration to
  drop the old, somewhat outdated macOS buildbot, and
  replace it with the Circle CI infrastructure which should
  cover that platform.


With this change, none of the buildbot builders define
  our gating-to-trunk continuous integration.  This is a
  great thing for the project, as it means external
  contributors will be able to get a "this is acceptable to
  merge" green checkmark without ./admin/pr_as_branch or any
  other similar repo:write-person-requiring shenanigans.


However, it also means that we are now spending a
  not-insignificant amount of contributor time maintaining a
  farm of machines that do tons of continuous integration
  work, which may not really be telling us anything
  interesting about Twisted's quality or correctness.  I
  think it might be worth considering decommissioning buildbot.twistedmatrix.com entirely,
  unless some of the vendors of the platforms and kernels
  covered there would like to step up to do some maintenance
  themselves.  It's been a decade or so since Twisted was
  spotting regular regressions in Linux, FreeBSD or Darwin,
  so I think this style of build infrastructure may be a
  relic of a bygone era.


For my part, I probably will start doing any
  contributions on my own fork, since that will mean I don't
  have to constantly kick random spurious RHEL7 buildbot
  failures to avoid getting a red "X" on my PRs.


Furthermore, if we decom'd buildbot as software
  infrastructure, we'd still have a significant amount of
  cloud / hardware resources we could potentially throw at
  *other* problems facing the project which cloud CI doesn't
  cover as well, like SpeedCenter.


So, do folks have any strong feelings, or would anyone
  like to volunteer to help with some aspect of this?  As
  always: we don't have enough folks to keep up with the
  operational demands of twistedmatrix.com,
  so if you want to dev some ops or ops some infra, please
  speak up :).

  
  
  -g

  
  

  

  
  ___
  Twisted-Python mailing list
  Twisted-Python@twistedmatrix.com
  https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

  
  
  
  ___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


  


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] http://twistedmatrix.com/ down?

[Chris Withers]

Looks to be...

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] why can't a callback be called with a deferred?

[Chris Withers]

On 21/02/2019 07:32, Glyph wrote:



Yeah, this is part of carly, that I posted earlier. It stems from the 
need to get the results of method calls when you have no reference to 
the object being calls, or sometimes a result that's a deferred you 
need to wait on, particularly in a test, but have no way of doing so.


I just can't parse this sentence.


Sorry, some typos in there are not helping ;-(


Breaking it down:


you have no reference to the object being calls,


How do you have no reference to the object being called? Aren't you 
calling it?


That test case I linked to shows one example, the original ones were 
from abstracting out JML's original post here:

https://jml.io/pages/how-to-disconnect-in-twisted-really.html

In any case you either have a Deferred or you don't; if you do, then 
it's clear you should wait on it,


Yes, but where to wait is the question here. I'd like to leave that 
choice to the user (see back in the thread for the details).



This is the test situation where I hit this issue:
https://github.com/cjw296/carly/blob/master/tests/test_untracked_deferred.py#L28-L35

I'd originally wanted to have that read:

   @inlineCallbacks
   def test1(self):
   ...
   result = yield pita.asyncMethod.called()
   with ShouldRaise(Exception(1)):
   yield result

Now, which I'm actually happier with the end result here, I think the 
above it legit, if unusual, and that assert trips it up.


Some type annotations might make it a bit clearer what the two states 
here are :).  


If those annotations were there, what would they look like? (I did link 
to the whole test file, which has all the detail I think you could need, 
it's pretty self contained and not that long)


As it is, it looks to me you want a Deferred to come *out* 
of a 'yield', which should definitely never happen. 


I disagree, in this specific case.

If this assert were 
to be removed, it would be done in such a way that would implicitly wait 
for the Deferred in question to fire: you should /never/ receive a 
Deferred as an argument to a function, or as the result of an 
(inlineCallbacks) 'yield' or (async def) 'await'.  It breaks the whole 
model of what 'awaiting' means.


Well, I'd agree 90% of the time, but test_untracked_deferred.py is where 
that's no always true.


Anyway, I'm happy with the API I have now, and it neatly works around 
that assert, so ¯\_(ツ)_/¯.


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] why can't a callback be called with a deferred?

[Chris Withers]

On 21/02/2019 06:55, Glyph wrote:




The methods being hooked don't necessarily return deferreds.


Glyph, this bit ^^^


I'd like it to be an explicit choice of the caller, ie:

result = yield SomeProtocol.onMessage.called()
# okay, we got here, we know onMessage was called,
# now we might want to tick a clock, or otherwise simulate
# async state manipulation.
# now I want to make sure the deferred chain on the onMessage result 
has been completed:

yield result


I'm not sure I understand your example here. 


Yeah, this is part of carly, that I posted earlier. It stems from the 
need to get the results of method calls when you have no reference to 
the object being calls, or sometimes a result that's a deferred you need 
to wait on, particularly in a test, but have no way of doing so.


If you're feeling brave, have a read of:
https://github.com/cjw296/carly/blob/master/carly/hook.py

The assertion in question only happens if you call returnValue or do a 
return with a Deferred directly; this example doesn't do either of 
those things.


This is the test situation where I hit this issue:
https://github.com/cjw296/carly/blob/master/tests/test_untracked_deferred.py#L28-L35

I'd originally wanted to have that read:

@inlineCallbacks
def test1(self):
...
result = yield pita.asyncMethod.called()
with ShouldRaise(Exception(1)):
yield result

Now, which I'm actually happier with the end result here, I think the 
above it legit, if unusual, and that assert trips it up.


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] why can't a callback be called with a deferred?

[Chris Withers]

On 19/02/2019 11:41, Adi Roiban wrote:


I think it was introduced to catch some common bad usage patterns ...
like yours :)


Not a massively helpful comment.


If you want to chain the deferreds, use the dedicated helper
https://twistedmatrix.com/documents/current/core/howto/defer.html#chaining-deferreds

Deferred are not always 100% resolved/called.
You might have a deferred called, but the current result might be
another deferred... so it has no final result yet.



so in your case, instead of `returnValue(result)` use

result = yield result
returnValue(result)


in this way, the result is resolved :)


The methods being hooked don't necessarily return deferreds.
I'd like it to be an explicit choice of the caller, ie:

result = yield SomeProtocol.onMessage.called()
# okay, we got here, we know onMessage was called,
# now we might want to tick a clock, or otherwise simulate
# async state manipulation.
# now I want to make sure the deferred chain on the onMessage result has 
been completed:

yield result

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] why can't a callback be called with a deferred?

[Chris Withers]

Hi All,

There's this assert:

https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/defer.py#L459

...and I'd like to understand why it's there.

I have what feels like a legitimate case where that assert trips me up, 
as I'd like to change this line:


https://github.com/cjw296/carly/blob/d39e316aa0f9e05613c263410e7b3ba5bcacba3a/carly/hook.py#L67

...to just return result.value. However, if the method being hooked 
returns a deferred, then I trip the assert.
Unfortunately, the commit that introduced that assert is just a one 
liner from Itamar back in 2003 without any context of what problem lead 
him to introduce the change.


Of course, I can work around it, was more curious than anything...

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] code coverage when code executed in a twisted worker thread

[Chris Withers]

On 01/12/2018 16:09, Kyle Altendorf wrote:



On November 29, 2018 8:10:41 AM EST, Chris Withers  wrote:

I've noticed that code coverage doesn't appear to be recorded when code

is executed in something that has been deferToThread'ed.

Is this a known issue? Does anyone have an explanation?


Are you using coverage.py?  I don't know offhand but does Twisted use `thread` 
or `threading`?

https://coverage.readthedocs.io/en/coverage-4.2/trouble.html#things-that-don-t-work


You have to be using one of those to use threads in Python. Does Twisted 
use the older/lower level `thread` library?


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] code coverage when code executed in a twisted worker thread

[Chris Withers]

Hi All,

I've noticed that code coverage doesn't appear to be recorded when code 
is executed in something that has been deferToThread'ed.


Is this a known issue? Does anyone have an explanation?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] leakage between tests with trial when code under test uses deferToThread

[Chris Withers]

Forgot to include the list in my reply...

On 23/11/2018 22:27, Chris Withers wrote:

On 23/11/2018 22:22, Glyph wrote:




On Nov 23, 2018, at 7:58 AM, Chris Withers  wrote:

Hi All,

Does trial do anything to clean up stuff that's been passed to 
deferToThread?


Nope.  It does its best to clean up stuff that it knows is "in" the 
reactor (sockets, timers, and the like), but since threads can kind 
of do ~whatever~ there hasn't been support for that.


Actually, looks like there's stuff in trial's Janitor class, but it 
only kicks in after the suite is finished, and I'm seeing leakage 
between tests within a suite...


deferToThread is a bit of a special case and you make a good point 
here: there should probably be special support for it in trial.


...which I plan to add in carly in the meantime: basically block with 
a timeout on everything in the threadpool finishing it's work.


Speaking of which, I'm happy with how carly is turning out, but would 
still welcome feedback, particularly on how the tests suites feel:


https://github.com/cjw296/carly/tree/master/tests

Just don't read hook.py unless you like head-bendy code ;-)


Okay, so here's what I came up with 
<https://github.com/cjw296/carly/commit/71a1d2bfd501f5c561712c75253fc23c28c3bba7#diff-3670fb1f3b913f6c1ec584fa64302c2dR17>. 
I'm not a fan of the sleep / busy loop pattern, but twisted makes it 
prettymuch impossible to get hold of the ThreadWorkers queue, so that I 
could block on them in this method. Any ideas on improvements would be 
very welcome!


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] leakage between tests with trial when code under test uses deferToThread

[Chris Withers]

Hi All,

Does trial do anything to clean up stuff that's been passed to 
deferToThread?


I'm seeing what looks like leakage between tests where stuff that is 
deferred to a thread from a LoopingCall is resulting in a DelayedCall 
ending up in the reactor for the next test.

Does that ring any bells?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Thread Consumption Problem in Daemon?

[Chris Withers]

On 22/11/2018 02:30, Glyph wrote:


On Nov 19, 2018, at 6:16 AM, Darren Govoni <mailto:darrenus...@gmail.com>> wrote:


I tried to find out if there is a way to limit the thread pool size 
from command line for twisted web and found nothing. Does it exist?


The thread pool is limited to 10. While this is configurable via the 
API, no command line option is exposed to tune it.  (This would be a 
great contribution if you were so inclined!)


It seems likely to me that Flask is spawning background threads for some 
reason; given the way Twisted's threadpool works, leaks like this are 
not common.  However, anything is possible: you probably want to gather 
some information about what all those threads are doing.


Some ideas on this front:

- pstree/ps and strace will tell you at a low level

- http://pyrasite.com/ and then use Python's thread introspection stuff.

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] carly: a tool for testing twisted protocols with real networking

[Chris Withers]

Hi All,

Okay, so, I've started working jml's "really disconnecting" blog post 
into a more generic set of tooling for testing protocols talking to each 
other over real networking like this:


https://github.com/cjw296/carly/blob/master/tests/test_line_receiver_server.py

Very interested in peoples thoughts, and especially if you can spot any 
bugs in these two:


https://github.com/cjw296/carly/blob/master/carly/hook.py
https://github.com/cjw296/carly/blob/master/carly/context.py

Thanks for all the help so far, it's certainly helping me get my head 
around this all :-)


Chris

PS: Real networking is bad, yes, I know: slow, unreliable, etc ;-) 
However, in most of the testing I do, I end up having to hit a real 
database server to check my code works with it, so a few more tcp 
connections to localhost is not the end of the world!


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "disconnecting properly" in tests still hangs on macOS

[Chris Withers]

On 14/11/2018 13:41, exvito here wrote:


Chris,

I played with this for a bit and quickly reproduced the "server side disconnect 
never seems to happen" behaviour you described, on my system running macOS 10.12.6 
and Twisted 18.9.0 using the SelectReactor.

Suspecting of an eventual race condition between "server stop listen" and "server 
disconnect", I tried this variation of tearDown which seems to work reliably:

 @defer.inlineCallbacks
 def tearDown(self):
 self.clientConnection.disconnect()
 yield defer.gatherResults([self.clientDisconnected, 
self.serverDisconnected])
 yield defer.maybeDeferred(self.serverPort.stopListening)

Do I have any motive to suspect such race condition? No, but after ensuring 
"disconnect first, stop listening later", things work much better here.

Also tested the CFReactor and KQueueReactor: CFReactor seems to exhibit a 
similar behaviour (original code hangs every now and then on cleanup, my code 
works); KQueueReactor always hangs on cleanup with the original code, and works 
reliably with my code.


Thanks, this has worked flawlessly!

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] "disconnecting properly" in tests still hangs on macOS

[Chris Withers]

Right, so, I've been trying to get the technique in 
https://jml.io/pages/how-to-disconnect-in-twisted-really.html to work 
for me.


No hating please, most of my testing in the past has involved hitting a 
relational database, so there's already a TCP connection flying around, 
one more won't make any difference.


jml's example, exactly as-is on that page, hangs around 30-40% of the 
time when running on my macOS laptop. From changing the teardown to look 
like this:


    def tearDown(self):
    ds = defer.maybeDeferred(self.serverPort.stopListening)
    dc = defer.maybeDeferred(self.clientConnection.disconnect)
    print()

    ds.addCallback(lambda _: print('serverPort.stopListening'))
    dc.addCallback(lambda _: print('self.clientConnection.disconnect'))
    self.clientDisconnected.addCallback(lambda _: 
print('self.clientDisconnected'))
    self.serverDisconnected.addCallback(lambda _: 
print('self.serverDisconnected'))
    self.serverDisconnected.addErrback(lambda _: 
print('self.serverDisconnected:', _))
    return defer.gatherResults([ds, dc, self.clientDisconnected, 
self.serverDisconnected])


...it appears that it's the serverDisconnected deferred that's failing 
to fire. I can't reproduce this on Linux as of yet, so I'm guessing this 
is a difference between the SelectReactor used on macOS and the 
EPollReactor used on Linux.


What's the best way to go about debugging a non-firing deferred like this?

Anyone know what this might be?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] pytest-twisted questions

[Chris Withers]

On 13/11/2018 20:31, Kyle Altendorf wrote:
I would just open an issue on GitHub.  Feel free to copy this in as the 
first message to keep the initial context.


Well, there's no specific issue to log here, so let's keep on going with 
email for now :-)


(in fact, copying in the twisted mailing list, as that's probably the 
right place for this)



On 2018-11-13 12:35, Chris Withers wrote:

- What's the intention of the plugin? Should all tests still subclass
twisted.trial.unittest.TestCase or should they *never* do so if using
this plugin?


I don't know what _should_ be done, 


Victor, what was your intention with the project?

but I know that I mostly don't have 
test classes and, for the class I do have, I didn't inherit.  Mostly I 
just @pytest.inlineCallbacks (I still don't like the namespace squashing 
into pytest though :] ) and I suppose in the probably-not-too-distant 
future I'll instead be using more @pytest_twisted.async_await  (ala 
#31/#34).


Okay, but twisted.trial.unittest.TestCase does a bunch of reactor 
management stuff, most notable making you aware when you've left the 
reactor in a bad state. As far as I can see from the code, 
pytest-twisted does not do that, correct?



- What's with the greenlet mentions? Is this plugin okay to use when
I'm just using a normal Twisted epoll reactor?


https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L4
https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L36-L46
https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L68-L83
https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L123-L126

What does greenlet have to do with twisted in this context? Would be 
great to have it as an optional thing, unless I'm missing something?



- What does pytest provide *without* this plugin, when it comes to
testing with Twisted?


I am not aware of anything twisted/pytest specific that is outside 
pytest-twisted, 


https://docs.pytest.org/en/latest/faq.html#how-does-pytest-relate-to-twisted-s-trial

So, my guess is that twisted.trial.unittest.TestCase subclasses 
unittest.TestCase and so pytest treats it in the same way. That means 
you get the management and checking of the reactor, along with the handy 
methods it provides, when you put your tests in class-based suites that 
subclass twisted.trial.unittest.TestCase.


Ronny, does pytest do anything else that's twisted-specific?

If you are just trying to get started with something that works, I'd 
skip the classes and inheritance and just let the reactor take care of 
itself. 


My experience with Twisted over the last 10 years or so is that this is 
an exceedingly dangerous approach to take...


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] probably stupid question about automated testing and reactor.run

[Chris Withers]

Hi All,

Sorry, the silly questions about testing keep coming...

Is it legit for a unit testable piece of an application's code to call 
reactor.run()?


How do you feed a reactor.stop() into such a test? Would that be a sane 
thing to try and do?


How do people deal with this kind of testing?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "blocking" on a response to a client

[Chris Withers]

On 06/11/2018 13:47, Jean-Paul Calderone wrote:

Thanks, but the special bit I was asking about is having my
application's logical flow pause until the websocket is successfully up
and a successful response has been received, all without blocking the
reactor.

How do I do that?


Put the rest of your code in a callback on a Deferred that fires when 
the event you're interested in happens.  When using inlineCallbacks, 
that's like:


     yield the_event_youre_interested_in
     # ... more code

At least, this is the extremely-localized solution to the problem.  
There are many possible ways to structure such a thing.  It's easy to 
describe this one because you said you're using inlineCallbacks and 
didn't say anything else about how your code is organized.


Right, but I'm still too stupid to join the dots :-(

More concretely, I want to spin up an autobahn websocket client, have it 
connect, send a login message, confirm that worked, send a 'create' 
message, confirm that worked, and *then* let the rest of my code progress...


I'm surprised there don't appear to be any example of this floating 
around, and disappointed at the lack of examples of good tests for these 
things.


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "blocking" on a response to a client

[Chris Withers]

On 05/11/2018 18:32, meejah wrote:

Chris Withers  writes:


So, separate question: I have a client that's making a connection to a
websocket from a server. The client needs to send an "auth" message
once the websocket is up, and the client code shouldn't progress until
it receives an affirmative response. How do I do this in a
non-blocking way when using twisted? (inside an inlineDeferred method
in this case!)


There's nothing really special about an @inlineCallbacks method -- it
just returns a Deferred so you can handle it like any other
Deferred-returning method.


Thanks, but the special bit I was asking about is having my 
application's logical flow pause until the websocket is successfully up 
and a successful response has been received, all without blocking the 
reactor.


How do I do that?

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] automated tests for a server application

[Chris Withers]

On 06/11/2018 05:43, Moshe Zadka wrote:


Some of the best advice depends on details of the application. One 
trick that is sometimes useful is passing in a "fake" reactor object. 
This, of course, is only useful if the application is structured in a 
way that functions/classes expect to get a reactor, instead of 
reaching for the global one. However, usually *that's* not a 
complicated refactoring to do.


Cool, do you have any example tests that do this?
Interesting, looks like pytest-twisted does away for the need for this 
by showing how to install a fake reactor globally:

https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L129-L142

You can look at 
https://twistedmatrix.com/documents/current/api/twisted.test.proto_helpers.MemoryReactor.html for 
inspiration although, as all code under `twisted.test`, it is not 
intended as end-user functionality, so using it directly is problematic. 


Not sure I fully understand this, why is the MemoryReactor bad to use? 
Where is it used?


It would be nice to have externally-useful reactors along the lines of 
Clock 
(https://twistedmatrix.com/documents/current/api/twisted.internet.task.Clock.html ) 
but, unfortunately, we do not have that yet.


So, clock is just a clock, right? How would that get worked into a 
"real" reactor for testing?


You can, of course, use a real reactor and a real client to pump data. 
However, in that case, you probably do want to switch to trial so that 
you can return a deferred from a test function and have the reactor 
run until the deferred fires. This is not great, but can work in a pinch.


pytest-twisted looks like it supports this pattern too, allowing test 
functions to return deferreds...


I guess I'm still really looking to see examples of all of this...

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] "blocking" on a response to a client

[Chris Withers]

Hi Again,

So, separate question: I have a client that's making a connection to a 
websocket from a server. The client needs to send an "auth" message once 
the websocket is up, and the client code shouldn't progress until it 
receives an affirmative response. How do I do this in a non-blocking way 
when using twisted? (inside an inlineDeferred method in this case!)


Similarly, how do I get this dance to happen again if the websocket 
drops? How would I do some kind of backoff on he connection retries?


And, icing on the cake, where can I find good examples of how to write 
automated tests for all of the above?


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] automated tests for a server application

[Chris Withers]

Hi All,

(copying in Moshe as he expressed an interest!)

I've inherited a twisted app for which I'd like to add some changes but 
want to improve the automated test coverage before I do.


The app itself listens on a web socket (autobahn), an rpc port (RPyC) 
and also connects to a mysql database.
So, for the automated tests, I want to check what happens when I send 
various messages to the websocket or rpc port, and then check what ends 
up in the db and what other state changes happen in the app itself.


What's the best/correct way to write automated tests for this kind of 
situation? I'm using pytest and would happily use pytest-twisted too. 
Not keen to use trial...


Any help is good help, pointers to nice existing examples would be 
fantastic!


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Forcing Twisted / Klein to use HTTPS

[Chris Norman]

Hi all,

I'm trying to force Klein to use HTTPS, and so far I have a custom error 
handler which redirects the user to the HTTPs version of the page via a 
check_secure function which takes the request object and raises the 
right error if request.isSecure() is False.



Is there a better global way to enforce HTTPS with any part of Klein or 
Twisted?



Cheers,


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Beginner

[Chris Norman]

On 12/04/2017 14:41, Jean-Paul Calderone wrote:
On Wed, Apr 12, 2017 at 9:37 AM, Chris Norman 
<chris.norm...@googlemail.com <mailto:chris.norm...@googlemail.com>> 
wrote:




On 12/04/2017 14:21, catch me wrote:

Hey! I don't know I should ask this question or not but as a
beginner I have to ask.
I have just started learn python network programming and
fortunately found such an amazing Twisted Framework.


Welcome! Glad you found Twisted, it really is awesome!



As a beginner I could not be able to decide where to start I mean
what should I make first using twisted framework. I would be very
thankful to you for guiding me.


I personally like telnet-like servers. They are very easy to make
with the LineReceiver protocol (hint hint), and you can do a lot
with them.


If you want /telnet/ then you should probably use 
/twisted.conch.telnet/.  It's a little bit more complicated than 
/LineReceiver/ but it will actually speak telnet for you.  
LineReceiver will get you some simple line-oriented interactions, though.




Ah, than you. I could never figure out how to make the proper 
TelnetProtocol to do anything useful, and line-based is all I've ever 
really needed.


Thank you for the tip though.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Beginner

[Chris Norman]

On 12/04/2017 14:21, catch me wrote:
Hey! I don't know I should ask this question or not but as a beginner 
I have to ask.
I have just started learn python network programming and fortunately 
found such an amazing Twisted Framework.


Welcome! Glad you found Twisted, it really is awesome!


As a beginner I could not be able to decide where to start I mean what 
should I make first using twisted framework. I would be very thankful 
to you for guiding me.


I personally like telnet-like servers. They are very easy to make with 
the LineReceiver protocol (hint hint), and you can do a lot with them.


I guess the most basic one is some kind of echo server / client that 
simply echos back whatever you type.


For more help, don't forget the Twisted documentation and the examples 
therein on http://www.twistedmatrix.com



Pardon me for bad english!


Not to worry at all, everyone is welcome.



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] in case there's downtime

[Chris Norman]

Hi,

What needs taking over on the mailing list side? I'd love to help out if 
possible.



Cheers,


On 25/03/2017 04:14, Glyph Lefkowitz wrote:
We've been very conservative about upgrading our operating system on 
twistedmatrix.com , since we don't want to 
disrupt anything.  Since our OS is one release too old to support 
Docker, that unfortunately means we haven't been able to migrate 
incrementally to containers, which creates a bit of a chicken-and-egg 
problem.


However, as we are days away from the end-of-life cliff for the 
operating system as a whole, and as I discovered a few minutes ago, 
dozens of packages are already unsupported and not receiving security 
updates, I'm going to upgrade the host OS manually, even if it results 
in some disruption.  As with the Twisted compatibility policy, 
security is a good enough reason to break stuff.


Maybe we'll even get a new version of Mailman that will let me turn 
automatic subscriptions back on while the star-crossed 
https://lists.twistedmatrix.com project waits for someone else to help 
out :-).


-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with Deferreds

[Chris Norman]

Hi,
Yes, that sorted the problem out no worries.

Thank you so much.



On 02/04/2016 15:00, Kevin Conway wrote:

Hi Chris,

tl;dr: Returning a value from 'dataReceived', or any of its extensions 
such as 'lineReceived' in the 'LineReceiver' Protocol subclass, 
triggers a disconnect and uses the returned value as the 'reason'. A 
'reason' must be an Exception or t.p.Failure object as other values 
will trigger this error.


Are you quite certain that your last line is not getting printed? I'm 
not sure exactly where this feature is documented, but returning any 
non-None value from a Protocol's 'dataReceived' method can result in 
this behaviour. The t.protocols.basic.LineReceiver calls 
'lineReceived' from 'dataReceived' and returns any value it gets from 
your implementation. The value returned from 'dataReceived' is passed 
along to the transport's 'doRead' which, again, returns it to the 
portion of the reactor handling selectables. The reactor assumes that 
anything returned from a transport during a read or write operation is 
a bad thing and disconnects the transport. During the disconnect 
process the reactor is generating a t.p.failure.Failure object and 
passing in your returned value as the 'why' which is expected to be an 
Exception or Failure and not a Deferred. Try returning None instead of 
your Deferred. That should resolve this particular issue.


On Sat, Apr 2, 2016 at 4:39 AM Chris Norman 
<chris.norm...@googlemail.com <mailto:chris.norm...@googlemail.com>> 
wrote:


Hi all,
I recently got over myself and forced myself to read about Deferreds
rather than using threading opperations.

I've used them successfully in a couple of places, but this one has me
flummoxed:

Here's the code for the Deferred and it's sub-commands:

  def do_command(self, cmd, **kwargs):
   """Process a command in true Deferred style."""
   if cmd.permissions(self.connection):
cmd(self.connection, **kwargs)
   else:
logger.warning('Blocked from running command %s which is secured
with %s.', cmd.__name__, cmd.permissions.__name__)
raise CommandError('You have insufficient privileges to
perform this
action. The staff have been notified.')

  def handle_error(self, err):
   """Handle an error from do_command."""
   if isinstance(err, CommandError):
return self.send_error(e.message, disconnect = e.disconnect)
   else:
self.log(e, level = 'exception')
self.send_error('Sorry, but a problem with the server means your
command was not executed. The staff have been notified.')

  def lineReceived(self, line):
   """Parse an incoming command."""
   global lines
   lines += 1 # Increment the line count.
   data = line.decode(settings.ENCODING)
   try:
command, kwargs = json.loads(data)
if not isinstance(command, string_types) or not isinstance(kwargs,
dict):
 raise TypeError('Expecting [str, dict]. Got [%s, %s] instead.' %
(type(command), type(kwargs)))
   except (TypeError, ValueError) as e:
self.log('Invalid command string: %s', data, level = 'error')
self.log(e, level = 'exception')
return self.send_error('Invalid command.', disconnect = True)
   cmd = commands.commands.get(command, None)
   if cmd  is None:
self.log('Unrecognised command: %s.', command, level = 'warning')
   elif self.connection.player or not cmd.login_required:
d = defer.Deferred()
print('Adding callback.')
d.addCallback(self.do_command, **kwargs)
print('Adding errback.')
d.addErrback(self.handle_error)
print('Calling callback.')
d.callback(cmd)
print('Called.') # Never gets this far.
return d
   else:
return self.send_error('Not authenticated.', disconnect = True)

Here's the traceback I get when the callback gets called:

Unhandled Error
Traceback (most recent call last):
   File "server/functions.py", line 88, in server_start
 reactor.run()
   File

"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1194, in run
 self.mainLoop()
   File

"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1206, in mainLoop
 self.doIteration(t)
   File

"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/epollreactor.py",
line 396, in doPoll
 log.callWithLogger(selectable, _drdw, selectable, fd, event)
---  ---
   File

"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py",
line 101, in callWithLogger
 return callWithContext({"system":

[Twisted-Python] Problem with Deferreds

[Chris Norman]

Hi all,
I recently got over myself and forced myself to read about Deferreds 
rather than using threading opperations.


I've used them successfully in a couple of places, but this one has me 
flummoxed:


Here's the code for the Deferred and it's sub-commands:

 def do_command(self, cmd, **kwargs):
  """Process a command in true Deferred style."""
  if cmd.permissions(self.connection):
   cmd(self.connection, **kwargs)
  else:
   logger.warning('Blocked from running command %s which is secured 
with %s.', cmd.__name__, cmd.permissions.__name__)
   raise CommandError('You have insufficient privileges to perform this 
action. The staff have been notified.')


 def handle_error(self, err):
  """Handle an error from do_command."""
  if isinstance(err, CommandError):
   return self.send_error(e.message, disconnect = e.disconnect)
  else:
   self.log(e, level = 'exception')
   self.send_error('Sorry, but a problem with the server means your 
command was not executed. The staff have been notified.')


 def lineReceived(self, line):
  """Parse an incoming command."""
  global lines
  lines += 1 # Increment the line count.
  data = line.decode(settings.ENCODING)
  try:
   command, kwargs = json.loads(data)
   if not isinstance(command, string_types) or not isinstance(kwargs, 
dict):
raise TypeError('Expecting [str, dict]. Got [%s, %s] instead.' % 
(type(command), type(kwargs)))

  except (TypeError, ValueError) as e:
   self.log('Invalid command string: %s', data, level = 'error')
   self.log(e, level = 'exception')
   return self.send_error('Invalid command.', disconnect = True)
  cmd = commands.commands.get(command, None)
  if cmd  is None:
   self.log('Unrecognised command: %s.', command, level = 'warning')
  elif self.connection.player or not cmd.login_required:
   d = defer.Deferred()
   print('Adding callback.')
   d.addCallback(self.do_command, **kwargs)
   print('Adding errback.')
   d.addErrback(self.handle_error)
   print('Calling callback.')
   d.callback(cmd)
   print('Called.') # Never gets this far.
   return d
  else:
   return self.send_error('Not authenticated.', disconnect = True)

Here's the traceback I get when the callback gets called:

Unhandled Error
Traceback (most recent call last):
  File "server/functions.py", line 88, in server_start
reactor.run()
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py", 
line 1194, in run

self.mainLoop()
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py", 
line 1206, in mainLoop

self.doIteration(t)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/epollreactor.py", 
line 396, in doPoll

log.callWithLogger(selectable, _drdw, selectable, fd, event)
---  ---
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py", 
line 101, in callWithLogger

return callWithContext({"system": lp}, func, *args, **kw)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py", 
line 84, in callWithContext

return context.call({ILogContext: newCtx}, func, *args, **kw)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py", 
line 118, in callWithContext

return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py", 
line 81, in callWithContext

return func(*args,**kw)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py", 
line 610, in _doReadOrWrite

self._disconnectSelectable(selectable, why, inRead)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py", 
line 258, in _disconnectSelectable

selectable.connectionLost(failure.Failure(why))
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/failure.py", 
line 232, in __init__

tb = self.value.__traceback__
builtins.AttributeError: 'Deferred' object has no attribute '__traceback__'

Anyone have any ideas?

Cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] 16.0.0 32-bit wheels for Windows

[Chris Norman]

Hi,

On 30/03/2016 19:20, anatoly techtonik wrote:

Hi,

I am having the same problem as this guy:
https://stackoverflow.com/questions/36279141/pip-doesnt-install-twisted-on-windows
Absence of 32-bit wheels breaks Python
projects that added Twisted>10 as a
dependency, for example.

Is that intentional change?
https://pypi.python.org/pypi/Twisted/16.0.0

No clue about any change, but which Python version are you using?

I have happily `pip install`ed twisted on Windows 10 (64 bit) with 32 
bit Python 3.5.1, and 2.7.11.


Only place where I had trouble was Debian 64 bit where a simple python 
setup.py installed did the trick.


Sorry you're having trouble.



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Sending disconnect messages to clients

[Chris Norman]

OK... ignore me.

I made your test program and it worked fine. So I tried connecting to my 
server with telnet and saw the expected message... So I checked the 
error function which should have been called on the client with the 
particular message to see it was as good as pass...


Sorry for all the messages, everything works fine, panic over!

Thank you for your help with this!


On 22/03/2016 20:37, L. Daniel Burr wrote:

Hi Chris,

On March 22, 2016 at 3:09:08 PM, Chris Norman 
(chris.norm...@googlemail.com <mailto:chris.norm...@googlemail.com>) 
wrote:



Hi Daniel,



[SNIP]

Actually it's been added to the callback. I think my mail client 
wrapped the lines, but issue self.transport.LoseConnection() is the 
comment, so self.deferred_disconnect is added to the callback chain.


Sorry for the confusion.

Anything else which can cause this?



Without seeing the rest of your code, I can’t suggest anything 
concrete.  You say the client never “sees” the message, but your call 
to loseConnection() still occurs, so you’re not encountering an 
exception during the callback chain.  Can you come up with an minimal 
example the demonstrates the problem?


Daniel
—
L. Daniel Burr
ldanielb...@me.com <mailto:ldanielb...@me.com>
(312) 656-8387



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Sending disconnect messages to clients

[Chris Norman]

Hi Daniel,

On 22/03/2016 19:49, L. Daniel Burr wrote:

Hi Chris,

On March 22, 2016 at 2:42:14 PM, Chris Norman 
(chris.norm...@googlemail.com <mailto:chris.norm...@googlemail.com>) 
wrote:



Hi all,
I'm sure I asked this question before, but I can't find any answers on
google, and I've changed my work flow a bit now, so thought it was worth
re-asking.

So on my server's protocol, I have a send method. This allows me to pass
arguments which get sent off to the client in the right format. The send
method looks something like this:

def send(self, command, **kwargs):
"""Cause the client to issue getattr(connection, command)(**kwargs).
If disconnect evaluates to True, disconnect the client after the message
is sent."""
disconnect = kwargs.get('disconnect', False)
try:
del kwargs['disconnect']
except KeyError:
pass # Argument not found.
d = defer.Deferred()
d.addCallback(self.prepare_command) Convert the command and kwargs to
json.
d.addCallback(self.deferred_write) # Write the json to the transport.
if disconnect:
d.addCallback(self.deferred_disconnect) # Issue
self.transport.loseConnection().


You are disconnecting right here, without waiting for your Deferred to 
fire.  It might make better sense to add the call the 
self.transport.loseConnection() to your Deferred’s callback chain.


Hope this helps,



Actually it's been added to the callback. I think my mail client wrapped 
the lines, but issue self.transport.LoseConnection() is the comment, so 
self.deferred_disconnect is added to the callback chain.


Sorry for the confusion.

Anything else which can cause this?

Cheers,


Daniel
--
L. Daniel Burr
ldanielb...@me.com <mailto:ldanielb...@me.com>
(312) 656-8387


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Sending disconnect messages to clients

[Chris Norman]

Hi all,
I'm sure I asked this question before, but I can't find any answers on 
google, and I've changed my work flow a bit now, so thought it was worth 
re-asking.


So on my server's protocol, I have a send method. This allows me to pass 
arguments which get sent off to the client in the right format. The send 
method looks something like this:


 def send(self, command, **kwargs):
  """Cause the client to issue getattr(connection, command)(**kwargs). 
If disconnect evaluates to True, disconnect the client after the message 
is sent."""

  disconnect = kwargs.get('disconnect', False)
  try:
   del kwargs['disconnect']
  except KeyError:
   pass # Argument not found.
  d = defer.Deferred()
  d.addCallback(self.prepare_command) Convert the command and kwargs to 
json.

  d.addCallback(self.deferred_write) # Write the json to the transport.
  if disconnect:
   d.addCallback(self.deferred_disconnect) # Issue 
self.transport.loseConnection().
  reactor.callFromThread(d.callback, [command, kwargs]) # Call the 
deferred's callback chain.

  return d # Return the deferred.

If I try something like:
protocol.send('alert', message = '*** Disconnected. ***', disconnect = True)
the client gets disconnected, but never sees the "*** Disconnected. ***" 
message.


I guess I could do a reactor.callLater and just wait for the transport 
to get the message, but that seems sloppy, and I can't help thinking 
there must be something I'm missing.


Any ideas welcome!

Cheers all,


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted serialport

[Chris West]

Adi,

Thanks for the reply, I've applied the patch and that has sorted out my problem.

Many thanks,
Chris

Date: Wed, 9 Mar 2016 09:59:21 +
From: Adi Roiban <a...@roiban.ro>
To: Twisted general discussion <twisted-python@twistedmatrix.com>
Subject: Re: [Twisted-Python] Twisted serialport
Message-ID:

[Twisted-Python] Twisted serialport

[Chris West]

I'm currently using the SerialPort connection type in twisted 15.5.0 with 
python 2.7.7 on a windows machine.

Twisted serial communications with the serial port fail when pyserial is 
upgraded from 2.7 to 3.0.1 and fails with the following error.

Traceback (most recent call last):
  File "C:\Anaconda\Scripts\fmv3dbg-script.py", line 9, in 
load_entry_point('fmv3tools==0.2.5', 'console_scripts', 'fmv3dbg')()
  File 
"C:\Anaconda\lib\site-packages\fmv3tools-0.2.5-py2.7.egg\fmv3tools\fmv3dbg.py", 
line 338, in main
server = create_system(args)
  File 
"C:\Anaconda\lib\site-packages\fmv3tools-0.2.5-py2.7.egg\fmv3tools\fmv3dbg.py", 
line 310, in create_system
server = SerialServer(com_port, outputs=outputs)
  File 
"C:\Anaconda\lib\site-packages\fmv3tools-0.2.5-py2.7.egg\fmv3tools\fmv3dbg.py", 
line 57, in __init__
FMv3SerialGateway.__init__(self, com_port, *args, **kwargs)
  File 
"C:\Anaconda\lib\site-packages\flatmesh-0.2.8-py2.7.egg\flatmesh\fmv3_datacoms.py",
 line 403, in __init
__
baudrate=baudrate)
  File 
"C:\Anaconda\lib\site-packages\twisted-15.5.0-py2.7-win32.egg\twisted\internet\_win32serialport.py",
 line 56, in __init__
self._finishPortSetup()
  File 
"C:\Anaconda\lib\site-packages\twisted-15.5.0-py2.7-win32.egg\twisted\internet\_win32serialport.py",
 line 65, in _finishPortSetup
flags, comstat = win32file.ClearCommError(self._serial.hComPort)
AttributeError: 'Serial' object has no attribute 'hComPort'

I've removed pyserial 3.0.1 from my system for the moment but I was wondering 
if there is a way to have both pyserial 2.7 and 3.0.1 on my system when using 
twisted?

Thanks,
Chris
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Waiting for transports to close

[Chris Norman]

Hello,

On 17/12/2015 13:03, Glyph Lefkowitz wrote:


On Dec 17, 2015, at 4:56 AM, Chris Norman 
<chris.norm...@googlemail.com <mailto:chris.norm...@googlemail.com>> 
wrote:


Hi,
It's a MUD server, so players type in commands and receive textual 
responses.


One of the admin commands is the ability to shutdown the server (or 
CTRL-C might be pressed on the console). I'd like this action to 
notify all connected transports that the server is going down for 
shutdown, so they're not rudely disconnected, then once the 
notifications have all gone through, then the server is free to shutdown.


Gotcha.  So you don't need to necessarily wait for all the messages to 
be delivered if there are slow clients waiting around; you just want 
to send everyone a farewell message and if they haven't responded 
within a reasonable timeout, go ahead and shut down anyway.


If your MUD server is already a Service 
<https://twistedmatrix.com/documents/15.5.0/api/twisted.application.service.IService.html> 
being launched by twistd, you just need to add a stopService 
<https://twistedmatrix.com/documents/15.5.0/api/twisted.application.service.IService.html#stopService> 
method that returns a Deferred.  When CTRL-C is hit (or anything else 
causes reactor.stop to be called), it will call this stopService 
method, and won't exit until a Deferred fires.


In your case, a simple deferLater 
<https://twistedmatrix.com/documents/15.5.0/api/twisted.internet.task.html#deferLater> 
will probably do the trick.  You can also speed things up when there 
are no connected clients left by cancelling that Deferred to make it 
finish firing immediately.


Will that work for you?


I hope all this makes sense.


It's not a service no... Should it be? I wasn't planning to use twistd, 
mainly because I don't know how to, and running

python main.py
is working fine, accepting command line arguments - the works.

It could be converted though, if there is an advantage with services?

Also, I've read quite a lot about Deferreds. I thought initially they 
were for multithreading your application, but I realise that's wrong, so 
I don't understand what the point in them is?


This isn't to say there isn't one mind you, I think I'm just majorly 
missing the point.




P.S.: For future reference, on this list the preferred style of reply 
is interleaved https://en.wikipedia.org 
<https://en.wikipedia.org/wiki/Posting_style#Interleaved_style>


Sorry, I'll do that in the future.

/wiki/Posting_style#Interleaved_style 
<https://en.wikipedia.org/wiki/Posting_style#Interleaved_style> or 
bottom-posting: https://en.wikipedia.org/wiki/Posting_style#Bottom-posting


-glyph



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Waiting for transports to close

[Chris Norman]

Hi,
It's a MUD server, so players type in commands and receive textual 
responses.


One of the admin commands is the ability to shutdown the server (or 
CTRL-C might be pressed on the console). I'd like this action to notify 
all connected transports that the server is going down for shutdown, so 
they're not rudely disconnected, then once the notifications have all 
gone through, then the server is free to shutdown.


I hope all this makes sense.

Cheers,

On 12/17/2015 11:49 AM, Glyph Lefkowitz wrote:


On Dec 16, 2015, at 9:25 AM, Chris Norman 
<chris.norm...@googlemail.com <mailto:chris.norm...@googlemail.com>> 
wrote:


Hi all,
I'm writing a MUD server, and I want a way for transports to be 
notified ofa shutdown before being disconnected, and the reactor 
being stopped.


I've tried:

for t in transports:
t.write('Shutting down.\r\n')
t.loseConnection()
reactor.stop()

This doesn't seem to notify the transports.

I also tried:
for t in transports:
t.write('Shutting down.\r\n')
t.loseConnection()
while t.connected:
 pass
reactor.stop()

That just blocked and did nothing, presumably something do with my 
while loop.


Is there a stopWhenEmpty function on the somewhere? I did look over 
the methods, and I couldn't find anything promising.


I'm just using the standard from twisted.internet import reactor 
reactor, so no special cases here. In case it matters the transports 
I'm using are twisted.protocols.basic.LineReceiver, and everything 
else works with them.


Cheers in advance for the help.


This is definitely doable, but before I explain it would help to know 
/why/ you want to do this.


The reason I ask is: servers crash; hardware fails.  The falcon cannot 
hear the falconer; things fall apart; the centre cannot hold.


When those servers /do/ crash (and they will), you don't get a clean 
notification of disconnects.  So if you're writing your application to 
rely very heavily on the ability to do a clean shutdown and get 
notifications of every disconnect at the time you expect 
reactor.stop() to be running, you are probably designing your system 
in a way that will be very fragile and prone to data loss.  I know, I 
have made this mistake more than once myself :).


So, before you continue: do you actually need to do this?  Could you 
just ignore the notification of the connection drop and exit 
gracefully, perhaps properly cleaning up whatever state is left over 
at next startup?  If you really need this, understanding why you need 
it would also help in determining which implementation technique to 
suggest (there are a few).


-glyph



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Waiting for transports to close

[Chris Norman]

Hi all,
I'm writing a MUD server, and I want a way for transports to be notified 
ofa shutdown before being disconnected, and the reactor being stopped.


I've tried:

for t in transports:
 t.write('Shutting down.\r\n')
 t.loseConnection()
reactor.stop()

This doesn't seem to notify the transports.

I also tried:
for t in transports:
 t.write('Shutting down.\r\n')
 t.loseConnection()
 while t.connected:
  pass
reactor.stop()

That just blocked and did nothing, presumably something do with my while 
loop.


Is there a stopWhenEmpty function on the somewhere? I did look over the 
methods, and I couldn't find anything promising.


I'm just using the standard from twisted.internet import reactor 
reactor, so no special cases here. In case it matters the transports I'm 
using are twisted.protocols.basic.LineReceiver, and everything else 
works with them.


Cheers in advance for the help.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Possible bug

[Chris Norman]

Hi Amber

On 31/10/2015 06:45, Amber "Hawkie" Brown wrote:

Hi Chris,

Twisted does not yet support Python 3 on Windows.


That's a shame. I've never contributed to an open source project before, 
but I'd be glad to try and help out. I feel I __may__ know enough Python 
now to certainly have a shot at it.


There's a ticket at https://twistedmatrix.com/trac/ticket/8025#ticket which 
makes all the tests pass on the platform; I just need to work on it some more. 
I expect Twisted 16.0 to have base support for Python 3 on Windows.


As above, if I can help, please tell me how!

Cheers,


- Amber


On 31 Oct 2015, at 14:41, Chris Norman <chris.norm...@googlemail.com> wrote:

Hi all,
Using windows 10 with Python 3.5, importing pretty much anything gives me the 
following error. Not sure if there's something wrong with my configuration or 
if it's a bug... I seem to remember something a while back, should I file a 
ticket?


Traceback (most recent call last):
  File "", line 1, in 
  File "C:\python35\lib\site-packages\twisted\internet\reactor.py", line 38, in 

from twisted.internet import default
  File "C:\python35\lib\site-packages\twisted\internet\default.py", line 56, in 

install = _getInstallFunction(platform)
  File "C:\python35\lib\site-packages\twisted\internet\default.py", line 50, in 
_getInstallFunction
from twisted.internet.selectreactor import install
  File "C:\python35\lib\site-packages\twisted\internet\selectreactor.py", line 18, in 

from twisted.internet import posixbase
  File "C:\python35\lib\site-packages\twisted\internet\posixbase.py", line 18, in 

from twisted.internet import error, udp, tcp
  File "C:\python35\lib\site-packages\twisted\internet\udp.py", line 53, in 

from twisted.internet import base, defer, address
  File "C:\python35\lib\site-packages\twisted\internet\base.py", line 23, in 

from twisted.internet import fdesc, main, error, abstract, defer, threads
  File "C:\python35\lib\site-packages\twisted\internet\defer.py", line 29, in 

from twisted.python import lockfile, failure
  File "C:\python35\lib\site-packages\twisted\python\lockfile.py", line 52, in 

_open = file
NameError: name 'file' is not defined

Cheers,


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Possible bug

[Chris Norman]

Hi all,
Using windows 10 with Python 3.5, importing pretty much anything gives 
me the following error. Not sure if there's something wrong with my 
configuration or if it's a bug... I seem to remember something a while 
back, should I file a ticket?



Traceback (most recent call last):
  File "", line 1, in 
  File "C:\python35\lib\site-packages\twisted\internet\reactor.py", 
line 38, in 

from twisted.internet import default
  File "C:\python35\lib\site-packages\twisted\internet\default.py", 
line 56, in 

install = _getInstallFunction(platform)
  File "C:\python35\lib\site-packages\twisted\internet\default.py", 
line 50, in _getInstallFunction

from twisted.internet.selectreactor import install
  File 
"C:\python35\lib\site-packages\twisted\internet\selectreactor.py", line 
18, in 

from twisted.internet import posixbase
  File "C:\python35\lib\site-packages\twisted\internet\posixbase.py", 
line 18, in 

from twisted.internet import error, udp, tcp
  File "C:\python35\lib\site-packages\twisted\internet\udp.py", line 
53, in 

from twisted.internet import base, defer, address
  File "C:\python35\lib\site-packages\twisted\internet\base.py", line 
23, in 
from twisted.internet import fdesc, main, error, abstract, defer, 
threads
  File "C:\python35\lib\site-packages\twisted\internet\defer.py", line 
29, in 

from twisted.python import lockfile, failure
  File "C:\python35\lib\site-packages\twisted\python\lockfile.py", line 
52, in 

_open = file
NameError: name 'file' is not defined

Cheers,


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Possible bug

[Chris Norman]

Hi,
I've used git extensively, and I can soon read up on how to contribute 
on git. So what's the next step? Is there a howto somewhere on the 
Twisted site?


Cheers,


On 31/10/2015 07:39, Glyph Lefkowitz wrote:


On Oct 30, 2015, at 11:49 PM, Chris Norman 
<chris.norm...@googlemail.com <mailto:chris.norm...@googlemail.com>> 
wrote:


Hi Amber

On 31/10/2015 06:45, Amber "Hawkie" Brown wrote:

Hi Chris,

Twisted does not yet support Python 3 on Windows.


That's a shame. I've never contributed to an open source project 
before, but I'd be glad to try and help out. I feel I __may__ know 
enough Python now to certainly have a shot at it.


If you feel like you might, you definitely do :-).  Often people feel 
like they need to be advanced rocket surgeons to work on Twisted, but 
in fact, our robust code-review process means that you don't have to 
worry about making mistakes at all.  Just submit your best effort, and 
you'll get a code review telling you how to improve it so that it can 
be integrated (and you _will_ get at least one round of review: I 
created Twisted, and even _I_ can't usually land a patch without one 
or two rounds of review ;-)).


Thanks for offering to help out!  If everybody who wanted Python 3 
support just signed on to work on a ticket like this, the port would 
have been finished a year ago ;-).

There's a ticket athttps://twistedmatrix.com/trac/ticket/8025#ticket  which 
makes all the tests pass on the platform; I just need to work on it some more. 
I expect Twisted 16.0 to have base support for Python 3 on Windows.

As above, if I can help, please tell me how!


Our toolchain is a little clunky (sorry about that) for hysterical 
raisins.


Before I launch into an explanation though, are you already familiar 
with Git and/or how to contribute on GitHub?


-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Running Trial [wav]: Re: Possible bug

[Chris Norman]

Hi,
So I went on the net on the off chance that the info I was looking for 
was easy to obtain, and I found out about trial.


I cloned the git and did:
python bin/trial twisted

I got the following:

Unhandled Error
Traceback (most recent call last):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 473, in postOptions

_BasicOptions.postOptions(self)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 383, in postOptions

self['reporter'] = self._loadReporterByName(self['reporter'])
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 370, in _loadReporterByName

for p in plugin.getPlugins(itrial.IReporter):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\plugin.py", line 
213, in getPlugins

allDropins = getCache(package)
---  ---
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\plugin.py", line 
171, in getCache

    provider = pluginModule.load()
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\modules.py", 
line 389, in load

return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\reflect.py", 
line 303, in namedAny

    topLevelPackage = _importAndCheckStack(trialname)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\reflect.py", 
line 242, in _importAndCheckStack

return __import__(importName)
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\plugins\twisted_core.py", 
line 5, in 
from twisted.internet.endpoints import _SystemdParser, 
_TCP6ServerParser, _StandardIOParser
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\endpoints.py", line 
34, in 

from twisted.internet.stdio import StandardIO, PipeAddress
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\stdio.py", 
line 30, in 

from twisted.internet import _win32stdio
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\_win32stdio.py", 
line 15, in 

from twisted.internet import _pollingfile, main
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\_pollingfile.py", 
line 106, in 

class _PollableReadPipe(_PollableResource):
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\_pollingfile.py", 
line 108, in _PollableReadPipe

implements(IPushProducer)
  File "C:\python35\lib\site-packages\zope\interface\declarations.py", 
line 412, in implements

raise TypeError(_ADVICE_ERROR % 'implementer')
builtins.TypeError: Class advice impossible in Python3.  Use the 
@implementer class decorator instead.


Unexpected error while writing cache file
Traceback (most recent call last):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 473, in postOptions

_BasicOptions.postOptions(self)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 383, in postOptions

self['reporter'] = self._loadReporterByName(self['reporter'])
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 370, in _loadReporterByName

for p in plugin.getPlugins(itrial.IReporter):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\plugin.py", line 
213, in getPlugins

allDropins = getCache(package)
---  ---
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\plugin.py", line 
185, in getCache

dropinPath.setContent(pickle.dumps(dropinDotCache))
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\filepath.py", 
line 1532, in setContent

os.rename(sib.path, self.path)
builtins.ValueError: rename: src and dst must be the same type

Traceback (most recent call last):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\lockfile.py", 
line 91, in symlink

    rename(newlinkname, filename)
FileExistsError: [WinError 183] Cannot create a file when that file 
already exists: 
'C:\\Users\\chris\\Dropbox\\SRC\\twisted\\_trial_temp.lock.1446280185217.newlink' 
-> 'C:\\Users\\chris\\Dropbox\\SRC\\twisted\\_trial_temp.lock'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/trial", line 22, in 
run()
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 616, in run

test_result = trialRunner.run(suite)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\trial\runner.py", 
line 959, in run

return self._runWithoutDecoration(test, self._forceGarbageCollection)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\trial\runner.py", 
line 983, in _runWithoutDecoration

oldDir = self._setUpTestdir()
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\trial\runner.py", 
line 886, in _setUpTestdir

testdir, self._testDirLock = util._unusedTestDirectory(base)
  File "C:\Users\ch

Re: [Twisted-Python] revisiting onboarding

[Chris Wolfe]

On 2 February 2015 at 19:18, Glyph Lefkowitz <gl...@twistedmatrix.com>
wrote:
[snip]
> Not all committers are actively involved in the project at all times, so
we should form a "committer committee"
> of people who are interested in evaluating candidates.  If you would like
to do that and you're already a committer,
> please contact me and I'll add you to the list.

It looks like this new process has stalled on the formation of the
committee to review new contributor requests. As of right now, the
committee has no members. :-)

This email is a call for volunteers. If you would like to serve on the
committee, please reply to this thread and state that you are interested in
serving on the committee. Once enough volunteers have signed up, new
contributor requests will be sent to the committee's mailing list.

Thanks!
-- 
Chris Wolfe / herrwolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Blacklisting hosts

[Chris Norman]

Hello,

 On 10 Aug 2015, at 03:32, Glyph gl...@twistedmatrix.com wrote:
 
 
 On Aug 9, 2015, at 9:07 AM, Cory Benfield c...@lukasa.co.uk 
 mailto:c...@lukasa.co.uk wrote:
 
 
 On 8 Aug 2015, at 08:07, Chris Norman chris.norm...@googlemail.com 
 mailto:chris.norm...@googlemail.com wrote:
 
 Hi all,
 I am using Twisted to make a game server. I want to be able to ban IP 
 addresses. Currently I check if the host is in a blacklist, and if it is, 
 call abortConnection on the transport. It works fine, but I'm thinking 
 there should be a better way, to actively refuse the connection in the 
 first place?
 
 I am not aware of any hook in the BSD socket API that lets you refuse a 
 connection entirely. Generally, you put a socket into ‘listen’ mode 
 (indicating to the OS that you’ll accept new connections), and then you call 
 accept() to get the new connection. In fact, the OS will accept the 
 connection even before you call accept(): it’ll do it asynchronously, and 
 you will just get the FD for the connection. IIRC Windows has a winsock 
 specific thing that might do what you want, but that’s pretty platform 
 specific and probably doesn’t actually prevent the connection getting 
 established anyway.
 
 If you really want to never allow the connection at all, you’ll probably 
 want to program iptables (or some other firewall if you aren’t on Linux) to 
 do the packet filtering for you. A combination of iptables and ipsets will 
 get you a high-performance IP address blacklist that will drop all packets 
 before they ever reach your application.
 
 
 There is a shortcut in Twisted, at least, although it does not actually 
 refuse the initial connection for the reasons listed above; you can examine 
 the addr passed to IProtocolFactory.buildProtocol and return None.

This is perfect, thanks. It would have been better to refuse the connection 
entirely, but as Corey said, I can use iptables if I get desperate.

 
 -glyph
 
 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com mailto:Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Blacklisting hosts

[Chris Norman]

Hello,
 On 9 Aug 2015, at 17:07, Cory Benfield c...@lukasa.co.uk wrote:
 
 
 On 8 Aug 2015, at 08:07, Chris Norman chris.norm...@googlemail.com wrote:
 
 Hi all,
 I am using Twisted to make a game server. I want to be able to ban IP 
 addresses. Currently I check if the host is in a blacklist, and if it is, 
 call abortConnection on the transport. It works fine, but I'm thinking there 
 should be a better way, to actively refuse the connection in the first place?
 
 I am not aware of any hook in the BSD socket API that lets you refuse a 
 connection entirely. Generally, you put a socket into ‘listen’ mode 
 (indicating to the OS that you’ll accept new connections), and then you call 
 accept() to get the new connection. In fact, the OS will accept the 
 connection even before you call accept(): it’ll do it asynchronously, and you 
 will just get the FD for the connection. IIRC Windows has a winsock specific 
 thing that might do what you want, but that’s pretty platform specific and 
 probably doesn’t actually prevent the connection getting established anyway.
 
 If you really want to never allow the connection at all, you’ll probably want 
 to program iptables (or some other firewall if you aren’t on Linux) to do the 
 packet filtering for you. A combination of iptables and ipsets will get you a 
 high-performance IP address blacklist that will drop all packets before they 
 ever reach your application.

Thanks for that. I was sort of hoping for a Pythonic solution that doesn't rely 
on SubProcess ETC, particularly as I want this server to run on any OS you 
throw at it. Thanks for the idea though, I'll certainly use that if I get 
something that little Python can't handle.
 
 Cory
 
 
 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revisiting onboarding

[Chris Wolfe]

On Tue, Apr 28, 2015 at 5:08 PM, Glyph Lefkowitz gl...@twistedmatrix.com
wrote:

 Can you send me an iCalendar invite of some kind to give you all the
 relevant permissions?  I would definitely like to volunteer to do it, but
 if we don't have a specific time, I will 100% guarantee you that I will
 forget :).

 -glyph

 P.S.: I've been setting a bad example, but we should generally be
 bottom-posting on this list, it makes the conversation easier to follow :).

 On Apr 28, 2015, at 2:54 PM, Chris Wolfe chriswwo...@gmail.com wrote:

 Sure! I can only think of three things that need to be done:

 1. Add a wiki page detailing the process you proposed. It may be helpful
 to display an example email showing what should be included in a new
 contributor request. The following pages should have links to the new
 policy:
   - https://twistedmatrix.com/trac/wiki/TwistedDevelopment#Policies
   - https://twistedmatrix.com/trac/wiki/ContributingToTwistedLabs

 I can create the wiki page and the example email. To do so, I'll need to
 get wiki permissions added to my trac account.

 2. Activate the email address com...@twistedmatrix.com and compile a list
 of people to whom new commit requests should be sent for review. I can't do
 this.

 3. Send an email to the general mailing list once the new pages are up to
 announce the new advancement path. I can do this.

 Is there anything I'm missing?

 - Chris // herrwolfe

 On Tue, Apr 28, 2015 at 3:25 PM, Glyph Lefkowitz gl...@twistedmatrix.com
 wrote:

 I think we can consider it tacitly accepted by the community (nobody
 seemed to object) but we still don't have anyone to implement it. Do you
 want to step up to do that? :)

 -g

 On Apr 27, 2015, at 6:04 PM, Chris Wolfe chriswwo...@gmail.com wrote:

 Hi,

 Has this proposal been accepted? Is there anything I can do to help
 implement it?

 -Chris

 On Sun, Feb 8, 2015 at 9:01 AM, Tom Prince tom.pri...@ualberta.net
 wrote:

 Glyph Lefkowitz gl...@twistedmatrix.com writes:

  So I have a proposal for a scaled back process that nevertheless would
 give us something official-ish:
  ..details...

 I support this proposal.

   Tom

 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




 --
 Chris Wolfe
 chriswwo...@gmail.com





 --
 Chris Wolfe
 chriswwo...@gmail.com



Hi,

I've added a draft wiki page detailing the contributor advancement path.
The page is located at
https://twistedmatrix.com/trac/wiki/Drafts/ContributorAdvancementPath.

If anyone has any feedback on the document, please feel free to either edit
the wiki or send me an email through the mailing list. If there aren't any
objections to what I've written by June 5, I will move it out of the drafts
section and link it up to the other documentation.

Thanks!
Chris

-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revisiting onboarding

[Chris Wolfe]

Sure! I can only think of three things that need to be done:

1. Add a wiki page detailing the process you proposed. It may be helpful to
display an example email showing what should be included in a new
contributor request. The following pages should have links to the new
policy:
  - https://twistedmatrix.com/trac/wiki/TwistedDevelopment#Policies
  - https://twistedmatrix.com/trac/wiki/ContributingToTwistedLabs

I can create the wiki page and the example email. To do so, I'll need to
get wiki permissions added to my trac account.

2. Activate the email address com...@twistedmatrix.com and compile a list
of people to whom new commit requests should be sent for review. I can't do
this.

3. Send an email to the general mailing list once the new pages are up to
announce the new advancement path. I can do this.

Is there anything I'm missing?

- Chris // herrwolfe

On Tue, Apr 28, 2015 at 3:25 PM, Glyph Lefkowitz gl...@twistedmatrix.com
wrote:

 I think we can consider it tacitly accepted by the community (nobody
 seemed to object) but we still don't have anyone to implement it. Do you
 want to step up to do that? :)

 -g

 On Apr 27, 2015, at 6:04 PM, Chris Wolfe chriswwo...@gmail.com wrote:

 Hi,

 Has this proposal been accepted? Is there anything I can do to help
 implement it?

 -Chris

 On Sun, Feb 8, 2015 at 9:01 AM, Tom Prince tom.pri...@ualberta.net
 wrote:

 Glyph Lefkowitz gl...@twistedmatrix.com writes:

  So I have a proposal for a scaled back process that nevertheless would
 give us something official-ish:
  ..details...

 I support this proposal.

   Tom

 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




 --
 Chris Wolfe
 chriswwo...@gmail.com





-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revisiting onboarding

[Chris Wolfe]

Hi,

Has this proposal been accepted? Is there anything I can do to help
implement it?

-Chris

On Sun, Feb 8, 2015 at 9:01 AM, Tom Prince tom.pri...@ualberta.net wrote:

 Glyph Lefkowitz gl...@twistedmatrix.com writes:

  So I have a proposal for a scaled back process that nevertheless would
 give us something official-ish:
  ..details...

 I support this proposal.

   Tom

 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted-web] Twisted 15.1 Release Announcement

[Chris Wolfe]

Hey Hawkie!

I gave Glyph all of the stickers that I had, as I thought he'd come into
contact with more Twisted people than I do. As such, you might want ping
him to get a hold of one.

Thanks for all your hard work!

Best,
Chris // herrwolfe

On Tue, Apr 14, 2015 at 12:28 AM, HawkOwl hawk...@atleastfornow.net wrote:

 zomg twisted stickers!!!

 Chris, let me know how much it is for you to send me some -- my laptop has
 Django stickers, but no Twisted. FOR SHAME

 - hawkie

  On 14 Apr 2015, at 13:23, Glyph gl...@twistedmatrix.com wrote:
 
 
  On Apr 13, 2015, at 04:17, HawkOwl hawk...@atleastfornow.net wrote:
 
  On behalf of Twisted Matrix Laboratories, I am honoured to announce the
 release of Twisted 15.1.0 -- just in time for the PyCon sprints!
 
  This is not a big release, but does have some nice-to-haves:
 
  - You can now install Twisted's optional dependencies easier -- for
 example, `pip install twisted[tls]` installs Twisted with TLS support.
  - twisted.web.static.File allows defining a custom resource for
 rendering forbidden pages.
  - Twisted's MSN support is now deprecated.
  - More documentation has been added on how Trial finds tests.
  - ...and 26 other closed tickets containing bug fixes, feature
 enhancements, and documentation.
 
  For more information, check the NEWS file (link provided below).
 
  You can find the downloads at https://pypi.python.org/pypi/Twisted
 (or alternatively http://twistedmatrix.com/trac/wiki/Downloads) . The
 NEWS file is also available at
  
 https://github.com/twisted/twisted/blob/releases/release-15.1.0-7758/NEWS
 .
 
  Many thanks to everyone who had a part in this release - the supporters
 of the Twisted Software Foundation, the developers who contributed code as
 well as documentation, and all the people building great things with
 Twisted!
 
  Twisted Regards,
  Hawkie Owl
 
  This is very exciting.  I am SUPER PUMPED to start telling people to
 `pip install twisted[tls]´ instead of the whole mess it used to be.  Thanks
 in particular to you, Hawkie, and to Chris Wolfe who landed that patch (and
 brought twisted stickers to PyCon!)
 
  -glyph
 
  ___
  Twisted-Python mailing list
  Twisted-Python@twistedmatrix.com
  http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 15.1 feedback

[Chris Wolfe]

Hawkowl,

The new functionality for installing optional dependencies does not on
Python3 (ticket: https://twistedmatrix.com/trac/ticket/7807).

This is due to two things:
1. some of the optional dependencies do not support python3
2. setup3.py isn't using the optional extras built in the main setup.py
file.

Thanks!
Chris (herrwolfe)

On Sat, Mar 21, 2015 at 10:29 AM, HawkOwl hawk...@atleastfornow.net wrote:

 Hi everyone,

 It's been about two weeks now -- does anyone have any feedback on whether
 15.1 worked/didn't work/caused a resonance cascade?

 If you missed the announcement, tarballs can be found at
 http://twistedmatrix.com/Releases/pre/15.1.0pre1/ , and the full NEWS
 file can be found at
 http://twistedmatrix.com/Releases/pre/15.1.0pre1/NEWS.txt .

 Once I'm happy that it's not completely broken, I'll push 15.1 out the
 door proper, and get 15.2 on the way... :)

 - Hawkie

 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 15.1 feedback

[Chris Wolfe]

Jean-Paul,

You are correct. No, this is not a regression compared to to 15.0. I missed
the words 'resonance cascade'.

Thanks,
Chris

On Sat, Mar 21, 2015 at 11:53 AM, exar...@twistedmatrix.com wrote:

 On 03:41 pm, chriswwo...@gmail.com wrote:

 Hawkowl,

 The new functionality for installing optional dependencies does not on
 Python3 (ticket: https://twistedmatrix.com/trac/ticket/7807).

 This is due to two things:
 1. some of the optional dependencies do not support python3
 2. setup3.py isn't using the optional extras built in the main setup.py
 file.


 Are these regressions compared to 15.0?  I don't think they are because
 these features are brand new (on Python 2) in 15.0.  #7807 should be fixed
 (and hopefully for 15.2) but it doesn't sound like it's a blocking issue
 for completing the 15.1 release because it doesn't remove something that
 was included in 15.0.

 Jean-Paul


 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revisiting onboarding

[Chris]

Glyph,

I support this idea. I like the fact that it gives a clear goal (5 submitted 
patches, 5 reviews, and 2 accepted submissions) towards which people interested 
in committing can work.

-Chris (herrwolfe)

 On Feb 2, 2015, at 1:18 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote:
 
 A few months ago, the question of an official process for obtaining commit 
 access was raised.
 
 The discussion lead to this proposal - 
 https://twistedmatrix.com/trac/wiki/Proposal/ContributorAdvancementPath 
 https://twistedmatrix.com/trac/wiki/Proposal/ContributorAdvancementPath.  I 
 like that proposal, but it is still incomplete: to wit, it defines a template 
 for how roles might be defined but does not actually define any roles.  It 
 also seems to be a bit overly ambitious, just from the observed reaction of 
 nobody having the time to implement it :).
 
 So I have a proposal for a scaled back process that nevertheless would give 
 us something official-ish:
 
 Not all committers are actively involved in the project at all times, so we 
 should form a committer committee of people who are interested in 
 evaluating candidates.  If you would like to do that and you're already a 
 committer, please contact me and I'll add you to the list.  I want to do this 
 so there's somewhere to apply to which isn't just a public mailing list or 
 discussion, since public discussions can create social pressure to grant 
 someone commit just to be nice, and rejections might be perceived as mean.
 
 Candidates should submit an application to this new list, 
 com...@twistedmatrix.com mailto:com...@twistedmatrix.com which is a list of 
 links to at least 10 tickets, at least 5 of which are patches they've 
 submitted, and at least 5 of which are code reviews they've done which have 
 been accepted by a committer.  At least 2 of their authored patches should 
 have gone all the way through the process to be landed.
 
 As with the other parts of our process, if there is at least one sponsor, and 
 no objections from anyone on the committee within 7 days, any member of the 
 committee may add the committer.
 
 New committers should then be announced on the mailing list.
 
 This is not really an ideal process - particularly, it lacks a good way to 
 give contributors something specific and useful to do - but it's something at 
 least.  If there is general assent that this is an improvement, I'll go make 
 a wiki page and a mailing list.
 
 -glyph
 
 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Call For Participation: a review rally before the race!

[Chris Wolfe]

Sure - I'm not a committer, but I'll do my best.
Am 09.10.2014 18:21 schrieb HawkOwl hawk...@atleastfornow.net:

 On 10 Oct 2014, at 6:56, Glyph Lefkowitz gl...@twistedmatrix.com wrote:

  On November 1, ClusterHQ is going to be hosting a Twisted sprint in
 Boston.
 
  One of the things that tends to put a damper on such events is the
 perpetual backlog of tickets in review, many of which have been there
 seemingly forever.
 
  I would like to propose that we - the whole Twisted community, but
 especially Twisted committers - give these sprinters a morale boost, by
 giving them a clean slate, review-queue wise.  This will let them focus on
 developing cool new stuff, getting a release out, fixing gnarly old bugs,
 or whatever strikes their fancy, instead of just grinding through reviews
 the whole time, or feeling bad about leaving those reviews outstanding.
 
  Right now, we have 58 tickets in review.
 
  https://twistedmatrix.com/trac/report/25
 
  Here's my proposal:
 
  I am looking for at least 4 volunteers for a review rally starting on
 Monday, October 13.  If each of 5 reviewers (the 4 volunteers, plus myself)
 can review 2 tickets per day for one week, until Sunday, October 19, we can
 clear 70 tickets, which should take care of the existing 58 tickets
 currently queued up, plus any that get resubmitted during that week.
 
  While some volunteers should be Twisted committers since we have plenty
 of pending contributions from non-committers that need review, there are at
 least 11 tickets in review from core contributors right now, so
 non-committer volunteers are equally appreciated!
 
  Of course, if we get 9 volunteers, then each reviewer only has to do one
 ticket per day, an even more tractable goal.  And so on into the higher
 numbers :-).
 
  Who is with me?!?!
 
  -glyph
 
  P.S.: While you should of course do this out of its self-evident merit,
 I do plan to recognize participation in some way, at least with a
 commemorative page on twistedmatrix.com, and possibly also with something
 a little goofy like a Mozilla Open Badge :-).
 

 Sure — I should be doing reviews anyway :)

 -hawkie


 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Call For Participation: a review rally before the race!

[Chris Wolfe]

Yes - I have, I'm herrwolfe
Am 09.10.2014 18:47 schrieb Glyph Lefkowitz gl...@twistedmatrix.com:

 On Oct 9, 2014, at 4:39 PM, Chris Wolfe chriswwo...@gmail.com wrote:

 Sure - I'm not a committer, but I'll do my best.\

 Thanks Chris, your participation is much appreciated!

 Now we've got:

 1. me
 2. hawkowl
 3. chris wolfe

 Have you done reviews before?  Before the week starts you should
 definitely familiarize yourself with 
 https://twistedmatrix.com/trac/wiki/ReviewProcess and make sure you know
 what to do.  Please feel free to reach out if this documentation is in any
 way unclear (we definitely know it's not the best).

 -glyph

 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] naming question before we end up committed...

[Chris]

+1 for twisted.logger.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] git?

[Chris Withers]

On 13/03/2014 19:38, Glyph Lefkowitz wrote:

On Mar 12, 2014, at 11:53 PM, Chris Withers ch...@simplistix.co.uk
mailto:ch...@simplistix.co.uk wrote:


Not to be too contentious, but when do you reckon you guys will switch
the main repo to git?


Just to be clear about the nature of my other answers: we do not
reckon such a thing will happen; we have no estimates.  Things get
done because volunteers do them.  You could be just such a volunteer :).


I've read up on the thread and it's interesting to note that Mike Bayer 
has poured a huge amount of effort into doing the move for SQLAlchemy, 
producing some great tools in the process. I believe he now has 3 git 
repos that keep each other in sync, as well as accepting pull requests 
via both bitbucket and github.


I asked him about it and checked that it was okay to forward on his 
reply, please find it attached.


Hope this helps,

Chris

--
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk
---BeginMessage---
well we were on mercurial before git, so our SVN migration was to mercurial, 
though it’s just as simple moving from SVN to git directly, we did that at work.

However: when I migrate VCS’s, I usually need to alter the migration utilities 
to support the following features that people usually don’t seem to care about:

1. author mapping.  Ensure that author names/emails from the old system are 
updated in the new system the way we want them to be.

2. commit message rewriting.   We make a lot of use of revision numbers in 
commit messages (e.g. like r1234), so when we do the migration we like to 
rewrite the commit messages with the new numbers as well (this of course means 
the transfer system needs to have those numbers available up front).

3. as far as tags and branches, I’m non-negotiable in that these must transfer 
*exactly* with no shenanigans, naming changes, loss of history, etc..  hg-git 
has some silly restrictions that it doesn’t support hg branches, only 
“bookmarks” which nobody uses, this is because hg-git is trying to do a 
hg-git bidirectional workflow, which is ludicrous.  I don’t need that, so I 
wrote https://bitbucket.org/zzzeek/hggitonce based on it to do what I want.

When you move VCS’s, you need to make sure you capture a file that is the 
translation map from old to new revision identifiers.  you then use this map to 
rewrite all of the issues and comments in your issue tracking system to refer 
to the new numbers.   Using trac, this is obviously pretty easy since you just 
access the database directly and do the work.I’ve done this twice with trac 
- once with svn- hg, and then with hg-git.

As far as github for issue tracking, you have much bigger issues there, since 
github does not allow a pure import of issues which include original timestamps 
and author names.  I tried to ask them about that once and I got a pretty 
dismissive response from some sleepy 20 year old kid.Again, it’s not an 
option to lose the timestamps on your issues nor to lose author names for 
people who have accounts in github.

That’s why I think bitbucket’s system is superior in many ways:

1. its workflow is much more similar to trac and is easier to map
2. their support is 150% top notch and I have direct email access to several 
people there 
3. they have a full blown import/export system that accepts author names and 
timestamps, you can rewrite your issue database as many times as you like.

To transfer issues from trac to bitbucket I wrote bbutils: 
https://bitbucket.org/zzzeek/bbutils.   It’s got some quirks but does the job.  
 When you transfer your issues to bitbucket, you need to be careful which 
author names you take over as a lot of them aren’t on bitbucket, and also you 
need to run through the issues about 30 minutes after import and clear the 
“spam” flags that their spam bot will invariably set up.  the BB devs can do 
this for you, but so I don’t have to bother them I wrote a “spam” command in 
bbutils that does this for you.

I’m not sure how twisted handles spam with their trac install, that’s 
ultimately the reason I had to move.  the spammers found me, and none of trac’s 
underdocumented and obtuse spam tools could do anything about it.super glad 
I don’t host any more applications or email anymore.

On Mar 13, 2014, at 4:56 AM, Chris Withers ch...@simplistix.co.uk wrote:

 Hi Mike,
 
 How much of what you did with sqlalchemy is a recipe that the Twisted 
 guys could follow?
 
 Chris
 
 
  Original Message 
 Subject: Re: [Twisted-Python] git?
 Date: Thu, 13 Mar 2014 15:14:53 +0800
 From: HawkOwl hawk...@atleastfornow.net
 Reply-To: Twisted general discussion twisted-python@twistedmatrix.com
 To: twisted-python@twistedmatrix.com
 
 Signed PGP part
 Hi Chris,
 
 I can assure you this topic has come up many times - and unless we
 have volunteers that can manage to get the several thousand tickets
 with comment history into GitHub (which

Re: [Twisted-Python] git?

[Chris Withers]

On 12/03/2014 23:52, Glyph Lefkowitz wrote:

On Mar 12, 2014, at 3:43 PM, Christopher Armstrong
ra...@twistedmatrix.com mailto:ra...@twistedmatrix.com wrote:


On March 12, 2014 at 5:39:36 PM, Chris Withers (ch...@simplistix.co.uk
mailto:ch...@simplistix.co.uk) wrote:

Hi All,

Is there a git mirror of the twisted repo anywhere?


Yes, it’s at

http://github.com/twisted/twisted


Also at https://code.twistedmatrix.com/git/Twisted.

However, if you actually want to /use/ this mirror to work on Twisted,
https://twistedmatrix.com/trac/wiki/GitMirror has some useful information.


Not to be too contentious, but when do you reckon you guys will switch 
the main repo to git?


The difficulty of using svn for the main repo is that it only has a 
subset of the features of a dvcs such as git.


The difficulty of using git for the main repo is if you have people who 
still want to interact via svn. Well, it would be if you didn't use github:


https://help.github.com/articles/support-for-subversion-clients

I do wonder how many people know about the above feature, it blows my 
mind a bit that it's actually there in the first place...


Anyway, just a thought :-)

Chris

--
Simplistix - Content Management, Batch Processing  Python Consulting
   - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] git?

[Chris Withers]

On 13/03/2014 11:17, HawkOwl wrote:

If you want git access, we have that already (both read and write!),\


So, if I hypothetically put in a pull request via GitHub, that would 
just work?


Chris

--
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] which async framework?

[Chris Withers]

Hi All,

Apologies for the cross-post, but this is a which framework question 
so seemed the most constructive way. Not interested in religious 
debates, just trying to pick the best tool for the job and didn't get 
much of a useful response from python-list...


So, I see python now has a plethora of async frameworks and I need to 
try and pick one to use from:


- asyncio/tulip
- tornado
- twisted

From my side, I'm looking to experimentally build a network testing 
tool that will need to speak a fair few protocols, both classic tcp and 
multicast-based, and have a web api living on top of it that most likely 
will have a websocket for pumping data to the browser. It'll also need 
to write out JUnit-compatible xml results, but that's like the easiest 
bit ;-)


I'd like to be able to serve the rest of the web api using a pyramid 
wsgi app if possible, and I'd like to be able to write the things that 
process requests in and validation out in a synchronous fashion, most 
likely spinning off a thread for each one.


The protocols are all financial (do we really not have a pure-python FIX 
library?!) but none are likely to have existing python implementations.


How should I pick between the options? What would people recommend and why?

cheers,

Chris

--
Simplistix - Content Management, Batch Processing  Python Consulting
   - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] which async framework?

[Chris Withers]

On 12/03/2014 08:53, Laurens Van Houtven wrote:

Twisted has a thing called Cyclone, which I hear (but that's only
hearsay) gives you Tornado's API on top of twisted, so you get all of
the stuff below for free.

 From my side, I'm looking to experimentally build a network testing
tool that will need to speak a fair few protocols, both classic tcp
and multicast-based, and have a web api living on top of it that
most likely will have a websocket for pumping data to the browser.
It'll also need to write out JUnit-compatible xml results, but
that's like the easiest bit ;-)

I don't know which protocols you are interested in specifically;


Aiming low (insert sarcastic look here), here's the first few:

http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_FIX_Specification.pdf

http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_Binary_Order_Entry_Specification.pdf

http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_PITCH_Specification.pdf

http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_MC_PITCH_Specification.pdf


Twisted has a third party project called txsockjs which works
excellently, both by itself and in the context of other (HTTP)
resources. sockjs is a protocol which is basically websockets, damnit,
even when the consumer is a bad browser like IE6.


Thankfully, I only have to support sane browsers...


Twisted comes with a threadpool-backed WSGI server. (When I say
threadpool-backed I mean that the WSGI requests are handled in threads;
the IO itself of course comes from the Twisted reactor).


Yeah, I remember that, and I remember liking it :-)


I find twisted to be a great tool for writing protocol implementations.
I have written tools for querying all sorts of gnarly proprietary
protocols over all sorts of gnarly transports (packet radio; it's
totally a thing), and more recently for doing crazy things like
multiplexing stream transports over existing RPC protocols. (Like, you
see a local port come up, and that actually creates a virtual stream
connection over an existing RPC thing to some virtual server on the
other end of the wire: https://www.youtube.com/watch?v=W_jEIvugwes).


Cool!


Twisted has had so many people write so many protocols in it that also
the testing tools (MemoryReactor, StringTransport) are great. Especially
if you are writing something very close to a wire protocol you will
undoubtedly enjoy those amenities. There's also tons of composable
things for receiving delimited lines, nestrings, etc. It's hard to tell
what you will be looking for since I don't know details about your
protocol, but having written more than a few protocol implementations
I'm going to wager a guess and say Twisted has it or a third party thing
for twisted has it.


Yep, certainly sounds good. I guess I have concerns about 
discoverability and making sure I'm picking the right things to use 
rather than re-implementing things when I don't need to.


What's the best way for me to find things I should be using? I guess my 
fallback position is to ask lots of clueless questions on this list and 
hope I don't annoy to many people, is that viable?



Twisted, emphatically and without reservation, for all the above
reasons. It's stable. All the stuff you need has been tried and tested
extensively in production. It runs on PyPy, and usually a damn sight
faster than on CPython, too.


Didn't know about the PyPy thing, that will be interesting if I ever hit 
performance problems...



A common criticism of Twisted is that it takes over your codebase. I
am speaking at PyCon in about a month to demonstrate that that isn't
true. (It just looks that way, because once people use it, they don't
want to go back... ;-))


Gutted I can't make it to PyCon this year, look forward to watching the 
video!


Chris

--
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] git?

[Chris Withers]

Hi All,

Is there a git mirror of the twisted repo anywhere?

Chris

--
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] building offline docs

[Chris Withers]

Sorry, more questions...

Unfortunately, most of the time I have for learning and research is offline.

How can I build the docs for browsing (html) from a subversion checkout?

Chris

--
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] ping?

[Chris Withers]

Sorry, not sure mail is getting through from the list...

--
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SURVEY: Have you submitted a patch to Twisted and it never got in?

[chris]

Hi all,

On 01.07.2011 18:36, Phil Mayers wrote:
 However, more constructively (less whiney!) some tickets languished in
 make these tiny cleanups and that's just incredibly painful in the
 current setup, with SVN and Trac mediating things.

 I've got absolutely no interest in pulling SVN head, writing a patch,
 submitting it as an attachment via Trac and *then* being told ok, I've
 created this branch. Go off and learn how to do branches in a crappy old
 centralised VCS, and in a way compatible with UQDS, re-do your patch in
 a branch, then send another diff in as a file

I absolutely agree with Phil here.
The twisted code and contribution standards are so high that patches 
from new contributors (like myself) are bound to be rejected/resubmitted 
at least once, maybe more. Don't get me wrong, I believe high standards 
are a good thing, but doing continuous development based on tools like 
svn and trac is really painful and it's really difficult to motivate 
yourself to work on a once rejected ticket.

That being said, I believe that the move to a DVCS is a smart move for 
any project looking for continuous community contribution, because IMHO 
they simply allow for a more developer friendly process for all 
involved, thus making the whole review process a more friendly and less 
discouraging thing.

Cheers,
Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] The Real Error (tm) [was Re: how to write a safe catch-all]

[Chris Withers]

On 30/09/2010 18:01, Phil Mayers wrote:
 It is more than a little confusing, and I'm sure frustrating.

 I've tried to produce something like this locally, but cannot.

The very first message in this thread (28th Sept, 14:48) and my email of 
30th Sept, 14:36 both had scripts attached which do exactly this..

 Let me see if I understand the problem in full.

 You have an @inlineCallbacks-decorated generator which is the target of
 a LoopingCall, like so:

 @inlineCallbacks
 def loop():
 try:
   somework()
 except:
   log.err()

 lc = task.LoopingCall(loop)

Almost:

 @inlineCallbacks
 def loop(self):
 # Check for files to send on schedule
 yield self.checkSchedule()

 @inlineCallbacks
 def checkSchedule(self):
try:
yield somework()
except Exception:
log.err(None,'Unhandled exception ...')

...although I'm currently changing the loop function to be:

 def loop(self):
self.checkSchedule()

...as this appears to give me what I want, until something proves 
otherwise...

 You want this loop function to catch  log all exceptions resulting from
 work it initiates.

Yep, errbacks, internal twisted bugs, whatever. As long as they're 
logged by something, preferabyl the try-except above, I don't mind.
What absolutely must not cannot ever happen is for the scheduler to die ;-)

 Your somework function calls, amongst other things, an
 @inlineCallbacks-decorated worker function:

 @inlineCallbacks
 def sendTransmission(...):
 try:
   yield maybeDeferred(feed.initiateTransmission)
 except:
   ...some handling

Yep.

 You are seeing two errors:


1. A GeneratorExit exception. This appears (if I'm reading your
 logging right) to be caught by your logging?

Yep.

2. A ConnectionLost exception. This is not caught by your logging, and
 is propagating up to the reactor, and giving Unhandled Error

Correct.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

Hi Glyph,

On 30/09/2010 19:52, Glyph Lefkowitz wrote:
 File Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/base.py,
 line 757, in runUntilCurrent
 call.func(*call.args, **call.kw)
 File test_looping.py, line 24, in __call__
 del self.connector
 exceptions.AttributeError: Break instance has no attribute 'connector'

 This traceback indicates a bug in Twisted.

 This is an error that other people have very occasionally spotted, and
 we have never been able to reliably reproduce it.

While I'm able to reproduce this, I can't really as it involves a 
customer ftp server also generating error messages and they've already 
had enough of them ;-)

 Over the years we have
 tried to diagnose it in various ways and we have always failed.

 It would be really valuable to everyone if you could write up a bug and

Is there a bug already tracking this?

 provide detailed reproduction instructions, ideally with some python
 code that triggers this error, so that we can address the bug. It would
 be super useful if you could write an example that reproduces the same
 bug on a recent Twisted version (8.2 is pretty old), especially the 10.2
 prerelease. But, if your example reproduces on 8.2 and not 10.0, that
 tells us something too.

Well, unfortunately I can't do any of the above :-( I've only seen this, 
as I said previously, when we use twisted.protocols.ftp.FTPClient to try 
to send a file to an ftp server that was only able to handle active ftp 
transfers when our setup only allowed passive ftp transfers.

It was happening in a steady, reproducible fashion (ie: every time we 
tried) but unfortunately we've had to bail and now send via sftp using a 
pexpect-based wrapper around the real sftp command client (we had buggy 
behaviour with twisted's sftp client elsewhere)

I hope this helps, do let me know if I should add it to an issue 
somewhere...

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 01/10/2010 15:00, exar...@twistedmatrix.com wrote:
 I hope this helps, do let me know if I should add it to an issue
 somewhere...

 Can you at least provide a traffic capture recording an instance of this
 happening?

I'm afraid not, I didn't have the forsight to do this at the time, and 
as I've said, the customer has had enough of us causing their ftp server 
to generate error messages :-S

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 30/09/2010 04:45, exar...@twistedmatrix.com wrote:
 On 12:52 am, ch...@simplistix.co.uk wrote:

 Because I haven't found any permutation that doesn't result in the
 LoopingCall's calls to `loop` from stopping after the exception.

 I would be more than ecstatic to be proved wrong ;-)

 You keep saying the LoopingCall stops.  It does not stop.  It is waiting
 for a result.

What result is it waiting for and how do I pass that from the `loop` 
function in my first example when the exception is caught?

 If you provide it with a result, it will proceed.

See above: how do I do that?

 Glyph
 suggested a number of (hackfully) mechanisms you might use to provide it
 with a result.

Why do you say hackfully? I'm trying to build a scheduler that is 
resilient to failure in a scheduled task. Yes, of course I want to fix 
that failure, but I don't want a failure in one scheduled task to 
prevent any further scheduled tasks from being executed...

 I suggested another way (fix the broken code, or at
 least supply enough information for someone else to fix it).

For all I know, it's already fixed in a newer version of Twisted 
(unfortunately, I can't move this project to a newer version of Twisted) 
but once I have a robust scheduler, I'll certainly do my best to report 
the underlying failure to you guys properly...

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 30/09/2010 15:33, exar...@twistedmatrix.com wrote:
 2010-09-30 15:07:03,161 ERROR   : log (22194|7f41910b26e0):
 Unhandled Error
 Traceback (most recent call last):
File test_looping.py, line 47, inmodule
  reactor.run()
File /twisted/internet/base.py, line 1166, in run
  self.mainLoop()
File /twisted/internet/base.py, line 1175, in mainLoop
  self.runUntilCurrent()
 ---exception caught here  ---
File /twisted/internet/base.py, line 779, in runUntilCurrent
  call.func(*call.args, **call.kw)
File test_looping.py, line 30, in __call__
  del self.connector
 exceptions.AttributeError: Break instance has no attribute 'connector'

 This is not logged by your code.  Do you recognize that?

Yes, this is what I'm complaining about ;-)

 /twisted/internet/defer.py:262: DeprecationWarning: Don't pass strings
 (like 'Break!') to failure.Failure (replacing with a DefaultException).
fail = failure.Failure(fail)
 2010-09-30 15:07:05,167 ERROR   : log (22194|7f41910b26e0):
 Unhandled scheduled exception
 Traceback (most recent call last):
 Failure: twisted.python.failure.DefaultException: Break!

 This comes from some code not included in the code you posted.

Sure it is, it was attached to the message I sent at 14:36.

 It looks
 like you're using Failure wrong though.

Sure, but that's hardly the issue at hand here...
What should I be passing to errback?

 So, how come my log.err doesn't get used for the AttributeError on
 connector?
 
 Your Deferred *never* fires with a Failure corresponding to that
 AttributeError.  This is the most important thing.  If you don't
 understand this, say so and we can talk about it some more.  Everything
 else is just confusing particulars.

Yes, I understand this, and this is what I'm talking about when I say I 
cannot gracefully handle the exception. Reading back, yes, it appears I 
was mistaken at some stage that my `loop` function was handling the 
exception, but I do understand now that it was not... which is 
frustrating...

 The AttributeError never becomes an error result on any Deferred.  It is
 caught inside the reactor implementation, logged there by Twisted
 itself, and then thrown away forever.

:-(

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 28/09/2010 15:21, exar...@twistedmatrix.com wrote:
 But, more crucially, the looping call then appears to stop.

 The function you're looping over returns a Deferred that never fires.
 The LoopingCall isn't stopped, it's waiting for the Deferred.

So, inlineCallbacks/generator things will only process errbacks, not 
actual exceptions raised inside asyncronous code called from them?!

 What can I do to get the exception logged and then everything handled
 sanely such that the looping call can continue and my lopp function
 will keep getting called once every second rather than stopping?

 When you do reactor.callLater(n, f), you put f into an execution context
 where the only thing that will happen to exceptions it raises is that
 they will be logged.

Okay, the script was a simplification of the real problem to try and 
give the list a smallest failing example to try and help with.

The real logging looks like this:

2010-09-27 15:30:16,340 ERROR   : log (24331|7f2e47b4d6e0): 
Unhandled exception sending schedule transmission
Traceback (most recent call last):
   File 
Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/python/context.py, line 
37, in callWithContext
 return func(*args,**kw)
   File 
Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/selectreactor.py, line 
146, in _doReadOrWrite
 why = getattr(selectable, method)()
   File Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py, 
line 631, in doConnect
 self.failIfNotConnected(error.getConnectError((err, strerror(err
   File Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py, 
line 588, in failIfNotConnected
 del self.connector
--- exception caught here ---
   File ourcode.py, line 180, in checkSchedule
 yield self.sendTransmissions(...)
exceptions.GeneratorExit:

2010-09-27 15:30:28,428 ERROR   : log (24331|7f2e47b4d6e0): 
Unhandled error in Deferred:
2010-09-27 15:30:28,584 ERROR   : log (24331|7f2e47b4d6e0): 
Unhandled Error
Traceback (most recent call last):
Failure: twisted.protocols.ftp.FTPError: ('Connection Failed', 
twisted.python.failure.Failure class 
'twisted.internet.error.ConnectError')

I don't quite follow what the above is trying to tell me, other than an 
FTP connection failed. However, I don't understand why that results in a 
GeneratorExit rather than an errback of the original exception being 
caught by the top level handler in the loop() function (switching back 
to the example terminology for simplicity). I also don't understand why 
an unhandled deferred is being logged rather than fed back into the 
handler I built for it!

 exception, then you have to arrange for that.  You can do this by
 wrapping f with another function that handles the exception and sends it
 where you want.

Well, as far as I can tell, that's what I'm trying to do. However, the 
thing failing in the real world is in code I don't own (looks like 
twisted's inards...) and I'd like to be able to cater for any failure, 
unforseen or not (barring say a SyntaxError ;-)) and still have the 
loop() call keep doing its thing.

How can I do that?

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 29/09/2010 17:16, Phil Mayers wrote:
 No.

 The problem is that your example is malformed.

Well, it's not, it's the reality of the situation and one I'd like to 
protect against; the scheduler must not die is the rule I need to make 
work...

 You do this:

1. Create a deferred on the Break class instance

The Break class, in reality, is the bowels of 
twisted.protocols.ftp.FTPClient, copying from my previous mail:

  self.failIfNotConnected(error.getConnectError((err, strerror(err
File Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py,
line 588, in failIfNotConnected
  del self.connector
--- exception caught here ---
File ourcode.py, line 180, in checkSchedule
  yield self.sendTransmissions(...)
exceptions.GeneratorExit:

How can I protect my scheduler against code which doesn't catch an 
exception when it should?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 29/09/2010 18:31, exar...@twistedmatrix.com wrote:
 Then you're talking about an API in Twisted which returns a Deferred
 that sometimes doesn't errback when the implementation encounters an
 error.

 Also, `failIfNotConnected` should never raise an exception.

 These sound like bugs.

 File a couple tickets.  With a unit tests please. :)

That's one side of things, sure, but how can I write a scheduler which 
handles the current situation?

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 30/09/2010 00:31, Glyph Lefkowitz wrote:

 There are so many ways.

Yes, that's what I thought, which is why I was confused to keep on 
getting fix the code responses when I'd already pointed out it wasn't 
my code to fix, and I'd like to build a solution that caters even for 
this eventually...

 You can add a log observer that handles isError=True log messages.

Okay, and this would mean that my `loop` function from the example I 
posted would keep getting called by LoopingCall?

 You can write a 'safetyBelt' function that catches the exception and does 
 something with it.

 From my original example, I'm pretty sure that's what my `loop` 
function is; I'm confused as to why it catches a GeneratorExit when the 
attribute error is indirectly raised in the `doStuff` function, and more 
confused that even thought the GeneratorExit has been caught, the calls 
to `loop` from LoopingCall cease...

 You can always invoke your code with maybeDeferred, which will turn 
 exceptions into failures for you.

In the real app, we have another layer or two between doStuff and 
Break, one of which does call down with maybeDeferred, doesn't make any 
difference...

 You can use inlineCallbacks, where in 'x = yield y()', y() raising an 
 exception and y() returning a failed Deferred are basically indistinguishable 
 to the caller.

Yes, as you can see, this is what `loop` does. In the real app, 
doStuff is actually @inlineCallbacks as well, the level below that calls 
with a maybeDeferred and so yes, doing this, same result...

 Why would you say there's no way?

Because I haven't found any permutation that doesn't result in the 
LoopingCall's calls to `loop` from stopping after the exception.

I would be more than ecstatic to be proved wrong ;-)

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python