Re: [Twisted-Python] trial.unitest-specific segfault with lxml

[Phil Mayers]

On 18/08/2017 07:05, Tristan Seligmann wrote:

Note that I don't think you're allowed to catch GeneratorExit like this; 


Interesting, I cribbed that from the lxml docs. Will look into it, 
thanks for the pointer.


Given that the crash only occurs at exit, I suspect this has something 
to do with finalizers running at a bad time during interpreter shutdown, 
but I don't have time at the moment to look into this deeply (I just 
spotted the GeneratorExit issue and wanted to check if that was the 
problem).


I suspect this is an lxml or Cython bug, ultimately, but changing the 
test base class changes some ordering of events arbitrarily resulting in 
this heisencrash. Alternatively, it might have something to do with logging.


You're spot on - I have managed to trim the test-case down to a plain 
python script, no twisted/trial. It seems to crash the interpreter on 
exit any time >1 xmlfile is live at that point, but it does not crash if 
you del the objects in certain orders - trial was obviously just 
revealing it.


Thanks for the attention and apologies for the noise, seems this is 
nothing to do with Twisted.


Regards,
Phil

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] WebSockets and QUIC for Twisted

[Phil Mayers]

On 08/07/17 03:59, Glyph wrote:


Ironically the goals of QUIC are, I believe, similar to Vertex ;o)


Really?  How so?


Sorry, memory not serving me well. I'm thinking of the PTCP bits, I'd 
managed to remember that, but forget about all the rest!


So no, not similar - my mistake.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] COMMERCIAL:Re: WebSockets and QUIC for Twisted

[Phil Mayers]

On 07/07/17 13:17, Barry Scott wrote:


As someone else noted, maybe you're thinking of SPDY?


I know abnout SPDY, it is indeed QUIC that I'm researching into.


I was talking to Glyph ;o) Sorry for any confusion.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] WebSockets and QUIC for Twisted

[Phil Mayers]

On 06/07/17 03:58, Glyph wrote:


On Jul 5, 2017, at 2:32 AM, Barry Scott > wrote:


I am interested in using Twisted with a couple of technologies: 
WebSockets and

QUIC. I'm researching what is already available.


Why are you interested in QUIC? My understanding was that Google used 
this protocol as an experiment, but HTTP/2 is the successful termination 
of the experiment, and nobody should really be using it.  (For http2 
support, `pip install twisted[tls,http2]`).


QUIC is definitely not dead. HTTP/2 is a clear win over HTTP/1 and was 
the easiest first step, but as you've noted there's substantial interest 
in running HTTP/2 over better-than-TCP transport (UDP being the only 
practical alternative in the near term, what with middleboxes being so 
horribly omnipresent).


As someone else noted, maybe you're thinking of SPDY?

Ironically the goals of QUIC are, I believe, similar to Vertex ;o)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP PUT a GET's streaming response with treq

[Phil Mayers]

On 05/05/17 12:40, Nagy, Attila wrote:


Cool. If you're happy using private members, knock yourself out.
I'm not happy with that of course. The really cool thing would be if 
Twisted could support this task without the need of complicated and 
error-prone code.


I don't disagree, of course. Cory seems to know the score here, and if I 
understand his email, right now you can't do it without buffering.


I wonder what the minimal implementation looks like right now, though?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP PUT a GET's streaming response with treq

[Phil Mayers]

On 05/05/17 12:10, Cory Benfield wrote:


As a second note, you may lock yourself out of HTTP/2. HTTP/2 is not
guaranteed to give you access to a raw transport object (though it


FWIW this kind of thing was, more or less, what I was thinking. It seems 
unlikely that stealing the TCP connection from Response for a few bytes 
was wise...



As a third note, your code does not handle the possibility that
original._transport may not implement IPushProducer. While *in


IIUC IResponse.deliverBody *does* guarantee (currently) to provide a 
transport implementing IPushProducer to the supplied IProtocol, at least 
as far as the docstring says?


Obviously that doesn't make accessing the private _transport safe - an 
implementation could wrap the transport to implement the IPushProducer 
semantics.



practice* it tends to, it needn’t. On top of that, it is not
forbidden for an IPushProducer implementation to call write() even
when paused, and code that wants to be correct in the face of all
situations will need to be able to buffer anyway.


Thanks for confirming this. I suspected as much, but was not certain (I 
make relatively little use of producer/consumer APIs in my own code).



use case. Right now it’s a bit of an annoyance that t.w._newclient
doesn’t allow the body receiving protocol to exert backpressure on
the data.


How does that square with deliverBody providing an IPushProducer? Is it 
the case that the transport may just disobey the pauseProducing requests 
as you've noted above?


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP PUT a GET's streaming response with treq

[Phil Mayers]

On 05/05/17 10:41, Nagy, Attila wrote:


What possible side effects do you see here? What problems could it cause?
So far I've tried to copy a lot of files in parallel, arbitrarily drop 
connections. Everything seems OK.
The file is either transferred in its entirety or (in case of dropped 
connections) not.


Cool. If you're happy using private members, knock yourself out.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Flask or Django

[Phil Mayers]

On 18/04/17 06:09, Hynek Schlawack wrote:


Oh and I’d like to propose to officially merge the twisted-web
mailing list into this one.  I don’t think it makes any sense to


Seconded.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Phil Mayers]

On 23/02/17 09:11, Glyph Lefkowitz wrote:


Yes, and building these binary artifacts is often harder than some
people (cough, alpine, cough) seem to think.  But there are better ways
to square this circle than restricting yourself to the versions of
/python/ libraries that happen to be available in your distro.


I don't disagree. I'm not defending the practice. But it is *clearly* a 
thing people do, and understanding the reasons is, IMO, important.



Windows, the reason they're not shipping manylinux1 wheels right now has
to do with the political implications of auto-shipping a second copy of
openssl to Linux distros that expect to manage security upgrades centrally).


Understandable. I'm sure anyone who was around at the time remembers the 
zlib fiasco.



It might seem weird to use Python-specific tooling and per-application
vendoring for Python dependencies, and yet use distro-global dynamic
linking for C dependencies.  But, this is actually a perfectly cromulent
strategy, and I think this bears a more in-depth explanation.


It's not at all weird. It's exactly what we do, and I agree with your 
rationale.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Phil Mayers]

On 23/02/17 10:27, Hynek Schlawack wrote:


What I don't quite understand is how people can be in love with Go’s
static linking but complaining about Virtualenvs in deployments.
Unwieldy as virtualenvs are: *for Python code* they are exactly that:
statically linked build artifacts.  The principles are very similar,
the execution is arguably better for Go.


Well, for one thing, Python virtualenv are problematic w.r.t. being 
relocatable. That alone makes them harder to use than a Go binary.


But honestly I think we're talking at cross purposes. I'm not 
complaining about virtualenv. We use them quite successfully.


I'm trying to point out that *some* might find the system Python 
attractive because they can use OS-supplied python packages in lieu of 
the effort of setting up a binary build infrastructure.


But I'm clearly not getting my point across, so I'm going to stop now.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Phil Mayers]

On 22/02/17 20:59, Cory Benfield wrote:


The permitted C dependencies *for dynamic linking* are tiny, sure.
But those libraries may statically link as they please. The likely
scenario for most such packages is that they statically link whatever
they need. This is the approach taken by some that I ship.


Excellent point, I hadn't considered that.

Tangentially I see there's also no-manylinux and associated github/pip 
issues w.r.t. people who *do* want to build local binary wheels on a 
local PyPI server...


All in all, not a simple area.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Phil Mayers]

On 22/02/17 17:42, Hynek Schlawack wrote:


I have to disagree here:  I don’t want build tools of any kind in my
final containers therefore I build my artifacts separately no matter
what language.  Of course you can just build the venv on your build


Agreed, 100%. Apologies if I gave you the impression I was advocating 
otherwise.



server without wheeling up a temporary container and then package it
using Docker or DEB or whatever.  You should be separating building
and running anyway so Python – as much as I’d like Go-style single
binaries too – is in no way special here.  The nice thing about
temporary containers though is that I can do all of that on my Mac.


I agree that you need to separate building and installation, and I've 
got no particular beef with using a container, chroot, throwaway VM or 
whatever works for people in doing the build phase.


(What people do with the resultant build output - and in particular 
whether there is a lot of ignoring of the hard-learned lessons of system 
package managers going on now - I will not comment on ;o)


What I was trying to say - badly, apparently - was that the system 
python *could* be attractive to someone because many dependencies may 
exist in the OS package list in suitable form, but conversely may not 
exist in PyPI in binary form for Linux.


As a very simple example: if you have a traditional (non-container) 
Linux system hosting a Python application in a virtualenv, and you 
deploy a Python app to a virtualenv e.g. using Puppet or Ansible, you 
either need to:


1. Use no C extensions
2. Hope there's a manylinux1 binary wheel
3. Use the OS package and --system-site-packages
4. Compile the C extensions and make them available to pip

#2 seems useful now that I know about it but - correct me if I'm wrong - 
the manylinux1 permitted C dependencies are super-tiny, and would not 
permit e.g. cryptography or psycopg2?


#4 is what you are advocating for I believe? But can we agree that for 
smaller projects, that might seem like a lot of repeated work if the 
package is already available in the OS repos?


Wondering out loud, I guess it would be possible for OS-compiled python 
extensions to be somehow virtualenv or relocation-compatible. One could 
envisage something like:


virtualenv t
. t/bin/activate
pip syspkg-install python-psycopg2

...and this going off and grabbing the OS-provided dependency of that 
name, extracting it, and deploying it into the virtualenv, rather than 
the system Python.


There are doubtless all sorts of reasons that is not practical.

Anyway, to be clear - I'm not advocating using the system Python. I'm 
trying to explain why, based on the efforts we expend locally, it could 
seem attractive to smaller sites.


Cheers,
Phil

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Phil Mayers]

On 22/02/17 17:35, Steve Waterbury wrote:

On 02/22/2017 12:23 PM, Phil Mayers wrote:

On 22/02/17 15:00, Steve Waterbury wrote:


Have you considered the 'conda' package manager?


I've never come across it. It looks too big for me to give a quick
opinion on, but I'll bear it in mind. Thanks for the pointer.


"Anaconda" is huge; conda is not.  You can get conda


I was referring to scope as opposed to size in bytes, but it's entirely 
possible I've misunderstood how it works.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Phil Mayers]

On 22/02/17 15:00, Steve Waterbury wrote:


Have you considered the 'conda' package manager?


I've never come across it. It looks too big for me to give a quick 
opinion on, but I'll bear it in mind. Thanks for the pointer.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Phil Mayers]

On 22/02/17 13:06, Jean-Paul Calderone wrote:


Ah!  They are now, maybe you haven't seen them?

Check out https://github.com/pypa/manylinux and
https://github.com/pypa/python-manylinux-demo


I had not seen this. Thanks, this looks like a big leap forward.


A lot of people seem to be thinking that way.  /Personally/, with my
experience with Go (about five months solid at previous employer), I
wouldn't go anywhere near that stack.  There are plenty of
/other /things with appealing features that Python lacks which would
seem to make a better move. :)


Very possibly!

Go as a language leaves me cold, and there are downsides to the 
one-big-binary approach. But it's undeniably easy to deploy in a 
container, and the concurrency is very attractive.



But check out the story for binary wheels.  The state of things in
Python may not be /quite/ as bad as you think.


Indeed, it does look like a promising improvement.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Phil Mayers]

On 22/02/17 12:49, Hynek Schlawack wrote:


That’s why you should use a build server and not ship build
environments.


Which we do.


I don’t see how that’s tedious since a compute does that for me.
Although I don’t see any value at wheeling them (and some packages
cannot be wheeled); my CI builds a venv and puts it into a container.
There’s nothing tedious about it at all.


I find the idea of running throwaway environments to generate a big blob 
of tarball'd python+libs, then copying said tarball to actual 
containers, a rather retrograde step by comparison with established 
package/build infrastructure tools.



It's a real shame that binary wheels on Linux/PyPI aren't a thing.


This is incorrect


Apparently so. The info JP provided was very useful.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Phil Mayers]

On 22/02/17 11:12, Glyph Lefkowitz wrote:


I'm tempted to launch into a diatribe about namespacing, containers, and
application isolation generally, but before I do - why is it that you
/want/ to use the system Python environment?  Had you just not
considered the option of virtual environments?


Awesome though it is, virtualenv can be very tedious if you need to 
install hundreds of megabytes of compiler and -devel packages. System 
packages are attractive precisely because you can avoid this.


I've had to do all sorts of tedious things with containers where I spin 
up a temporary container to build a bunch of .whl files, then actually 
install them in the final container - all to avoid bloating the 
container image with the build tools.


It's a real shame that binary wheels on Linux/PyPI aren't a thing. I 
understand the reasons why, but TBH it substantially reduces the value 
of the Python ecosystem for me.


Go is looking like a more and more attractive competitor to Python, 
frankly, having sidestepped this problem entirely.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 17.1 Release Announcement

[Phil Mayers]

On 12/02/2017 01:44, Glyph Lefkowitz wrote:


Yeah, this is a lack of test coverage for the 'default' case.
 Technically it /is/ a regression, but it's probably pretty unusual to
use the default; point being, I don't think we need an emergency release
here.


Agreed. It's completely tolerable.


Nevertheless - please file a ticket!


#9033 done

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 17.1 Release Announcement

[Phil Mayers]

On 11/02/17 18:12, Phil Mayers wrote:

On 11/02/2017 17:28, Tristan Seligmann wrote:


String endpoint descriptions, as used by twist/twistd among other
things, for listening on tcp ports now require the "tcp:": that is, "80"
is no longer accepted, you need to use "tcp:80".


In which case there's a regression in the "portforward" twistd plugin:


Actually this works:

twistd portforward -p tcp: -d 6665

Odd but meh, I'll take it

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 17.1 Release Announcement

[Phil Mayers]

On 11/02/2017 17:28, Tristan Seligmann wrote:


String endpoint descriptions, as used by twist/twistd among other
things, for listening on tcp ports now require the "tcp:": that is, "80"
is no longer accepted, you need to use "tcp:80".


In which case there's a regression in the "portforward" twistd plugin:

$ v/bin/twistd portforward
Traceback (most recent call last):
...
ValueError: Unknown endpoint type: ''

$ v/bin/twistd portforward -p tcp: -d tcp:6665
...
ValueError: invalid literal for int() with base 10: 'tcp:6665'

:o(

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problems with inlineCallback, Deferred, yield and Python 3 in buildbot

[Phil Mayers]

On 24/01/2017 21:14, Glyph Lefkowitz wrote:


I've encountered this before and quickly worked around it, but I think
this might actually be a bug in python 3, or at least its documentation.

The language docs officially say that a "list display" (which is what I
believe we're looking at here) "yields a new list object".  But that is
not what I see here:

>>> [(yield 1) for x in range(10)]


Related, see:

http://stackoverflow.com/questions/32139885/yield-in-list-comprehensions-and-generator-expressions

http://bugs.python.org/issue10544

Basically, don't use yield inside comprehensions if you don't want 
weirdness, AFAICT :o/


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How do you determine the buffer size of a transport - a use-case for not using back pressure

[Phil Mayers]

On 17/08/16 23:43, Steve Morin wrote:

Twisted Community

Problem: How do you determine the buffer size of a transport, to know
how much data is waiting to be transmitted from using transport.write?


You haven't received many responses, so I'll add a short note here:

The ideal way to do this IMO is to have the receiver send 
acknowledgements with a sequence number to the sender once a message has 
been received *and processed*. Keep track of these at the sender and 
adjust the sending rate accordingly.


Another complementary approach is for the client to keep track of the 
length (in space or in time/age) of it's local 
received-but-not-processed queue and report this. Keeping track of the 
queue in terms of delay is close to the CoDel approach for AQM.


Depending on the network portion of the implementation, the mere fact of 
a message having been received and TCP-ack'd doesn't tell you the client 
is keeping up - it might be receiving and buffering into RAM and getting 
more and more backlogged.


If you can't alter the protocol to include protocol-level ack messages 
then obviously you have to go the less satisfactory route and estimate 
it from the lower-layer transport. As JP has noted, you can get this 
from the kernel.


Flow control is hard ;o)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Style/testing for log-related changes

[Phil Mayers]

All,

I'd like to submit a patch to convert t.conch.ssh to the new logging. 
The main reason is that the conch code logs a *lot* of really, really, 
really boring crap that I want to throw away because it just clutters up 
the logs e.g.


https://github.com/twisted/twisted/blob/twisted-16.2.0/twisted/conch/ssh/connection.py#L454

Moving it to the new logging would, at very least, let me trivially 
write an observer which throws away these by module.


Does anyone have an example ticket/commit for a conversion to the new 
logging showing the general style, and the technique used for writing 
tests for that?


Cheers,
Phil

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted listening on UDP port, why?

[Phil Mayers]

On 23/04/16 03:36, Jonathan Vanasco wrote:

Thanks for all this.

A few quick notes:

On Apr 20, 2016, at 9:15 PM, Glyph wrote:


This isn't so much a feature of Python as it is a feature of the BSD
sockets API.  Sending traffic through a socket, whether it's TCP or
UDP, has to bind a client port.  Given the nature of UDP, binding on
all interfaces is the expectation unless you specify.

I didn't have time to test a simple C program before sending this
message, but
https://github.com/python/cpython/blob/master/Modules/socketmodule.c only
calls "bind()" from sock_bind, not from send(), nor does
https://github.com/python/cpython/blob/master/Lib/socket.py engage in
any such shenanigans.


The 'feature' of Python is a few things:

 From what I could tell, the actual communication and binding happens
somewhere in the c module.



Not so. It's down inside the kernel. All applications using the socket 
API in this way will display this behaviour, regardless of language.


Seriously, try it and see:

#include 
#include 
#include 
#include 
#include 

int main(int argc, char* argv[]) {
int s,r;
struct sockaddr_in dst;
dst.sin_family = AF_INET;
dst.sin_port = htons(37);
dst.sin_addr.s_addr = INADDR_LOOPBACK;

s = socket(AF_INET, SOCK_DGRAM, 0);
printf("socket created\n");
sleep(30);
sendto(s, "foo", 3, 0, , sizeof(dst));
printf("socket used\n");
sleep(30);
return 0;
}

Compile & run the program and quickly lsof the process, you'll see:

test16258 pjm33u  sock0,8  0t0 87111053 protocol: UDP

...wait until it has printed that it has used the socket, repeat and 
you'll see:


test16258 pjm33u  IPv4 87111053  0t0 UDP *:51669

As glyph says, this is an inherent feature of the socket API. When you 
create a socket, it is unbound because you might be about to call bind() 
yourself.


If you then use it without binding it, the kernel has to allocate a 
source port, and in turn an interface, and the only sensible choice 
absent any instructions from userland is INADDR_ANY.


This is definitely not Python doing this.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted listening on UDP port, why?

[Phil Mayers]

On 20/04/16 01:01, Glyph wrote:


Perhaps this is libc's DNS client?  Twisted doesn't do anything like this.


It does something similar with win32reactor IIRC?

http://twistedmatrix.com/trac/browser/tags/releases/twisted-16.1.1/twisted/internet/posixbase.py#L60

...but that binds to 127.0.0.1 not 0.0.0.0

glibc doesn't hold it's DNS sockets open AFAIK - it closes them once the 
reply is done.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted-web] upcoming changes to twistedmatrix.com mail infrastructure

[Phil Mayers]

On 16/03/16 18:52, Glyph wrote:

Over the last few months, twistedmatrix.com 's
mailman installation has been used increasingly frequently to execute
denial-of-service attacks against people's mailboxes.  This is


My sympathies; this exact problem was the reason we CAPTCHA-ised our 
install of mailman and have to keep a very close eye on it.


It's really a shame there's so little open-source competition in the 
email sector these days; it all appears to have been hoovered up by 
Gmail, Office 365 and various spam (sorry - bulk email) providers.



There will be a couple of inconveniences immediately after the transition:


Couple of random thoughts:

Does mailgun actually contain a mailman-alike product or are you 
effectively building one on top of it?


Will the mailman-style List-X headers remain?

Will the behaviour of the list w.r.t. things like routing of To:/Cc:'ed 
people change.


Good luck with the migration.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted + OpenSSL 1.0.2f issues resolved

[Phil Mayers]

On 22/02/16 21:18, Glyph Lefkowitz wrote:



On Feb 22, 2016, at 09:26, Phil Mayers <p.may...@imperial.ac.uk>
wrote:

On 20/02/16 09:33, Glyph Lefkowitz wrote:


I assume that they subscribed by accident (spammers seem to
periodically attempt to mass-subscribe people using our listserv,
I have no idea why, and some of them click the "yes subscribe me"
link for some reason).


FWIW, our local mailman install is patched to require a captcha
(and to disable email-based subscribe) to solve this exact
problem.


Is this simple to do?


It was pretty easy. I originally followed this recipe:

https://www.dragonsreach.it/2014/05/03/adding-recaptcha-support-to-mailman/

However, I had to tweak it slightly - some of the mailman "HTML" (I use 
that term loosely) had weird tag nesting that broke the captcha form 
submission on newer browsers (some monstrosity like 
...)


I'll sling you the diff against 2.1.12 that we use off-list, feel free 
to use or modify as appropriate.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted + OpenSSL 1.0.2f issues resolved

[Phil Mayers]

On 20/02/16 09:33, Glyph Lefkowitz wrote:


I assume that they subscribed by accident (spammers seem to
periodically attempt to mass-subscribe people using our listserv, I
have no idea why, and some of them click the "yes subscribe me" link
for some reason).


FWIW, our local mailman install is patched to require a captcha (and to 
disable email-based subscribe) to solve this exact problem.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python 2.7 on CentOS 6

[Phil Mayers]

On 15/12/2015 16:40, Ray Cote wrote:

We have a fair bit of Python 2.7 Twisted code deployed on RHEL and
CentOS 5 and 6.
In each case, we build from source and do a make altinstall so we’re
running a Python separate from the system’s.


We build an RPM, but basically yes; put it in a different path and it's 
fine.


Minor hassle though it is, it's one more thing that I would prefer to 
not need. But reinstalling upwards of 60 RHEL6 production machines is 
vastly more hassle ;o)


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] doc breakage

[Phil Mayers]

Does this look right?

https://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReadWriteDescriptor.html

Looks like docs for logPrefix has stomped on docs for all the other 
interface methods?


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Writing a low-level network debugging tool

[Phil Mayers]

On 27/11/15 14:05, Jonathan Ballet wrote:


* how long does it take to resolv the domain name to (at least) one of
its IP address
   - against a specified name server or using the system configured servers


That is relatively straightforward.


   - how many tries did it require
 * if there were several tries, the timing of each ones


Typically, application-layer code doesn't retry a DNS lookup; rather the 
c or other runtime will handle this, for example getaddrinfo() in glibc, 
according to settings read from /etc/resolv.conf or compiled-in defaults.


So it depends on whether you want to emulate "typical" application code, 
a specific application stack that may or may not do it's own resolution 
(e.g. modern browsers) or something else.




* how long does it take to get the first bytes of the endpoint
   - how long does it take to complete the TCP connection handshake
   - the status of the packets exchanged (how many retries, how many
packets lost, etc.)


Some of this is available in platform-specific APIs e.g. SIOCGSTAMP and 
TCP_INFO socket options available on Linux.


In general, any timings you make based on return of control from kernel 
will include error relating to system/scheduling issues. If you're 
concerned about getting raw, on-the-wire timings, this is extremely 
difficult without being in-kernel, and even then various issues - TCP 
offload for example - can end up hiding data from you.



How far can I do this kind of things with Twisted? I know I can somewhat
easily get the timings of the name resolution, the TCP connection
handshake also and the time to first byte(s), but what about the
packets? I haven't look at the code of Twisted Names yet, but if it's
doing the DNS request by itself, I may be able to plug-in somewhere and
have my request counter and the timers associated, but I'm not sure if
the underlying details of the TCP protocol are exposed to the upper
layer such as Twisted?


Only via platform-specific options.

To do this kind of thing "reliably", you'd need to reimplement TCP in 
user-space.


But the info above may be a helpful start.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Writing a low-level network debugging tool

[Phil Mayers]

On 27/11/15 14:39, Phil Mayers wrote:


But the info above may be a helpful start.



You may also want to look at TCP_CC_INFO. See the kernel source:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/net/ipv4/tcp.c?id=refs/tags/v4.4-rc2

...for an idea of what else is available. Be aware some of these are 
relatively new.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] List admin attention

[Phil Mayers]

Someone please un-sub this person?


 Forwarded Message 
Subject:Re: [Twisted-Python] ANN: Eliot 0.8, the logging system with
causality
Date:   Mon, 10 Aug 2015 12:59:48 -0600
From:   Judy Craig jcr...@gsacrd.ab.ca
Reply-To:   Twisted general discussion twisted-python@twistedmatrix.com
To: twisted-python@twistedmatrix.com



This is an automatic reply. I am away from the office for the summer,
returning on Tuesday, August 25, 2015.  I will be happy to get back to
you then.

Enjoy your summer!

Judy Craig
Student Services

This email and any files transmitted with it may be confidential and are
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the
system manager. If you are not the named addressee you should not
disseminate, distribute or copy this email.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[Phil Mayers]

On 22/04/15 09:04, HawkOwl wrote:

Hi,

I think that Twisted should deprecate subproject packaging -- they


Agreed. It's largely pointless.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Add support for Next Protocol Negotiation and Application Layer Protocol Negotiation

[Phil Mayers]

On 21/04/15 16:25, Adi Roiban wrote:

Hi,

I just want to let you know about this patch which is in work :
https://twistedmatrix.com/trac/ticket/7860


Nice.



I did an initial review but it would help to get more feedback for this new API.


I would very, very much like to see something along the lines of a 
factory-of-factories which handles creating the protocol instances on 
connections automatically.


Straw-man API:

class NegoFactory(tls.NegoFactory):
nextProtocols = ('h2', 'http/1.1')

def buildProtocol(self, addr, nextproto):
  if nextproto=='h2':
return H2Proto()
  elif nextproto=='http/1.1':
return HttpProto()
  else:
raise NotImplemented()

f = NegoFactory()
listenSSL(port, f)

The basic protocol level support is great, but it'll be much easier to 
use if there's a Twisted-like API on top of it.


But don't block the main ticket - open another one up maybe?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Adding support for Diameter protocol

[Phil Mayers]

On 14/04/15 06:42, Glyph wrote:



On Apr 14, 2015, at 00:22, Hamza Sheikh fehr...@codeghar.com
mailto:fehr...@codeghar.com wrote:

I am new to Twisted and its community


Welcome, we are happy to have you!


and exploring adding Diameter support.


What is Diameter?


offtopic

It's a bit like RADIUS-over-TCP but with knobs on.

The only place it really took off - though it's a big place - is 
3G/mobile network AAA.


RADIUS has such a large installed user-base that DIAMETER had trouble 
reaching a critical mass anywhere that backwards compatibility was needed.


Kind of a shame, since it means AAA signalling for most encrypted 
wireless networks takes place over a crufty old protocol with a bunch of 
problems, not least lack of reliable transport, which got solved by... 
running radius-over-TLS-over-TCP, thus proving:


https://xkcd.com/927/

/offtopic

;o)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Removing Python 2.6 Support after Twisted 15.1

[Phil Mayers]

On 20/03/15 00:23, David Ripton wrote:

On 03/19/2015 08:05 AM, Phil Mayers wrote:

On 18/03/15 23:57, Glyph Lefkowitz wrote:


Rather than just suggest we preserve the status quo and stay on 2.6
forever to do indefinite free work to support Red Hat's obsolescence


You *definitely* shouldn't do that. Push back on RedHat, and tell
customers to push back on RedHat. I say this to other projects.


Red Hat supports old versions of RHEL forever because they have
customers that want that (because they have legacy apps that they want
to keep working with minimal maintenance, but who still want security
patches), and are willing to pay for it.  It's a good thing that someone
provides that option, even if we don't all need it.


Absolutely, and we use it for that reason. But since RedHat charge for 
this support, it seems fair they bear the maintenance burden ;o)


Push back could be ask for an appropriate SCL or some other way of 
providing newer packages without violating the stability. The samba 
versus samba3x packages on RHEL6 are a good example.




That doesn't mean Twisted needs to support old versions of RHEL forever.
  RHEL 7 has been out for a while, and comes with Python 2.7 as the
default.  So even if Twisted drops support for Python 2.6, it'll still
work on RHEL.  (Just not on old versions of RHEL.)

Furthermore, SCL makes Python 2.7 available (in /opt) on RHEL 5 and 6
and CentOS 6, without users needing to build it themselves.  So even
those who are stuck with old versions can install SCL, then use pip to
install new Twisted.

IOW, I wouldn't worry about it.  Drop support for Python 2.6 when it
makes sense for Twisted.


+1

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Removing Python 2.6 Support after Twisted 15.1

[Phil Mayers]

On 19/03/15 00:42, Donald Stufft wrote:


If you want newer than that you could possibly convince PyPy to build
RPMs with COPR. I’m doing that for python-pip and co and it’s not
very hard once you have the RPM spec file originally written, which
could probably just be taken from Fedora.


IME that is usually a lot harder than it looks. We (try to) do it a lot 
and it's very common to need a lot of work backporting newer Fedora 
.spec to EL6. It's often easier to just roll a new one.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Removing Python 2.6 Support after Twisted 15.1

[Phil Mayers]

On 18/03/15 23:57, Glyph Lefkowitz wrote:


Rather than just suggest we preserve the status quo and stay on 2.6
forever to do indefinite free work to support Red Hat's obsolescence


You *definitely* shouldn't do that. Push back on RedHat, and tell 
customers to push back on RedHat. I say this to other projects.


As mentioned, if maintaining 2.6 support is anything other than a 
trivial amount of work, then it's totally reasonable to drop it.



business, perhaps we should make a condition of dropping 2.6 support
being a clear guide for getting a Twisted environment with py2.7 up
and running on whatever appropriately decrepit environment is popular
in the CentOS/RHEL user community?  I am annoyed with RH for lagging
so much, but I don't want to make their customers' employees suffer
for it.

Phil - I do have a question though, since you seem to be a real life
user with this use-case :).  If you want to use an old, unsupported
version of Python, why do you want to deploy a new, updated version
of Twisted on it?


Well, I don't want to use Python 2.6 per-se. It's what a large number of 
the systems we run have bundled, I have no need for newer 2.7 language 
features, and thus if I can *possibly* avoid it, I don't want to 
maintain the burden of deploying another version, even less so the 
burden of packaging it. We've done that in the past, it's non-trivial 
effort.


I've no specific need for a newer Twisted right now, but I can imagine 
things that might appear - support in listenSSL for ALPN for example, or 
HTTP/2 - that we could use if they appeared.


So, it's purely speculative. I have no current specific need.


dstufft - is there a PyPy EPEL?  As long as we're telling people to
change which Python to use, perhaps we should point them at an
actually good one ;).


PyPy in EPEL 6 is currently sitting at 2.0.2. I've only tried it a few 
times and had mixed results; having to use psycopg2cffi was a bit of a 
pain, and I had some pathological performance issues, but it did 
basically work.


The RHEL SCLs OTOH... After Donald's pointer I took a look at them. They 
are... not great.


[Basically, they're built to think they run in one path, but installed 
in another, and you have to run it with a wrapper script that sets up 
LD_LIBRARY_PATH, PATH, PYTHONPATH, etc.


As opposed to what you'd think, a Python 2.7 compiled and installed 
into, say, /opt/python2.7, which you can just run directly.


Sigh...]

So my £0.02 is that recommending SCL for Python 2.7 on EL6 is something 
you would want to do with caution ;o)


I honestly wouldn't worry about it too much; RedHat's policies aren't 
and shouldn't be Twisted's problem. Do what's best for your development.


Cheers,
Phil

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Removing Python 2.6 Support after Twisted 15.1

[Phil Mayers]

On 18/03/15 12:29, Donald Stufft wrote:


Isn’t the RHEL answer here to just use SCLs to install Python 2.7?


I've never actually come across the SCL stuff before. Thanks for the 
pointer.


My latter point still stands - don't do it because others do it, do it 
because it's good for Twisted (or not ;o)


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Removing Python 2.6 Support after Twisted 15.1

[Phil Mayers]

On 18/03/15 10:45, HawkOwl wrote:

Hi,

As Python 2.6 has been officially abandoned for a year+ (and will
receive no further official security updates) and Python 2.7 has been
available for almost five years, it might be time to put Python 2.6
support to bed. The only currently supported distro that has Python
2.6 is RHEL6, and RHEL7 with Python 2.7 has now been out for a year.

This would be in line with other major projects (eg. Django, Plone)
and Python's porting guide
(https://docs.python.org/3/howto/pyporting.html#drop-support-for-python-2-6-and-older).
We can also remove some things from TestCase, and there's random
smatterings of 2.6 support workarounds/comments that we could also
remove.

As such, I think we should make 15.1 the final release which
officially supports Python 2.6.

Any objections?


Objectively it seems reasonable.

Subjectively, it'll substantially inconvenience me (and others who have 
RH/CentOS 6 systems which will be in service for years more).


In the past, we've built and deployed a newer Python when this has 
happened e.g. RHEL4/5. That was a huge pain in the arse, and we don't 
intend to repeat it. Absent an EPEL package for python27, I suspect many 
people will do what we intend to do - stop updating Twisted.


Expect to see queries from people on older versions they can't update 
for some time ;o)


In all seriousness, I guess it depends on how useful this is. Being in 
line with other projects doesn't seem like a useful goal - who cares 
what Plone does? If it's useful for Twisted, then do it.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Process pools and logging

[Phil Mayers]

Before I re-invent the wheel, is there a twisted library that will let me:

 1. Fire up child worker processes
 2. That log to a logObserver coming out in the parent
 3. Manage the whole lot as a .tac file

Note I *don't* need deferToProcess or similar in this situation. I just 
need to fire up N processes but have them managed by and logging to a 
single parent controlled by twistd.


(FWIW, they're stateless workers that pull from a work queue).

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] some competition and hopefully inspiration for conch

[Phil Mayers]

On 16/07/14 22:40, Glyph Lefkowitz wrote:

There's a new Python-based asynchronous SSH client and server in town:

http://asyncssh.timeheart.net/#


This looks to have a much more straightforward API and /way/ better


The client endpoint API is great. It is what Conch should always have 
been and is a pleasure to work with, so congrats to the authors of that. 
It'll be even better when #6617 lands!


It would be great to have a conch server endpoint API.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] FTP box disconnecting after 60 seconds

[Phil Mayers]

On 08/07/14 15:45, Paul Wiseman wrote:

Hey,

This is a bit of a stab in the dark, but I'm really not sure what to try next.

Basically I've got a custom twisted FTP server, recently (not sure
when it started) I've noticed that connections to it seem to
disconnect after 60 seconds if they haven't completed. This could be a


Are there any middlebox devices (firewall, NAT) in the path?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] adbapi.ConnectionPool - reconnecting to postgresql

[Phil Mayers]

On 10/06/2014 19:24, Jonathan Vanasco wrote:


I had to restart postgresql , and that triggered some endless errors in my logs.

i don't have anything that can handle a suddenly dropped / resumed connection.

does anyone have a recommendation for trying to reconnect or handle this in 
general

from what I can tell from my logs...

When I catch an error, my cleanup code tries to

self._connection.rollback()

which raises

psycopg2.InterfaceError: connection already closed


Yes, this is irritating.

Related tickets see:

http://twistedmatrix.com/trac/ticket/4404
http://twistedmatrix.com/trac/ticket/4964

You absolutely want to be setting cp_reconnect=1 on ConnectionPool. 
Anything else is asking for major trouble, and I really think this 
should be the default.


cp_reconnect will not avoid all problems - for a thread pool of size N, 
the next N transactions will have to fail to throw away all the dead 
connections - but it might stop them happing forever.


In some other, non-Twisted contexts, people have had problem even with 
reconnecting, because some versions of some DBAPI drivers emit a 
connection from .close() if it's already closed, which interrupts the 
reconnection logic :o(


It seems your version of psycopg is doing this - how old is it? I 
thought that was fixed these days?


You might also look at something like pgbouncer, which you can configure 
to try a good SQL query on the outbound connections before handing to 
an inbound client; this probes the connections. It adds latency, but 
personally I would far rather pay the small number of milliseconds on 
select 1 than bomb out to the upper layers.


An annoying problem.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Fwd: Trouble with session id reuse/disabling with twisted TLS

[Phil Mayers]

On 30/04/14 17:43, Henrik Thostrup Jensen wrote:

Hi

I have a twisted service, which uses TLS and I seeing some odd behaviour.

New connections are accepted fine, but if a client tries to re-use a
TLS session id with a new connection, the services rejects the
connection.


This example:

https://twistedmatrix.com/documents/12.0.0/core/howto/ssl.html#auto2

...seems to work fine for me on both Twisted 12.1.0 / pyOpenSSL 0.13.1 
and Twisted 13.2.0 / pyOpenSSL 0.14. I'm testing using:


openssl s_client -connect host:port -reconnect

...and seeing:

drop connection and then reconnect
CONNECTED(0003)
---
Reused, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384

^^ looks ok.

What do you see if you use that openssl command to test? Can you reduce 
it to a sscce with a fake self-signed cert?


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] t.w.client.Agent exceptions

[Phil Mayers]

On 05/03/2014 22:08, Glyph Lefkowitz wrote:


You're right; this is a bug in Agent.

http://twistedmatrix.com/trac/ticket/5310


And it must be said, really quite an infuriating one ;o)



For right now, yes, you're going to have to import some crappy private
APIs.  Any chance I could convince you to submit some fixes to that
ticket so we won't have to do that in the future? :-)


I'll have a look tomorrow; though it seems like testing that would be 
tricky, and pointers to existing examples of how to test that 
Failure-with-traceback is being raised?


Do we know what the rationale for _WrapperException was? The only place 
it's used, it wraps a single exception, which is singularly pointless 
(ha ha marvel at my comedy).


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Configuration options for twisted code

[Phil Mayers]

On 04/03/2014 00:52, Itamar Turner-Trauring wrote:

On 03/03/2014 01:09 PM, Adi Roiban wrote:


What do you think? Should I go ahead and add options as __init__
arguments or think of a configuration system?



A configuration system would take a while to design and meanwhile these
unrelated features would languish, so I'd suggest you start with options
to __init__.


-1

I hate __init__ methods with a hojillion parameters, and that's where 
this leads.


+1 instead for class variables which can be overridden in sub-classes to 
achieve DRY and/or set after instantiation if necessary e.g.


class Foo:
  maxThingLen = 1

class MyFoo(Foo):
  maxThingLen = 100

This is also somewhat consistent with what things like t.w.s.Site 
already do.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Exit all threads upon KeyboardInterrupt

[Phil Mayers]

On 14/01/14 14:27, Dustin J. Mitchell wrote:

There are two things wrong with your scenario.

First, in any production or production-ready environment, your
software needs to correctly handle stuck TCP connections.  With the
proliferation of smart network equipment (session-tracking
firewalls, etc.), this is much more common than it was five or ten
years ago.  Some DBAPI backends are better about this than others -
for example, MySQL-Python gets it right, but PyMySQL does not.


That is certainly a fine sentiment with which I agree in principle, but 
an ENORMOUSLY LARGE amount of (really very expensive and business 
critical) software doesn't handle this *at all* as anyone who runs a 
firewall in front of an e-Business app can tell you.


[A well known vendor whose name rhymes with a type of traditional small 
boat used extensively in the British Isles is a particular offender]


So if you're handling this correctly, give yourself a pat on the back - 
there is software costing millions a year in licence fees that assumes 
TCP is as reliable as CPU-CPU cache communications :o(



Second, this has nothing whatsoever to do with whether the I/O is
blocking or not.  Twisted apps can and, believe me from experience, do
suffer from stuck TCP connections.  Sure, the wasted resources in this
case are smaller (just a file descriptor, not a thread), but the user
experience is the same (part of the application hangs irreversibly).
In fact, this is about the only thing the manhole is used for at
Mozilla: running `os.close` on stuck fd's.


I'm wondering if TCP keepalive isn't any use to you in this regard?

(and what do Mozilla do with Twisted? :o)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Exit all threads upon KeyboardInterrupt

[Phil Mayers]

On 14/01/14 15:10, Dustin J. Mitchell wrote:

On Tue, Jan 14, 2014 at 10:05 AM, Phil Mayers p.may...@imperial.ac.uk wrote:

I'm wondering if TCP keepalive isn't any use to you in this regard?


It is, but only in cases where we can tweak kernel parameters
appropriately so that the timeout is not 2h.


Hmm. I see Twisted has no API to set this using the socket options, 
which is unfortunate. I've opened #6897 for that.


Presumably there should be endpoint syntax for it too...


(and what do Mozilla do with Twisted? :o)


Buildbot, among other things.


Doh yes of course.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted names client.getHostbyName dual-stack operation

[Phil Mayers]

On 08/01/14 23:05, Glyph Lefkowitz wrote:


However, if you’re building a big service to run in the cloud, or on a
VPN, on specially-tuned machines that you configured, using
twisted.names can be more scalable and allow you to react more
reasonably to high load situations.  Or, at least, it will give you a
level of control so that you can tune it, since just doing potentially
blocking platform stuff in a thread is not amenable to tuning.


Sure, agreed - I folded those concerns into the user-supplied names 
bit ;o)


Using a resolver directly is totally appropriate in certain use-cases, 
and t.names is a good choice if you're using Twisted[1] but I think of 
it like should I use UDP instead of TCP for my protocol - if you have 
to ask, the answer is no.


(In fairness to the Chrome guys, they've probably got the chops to 
reimplement it properly and are doing enough innovative stuff in Chrome 
that having an in-browser resolver is reasonable, so my dig at them is 
probably unfair ;o)


[1] Modulo a couple of rather big holes that I know people are already 
working on, like source-port randomisation for Kaminsky-resistance, and 
so on.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted names client.getHostbyName dual-stack operation

[Phil Mayers]

On 06/01/14 08:26, Maciej Wasilak wrote:

Hello,

I've tried to use Twisted names.client.getHostByName() method to resolve
some addresses for my UDP based app. By default getHostByName() always
returns IPv6 address. I don't always have end-to-end IPv6 connectivity,


I'm not sure if t.n.client implements the same stuff that getaddrinfo 
does, namely RFC 6724 address selection rules and behaviour, and in 
particular if it has an equivalent of the AI_ADDRCONFIG flag.



After Twisted 13.2 announcement I've checked the new HostnameEndpoint
implementation to find possible solution (it seems to be dedicated to
solve similar problems, but for TCP-based protocols). However it seems
to use deferToThread(), and built-in Python getaddrinfo().

My questions:
1. Is using deferToThread() a preferred solution?


IMO *all* applications should *always* use getaddrinfo() for resolution 
of user-supplied names. It really really annoys me when people decide to 
re-invent it - this is hard to get right, and apps like Chrome having 
their own implementation precludes things like DHCP-based 6724 tables in 
the future.


So yes, use deferToThread and getaddrinfo. Note that you should not, 
in general, specify the address family; let the AI_ADDRCONFIG flag sort 
that out for you.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Reliable way to check if Twisted has IPv6 support?

[Phil Mayers]

On 13/12/13 03:22, Amit Saha wrote:


Is there a more reliable way (which works with the Twisted-8.0+) to check this?


Check the Twisted version.

You should also note that IPv6 features have appeared in multiple 
versions; HostnameEndpoint was added in 13.2.0 for example, whereas 
basic connectTCP to IPv6 address was added in 12.1.0, with listenTCP 
in 12.0.0


So:

import twisted

vt = (twisted.version.major, twisted.version.minor)

if vt  (13,2):
  def connectv6name(n):
return HostnameEndpoint(...)
elif vt  (12,1):
  def connectv6name(n):
d = myGetAddrInfo(...)
d.addCallback(reactor.connectTCP, ...)
else:
  def connectv6name(n):
raise Exception('twisted too old for IPv6')

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Queries about connecting to a XML-RPC server over IPv6

[Phil Mayers]

On 12/12/13 12:23, Amit Saha wrote:



- Original Message -

From: Amit Saha as...@redhat.com
To: Twisted general discussion twisted-python@twistedmatrix.com
Sent: Tuesday, December 10, 2013 11:59:27 AM
Subject: Re: [Twisted-Python] Queries about connecting to a XML-RPC server over 
IPv6



- Original Message -

From: Phil Mayers p.may...@imperial.ac.uk
To: twisted-python@twistedmatrix.com
Sent: Monday, December 9, 2013 7:33:17 PM
Subject: Re: [Twisted-Python] Queries about connecting to a XML-RPC server
over IPv6

On 09/12/13 05:28, Amit Saha wrote:


proxy = Proxy('http://localhost6:8000')
proxy.callRemote('my_proxy_method').addCallbacks(printValue, printError)


When I run it, i get No route to host: 101, Network is unreachable.

However, 'curl -6 localhost:8000' succeeds. What could be going on here?


The IPv6 support in Twisted is very new - check your version even *has*
it - and it's still a work in progress. You might find that you can't do
this. I'd have to read the code to be sure and I don't have time right
now, but my guess is that t.w.xmlrpc isn't getaddrinfo-ised so won't
connect to IPv6 names.

If your version of Twisted supports it, you could probably work around
this by sub-classing Proxy and calling reactor.connectTCP('::1', 8000)
to attach the protocol yourself.


Thanks for your reply, Phil. So, I basically modified twisted/web/xmlrpc.py
so that
when it was calling connectTCP(), i substituted '::1' in place of self.host
and yes the method is called successfully.

I will consider your other hints and see if there is an easy way to work
around this
in my particular use case.


I finally ended up doing this:


Just to point out this is obviously a pretty hackish way of doing it; 
for starters, getaddrinfo() is blocking, so you should use deferToThread 
at minimum.


You might want to look at:

http://twistedmatrix.com/trac/browser/tags/releases/twisted-13.2.0/twisted/internet/endpoints.py#L707

...for inspiration.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] ISSLTransport.getPeerCertificate returning None?

[Phil Mayers]

On 09/12/13 22:19, Laurens Van Houtven wrote:


... but in the server's AMP protocol,
self.transport.getPeerCertificate() is still None :/


Unless you set the appropriate verify options on the server side too, 
the client cert isn't available for inspection.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] ISSLTransport.getPeerCertificate returning None?

[Phil Mayers]

On 10/12/13 13:21, exar...@twistedmatrix.com wrote:

On 08:41 am, p.may...@imperial.ac.uk wrote:

On 09/12/13 22:19, Laurens Van Houtven wrote:

... but in the server's AMP protocol,
self.transport.getPeerCertificate() is still None :/


Unless you set the appropriate verify options on the server side too,
the client cert isn't available for inspection.


twisted.internet.ssl.CertificateOptions sets these by default.  If you
use it, you shouldn't need to do anything else.


Are you sure about that? It does not match my recollection.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Queries about connecting to a XML-RPC server over IPv6

[Phil Mayers]

On 09/12/13 05:28, Amit Saha wrote:


proxy = Proxy('http://localhost6:8000')
proxy.callRemote('my_proxy_method').addCallbacks(printValue, printError)


When I run it, i get No route to host: 101, Network is unreachable.

However, 'curl -6 localhost:8000' succeeds. What could be going on here?


The IPv6 support in Twisted is very new - check your version even *has* 
it - and it's still a work in progress. You might find that you can't do 
this. I'd have to read the code to be sure and I don't have time right 
now, but my guess is that t.w.xmlrpc isn't getaddrinfo-ised so won't 
connect to IPv6 names.


If your version of Twisted supports it, you could probably work around 
this by sub-classing Proxy and calling reactor.connectTCP('::1', 8000) 
to attach the protocol yourself.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Queries about connecting to a XML-RPC server over IPv6

[Phil Mayers]

On 09/12/13 13:28, Glyph wrote:

On Dec 9, 2013, at 1:33 AM, Phil Mayers p.may...@imperial.ac.uk
mailto:p.may...@imperial.ac.uk wrote:


The IPv6 support in Twisted is very new - check your version even
*has* it - and it's still a work in progress. You might find that you
can't do this. I'd have to read the code to be sure and I don't have
time right now, but my guess is that t.w.xmlrpc isn't
getaddrinfo-ised so won't connect to IPv6 names.

If your version of Twisted supports it, you could probably work around
this by sub-classing Proxy and calling reactor.connectTCP('::1', 8000)
to attach the protocol yourself.


Worth noting, of course, that Proxy not working with IPv6 is a bug.


Opened #6870

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Queries about connecting to a XML-RPC server over IPv6

[Phil Mayers]

On 09/12/13 14:54, Phil Mayers wrote:

On 09/12/13 13:28, Glyph wrote:

On Dec 9, 2013, at 1:33 AM, Phil Mayers p.may...@imperial.ac.uk
mailto:p.may...@imperial.ac.uk wrote:


The IPv6 support in Twisted is very new - check your version even
*has* it - and it's still a work in progress. You might find that you
can't do this. I'd have to read the code to be sure and I don't have
time right now, but my guess is that t.w.xmlrpc isn't
getaddrinfo-ised so won't connect to IPv6 names.

If your version of Twisted supports it, you could probably work around
this by sub-classing Proxy and calling reactor.connectTCP('::1', 8000)
to attach the protocol yourself.


Worth noting, of course, that Proxy not working with IPv6 is a bug.


Opened #6870



Also #6871 for IPv6 literal URLs, but maybe the whole xmlrpc code should 
be moved to the new web.client code, and move all that logic in there?


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] carbon-relay eating CPU - EAGAIN (Resource temporarily unavailable)?

[Phil Mayers]

On 26/11/13 04:05, Quan Tong Anh wrote:

  - Python 2.7.3
  - [twisted, version 13.1.0]
  - xen-domU


I assume xen-domU means Linux as the guest OS; which kernel version, 
out of interest.




`atop` shows that `carbon-relay` is eating 80, 90% USRCPU. From the


I don't know about anyone else, but I don't know what carbon-relay is. A 
quick google suggests it's part of that graphite monitoring/RRD 
replacement thing; which version are you running (Or: what reactor is it 
using)?


How many open files do you have?



`strace`:


Without timestamps it's difficult to interpret this unambiguously; that 
could be a microsecond or minute we're seeing ;o)



The strange thing is: even restart the service, it seems stuck at fd 7
everytime running `strace`. Does it mean this fd is not being cleanup
properly?


No. fd numbers are allocated starting from 0 and going upwards. The same 
process running the same startup code every time will tend (not always, 
since it's order-dependent) to use the same file descriptors for the 
same things, modulo close/open events.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] txThings - simple library for CoAP protocol

[Phil Mayers]

On 23/11/13 12:03, Maciej Wasilak wrote:


3. There is a problem with txThings IPv6 server, because in most
operating systems source address of datagram is determined in round
robin fashion. It may cause the source address of the response to be
different than destination address of the request.


This is in general a bit of a pain with UDP apps.

In Twisted, the easiest solution is to specifically bind the UDP sockets 
to the IP address(es) on the host, and always reply on the same 
protocol/transport you received on.


You may find the netifaces package helpful for getting a list of IPs 
in a cross-platform manner.


In theory you can avoid doing this if you have sendmsg/recvmsg support, 
but Twisted doesn't have this (yet).


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] epoll keep sharing state between process even after fork.

[Phil Mayers]

On 23/10/13 16:46, Flint wrote:

Hi everybody

I a came across a surprising problem when using the epoll based reactor.
(ticket here https://twistedmatrix.com/trac/ticket/6796)

As you can see on the ticket, the epoll object seems to share some state
even after fork.
Which means even after having forked the process, making some changes to
the inherited epoll object in one process may impact the one existing in
another process !

This problem is only related to epoll based reactor, poll and select
behave correctly.
I'm aware that some may say that this in not a twisted related problem
(but an epoll issue), but I'm addressing my question here to try to
figure out what would be the best workaround for this problem in a
twisted based project ?


The problem is that you're sharing the epoll object (and thus the 
underlying epoll FD, and associated state) with the multiprocessing 
child processes.


Either:

 1. Don't use multiprocessing
 2. Arrange for the epoll object (or FD) to be closed after fork, but 
before exec, so that the child process can't fiddle with it

 3. Create the epoll object after starting the multiprocessing children

IMHO Twisted and multiprocessing are not good together.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] epoll keep sharing state between process even after fork.

[Phil Mayers]

On 23/10/13 17:39, Phil Mayers wrote:


  2. Arrange for the epoll object (or FD) to be closed after fork, but
before exec, so that the child process can't fiddle with it


See also:

http://bugs.python.org/issue8713

...which suggests Python 3.4 added fork+exec support to multiprocessing. 
On Unix and older Python versions, you're stuck with plain fork and all 
the attendant horribleness.


This is a multiprocessing bug IMHO.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted as a full solution for web hosting [WSGI + other]

[Phil Mayers]

Does the Twisted wsgi runner have support for process rather than threading 
models to avoid the gil? Because if so I should take a look at it...
-- 
Sent from my phone with, please excuse brevity and typos___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] AutobahnPython 0.6.3 - WebSocket compression and more

[Phil Mayers]

On 10/07/2013 08:51 AM, Tobias Oberstein wrote:


I did some further looking around: turns out there is TLS-PGP

http://tools.ietf.org/html/rfc6091

Does someone know whether OpenSSL supports that?



There are *lots* of TLS extensions that eliminate or obviate the need 
for the (horrible) PKIX trust model as deployed. For example, TLS PSK, 
TLS-SRP, the PGP method you've found, and others.


Right now, none are useful in a browser, but personally I have high 
hopes for raw keys, trust-anchored by DNSSEC via RFC 6698. In this 
model, X.509 is essentially just a payload format for certs - the entire 
trust model is unused.



[Sidenote: if not, one more reason why a pure Python TLS


Such as tlslite?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] AutobahnPython 0.6.3 - WebSocket compression and more

[Phil Mayers]

On 10/07/2013 09:50 AM, Phil Mayers wrote:


Right now, none are useful in a browser, but personally I have high
hopes for raw keys, trust-anchored by DNSSEC via RFC 6698. In this
model, X.509 is essentially just a payload format for certs


Sorry, payload format for keys.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] AutobahnPython 0.6.3 - WebSocket compression and more

[Phil Mayers]

On 07/10/13 11:56, Donald Stufft wrote:

DNSSEC solves none of the problems with the CA system. It just moves
the problem around.


Disagree.

However - there are other, better forums to have this argument in (and 
to be frank, I've no interest in having it at all) so I won't respond 
further.


I would however urge other people reading this to look into the issues 
and decide for themselves.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] AutobahnPython 0.6.3 - WebSocket compression and more

[Phil Mayers]

On 07/10/13 12:35, Tobias Oberstein wrote:


DNSSEC seems to follow a centralized/hierachical trust model. Won't
help. The NSA will (does?) own those.


The default trust model is to have parent sign the child. Other models 
are not only possible, they're deployed. Google DLV and trust anchor.


As to whether the NSA has the root keys; given recent revelations I 
rule nothing out. But if this is a concern, I would urge you to 
investigate and get involved in the root key generation and rollover 
procedures - there is a rollover coming soon, and more eyes make 
subversion less likely.



That could be a good start: it would take a community effort to
scrutinize, security review and robustify for production.

The monoculture of OpenSSL is no good IMHO.


I agree, but there are other options - gnutls, NSS - which have received 
this scrutiny, if you want to move away from OpenSSL.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] AutobahnPython 0.6.3 - WebSocket compression and more

[Phil Mayers]

On 07/10/2013 18:58, Glyph wrote:


If you have a disagreement, please say /what the disagreement is/ (not
just disagree) and then link to resources instead of abstractly
claiming people may find them themselves somehow.  You don't have to get
into a big back-and-forth, but I believe DNSSEC implementation work is
proceeding in Twisted so it would be good for the community to be aware
of these issues.


Ok. Don't say I didn't warn you. Apologies to anyone who forces 
themselves to read all this, and please note I DO NOT claim authority on 
any of this - I'm just a random guy. People should decide for themselves.


Donald wrote:


DNSSEC solves none of the problems with the CA system. It just moves the 
problem around.



I believe I understand why he said this, but I think it's incorrect to 
say that DNSSEC solves none of the problems. I think it solves many 
(though not all) of them, and that supplementary systems - which are 
desirable in and of themselves - can take us the rest of the way.


Here's my reasoning:

I think it's fair to say that the PKIX trust model is known to have some 
serious flaws. They basically stem from the existence of far, far too 
many CAs, and the lack of constraint on what a CA can issue for.


There are various proposed solutions to this problem. DNSSEC-signed TLSA 
records (DANE; RFC 6698) offer one solution to the problem - to put 
hashes of certs/keys or issuer/chain certs/keys in DNS, and sign then 
with DNSSEC. This accomplishes two things:


 1. Reduction in the number of trusted CAs. To what degree depends on 
what deployment model you use - you can put a specific key/cert into 
DNS, or one or more traditional X.509 issuers that allowed to sign for 
you. The latter in particular seems likely to be common - reduce the CAs 
that can sign for blah.example.com from ~1500 by putting the hashes of 
the 1/2/N good CA certs (most specific parent!) into the 
blah.example.com TLSA records.


 2. Constraint - a DNS zone-signing key can only sign records below it, 
terminating at secure child delegations (DS keys).


The obvious objection to this solution is the necessary trust in the DNS 
root / parent zones. There are, it seems, people who are not willing to 
grant this trust. I understand that - particularly in light of recent 
revelations.


However: government agencies are not the only people who might want to 
get certs in the name of a 3rd party. Crime syndicates attacking the 
latest race-to-the-bottom CA to get esome unicode glyph like 
xample.com certs are a real issue, and DANE can handle these just fine.


There are arguments that DANE moves the trust problem from CAs to 
registrars and registries. I must admit, I don't understand the threat 
model here - it's asserted that registrars are cheapcheerful (true) but 
they only handle public key material, and don't run the DNS; the 
registry is a more fruitful target, but validation of the public key 
material they publish for you is trivial (whereas proving that a CA 
somewhere hasn't signed a cert for your domain is not).


In short, I think it's a significant net win, and as a side benefit 
offers greatly reduced key management burden. The ability to publish and 
revoke TLSA records at significantly lower cost than X.509 cert 
issuance, and without need to interact with a 3rd party, would be 
valuable even without the above. It could in theory help decouple TLS 
from X.509 - useful if you want to move to PGP, for example.


However, some people don't agree. Moxie Marlinspike discusses the 
general issues and makes an argument against DANE - see:


http://www.thoughtcrime.org/blog/ssl-and-the-future-of-authenticity/

...and the video of the talk here:

http://www.youtube.com/watch?v=Z7Wl2FW2TcA

He essentially suggests a trusted notary system - a working 
implementation of which you can see here:


http://convergence.io/

I agree that this approach is promising. I am not super-confident that 
it will take off - end-users literally DO NOT CARE about these issues 
until it's too late - but if someone (Chrome, Firefox) starts bundling 
notary functionality and prompting users to trust one or more of a 
(randomly shuffled) set of notaries on startup, it might... However, I'm 
not sure how likely that is - see:


https://www.imperialviolet.org/2011/09/07/convergence.html

[Note Adam Langley is Google TLS person]

For an alternative take on the problem, see:

http://www.certificate-transparency.org/

People interested in reading a pro-DNSSEC PoV should look here (warning 
- sequence of posts):


http://dankaminsky.com/2010/12/13/dnssec-ch1/

...and here:

http://www.slideshare.net/dakami/black-opspki-2

So - not as short as I'd like, but as short as I could make it. I hope 
that makes my position clear.


Finally a note on why I was reluctant to respond...

In my experience, discussions on TLS, X.509, DNS and DNSSEC can become 
very emotive, very quickly. There are people who care very, very deeply 
about the minutae of these 

Re: [Twisted-Python] twisted linux netfilter_log protocol

[Phil Mayers]

On 10/02/2013 07:14 AM, David Stainton wrote:

Hi,

I am wondering how I should combine various twisted interfaces
in a way that makes sense. It probably shows here... that I'm new to
twisted.

The Linux iptables can log packets to the netfilter_log which can give
access to user space.
I wrote a simple twisted Reader (IReadDescriptor implementation) that is
working functional code...
https://gist.github.com/david415/6789612

But since these are packets it returns... should I implement a read
only Protocol?


Well, if your transport is not writable, just don't implement that - 
Exceptions will be raised if you mistakenly try to write, so nothing bad 
will happen.



The protocol's dataReceive() method could be called from the
NFLogReader's doRead() method.


Since they're packets it should really be a DatagramProtocol and call 
datagramReceived.



Does this mean that NFLogReader would be responsible for calling
buildProtocol to construct the NFLogProtocol?


Typically a factory is responsible for calling buildProtocol, but in 
most datagram uses, there is only one protocol instance per port, so you 
just instantiate it. See e.g.


http://twistedmatrix.com/documents/current/core/howto/udp.html


In the normal Twisted examples the buildProtocol seems to be called from
the react loop..
Should I pass the protocol factory and the reader to the service?


Sorry I don't know what this means.



If I want to drop privileges right after retrieving the filedescriptor
for netfilter_log, shall I use a Twisted Service/Application to drop the
root privs to a non-superuser?


That would be the Twisted way of doing it, yes.


Should the privileges be dropped by the privilegedStartService() method?
The manual says that method is for preparing to drop services...

What I want out of all of this is the most high performance and general
solution to utilizing netfilter_log in twisted...


Couple of general points:

The nflog_cffi code is... hard to read. But it's not obvious to me that 
it sets the FD to nonblocking anywhere, or that it's (frankly weird) 
generator construct will actually yield the nonblock marker object at 
the right times. You might want to test this.


On a different note, a colleague wrote some code to process the IPQUEUE 
target of iptables a while back. This code did something similar to your 
example above, but it didn't bother integrating with the 
protocol/factory machinery, and you could consider whether it makes 
sense to do so - unlike a TCP or UDP port, you're unlikely to ever have 
1 nflog connection per-process, so it's not obvious that using the 
full generality of the factory/protocol machinery is appropriate.


However if you do that, I would just copy what Twisted does for UDP; 
don't model it on TCP.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted linux netfilter_log protocol

[Phil Mayers]

On 02/10/13 09:25, David Stainton wrote:


In examples of Twisted UDP clients they all do this:

reactor.listenUDP(3000,MyProtocol())

What would I do instead?
reactor.addReader makes sense to me since what I have is a file descriptor,
and the reactor loop and use select or epoll find out when there is data
to read...
and call our doRead() method of the Reader.

I don't know about implementing this as a Protocol because I do not have any
equivalent to ReactorUDP. ReactorUDP's listenUDP returns a ListeningPort.
And I'm not sure how a Transport is related to my Reader... but it seems
to me
that there is some abstraction which sets callbacks for io and also
calls buildProtocol...


Sorry, maybe I was unclear: you were asking about protocols, so I was 
merely pointing out that *if* you were to use a protocol, you should 
copy the approach DatagramProtocol and udp.Port take.


However as I noted, it *might not* make sense to do this, if there's 
only ever going to be one socket listening on this thing, your current 
approach might be fine.



If I am going to be using a Twisted Service to drop privileges after
gaining access to the fd...
How would I do this? Would I have startService() instantiate a Protocol
instance?


As above, it depends. If you're using Protocol instances, then yes. If 
not, then no.


Services are just things which start and stop; they may or may not use 
objects implementing the transport/factory/protocol pattern.




I don't know about the nflog file descriptor...
but the nflog generator interface is non-blocking if used like this:


Ok. Weird API, but if it works...


My nflog reader seems to work great.


If it works for you, and you're using documented Twisted interfaces, 
then there's probably no need to fiddle with it.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Conch - Why does one piece of code fail while another works

[Phil Mayers]

On 02/10/13 15:00, exar...@twistedmatrix.com wrote:


BTW, I'm not sure what you plan to do with this connection, but if you
haven't seen `twisted.conch.endpoints` (and https://tm.tl/6617) then
you probably want to look.


ZOMG I had not seen the (existing) Conch endpoints before. Amazing, and 
many thanks to whoever did this work.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] raw socket IWriteDescriptor

[Phil Mayers]

On 10/01/2013 08:30 AM, David Stainton wrote:


Hi,

I'm trying to implement an IWriteDescriptor using raw sockets.
So far this is what I wrote... but it doesn't work:
https://github.com/david415/hushTCP/blob/master/hush_writer.py

Am I doing something obviously wrong here?


I don't see you trying to add the raw socket to the reactor anywhere; 
typically you'll want to do at least reactor.addWriter(hush), but for a 
proper implementation there's more - see the IReactorFDSet docs.


You may want to look at the udp code in Twisted to see how this is done 
under the hood (although TBH it can be hard to follow!)



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] question about subprocess

[Phil Mayers]

On 09/27/2013 02:51 AM, Jonathan Vanasco wrote:


i was looking at reactor.SpawnProcess -- which I know is more
correct.  it just seems to be a bit annoying to use


Are you aware of twisted.internet.utils.getProcessOutputAndValue? For 
simple spawn/check cases, that's pretty much always the best option IMO.


If you have a case where you need to write to a child process - well, I 
don't see how you hope to avoid using spawnProcess and a 
ProcessProtocol; like others I'd be interested to hear what you dislike 
about the API (I have problems with a bunch of Twisted APIs, but that 
one I quite like ;o)


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Clean pb solution for two-way object sync?

[Phil Mayers]

On 09/27/2013 05:48 AM, Daniel Sank wrote:

Upon re-reading the Cacheable docs I still don't understand how to use
it.


Have you seen this:

http://twistedmatrix.com/documents/current/core/howto/pb-copyable.html#auto9

Essentially, you move all attribute access to accessor methods and do 
callRemote to propagate the changes out to observers; new observers are 
passed to you in getStateToCacheAndObserveFor.


Observers respond to observe_xxx methods, and implement a 
setCopyableState method.


Then you map the cacheable and observer with pb.setUnjellyableForClass

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Passing additional arguments to errback

[Phil Mayers]

Maciej Wasilak wasi...@gmail.com wrote:
Hello,

just one more question:

Well, yes, you should, but still keep in mind that you can actually
just
 pass the request like so:

 .addErrback(handleErrors, request)

 You don't really have to save the request itself anywhere as an
attribute.


if I understand correctly, when I put this instruction into my
callback,
then the errback will catch errors only from the callback, and not from
agent.request, right? Such code seems hard to debug. Is this the
standard
procedure in Twisted to add callbacks/errbacks inside other callbacks?
Best Regards
Maciek




___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

No it'll catch errors from anything previous in the chain - either the agent 
call or the callback. If you read the docs carefully this should be clear.
-- 
Sent from my phone with, please excuse brevity and typos___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted serialport dataReceived() provides fragmented data

[Phil Mayers]

On 29/08/13 17:15, Sivakumar Balasubramanian wrote:


Dear Phil,
I replaced thye dataReceived function by stringReceived, and ran the code.
But when I do this, the program does not even enter the stringReceived function.


That implies your protocol isn't an int16string-prefixed one. I assumed 
it was, since you'd inherited from that class.


Obviously you will need to pick a base class that matches the actual 
structure of your protocol.




I also tried a modified version of the programs above with LineReceived
protcol, and here too I have the same problem of fragmented data from the
dataReceived function.


Well, *again* you shouldn't be overriding dataReceived if you're 
inheriting from LineReceiver. You should be overriding lineReceived.


To be clear: dataReceived gets raw data. It can, and *will*, get 
incomplete messages, because it doesn't know anything about messages.


If you have a protocol with a message structure, you either need to:

 1. Inherit from an existing base-class which handles the message 
framing, and implement the msgReceived handler specific to that 
protocol.


 2. Or, if there's no base-class matching your message framing, write 
your own - this will entail buffering the data, then extracting messages.


See here:

http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#Whyisprotocol.dataReceivedcalledwithonlypartofthedataIcalledtransport.writewith

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Advice on porting Python application to Twisted

[Phil Mayers]

On 28/08/13 14:40, Tom Sheffler wrote:


Hi Matthew -

I have a couple of idioms I use for #2 and #3 in your message.  Here
they are.

#2)

For timer events, I create a function that when called, continuously
schedules itself again in the reactor, does some work for the curren


Are you aware of t.i.task, specifically LoopingCall and cooperate/coiterate?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Serial transport protocol logging on Raspberry Pi

[Phil Mayers]

On 08/24/2013 06:37 PM, Lucas Taylor wrote:


9600bps, using USB-Serial adapters (tried 3 different supported
chipsets). Logging to local storage and also to stdout.


But presumably stdout is going somewhere - either a local or network 
terminal or file redirect.


Could you try a simple RAM-based log observer (append all log messages 
to python list)? Just out of curiosity ;o)



At any rate, I don't think there is much to be done on the Twisted
side. I have a few minor things left to try, but I have a feeling my
curiosity about the root cause of this is going to disappear in favor
of ponying up for a more substantial piece of hardware...


I hear good things about the beagleboard, but haven't used one myself.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Serial transport protocol logging on Raspberry Pi

[Phil Mayers]

On 08/23/2013 06:39 PM, Lucas Taylor wrote:


I suppose the obvious answer may be that the Pi is underpowered for
this task, but I am hoping to better understand why.  I may be able
to defer logging to a queue or find other workarounds, but I'd like
to figure out why basic usage of logging could be causing this.  Any
thoughts as to what I could be looking at next would be appreciated.


What is the data rate, and the nature of the serial ports (e.g. UART or 
USB?)


Your suggestion of CPU rings a little bell for me; IIRC the Pi had some 
(ahem) sub-optimal behaviour w.r.t. the USB controller in some hardware 
versions. There was some mild controversy over this, with various 
accusations slung back and forth in the forums, people getting banned 
for continuing to talk about it and so on. But I seem to recall timely 
data delivery under CPU load was a symptom, and it might be the overhead 
of Twisted/Python/logging drives the Pi too hard.


Where is the logging directed - I'm assuming NV storage which is the SD 
card? - you might try logging to RAM and not touching the storage or bus.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Setting socket options before connect

[Phil Mayers]

On 18/07/13 13:10, exar...@twistedmatrix.com wrote:


It wasn't so much an API as a Hey, I have an idea... endpoints... here
is an example I can think of in 10 seconds. :)


Understood!


I guess it might be ok if there was a way to reliably inject unknown
options with arbitrary payloads, but I'm struggling to see a clean way
to do this with a parse a string-style API.


I think you're talking about the fact that sockopts are random
integers associated with other big piles of random integers.  Some of
them are flags you turn on, but some come with random payloads of
basically no possible known shape.


Yep.


It sounds like you're trying to think of an API that will support any
and all socket options without understanding them.  This API exists
already.  It is `setsockopt`.  There's no reason to re-invent it.


Sure.


I prefer the approach taken elsewhere in Twisted, where a particular
option is given some consideration and an API that understands the
option is introduced.  This approach certainly has its shortcomings -
for example, it doesn't support arbitrary options. :)  Do people really
like using `setsockopt` though?


No-one likes it, but if you need to set a socket option, you don't 
really have much choice (absent a higher-layer API).



So maybe the correct way to handle this is implement your own FD
object?


Nothing stops anyone from doing this already, today.  Except that it's a
lot of work and no one seems to want any of these features badly enough
to do it.


Well, exactly. It's currently easier for me to sub-class a Twisted 
internal/private class. But if that goes away, that option is 
unavailable and a replacement would be nice - even if it's a convenience 
API like:


connectExistingSocket(...)

Isn't there code to do this as part of the systemd activation?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] logging

[Phil Mayers]

On 06/15/2013 06:48 AM, Christopher Armstrong wrote:


There's... a lot here. I'm overwhelmed by how complex this system
would be, and I think that end users would be as well. I don't really
want to put UUIDs into my source code, whether I type them or paste
them, and I don't want to have to invoke command line tools to figure
out what those UUIDs in order to be a good developer.


+1. I like the general idea of better logging via keyword args and an 
id, but I strongly dislike the idea of using UUIDs as log message 
identifiers.


In fact, I'll go so far as to say I defintely wouldn't use such a 
system, and for Twisted to migrate to it would actually reduce the 
utility of the in-built logging, because I'd have to build and maintain 
something to translate Twisted log messages I cared about into whatever 
system I built to do my own logging :o(



I think there's basically no practical benefit over the hierarchical
system + event-identifier system, where:


+1. Hierarchial IDs also permit prefix-based wildcarding:

log.observe(twisted.factory.*)
log.exclude(twisted.factory.http.*)
log.observe(myapp.thing)
log.observe(txSomeProject.event.*)

Obviously this is just an example; I haven't put any real thought into it.



1. it's trivial to specify a hierarchical, easy-to-read system key
that has a small-ish scope
2. you think of a unique event name in that small-ish scope and type it out.

I'm beginning to think #1 is the most important of all, but I think
I'll continue to use event-names to describe all my log statements.


I pretty much agree with all of this.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] github, again

[Phil Mayers]

On 06/06/13 17:08, Jonathan Vanasco wrote:


the only downside to git, is that once something goes onto the
server... it's there for good.  it's possible to rebase a repo back
to a specific commit , then replay without specific commits, and
push -f to overwrite the history... but if anyone updated against
the server, those commits will come back and haunt you.  over and
over and over again.


Yes. Never do this. It's hateful to fix, and rude to contributors!

(Obviously not an issue if your git repo will just be a read-only copy 
of SVN used to drive git-based code review tools)


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted server blocking on MySQL read in main thread when using adbapi.ConnectionPool

[Phil Mayers]

On 05/21/2013 01:47 AM, Dan Williams wrote:
 The invocation is simply py-bt, but it only works if symbols are
 available, which aren't in my case.

Really, surely Ubuntu LTS has debug symbol packages or something? A 
quick google suggests its packagename-dbg for stripped symbols.

I meant to reply with this last night: knowing very little about MySQL 
or it's python bindings, is it possible you're returning an object from 
runInteraction that's still attached to the DB e.g. a BLOB that 
lazy-loads on references, or something?

In the absence of a python traceback it might be the age-old add prints 
everywhere and see where it stops dance...

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted server blocking on MySQL read in main thread when using adbapi.ConnectionPool

[Phil Mayers]

On 05/21/2013 08:02 AM, Phil Mayers wrote:
 or it's python

its/it's agh. It's early for me...

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Release questions

[Phil Mayers]

On 04/03/2013 10:58 PM, Laurens Van Houtven wrote:

 Is the accidental corruption thing a real risk? I thought that was the
 point of, say, TCP checksums :) Perhaps I'm just mistaken as to how
 often his happens in the wild...

TCP checksums absolutely can and do fail to protect you from in-flight 
corruption of data. I've seen this on several occasions over links with 
non-zero BER (i.e. any real-world link). A two-bit error in a single 
packet will not be identified unless you're using HTTPS or SSH or 
similar, with strong transport-layer hashes.

FWIW I saw it with very, very large downloads but it can obviously 
happen with smaller data, just with a lower probability.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to dispatch message to different servers

[Phil Mayers]

On 03/01/2013 10:27 PM, Glyph wrote:

   What is AMP too much of?  Memory?  CPU?  Bandwidth?  API complexity?
   Code size?  As compared to what?

FWIW, every time (all three of them...) I've looked at AMP I got rapidly 
bored and ended up using PB. My use-cases have usually been 
Twisted-Twisted, and the AMP Command definition, and syncing said 
defs up at both ends, was hassle I didn't need.

When I need to talk to external systems I almost always use some REST, 
XMLRPC or JSON-RPC interface, because they're usually available 
out-of-the-box.

I also don't care for the idea of statically typing the command defs; 
I'm sure this helps when talking to languages with crappy typing 
systems, but honestly - who uses Java these days? ;o)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to dispatch message to different servers

[Phil Mayers]

On 03/02/2013 12:19 PM, Glyph wrote:

 I wrote a blog post some time ago explaining why you want static
 declarations, even if your programming language is all nice and dynamic:

Being able to early drop bad PDUs is a fine notion, but it's not a 
property that comes with zero cost. If you feel that the cost/benefit 
tradeoff is not useful in your case (and I do) then it doesn't make 
sense to pay it.

AMP would be somewhat more attractive (to me) if it were possible to run 
with the IDL just on the server (i.e. callee); this would achieve the 
goal of protocol-level sanity checking, without the cost of having to 
distribute the IDL to the clients. Presumably it would be possible to 
implement this now - have an AMP discovery command that dynamically 
builds the client-side IDLs. But there's no incentive for me to do that.

For the use-cases I have, the protocol version/interop problem is not 
relevant.

 not full replacements for AMP because they're asymmetrical and don't
 support pipelining.  With AMP, a client can receive requests from a
 server, you don't need to have a TCP handshake for every request, you

You don't need to do a TCP handshake for every request in any of those 
protocols either. Long-lived TCP/HTTP sessions are quite common, though 
not universal.

It will be interesting to see if and how HTTP/2.0 solves a lot of these 
issues.

 can receive responses out of order, and you can receive a request while
 waiting for a response, and so on.

Obviously. But I would tend to use those protocols when either those 
properties are not needed (perhaps even undesireable) and may be 
impossible to make use of (if the far end is a synchronous, blocking 
client unable to make use of out-of-order responses or server-initiated 
messages).

In those cases, it would also be rare for AMP to be available, and it 
certainly wouldn't be pre-installed.

Install this Java library on your Oracle servers then...
Ha ha no go away.

You seemed curious why someone wouldn't use AMP. Personally I have two 
common use-cases:

  1. Communicating between two trusted Twisted processes, for which PB 
is good enough.

  2. Communicating with a semi-trusted third party process, for which a 
RESTful interface or XML/JSON RPCs are *available* and sufficiently 
secure and capable for my needs.

Obviously other people will have radically different needs, but for me, 
AMP adds little.

Cheers,
Phil

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to dispatch message to different servers

[Phil Mayers]

On 03/02/2013 03:24 PM, Tristan Seligmann wrote:

 I don't really understand how this is any easier with PB than with
 AMP, though. With AMP, you just need the same command definitions on
 both sides. With PB, you need the exact same Python code versions for
 every class you are sending over the wire;

I don't use PB like that. I only ever send simple data structures made 
of builtin types (int, str, list, dict).

 You don't quite have to do a TCP handshake for every request you want
 to send over an HTTP channel, but there are still some problems.

You're not telling me anything I don't already know.

My point is that, for the use-cases *I* have, those limitations have not 
proven to be a problem, so the effort

 asymmetric (client makes requests to the server, server cannot make
 requests to the client, so you need to do some kind of long polling /
 inversion of control to simulate this).

Yes.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to dispatch message to different servers

[Phil Mayers]

On 03/02/2013 04:36 PM, Phil Mayers wrote:

 My point is that, for the use-cases *I* have, those limitations have not
 proven to be a problem, so the effort

Sorry, should have been:

...so the effort of deploying a protocol which avoids those limitations 
is not warranted.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Building a TCP server, having issues

[Phil Mayers]

On 02/17/2013 01:09 AM, Adam Heller wrote:
 Glyph,

 I will definately look into that.  As far as the server spec, it's over
 TOR (This part of the 'competition' has since closed, but feel free to
 catch up: http://uncovering-cicada.wikia.com/wiki/Uncovering_Cicada_Wiki
 and
 http://mentalfloss.com/article/31932/chasing-cicada-exploring-darkest-corridors-internet
 (from last year's 3301).  I'm simply trying to do the python thing and
 move classes around instead of trying to reinvent the wheel [dh() is a
 rudimentary Diffie Helman key exchange] : 

 In the programming language of your choice build a TCP server
 that implements the protocol below. The server code must be written by
 you and you alone, although you are free to use any modules or libraries
 publicly available for the selected programming language. Once you have
 done this, make it accessible as a Tor hidden service. Then provide us
 with the onion address and port via a GPG-encrypted email to this
 address. You have until 0:00 UTC on 3 Feb, 2013. Any emails received
 after that time will be ignored. Good luck. 3301
  1.
 INTRODUCTION The TCP server MUST listen on an arbitrary port, and send
 and receive plain text with lines separated by CRLF

Glyph has already mentioned the buffering and state machine options, but 
since it's a line-based protocol, you chould also look at 
t.p.basic.LineReceiver:

http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.LineReceiver.html

...which does the buffering for you and calls a method with full 
lines. You may want to look at the source for that class' dataReceived 
method, for an example of how it does the buffering Glyph talks about.

There are other examples of doing state machine dispatch in the Twisted 
sources, but it's a pretty simple technique - set a state variable, 
and a big if/then clause (or, for performance, a dict of state-handler)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted client memory leak

[Phil Mayers]

On 22/01/13 13:54, Diego Woitasen wrote:

 Ok, that's work. Thanks.

 My question is now, why is this done automatically for EchoProtocol()
 and not for EchoFactory()? Looks like the references are dropped why
 Python is taking too much time to frees the memory.

 An explanation around this is welcome :)

This kind of thing has been discussed on the list before, I think. If I 
recall, it's due to the behaviour of the Python GC under load.

It would help if you could say which version of Python and Twisted 
you're using.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted client memory leak

[Phil Mayers]

On 22/01/13 15:15, Tristan Seligmann wrote:
 On Tue, Jan 22, 2013 at 3:06 PM, Diego Woitasen di...@woitasen.com.ar wrote:
 I reproduce the problem with a modification of Echo client example
 from Twisted documentation:

 Adding __del__ methods causes the object to become uncollectable if it

Ah yes, well spotted.

Personally I tend to avoid __del__ in almost all circumstances, but 
particularly using Twisted (not because of anything Twisted-specific, 
but because my Twisted code tends to be *very* long-running, and because 
it's got sane died callbacks on most interfaces).

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted client memory leak

[Phil Mayers]

On 22/01/13 16:45, Facundo Batista wrote:

 Yes, but note that without the __del__ it had the same behaviour...


Not quite. The OP said that:

  a) He had a problem with a Twisted app not freeing memory under load
  b) He had reproduced that problem with his example, that included __del__

Nowhere did he say it does the same without __del__. He was in fact 
not specific about whether the original/real code uses __del__ or not.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted client memory leak

[Phil Mayers]

On 22/01/13 17:06, Facundo Batista wrote:
 On Tue, Jan 22, 2013 at 2:02 PM, Phil Mayers p.may...@imperial.ac.uk wrote:

 On 22/01/13 16:45, Facundo Batista wrote:

 Yes, but note that without the __del__ it had the same behaviour...


 Not quite. The OP said that:

a) He had a problem with a Twisted app not freeing memory under load
b) He had reproduced that problem with his example, that included __del__

 Nowhere did he say it does the same without __del__. He was in fact
 not specific about whether the original/real code uses __del__ or not.

 He didn't say it. I'm saying it, after testing it.


When you say it, you mean his example code as posted to the list, right?

Because having just tested it, I don't see any problem - his example 
code has stable memory usage for both client/server processes, and with 
the client making 1, 100 or 1000 connections/sec, and the debug shows 
protocol and factory instances being freed as I would expect i.e. in a 
timely fashion, not just at process close.

Tested on both:

Python 2.7.3 / Twisted 11.1.0 / Linux 64-bit
Python 2.6.8 / Twisted 12.2.0 / Linux 64-bit

Odd...

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Making ConnectionPool not pool

[Phil Mayers]

On 27/11/12 12:49, Hynek Schlawack wrote:

 Yep, and the connections *do* heal after dying. I don't have to restart
 twistd, it just takes a few failed requests to warm up again.

Yes - one attempt per thread in the pool.

It is a pain. But as per recent discussions on the list, there's no 
consensus about how to handle this.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Request for help: throttling serving of static content

[Phil Mayers]

On 05/11/12 22:26, Itamar Turner-Trauring wrote:

 There's also twisted.protocols.htb, although again it's old and possibly
 buggy and may not work. For demo purposes something like
 https://github.com/rflynn/dragnet might do.

If you're on Linux, netem is also an option.

http://www.linuxfoundation.org/collaborate/workgroups/networking/netem

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Create client in Server factory

[Phil Mayers]

On 10/31/2012 07:14 AM, Grégoire Leroy wrote:
 Hi,

 I have a server who receive data from clients A/B/C (remote machines). I want
 to use a client D (on the same machine than the server) to send this data to
 another server (remote).

 The difficulty is, I want to use the same client D connection for any client. 
 I
 don't want open an new connection each time.

 First thing I would think is to create the connection in my server factory,
 and use client's methods in my server protocol, for example with
 [...]
 class LocalProxyFactory(Factory):
  def __init__(self):
  f = LocalProxyClientFactory()
  reactor.connectTCP(retenodus.net, 4242, f)
  reactor.run()

The reactor.run is just wrong - remove it.

You need to connect to a server, and share this connection amongst some 
protocols. But you can't control the order in which these connections 
complete, so A/B/C might connect before D is ready.

You've really got two choices - accept the connections from A/B/C but 
have your server protocol wait until D is ready - something like this:

from twisted.internet.protocol import ClientCreator

class Server(...):
   def connectionMade(self):
 if self.factory.connD is None:
   self.factory.waitFor(self._ready)
   self.transport.pauseProducing()

   def _ready(self):
 self.transport.resumeProducing()

d_connect = ClientCreator(reactor, DProtocol)

class ServerFactory(...):
   def __init__(self):
 self.connD = None
 self._wait = []
 d_connect.connectTCP(Dhost, port).addCallback(self.dReady)

   def waitFor(self, _cb):
 if self.connD:
   _cb(self.connD)
 else:
   d = defer.Deferred()
   d.addCallback(_cb)
   self._wait.append(d)

   def dReady(self, proto):
 self.connD = proto
 cb = self._wait
 self._wait = []
 for c in cb:
   c.callback(proto)

def main():
   reactor.listenTCP(..., ServerFactory())
   reactor.run()

...or don't start listening until D has connected, like this:

class ServerFactory(...):
   def __init__(self, dProto):
 self.connD = dProto

def startListen(dProto):
   reactor.listenTCP(..., ServerFactory(dProto))

def main():
   d_connect.connectTCP(Dhost, port).addCallback(startListen)
   reactor.run()

The latter is simpler, but which is appropriate depends on your needs.

Note that I've used ClientCreator to get a callback when the connection 
to D is ready - remember that doesn't happen immediately.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Create client in Server factory

[Phil Mayers]

On 10/31/2012 02:31 PM, Kevin Horn wrote:

 See also:
 http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputononeconnectionresultinoutputonanother

 That example only uses a single connection in place of your clients
 A/B/C, but should be adaptable to your situation.


In fairness to the OP, he did say he'd seen that ;o)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] a question about monotonic clock

[Phil Mayers]

On 10/28/2012 10:16 PM, Tristan Seligmann wrote:
 On Mon, Oct 29, 2012 at 12:05 AM, Phil Mayers p.may...@imperial.ac.uk wrote:
 This depends on how you're running ntpd. If you have -x on the command
 line, yes - ntpd will not step.

 If not, there are circumstances it will step - clock diffs in excess of
 128ms iirc?

 If an offset of 128ms occurs at any time other than initial ntpd
 startup (which will presumably occur at system startup), that means
 you've either experienced a significant period of time without
 connectivity to time servers[1], you have a hardware / kernel issue
 that should be resolved, or some other software on the system is
 messing with the clock. Aside from network issues, the other
 possibilities are all serious issues that should be corrected, not
 tolerated as a normal situation.

Sadly, this is not the case. As has already been pointed out, virtual 
machine clocks can undergo stepping in normal oepration. A specfic 
example: if a VMWare installation performs live migration of a host. We 
often see:

ntpd[1793]: time reset +0.263757 s

...when this happens. This can occur several times a day, as we're 
running vCenter-controlled auto-migration - a very common setup. This is 
forward stepping of course, so is relatively harmless (backward is a pain).


 Who knows what newer implementations like chrony or openntpd do!

 If they're doing something silly, then maybe you shouldn't use them.

I have no reason to suspect they are doing somthing silly. I merely 
point out that there are other implmentations than the common ntpd, and 
that I don't know if they step or not. In fact, a little research 
suggests that chrony has *better* behaviour w.r.t stepping than ntpd:

http://lists.fedoraproject.org/pipermail/devel/2010-May/135679.html

...which is nice.

I sense a bit of defensiveness to this reply, TBH. Maybe I'm imagniing 
it, but if so that's unncessary. I don't hold a strong position about 
Twisted having a monotonic clock.

When the original ticket was discussed here months ago, I was quite 
alarmed because the symptoms sounded dire. Further discussion clarified 
what the issues were, and I decided they weren't significant (for us).

Maybe Twisted doesn't need a monotonic clock is the right reply, but 
it would be wrong to base that on the assumpton that ntpd doesn't step 
- that's all I was trying to say.

Cheers,
Phil

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python