Re: [Twisted-Python] Async-pep (again)

[Glyph Lefkowitz]

On Jul 14, 2011, at 4:05 AM, Laurens Van Houtven wrote:

> At least in Twisted, transports and protocols do not exist in a vacuum;
> they have to be hooked up via the reactor. Will this PEP define
> a (skeletal) API to be implemented by potential reactors, or is that
> going to left entirely unspecified, like WSGI?
> 
> Entirely unspecified, because different implementations have to do pretty 
> different things.

I thought the idea was to include an asyncore reactor interface?  My assumption 
was that we'd provide an adapter around the Twisted reactor which would provide 
some basic functionality, like listenTCP, connectTCP, and callLater.

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Async-pep (again)

[Glyph Lefkowitz]

On Jul 14, 2011, at 2:48 AM, Tim Allen wrote:

> On Wed, Jul 13, 2011 at 02:03:03PM +0200, Laurens Van Houtven wrote:
>> So, some of you might remember my async-pep post a while ago. Some people
>> correctly complained there was no code or text. There's some code and quite
>> a bit of text now. In fact, it even has a PEP number (3153)! So I'm
>> soliciting feedback again.
> 
> The idea of Protocols implementing Transports is vaguely gestured at as
> a Useful Thing, but not much detail is given. I think it would be useful
> for the final PEP to address that topic more rigorously - partially
> because it's good to have a firm basis on which to model SOCKS and SSH
> libraries, but mostly because figuring out how SSL should interact with
> TCP is going to give people headaches. Twisted, so far as I can see,
> just sort of punts and says "Yeah, SSL is just another transport like
> TCP", but then you have to make the SSL transport support all the same
> options that the TCP transport supports (socket options? IPv6?), but
> then what if you want to run SSL over a serial port or a SOCKS
> connection... A!

> In practice, it might be simpler because "SSL" means "whatever subset of
> TCP functionality we can coax OpenSSL into providing" rather than
> a fully stackable protocol-providing-a-transport.

Actually, you might be interested in .  This will be in 
11.1.  TLS _is_ a protocol-that-is-a-transport now (in trunk).  This was the 
case in 11.0, too, but only for the IOCP reactor.  We've been smoothing out 
some interesting quirks that occurred as a result, mostly test-related, but 
it's looking good for the release; more robust, actually, because it's easier 
to test the stacked version than to try to trick sockets into returning 
specific values in C.

> The thing with Consumers and Producers seems... very abstract. If I'm
> sitting down to retrieve email via POP3 (to pick a random protocol),
> 'transports' and 'protocols' are tools that nestle very comfortably in
> my mental model of the task in front of me; "consumers" and "producers"
> are not.

The APIs definitely aren't as nice, and that's where I predict the most 
discussion in the PEP.

> Are they concepts that should be handled by transport implementors?

Yes, pretty much always.

> Protocol implementors?

Yes, if you need them.

> Protocol users?

It depends.  Ideally you should be able to rely on the protocol providing a 
reasonable stream-friendly API.  (You probably only care about this if you're 
writing a proxy.)

> Should they be mapped onto XON/XOFF or RTS/CTS by serial transports?

Either or.  Probably an option to the serial transport.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] how to implement a perspective broker client with reconnection

[gelin yan]

Hi All

I plan to use pb to communicate between my processes. The question is
how i am able to implement a pb client with some degrees of reliability.
What i want is when getRootObject() fail sometimes, it can retry several
times automatically. It seems docs doesn't mention this problem and I didn't
find any hint from source code.


Thanks

gelin yan
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] FilesystemLock/DeferredFilesystemLock with NFS and multiple processes

[Alberto Donato]

Hi,
  I have a scenario in which I need to use a lock file to prevent
multiple processes to run on the same directory concurrently.
The processes could run on the same machine or on different machine,
with a NFS filesystem.

If I understand correctly FilesystemLock works fine with different
processes on the same machine, but since it uses a symlink to PIDs to
identify the lock owner, it is not guaranteed to be safe on NFS.
Is this the only issue with this scenario? In this case I guess it
could be fixed by adding a machine identifier (hostname or some hash)
to the name.
Also, is there any possible concurrency problem with NFS atomicity?

Thanks,

Alberto

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] what is the status of storm integration with twisted?

[Free Ekanayaka]

Hi Ilja,

|--==> On Tue, 12 Jul 2011 13:02:43 +0200, Jamu Kakar  said:

  JK> Hi Ilja,
  JK> On Tue, Jul 12, 2011 at 9:38 AM, Ilja Livenson  
wrote:
  >>basically, $subj. I really like the DeferredStore concept introduced
  >>in storm/twisted-integration, but it seems that that branch is lagging
  >>behind on merging (for 2 years now). Any inside info, perhaps?

  JK> The branch is not being actively maintained, though I do believe some
  JK> folks are using it in production systems.  The way we've been
  JK> recommending people use Storm with Twisted is by running transactions
  JK> in threads with deferToThread.  Free recently landed some code in
  JK> storm.twisted.transact that makes pursuing this model, with a thread
  JK> pool, fairly straightforward.

What Jamu said. Considering that the branch you're mentioning does use
threads as well, the storm.twisted.transact is probably easier and
virtually identical as far as performance goes.

Cheers,

Free

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Async-pep (again)

[exarkun]

On 10:26 am, screwt...@froup.com wrote:
>On Thu, Jul 14, 2011 at 07:05:38PM +1000, Tim Allen wrote:
>>- You might also want to create an optional Transport method to wrap 
>>the
>>   sendfile(2) and/or splice(2) functions.
>
>I suggested this not knowing whether Python would ever grow support for
>sendfile(), since it seemed like the sort of thing that
>performance-oriented async-io frameworks might want to set up with
>ctypes or similar. However, I've just discovered that os.sendfile() 
>will
>be in Python 3.3:
>
>http://docs.python.org/dev/library/os.html#os.sendfile
>
>Since your PEP has a 3000-series number anyway, os.sendfile() might
>potentially be quite useful.

Since this non-trivially complicates the implementation of a transport, 
I suggest that it be an optional extension proposed separately.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Async-pep (again)

[Tim Allen]

On Thu, Jul 14, 2011 at 07:05:38PM +1000, Tim Allen wrote:
> - You might also want to create an optional Transport method to wrap the
>   sendfile(2) and/or splice(2) functions.

I suggested this not knowing whether Python would ever grow support for
sendfile(), since it seemed like the sort of thing that
performance-oriented async-io frameworks might want to set up with
ctypes or similar. However, I've just discovered that os.sendfile() will
be in Python 3.3:

http://docs.python.org/dev/library/os.html#os.sendfile

Since your PEP has a 3000-series number anyway, os.sendfile() might
potentially be quite useful.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Async-pep (again)

[Tim Allen]

On Thu, Jul 14, 2011 at 10:05:00AM +0200, Laurens Van Houtven wrote:
> On Thu, Jul 14, 2011 at 8:48 AM, Tim Allen  wrote:
> > The idea of Protocols implementing Transports is vaguely gestured at as
> > a Useful Thing, but not much detail is given. I think it would be useful
> > for the final PEP to address that topic more rigorously - partially
> > because it's good to have a firm basis on which to model SOCKS and SSH
> > libraries, but mostly because figuring out how SSL should interact with
> > TCP is going to give people headaches. Twisted, so far as I can see,
> > just sort of punts and says "Yeah, SSL is just another transport like
> > TCP", but then you have to make the SSL transport support all the same
> > options that the TCP transport supports (socket options? IPv6?), but
> > then what if you want to run SSL over a serial port or a SOCKS
> > connection... A!
> >
> > In practice, it might be simpler because "SSL" means "whatever subset of
> > TCP functionality we can coax OpenSSL into providing" rather than
> > a fully stackable protocol-providing-a-transport.
> 
> Cool. Can I shove those 2 paragraphs into a ticket or will the copyright
> monster haunt me?

Go right ahead! I guess most of these things should be tickets, but
I don't have a GitHub account and I'm not particularly looking to
register on more websites at the moment.

> > The thing with Consumers and Producers seems... very abstract. If I'm
> > sitting down to retrieve email via POP3 (to pick a random protocol),
> > 'transports' and 'protocols' are tools that nestle very comfortably in
> > my mental model of the task in front of me; "consumers" and "producers"
> > are not. Are they concepts that should be handled by transport
> > implementors? Protocol implementors? Protocol users? Should they be
> > mapped onto XON/XOFF or RTS/CTS by serial transports?
> 
> Yes, Consumers and Producers are about flow control, and most Transports
> probably are producers.

Having looked at the issues list after sending that message, I see this
is basically issue 13, "Why are producers/consumers important, how are
they different from protocols/transports?"

If your PEP includes producers and consumers (and I note that the
current example code doesn't, it just has a "FlowControl" class), you'll
want to have an example Protocol that uses producers and consumers in
some useful, illustrative fashion.

> > At least in Twisted, transports and protocols do not exist in a vacuum;
> > they have to be hooked up via the reactor. Will this PEP define
> > a (skeletal) API to be implemented by potential reactors, or is that
> > going to left entirely unspecified, like WSGI?
> 
> Entirely unspecified, because different implementations have to do pretty
> different things.

I guess the selection of available Transports is up to the hosting
event-loop, too - it might be worth noting that in the section on
"Transports". Unless, of course, the Transport in question is
implemented by another Protocol, in which case I guess it's anybody's
guess how you might hook your Protocol up.

I almost think that, for pedagogy's sake, there should be an additional
Encapsulator or Framer abstract class, that inherits from Protocol, but
adds a .connectProtocol() method that takes another Protocol instance,
and hooks itself up as that Protocol's transport. Sure, anyone who
understands what's going on should be able to figure out what's going
on, but I think an extra class would make it blindingly obvious, and
I like APIs that save me from having to think too hard.

While there's still people listening to my half-formed opinions:

- Issue 7 seems to have settled on removing support for half_closing
  transports. I seem to recall somebody mentioning support for
  half-close as being one of those weird-corner cases that nobody thinks
  they need until they're trying to figure out why their SSH sessions
  always die with "broken pipe" errors. While it probably would
  complicate the documentation to include it, I'd hope that many
  frameworks that implement this PEP would want to include support for
  half-closing transports, and it'd be nice if there was a standard API
  for it instead of everybody adding their own methods with their own
  semantics. Perhaps there could be an HalfClosableTransport(Transport)
  ABC, that's optional in the same way that, say, DB-API's
  "connection.rollback()" method is defined but optional.

- For issue 6 ("Scatter/gather IO API doesn't make sense yet"), I can't
  see much of a use for readv/scatter, because I imagine the benefits
  come from having a bunch of pre-allocated buffers lying around, and
  even if the Python VM had such buffers, they probably wouldn't be
  visible or useful to running Python code. On the other hand, I can
  easily imagine Python code having a bunch of independently-generated
  buffers that need to be written in a particular order (framing data
  and framed data, for example), and being able to avoid ''.join() could
  b

Re: [Twisted-Python] Async-pep (again)

[Laurens Van Houtven]

On Thu, Jul 14, 2011 at 8:48 AM, Tim Allen  wrote:

> On Wed, Jul 13, 2011 at 02:03:03PM +0200, Laurens Van Houtven wrote:
> > So, some of you might remember my async-pep post a while ago. Some people
> > correctly complained there was no code or text. There's some code and
> quite
> > a bit of text now. In fact, it even has a PEP number (3153)! So I'm
> > soliciting feedback again.
>
> The idea of Protocols implementing Transports is vaguely gestured at as
> a Useful Thing, but not much detail is given. I think it would be useful
> for the final PEP to address that topic more rigorously - partially
> because it's good to have a firm basis on which to model SOCKS and SSH
> libraries, but mostly because figuring out how SSL should interact with
> TCP is going to give people headaches. Twisted, so far as I can see,
> just sort of punts and says "Yeah, SSL is just another transport like
> TCP", but then you have to make the SSL transport support all the same
> options that the TCP transport supports (socket options? IPv6?), but
> then what if you want to run SSL over a serial port or a SOCKS
> connection... A!
>
> In practice, it might be simpler because "SSL" means "whatever subset of
> TCP functionality we can coax OpenSSL into providing" rather than
> a fully stackable protocol-providing-a-transport.
>


Cool. Can I shove those 2 paragraphs into a ticket or will the copyright
monster haunt me?


>
> The thing with Consumers and Producers seems... very abstract. If I'm
> sitting down to retrieve email via POP3 (to pick a random protocol),
> 'transports' and 'protocols' are tools that nestle very comfortably in
> my mental model of the task in front of me; "consumers" and "producers"
> are not. Are they concepts that should be handled by transport
> implementors? Protocol implementors? Protocol users? Should they be
> mapped onto XON/XOFF or RTS/CTS by serial transports?
>

Yes, Consumers and Producers are about flow control, and most Transports
probably are producers.


>
> At least in Twisted, transports and protocols do not exist in a vacuum;
> they have to be hooked up via the reactor. Will this PEP define
> a (skeletal) API to be implemented by potential reactors, or is that
> going to left entirely unspecified, like WSGI?
>

Entirely unspecified, because different implementations have to do pretty
different things.


>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python