Re: [Python-Dev] threadsafe patch for asynchat

[Guido van Rossum]

On 2/7/06, Mark Edgington <[EMAIL PROTECTED]> wrote:
> Ok, perhaps the notation could be improved, but the idea of the
> semaphore in the patch is "Does it run inside of a multithreaded
> environment, and could its push() functions be called from a different
> thread?"

The long-term fate of asyncore/asynchat aside, instead of wanting to
patch asynchat, you should be able to subclass it easily to introduce
the functionality you want. Given the disagreement over whether this
is a good thing, I suggest that that's a much better way for you to
solve your problem than to introduce yet another obscure confusing
optional parameter. And you won't have to wait for Python 2.5.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Donovan Baarda]

On Thu, 2006-02-09 at 13:12 +0100, Fredrik Lundh wrote:
> Donovan Baarda wrote:
> 
> >> Here I think you meant that medusa didn't handle computation in separate
> >> threads instead.
> >
> > No, I pretty much meant what I said :-)
> >
> > Medusa didn't have any concept of a deferred, hence the idea of using
> > one to collect the results of a long computation in another thread never
> > occurred to them... remember the highly refactored OO beauty that is
> > twisted was not even a twinkle in anyone's eye yet.
> >
> > In theory it would be just as easy to add twisted style deferToThread to
> > Medusa, and IMHO it is a much better approach. Unfortunately at the time
> > they went the other way and implemented multiple async-loops in separate
> > threads.
> 
> that doesn't mean that everyone using Medusa has done things in the wrong
> way, of course ;-)

Of course... and even Zope2 was not necessarily the "wrong way"... it
was a perfectly valid design decision, given that it was all new ground
at the time. And it works really well... there were many consequences of
that design that probably contributed to the robustness of other Zope
components like ZODB...

-- 
Donovan Baarda <[EMAIL PROTECTED]>
http://minkirri.apana.org.au/~abo/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Fredrik Lundh]

Donovan Baarda wrote:

>> Here I think you meant that medusa didn't handle computation in separate
>> threads instead.
>
> No, I pretty much meant what I said :-)
>
> Medusa didn't have any concept of a deferred, hence the idea of using
> one to collect the results of a long computation in another thread never
> occurred to them... remember the highly refactored OO beauty that is
> twisted was not even a twinkle in anyone's eye yet.
>
> In theory it would be just as easy to add twisted style deferToThread to
> Medusa, and IMHO it is a much better approach. Unfortunately at the time
> they went the other way and implemented multiple async-loops in separate
> threads.

that doesn't mean that everyone using Medusa has done things in the wrong
way, of course ;-)

 



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Donovan Baarda]

On Wed, 2006-02-08 at 15:14 +0100, Valentino Volonghi aka Dialtone
wrote:
> On Wed, Feb 08, 2006 at 01:23:26PM +, Donovan Baarda wrote:
> > I believe that Twisted does pretty much this with it's "deferred" stuff.
> > It shoves slow stuff off for processing in a separate thread that
> > re-syncs with the event loop when it's finished.
> 
> Deferreds are only an elaborate way to deal with a bunch of callbacks.
> It's Twisted itself that provides a way to run something in a separate thread
> and then fire a deferred (from the main thread) when the child thread
> finishes (reactor.callInThread() to call stuff in a different thread,
[...]

I know they are more than just a way to run slow stuff in threads, but
once you have them, simple as they are, they present an obvious solution
to all sorts of things, including long computations in a thread.

Note that once zope2 took the approach it did, blocking the async-loop
didn't hurt so bad, so lots of zope add-ons just did it gratuitously. In
many cases the slow event handlers were slow because they are waiting on
IO that could in theory be serviced as yet another event handler in the
async-loop. However, the Zope/Medusa async framework had become so scary
hardly anyone knew how to do this without breaking Zope itself.

> > In the case of Zope/ZEO I'm not entirely sure but I think what happened
> > was medusa (asyncore/asynchat based stuff Zope2 was based on) didn't
> > have this deferred handler support. When they found some of the stuff
> 
> Here I think you meant that medusa didn't handle computation in separate
> threads instead.

No, I pretty much meant what I said :-)

Medusa didn't have any concept of a deferred, hence the idea of using
one to collect the results of a long computation in another thread never
occurred to them... remember the highly refactored OO beauty that is
twisted was not even a twinkle in anyone's eye yet.

In theory it would be just as easy to add twisted style deferToThread to
Medusa, and IMHO it is a much better approach. Unfortunately at the time
they went the other way and implemented multiple async-loops in separate
threads.

-- 
Donovan Baarda <[EMAIL PROTECTED]>
http://minkirri.apana.org.au/~abo/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Josiah Carlson]

Mark Edgington <[EMAIL PROTECTED]> wrote:
> 
> Martin v. Löwis wrote:
>  > That patch looks wrong. What does it mean to "run in a thread"?
>  > All code runs in a thread, all the time: sometime, that thread
>  > is the main thread.
>  >
>  > Furthermore, I can't see any presumed thread-unsafety in asynchat.
> 
> Ok, perhaps the notation could be improved, but the idea of the 
> semaphore in the patch is "Does it run inside of a multithreaded 
> environment, and could its push() functions be called from a different 
> thread?"

Asyncore is not threadsafe.  The reason it is not threadsafe is because
there was no effort made to make it threadsafe, because it is not
uncommon for the idea of asynchronous sockets to be the antithesis of
threaded socket servers.

In any case, one must be very careful as (at least in older versions of
Python on certain platforms), running sock.send(data) on two threads
simultaneously for the same socket was a segfault.  I understand that
this is what you are trying to avoid, but have you considered just doing...

q = Queue.Queue()
def push(sock, data):
q.put((sock, data))

def mainloop():
...
while not q.empty():
sock, data = q.get()
sock.push(data)
...

Wow, now we don't have to update the standard library to introduce a
false sense of thread-safety into asyncore!

 - Josiah

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Bill Janssen]

Not terrible.  I think I may try re-working Medusa to use this.

Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Pedro Werneck]

On Wed, 08 Feb 2006 15:42:39 +0100
Georg Brandl <[EMAIL PROTECTED]> wrote:

> Neal Norwitz wrote:
> > On 2/7/06, Christopher Armstrong <[EMAIL PROTECTED]> wrote:
> >>
> >> Twisted is wonderful, powerful, rich, and very large.  Perhaps a
> >small > > subset could be carefully extracted
> >>
> >The subject of putting (parts of) Twisted into the standard library
> >comes up once every 6 months or so, at least on our mailing list.
> >For all that I think asyncore is worthless, I'm still against
> >copying Twisted into the stdlib. Or at least I'm not willing to
> >maintain the necessary fork, and I fear the nightmares about
> >versioning that can easily occur when you've got both standard
> >library and third party versions of a project.
> > 
> > I wouldn't be enthusiastic about putting all of Twisted in the
> > stdlib either.  Twisted is on a different release schedule than
> > Python.  However, isn't there a relatively small core subset like
> > Alex mentioned that isn't changing much?  Could we split up those
> > components and have those live in the core, but the vast majority of
> > Twisted live outside as it does now?
> 
> +1. This would be very useful for simple networking applications.


I have a simple library I wrote some time ago to make asynchronous TCP
servers (honeypots), and I wrote it exactly for the reasons being
discussed on this thread: the other developers were not very familiar
with Python (they were planning to use Perl on the project) and a bit
confused with asyncore. Twisted was the obvious answer, but I could not
convince them to put it in the project because of the size and the work
needed to put it in all machines they were planning to use.

I used this library several times the last two years. The last two weeks
I've been using it with another project, but yesterday (a coincidence ?)
I decided to reduce all of it to a single module. 

It is roughly based on Twisted, the interface is similar, some parts are
a copy of Twisted code (select code, LineProtocol is a copy of twisted's
LineReceiver) but only 16k in size, everything is covered by unittests.
It's intended for servers, but client support can be added with some
effort too. Maybe it fits the needs of what is being discussed on this
thread.


It's available here: 

http://www.pythonbrasil.com.br/moin.cgi/HoneyPython?action=AttachFile&do=get&target=asiel.tar.bz2

-- 
Pedro Werneck
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Mark Edgington]

Martin v. Löwis wrote:
 > That patch looks wrong. What does it mean to "run in a thread"?
 > All code runs in a thread, all the time: sometime, that thread
 > is the main thread.
 >
 > Furthermore, I can't see any presumed thread-unsafety in asynchat.

Ok, perhaps the notation could be improved, but the idea of the 
semaphore in the patch is "Does it run inside of a multithreaded 
environment, and could its push() functions be called from a different 
thread?"

I have verified that there is a problem with running it in such an 
environment.  My results are more or less identical to those described 
in the following thread:
http://mail.python.org/pipermail/medusa-dev/1999/000431.html
(see also the reply message to this one regarding the solution -- if you 
look at the Zope source, Zope deals with the problem in the way I am 
suggesting asynchat be patched)

It seems that somehow in line 271 (python 2.4) of asynchat.py, 
producer_fifo.list is not empty, and thus popleft() is executed. 
However, popleft() finds the deque empty.  This means that somehow the 
deque (or list -- the bug is identical in python 2.3) is emptied between 
the if() and the popleft(), so perhaps asyncore.loop(), running in a 
different thread from the thread which calls async_chat.push(), empties it.

The problem is typically exhibited when running in a multithreaded 
environment, and when calling the async_chat.push() function many (i.e. 
perhaps tens of thousands) times quickly in a row from a different 
thread.  However, this behavior is avoided by creating a Lock for 
refill_buffer(), so that it cannot be executed simultaneously.  It is 
also avoided by not executing initiate_send() at all (as is done by Zope 
in ZHTTPServer.zhttp_channel).

 > Sure, there is a lot of member variables in asynchat which aren't
 > specifically protected against mutual access from different threads.
 > So you shouldn't be accessing the same async_chat object from multiple
 > threads.

If applying this patch does indeed make it safe to use async_chat.push() 
from other threads, why would it be a bad thing to have?  It seems to 
make the code less cryptic (i.e. I don't need to override base classes 
in order to include code which processes a nonempty Queue object -- I 
simply make a call to the push() function of my instance of async_chat, 
and I'm done).

-Mark

(also, of course push_with_producer() would probably also need the same 
changes that would be made to push() )

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Barry Warsaw]

On Wed, 2006-02-08 at 11:45 -0800, Keith Dart wrote:

> There are other, third-party, SMTP server objects available. You could
> always use one of those. 

Very true.  In fact, Twisted comes to the rescue again here.  When I
needed to test Mailman's NNTP integration I could either spend several
weeks figuring out how to install and configure some traditional NNTP
server, or I could just install Twisted and run exactly three commands
(one of which was "sudo" :).

> Once the "Python egg" and PyPI improve and start widespread use perhaps
> the question of what is in the core library and what is not will become
> moot.

Indeed.

> Being a Gentoo Linux user I already enjoy having many modules
> available, with automatic dependency installation, on demand. So the
> idea of "core" library is already blurred for me.

Although I'm doing a lot more dev on the Mac these days, I definitely
agree that this is what makes Gentoo so cool for Linux, and I can't wait
for Gentoo-on-OSX to switch to doing things the Right Way (can you say
bye-bye DarwinPorts?).

-Barry



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Keith Dart]

Barry Warsaw wrote the following on 2006-02-08 at 13:45 PST:
===
> Or the guy who needs to whip together an RFC-compliant minimal SMTP
> server to use in unit tests of some random Python implemented mailing
> list manager.  Just fer instance.  But still...
> 
> > But I can't speak for how often this need comes up among users.
> 
> Yeah, there is that. ;)
===

There are other, third-party, SMTP server objects available. You could
always use one of those. 

Once the "Python egg" and PyPI improve and start widespread use perhaps
the question of what is in the core library and what is not will become
moot.

Being a Gentoo Linux user I already enjoy having many modules
available, with automatic dependency installation, on demand. So the
idea of "core" library is already blurred for me.


-- 

-- ~
   Keith Dart <[EMAIL PROTECTED]>
   public key: ID: 19017044
   
   =


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Barry Warsaw]

On Wed, 2006-02-08 at 10:24 -0800, Robert Brewer wrote:

> It'd be a huge loss for the random fellow who needs to write an email
> fixup proxy between a broken client and Exim in a couple of hours. ;)

Or the guy who needs to whip together an RFC-compliant minimal SMTP
server to use in unit tests of some random Python implemented mailing
list manager.  Just fer instance.  But still...

> But I can't speak for how often this need comes up among users.

Yeah, there is that. ;)

-Barry



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Martin v. Löwis]

Steve Holden wrote:
> In case the processing of events needed to block? If I'm processing web 
> requests in an async* dispatch loop and a request needs the results of a 
> (probably lengthy) database query in order to generate its output, how 
> do I give the dispatcher control again to process the next asynchronous 
> network event?

I see. Ideally, you should obtain the socket for the connection to
the database, and add it to the asyncore loop. That would require
you have an async database API, of course.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Robert Brewer]

Barry Warsaw wrote:
> On Tue, 2006-02-07 at 16:01 -0800, Robert Brewer wrote:
> 
> > Perhaps, but please keep in mind that the smtpd module uses 
> > both, currently, and would have to be rewritten if either is 
> > "removed".
> 
> Would that really be a huge loss?

It'd be a huge loss for the random fellow who needs to write an email
fixup proxy between a broken client and Exim in a couple of hours. ;)
But I can't speak for how often this need comes up among users.


Robert Brewer
System Architect
Amor Ministries
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Pedro Werneck]

On Wed, 08 Feb 2006 15:42:39 +0100
Georg Brandl <[EMAIL PROTECTED]> wrote:

> Neal Norwitz wrote:
> > On 2/7/06, Christopher Armstrong <[EMAIL PROTECTED]> wrote:
> >>
> >> Twisted is wonderful, powerful, rich, and very large.  Perhaps a
> >small > > subset could be carefully extracted
> >>
> >The subject of putting (parts of) Twisted into the standard library
> >comes up once every 6 months or so, at least on our mailing list.
> >For all that I think asyncore is worthless, I'm still against
> >copying Twisted into the stdlib. Or at least I'm not willing to
> >maintain the necessary fork, and I fear the nightmares about
> >versioning that can easily occur when you've got both standard
> >library and third party versions of a project.
> > 
> > I wouldn't be enthusiastic about putting all of Twisted in the
> > stdlib either.  Twisted is on a different release schedule than
> > Python.  However, isn't there a relatively small core subset like
> > Alex mentioned that isn't changing much?  Could we split up those
> > components and have those live in the core, but the vast majority of
> > Twisted live outside as it does now?
> 
> +1. This would be very useful for simple networking applications.


I have a simple library I wrote some time ago to make asynchronous TCP
servers (honeypots), and I wrote it exactly for the reasons being
discussed on this thread: the other developers were not very familiar
with Python (they were planning to use Perl on the project) and a bit
confused with asyncore. Twisted was the obvious answer, but I could not
convince them to put it in the project because of the size and the work
needed to put it in all machines they were planning to use.

I used this library several times the last two years. The last two weeks
I've been using it with another project, but yesterday (a coincidence ?)
I decided to reduce all of it to a single module. 

It is roughly based on Twisted, the interface is similar, some parts are
a copy of Twisted code (select code, LineProtocol is a copy of twisted's
LineReceiver) but only 16k in size, everything is covered by unittests.
It's intended for servers, but client support can be added with some
effort too. Maybe it fits the needs of what is being discussed on this
thread.


It's available here: 

http://www.pythonbrasil.com.br/moin.cgi/HoneyPython?action=AttachFile&do=get&target=asiel.tar.bz2

-- 
Pedro Werneck
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Georg Brandl]

Neal Norwitz wrote:
> On 2/7/06, Christopher Armstrong <[EMAIL PROTECTED]> wrote:
>>
>> > Twisted is wonderful, powerful, rich, and very large.  Perhaps a small
>> > subset could be carefully extracted
>>
>> The subject of putting (parts of) Twisted into the standard library
>> comes up once every 6 months or so, at least on our mailing list. For
>> all that I think asyncore is worthless, I'm still against copying
>> Twisted into the stdlib. Or at least I'm not willing to maintain the
>> necessary fork, and I fear the nightmares about versioning that can
>> easily occur when you've got both standard library and third party
>> versions of a project.
> 
> I wouldn't be enthusiastic about putting all of Twisted in the stdlib
> either.  Twisted is on a different release schedule than Python. 
> However, isn't there a relatively small core subset like Alex
> mentioned that isn't changing much?  Could we split up those
> components and have those live in the core, but the vast majority of
> Twisted live outside as it does now?

+1. This would be very useful for simple networking applications.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Aahz]

On Wed, Feb 08, 2006, Thomas Wouters wrote:
>
> Anything beyond simple bugfixes on asyncore/asynchat seems like a terrible
> waste of effort, to me. And I hardly ever use Twisted.

+1
-- 
Aahz ([EMAIL PROTECTED])   <*> http://www.pythoncraft.com/

"19. A language that doesn't affect the way you think about programming,
is not worth knowing."  --Alan Perlis
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Valentino Volonghi aka Dialtone]

On Wed, Feb 08, 2006 at 01:23:26PM +, Donovan Baarda wrote:
> I believe that Twisted does pretty much this with it's "deferred" stuff.
> It shoves slow stuff off for processing in a separate thread that
> re-syncs with the event loop when it's finished.

Deferreds are only an elaborate way to deal with a bunch of callbacks.
It's Twisted itself that provides a way to run something in a separate thread
and then fire a deferred (from the main thread) when the child thread
finishes (reactor.callInThread() to call stuff in a different thread,
reactor.callFromThread() to call reactor APIs from a different thread)
Deferreds are just a bit more than:

class Deferred(object):
def __init__(self):
self.callbacks = []

def addCallback(self, callback):
self.callbacks.append(callback)

def callback(self, value):
for callback in self.callbacks:
value = callback(value)

This is mostly what a deferred is (without error handling, extra argument
passing, 'nested' deferreds handling and blabla, the core concept however
is there). As you see there is no extra magic in deferreds (or weird
dependency on Twisted, they are pure python and could be used everywhere,
you can implement them in any language that supports first class functions).

> In the case of Zope/ZEO I'm not entirely sure but I think what happened
> was medusa (asyncore/asynchat based stuff Zope2 was based on) didn't
> have this deferred handler support. When they found some of the stuff

Here I think you meant that medusa didn't handle computation in separate
threads instead.

-- 
Valentino Volonghi aka Dialtone
Now Running MacOSX 10.4
Blog: http://vvolonghi.blogspot.com
New Pet: http://www.stiq.it


pgpMxTGEMNS2c.pgp
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Andrew Bennetts]

Donovan Baarda wrote:
> On Wed, 2006-02-08 at 02:33 -0500, Steve Holden wrote:
> > Martin v. Löwis wrote:
> > > Tim Peters wrote:
> [...]
> > > What is the reason that people want to use threads when they can have
> > > poll/select-style message processing? Why does Zope require threads?
> > > IOW, why would anybody *want* a "threadsafe patch for asynchat"?
> > > 
> > In case the processing of events needed to block? If I'm processing web 
> > requests in an async* dispatch loop and a request needs the results of a 
> > (probably lengthy) database query in order to generate its output, how 
> > do I give the dispatcher control again to process the next asynchronous 
> > network event?
> > 
> > The usual answer is "process the request in a thread". That way the 
> > dispatcher can spring to life for each event as quickly as needed.
> 
> I believe that Twisted does pretty much this with it's "deferred" stuff.
> It shoves slow stuff off for processing in a separate thread that
> re-syncs with the event loop when it's finished.

Argh!  No.  Threading is completely orthogonal to Deferreds.

Deferreds are just an abstraction for managing callbacks for an asychronous
operation.  They don't magically invoke threads, or otherwise turn synchronous
code into asynchronous code for you.

This seems to be a depressingly common misconception.  I wish I knew how to stop
it.

They're much simpler than people seem to think.  They're an object a function
returns to say "I don't have a result for you yet, but if you attach callbacks
to this I'll run those when I do."  We've do this because it's much nicer than
having to pass callbacks into functions, particularly when you want to deal with
chains of callbacks and error handling.

There is a single utility function in Twisted called "deferToThread" that will
run a function in a threadpool, and arrange for a Deferred to be fired with the
result (in the event loop thread, of course).  This is just one of many possible
uses for Deferreds, and not an especially common one.

I'm happy to provide pointers to several Twisted docs if anyone is at all
unclear on this.

While they are very useful, I don't think they're an essential part of a minimal
Twisted replacement for asyncore/asynchat -- in fact, they'd work just fine with
asyncore/asynchat, because they do so little.

-Andrew.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Donovan Baarda]

On Wed, 2006-02-08 at 02:33 -0500, Steve Holden wrote:
> Martin v. Löwis wrote:
> > Tim Peters wrote:
[...]
> > What is the reason that people want to use threads when they can have
> > poll/select-style message processing? Why does Zope require threads?
> > IOW, why would anybody *want* a "threadsafe patch for asynchat"?
> > 
> In case the processing of events needed to block? If I'm processing web 
> requests in an async* dispatch loop and a request needs the results of a 
> (probably lengthy) database query in order to generate its output, how 
> do I give the dispatcher control again to process the next asynchronous 
> network event?
> 
> The usual answer is "process the request in a thread". That way the 
> dispatcher can spring to life for each event as quickly as needed.

I believe that Twisted does pretty much this with it's "deferred" stuff.
It shoves slow stuff off for processing in a separate thread that
re-syncs with the event loop when it's finished.

In the case of Zope/ZEO I'm not entirely sure but I think what happened
was medusa (asyncore/asynchat based stuff Zope2 was based on) didn't
have this deferred handler support. When they found some of the stuff
Zope was doing took a long time, they came up with an initially simpler
but IMHO uglier solution of running multiple async loops in separate
threads and using a front-end dispatcher to distribute connections to
them. This way it wasn't too bad if an async loop stalled, because the
other loops in other threads could continue to process stuff.

If ZEO is still using this approach I think switching to a twisted style
approach would be a good idea. However, I suspect this would be a very
painful refactor...

-- 
Donovan Baarda <[EMAIL PROTECTED]>
http://minkirri.apana.org.au/~abo/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Steve Holden]

Josiah Carlson wrote:
> "Fredrik Lundh" <[EMAIL PROTECTED]> wrote:
> 
>>Steve Holden wrote:
>>
>>
What is the reason that people want to use threads when they can have
poll/select-style message processing? Why does Zope require threads?
IOW, why would anybody *want* a "threadsafe patch for asynchat"?

>>>
>>>In case the processing of events needed to block? If I'm processing web
>>>requests in an async* dispatch loop and a request needs the results of a
>>>(probably lengthy) database query in order to generate its output, how
>>>do I give the dispatcher control again to process the next asynchronous
>>>network event?
>>>
>>>The usual answer is "process the request in a thread". That way the
>>>dispatcher can spring to life for each event as quickly as needed.
>>
>>but why do such threads have to talk to asyncore directly ?
> 
Good question.
> 
> Indeed.  I seem to remember a discussion a few months ago about "easy"
> thread programming, which invariably directed people off to use the
> simplest abstractions necessary: Queues.
> 
Maybe people are finding Python too easy and they just want to 
complicate their code to the point where it contains interesting bugs? I 
dunno 

regards
  Steve
-- 
Steve Holden   +44 150 684 7255  +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006  www.python.org/pycon/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Thomas Wouters]

On Tue, Feb 07, 2006 at 08:53:46PM -0800, Bill Janssen wrote:

> Perhaps the right idea is to fix the various problems of asyncore.

The problem with making asyncore more useful is that you end up with (a cut
down version of) Twisted, although not one that would be able to integrate
with Twisted. asyncore/asynchat and Twisted are really not that different,
and anything you do to enhance the former will make it look more like the
latter. I'd personally rather fork parts of Twisted, in spite of the
maintenance issues, than re-invent Twisted, fix all the issues Twisted
already solves and face the same kind of maintenance issues. It would be
perfect if the twisted-light in the stdlib would integrate with the 'real'
Twisted, so that users can 'upgrade' their programs just by installing
Twisted and using the extra features.

Not that I think we should stop at the event core and the TCP/SSL parts of
Twisted; imaplib, poplib, httplib, xmlrpclib, for instance, could all do
with Twisted-inspired alternatives (or even replacements, if the synchronous
API was kept the same.) The synchronous versions are fine for simple scripts
(or complex scripts that don't mind long blocking operations.) If we start
exporting a really useful asynchronous framework, I would expect
asynchronous counterparts to the useful higher-level networking modules,
too. But that doesn't have to come right away ;)

Anything beyond simple bugfixes on asyncore/asynchat seems like a terrible
waste of effort, to me. And I hardly ever use Twisted.

-- 
Thomas Wouters <[EMAIL PROTECTED]>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Josiah Carlson]

"Fredrik Lundh" <[EMAIL PROTECTED]> wrote:
> 
> Steve Holden wrote:
> 
> > > What is the reason that people want to use threads when they can have
> > > poll/select-style message processing? Why does Zope require threads?
> > > IOW, why would anybody *want* a "threadsafe patch for asynchat"?
> > >
> > In case the processing of events needed to block? If I'm processing web
> > requests in an async* dispatch loop and a request needs the results of a
> > (probably lengthy) database query in order to generate its output, how
> > do I give the dispatcher control again to process the next asynchronous
> > network event?
> >
> > The usual answer is "process the request in a thread". That way the
> > dispatcher can spring to life for each event as quickly as needed.
> 
> but why do such threads have to talk to asyncore directly ?

Indeed.  I seem to remember a discussion a few months ago about "easy"
thread programming, which invariably directed people off to use the
simplest abstractions necessary: Queues.

 - Josiah

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Fredrik Lundh]

Steve Holden wrote:

> > What is the reason that people want to use threads when they can have
> > poll/select-style message processing? Why does Zope require threads?
> > IOW, why would anybody *want* a "threadsafe patch for asynchat"?
> >
> In case the processing of events needed to block? If I'm processing web
> requests in an async* dispatch loop and a request needs the results of a
> (probably lengthy) database query in order to generate its output, how
> do I give the dispatcher control again to process the next asynchronous
> network event?
>
> The usual answer is "process the request in a thread". That way the
> dispatcher can spring to life for each event as quickly as needed.

but why do such threads have to talk to asyncore directly ?





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Steve Holden]

Martin v. Löwis wrote:
> Tim Peters wrote:
> 
>>Bugs and "missing features" in asyncore.  For ZEO's purposes, if I had
>>designed it, I expect it would have used threads (without asyncore). 
>>However, bits of code still sitting around suggest that it was at
>>least the _intent_ at one time that ZEO be able to run without threads
>>at all.  That's certainly not possible now.
> 
> 
> What is the reason that people want to use threads when they can have
> poll/select-style message processing? Why does Zope require threads?
> IOW, why would anybody *want* a "threadsafe patch for asynchat"?
> 
In case the processing of events needed to block? If I'm processing web 
requests in an async* dispatch loop and a request needs the results of a 
(probably lengthy) database query in order to generate its output, how 
do I give the dispatcher control again to process the next asynchronous 
network event?

The usual answer is "process the request in a thread". That way the 
dispatcher can spring to life for each event as quickly as needed.

regards
  Steve
-- 
Steve Holden   +44 150 684 7255  +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006  www.python.org/pycon/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Martin v. Löwis]

Tim Peters wrote:
> Bugs and "missing features" in asyncore.  For ZEO's purposes, if I had
> designed it, I expect it would have used threads (without asyncore). 
> However, bits of code still sitting around suggest that it was at
> least the _intent_ at one time that ZEO be able to run without threads
> at all.  That's certainly not possible now.

What is the reason that people want to use threads when they can have
poll/select-style message processing? Why does Zope require threads?
IOW, why would anybody *want* a "threadsafe patch for asynchat"?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Tim Peters]

[Josiah Carlson]
> ...
> Back to the topic that Guido was really complaining about: Zope +
> asyncore.  I don't doubt that getting Zope to play nicely with asyncore
> was difficult,

It's more that mixing asyncore with threads is a bloody nightmare, and
ZEO and Zope both do that.  Zope (but not ZEO) goes on to mix threads
with asynchat too.  In addition, ZEO makes life much harder than
should be necessary by running in two different modes and
auto-switching between them, depending on whether "the app" is or is
not running an asyncore mainloop itself.  In order to _detect_ when
"the app" fires up an asyncore mainloop, ZEO monkey-patches asyncore's
loop() function and physically replaces it with its own loop()
function.  It goes downhill from there.

Guido's memories are partly out of date now:  ZEO used to replace a
lot more of asyncore than it does now, because of bugs in the asyncore
distributed with older Python versions.  The _needs_ for that went
away little by little over the years, but the code in ZEO stuck around
much longer.  ZEO's ThreadedAsync/LoopCallback.py is much smaller now
(ZODB 3.6) than Guido remembers.

For a brief while, I even ripped out ZEO's monkey-patching of Python's
asyncore loop(), but it turned out that newer code in Zope3 (but not
Zope2) relied on, in turn, poking values into ZEO's module globals to
cause ZEO's loop() replacement to shut down (that's the kind of
"expedient" joy you get when mixing asyncore with threads).

Every piece of it remains "underdocumented" and, IMO, highly obscure.

> but it begs the questions: what would have been done if asyncore didn't exist,

Who knows?  What would python-dev be like if you didn't exist :-)?

> and why wasn't that done instead of trying to play nicely with asyncore?

Bugs and "missing features" in asyncore.  For ZEO's purposes, if I had
designed it, I expect it would have used threads (without asyncore). 
However, bits of code still sitting around suggest that it was at
least the _intent_ at one time that ZEO be able to run without threads
at all.  That's certainly not possible now.

If you look at asyncore's revision history, you'll note that Jeremy
and Guido made many changes when they worked at Zope Corp.  Those
largely reflect the history of moving ZEO's asyncore monkey-patches
into the Python core.

BTW, if you don't use ZEO, I believe it's possible to run Zope3
without asyncore (you can use Twisted in Zope3 instead).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Bill Janssen]

> what other reactive socket framework is there that would fit well into
> the standard library ?  is twisted really simple enough ?

I've been very happy with Medusa, which is asyncore-based.

Perhaps the right idea is to fix the various problems of asyncore.  We
might lift the similar code from the kernel of ILU, for example, which
carefully addresses the various issues around this style of action loop.

Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Neal Norwitz]

On 2/7/06, Christopher Armstrong <[EMAIL PROTECTED]> wrote:
>
> > Twisted is wonderful, powerful, rich, and very large.  Perhaps a small
> > subset could be carefully extracted
>
> The subject of putting (parts of) Twisted into the standard library
> comes up once every 6 months or so, at least on our mailing list. For
> all that I think asyncore is worthless, I'm still against copying
> Twisted into the stdlib. Or at least I'm not willing to maintain the
> necessary fork, and I fear the nightmares about versioning that can
> easily occur when you've got both standard library and third party
> versions of a project.

I wouldn't be enthusiastic about putting all of Twisted in the stdlib
either.  Twisted is on a different release schedule than Python. 
However, isn't there a relatively small core subset like Alex
mentioned that isn't changing much?  Could we split up those
components and have those live in the core, but the vast majority of
Twisted live outside as it does now?

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Christopher Armstrong]

On 2/8/06, Alex Martelli <[EMAIL PROTECTED]> wrote:
> On 2/7/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
>...
> > what other reactive socket framework is there that would fit well into
> > the standard library ?  is twisted really simple enough ?
>
> Twisted is wonderful, powerful, rich, and very large.  Perhaps a small
> subset could be carefully extracted that (given suitable volunteers to
> maintain it in the future) might fit in the standard library, but [a]
> that extraction is not going to be a simple or fast job, and [b] I
> suspect that the minimum sensible subset would still be much larger
> (and richer / more powerful) than asyncore.

The subject of putting (parts of) Twisted into the standard library
comes up once every 6 months or so, at least on our mailing list. For
all that I think asyncore is worthless, I'm still against copying
Twisted into the stdlib. Or at least I'm not willing to maintain the
necessary fork, and I fear the nightmares about versioning that can
easily occur when you've got both standard library and third party
versions of a project.

But, for the record, to the people who argue not to put Twisted into
the stdlib because of its size: The parts of it that would actually be
applicable (i.e. those that obselete async* in the stdlib) are only a
few kilobytes of code. At a quick run of "wc", the parts that support
event loops, accurate timed calls, SSL, Unix sockets, TCP, UDP,
arbitrary file descriptors, processes, and threads sums up to about
5300 lines of code. asynchat and asyncore are about 1200.

--
  Twisted   |  Christopher Armstrong: International Man of Twistery
   Radix|-- http://radix.twistedmatrix.com
|  Release Manager, Twisted Project
  \\\V///   |-- http://twistedmatrix.com
   |o O||
wvw-+
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Barry Warsaw]

On Tue, 2006-02-07 at 16:01 -0800, Robert Brewer wrote:

> Perhaps, but please keep in mind that the smtpd module uses both, currently, 
> and would have to be rewritten if either is "removed".

Would that really be a huge loss?

-Barry



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Josiah Carlson]

Guido van Rossum <[EMAIL PROTECTED]> wrote:
> IMO asynchat and asyncore are braindead. The should really be removed
> from the standard library. The code is 10 years old and represents at
> least 10-year-old thinking about how to do this. The amount of hackery
> in Zope related to asyncore was outrageous -- basically most of
> asyncore's guts were replaced with more advanced Zope code, but the
> API was maintained for compatibility reasons. A nightmare.

I'm going to go ahead and disagree with Guido on this one.  Before
removing asyncore (and asynchat) from the standard library, I believe
that there would necessarily need to be a viable replacement already
in place. The SocketServer module and its derivatives are wholly
unscalable for server-oriented applications once you get past a few
dozen threads (where properly designed asyncore derivatives will do
quite well all the way to your platform file handle limit).

Every once and a while I hear about people pushing for Twisted to be
included with Python, but at 2 megs for the base bz2 package, it seems a
little...hefty.  I'm not aware of any other community-accepted package
for asynchronous socket clients and servers, but I'm always looking.

Now, don't get me wrong, writing servers and clients using asyncore or
asynchat can be a beast, but it does get one into the callback/reactor
method of programming, which seems to have invaded other parts of Python
and 3rd party libraries (xml.sax, tk, Twisted, wxPython, ...).

Back to the topic that Guido was really complaining about: Zope +
asyncore.  I don't doubt that getting Zope to play nicely with asyncore
was difficult, but it begs the questions: what would have been done if
asyncore didn't exist, and why wasn't that done instead of trying to
play nicely with asyncore?

 - Josiah

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Alex Martelli]

On 2/7/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
   ...
> what other reactive socket framework is there that would fit well into
> the standard library ?  is twisted really simple enough ?

Twisted is wonderful, powerful, rich, and very large.  Perhaps a small
subset could be carefully extracted that (given suitable volunteers to
maintain it in the future) might fit in the standard library, but [a]
that extraction is not going to be a simple or fast job, and [b] I
suspect that the minimum sensible subset would still be much larger
(and richer / more powerful) than asyncore.


Alex
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Robert Brewer]

Guido van Rossum wrote:
> IMO asynchat and asyncore are braindead. The should really be removed
> from the standard library. The code is 10 years old and represents at
> least 10-year-old thinking about how to do this. The amount of hackery
> in Zope related to asyncore was outrageous -- basically most of
> asyncore's guts were replaced with more advanced Zope code, but the
> API was maintained for compatibility reasons. A nightmare.

Perhaps, but please keep in mind that the smtpd module uses both, currently, 
and would have to be rewritten if either is "removed".


Robert Brewer
System Architect
Amor Ministries
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Fredrik Lundh]

Guido van Rossum wrote:

> IMO asynchat and asyncore are braindead. The should really be removed
> from the standard library. The code is 10 years old and represents at
> least 10-year-old thinking about how to do this.

strange.  I'd say it works perfectly fine for what it was designed for
(after all, sockets haven't changed much in 10 years either).

what other reactive socket framework is there that would fit well into
the standard library ?  is twisted really simple enough ?





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Guido van Rossum]

IMO asynchat and asyncore are braindead. The should really be removed
from the standard library. The code is 10 years old and represents at
least 10-year-old thinking about how to do this. The amount of hackery
in Zope related to asyncore was outrageous -- basically most of
asyncore's guts were replaced with more advanced Zope code, but the
API was maintained for compatibility reasons. A nightmare.

--Guido

On 2/6/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Mark Edgington wrote:
> > Does anyone have any comments about applying the following patch to
> > asynchat?
>
> That patch looks wrong. What does it mean to "run in a thread"?
> All code runs in a thread, all the time: sometime, that thread
> is the main thread.
>
> Furthermore, I can't see any presumed thread-unsafety in asynchat.
>
> Sure, there is a lot of member variables in asynchat which aren't
> specifically protected against mutual access from different threads.
> So you shouldn't be accessing the same async_chat object from multiple
> threads. I cannot see why using a creating and using
> an async_chat object in a thread that is not the main thread
> could cause any problems. I also cannot see how this patch could
> have significant effect on asyn_chat's behaviour when used in
> multiple threads.
>
> Regards,
> Martin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] threadsafe patch for asynchat

[Martin v. Löwis]

Mark Edgington wrote:
> Does anyone have any comments about applying the following patch to 
> asynchat? 

That patch looks wrong. What does it mean to "run in a thread"?
All code runs in a thread, all the time: sometime, that thread
is the main thread.

Furthermore, I can't see any presumed thread-unsafety in asynchat.

Sure, there is a lot of member variables in asynchat which aren't
specifically protected against mutual access from different threads.
So you shouldn't be accessing the same async_chat object from multiple
threads. I cannot see why using a creating and using
an async_chat object in a thread that is not the main thread
could cause any problems. I also cannot see how this patch could
have significant effect on asyn_chat's behaviour when used in
multiple threads.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com