Re: Event Driven programming - Doubts
Kottiyath wrote: [...] I have not yet understood the implementation of deferred. I went through a lot of tutorials, but I guess most places they expect that the user already understands how events are generated. The tutorials mention that there is no more threads once twisted is used. My question is as follows: I have not understood how the callbacks are hit without (a) blocking the code or (b) having new threads. The usual example given is that of a program waiting for data coming through a socket. In the tutorials, it is mentioned that -in an event driven program, we schedule the code to hit when the remote server gets back to us - . Now, my question is - somebody has to still wait on that socket and check whether the data is received, and once all the data is received, call the appropriate callbacks. Is twisted creating a micro-thread which just waits on the socket and once the data is received, calls callFromThread for it to run on the main loop? No. Event-driven frameworks are built upon some system call that waits for events from multiple sources at once. The Python standard library exposes several such calls, the most portable of which is select(), in the module also named "select". http://docs.python.org/library/select.html So how is something like deferred implemented? When you schedule action on your socket, you are telling the framework to include your socket in in subsequent calls to select(), until the socket selects as ready. The framework keeps a single global list of sockets on which clients are waiting, and from this list it builds the select() parameters. A single call to select() waits on behalf of all the callbacks. The select() call returns a list of sockets ready for I/0; the framework iterates over the ready list, invoking the corresponding callbacks one by one. After the last callback returns, the framework loops back to select() again. select() is not the only call to do multi-source I/O, and I'm not an expert on these frameworks, so take the above as a simplified general description. -- --Bryan -- http://mail.python.org/mailman/listinfo/python-list
Re: Event Driven programming - Doubts
On Tue, Dec 23, 2008 at 12:57 AM, Kottiyath wrote: > Hi, >I have been looking at Twisted and lately Circuits as examples for > event driven programming in Python. Wonderful! :) "circuits" that is :) >Even though I understood how to implement the code in these and > what is deferred etc, I have not yet understood the implementation of > deferred. I went through a lot of tutorials, but I guess most places > they expect that the user already understands how events are > generated. The tutorials mention that there is no more threads once > twisted is used. deferred to me is nothing more than scheduling "things" to happen "later". ie: Timers, or Timed Events. I clearly biased here as I'm the developer of circuits, so I can't comment on Twisted's design or architecture, but the way to achieve Twisted's so-called deferred (events?) is to use the Timer component, Timed Events. >My question is as follows: >I have not understood how the callbacks are hit without (a) > blocking the code or (b) having new threads. Again speaking in terms of circuits, but also in the event-driven paradigm: a) Event Handlers _can_ block - but ideally shouldn't (ihmo). b) Forking new threads/processes per event is mostly not necessary. (There are exceptions). > The usual example given is that of a program waiting for data coming > through a socket. In the tutorials, it is mentioned that -in an event > driven program, we schedule the code to hit when the remote server > gets back to us - . > Now, my question is - somebody has to still wait on that socket and > check whether the data is received, and once all the data is received, > call the appropriate callbacks. In the case of circuits' TCPServer component you simply poll it in your main event-loop. The pattern looks like this: from circuits.lib.sockets import TCPServer server = TCPServer(8000) while True: server.poll() server.flush() The TCPServer component has various builtin event handlers that do all the work for you and expose other more useful events to the application, such as: * connect * disconnect * read * error > Is twisted creating a micro-thread which just waits on the socket and > once the data is received, calls callFromThread for it to run on the > main loop? I can't comment - I tend to avoid threads where ever possible. > If so, Even though data locking etc is not a problem, are we not still > having threads? Will it not still cause scalability problems in high > traffic? Regarding scalability btw ... (in case you're interested) circuits comes with circuits.lib.web and several sets of Web Components. A lot of code was borrowed from the BaseHTTPServer from the python standard library and bits and pieces from CherryPy and the cgi module In terms of performance, it has a "raw" performance ~3k req/s on good hardware. Do have fun in your endeavours :) cheers James Circuits: http://trac.softcircuit.com.au/circuits/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Event Driven programming - Doubts
On Mon, Dec 22, 2008 at 9:57 AM, Kottiyath wrote: > > If so, Even though data locking etc is not a problem, are we not still > having threads? Will it not still cause scalability problems in high > traffic? > If not, could somebody let me know how it is done? This somewhat depends on the application. Is it IO bound or CPU bound? >From what I understand about twisted you will only have one process and one thread which means you will only be using one CPU. -- David http://www.traceback.org -- http://mail.python.org/mailman/listinfo/python-list
Re: Event Driven programming - Doubts
On Mon, 22 Dec 2008 06:57:55 -0800 (PST), Kottiyath wrote: Hi, I have been looking at Twisted and lately Circuits as examples for event driven programming in Python. [snip] My question is as follows: I have not understood how the callbacks are hit without (a) blocking the code or (b) having new threads. There is blocking code - but just in one place. For example, http://twistedmatrix.com/trac/browser/trunk/twisted/internet/selectreactor.py#L93 Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Event Driven programming - Doubts
Hi, I have been looking at Twisted and lately Circuits as examples for event driven programming in Python. Even though I understood how to implement the code in these and what is deferred etc, I have not yet understood the implementation of deferred. I went through a lot of tutorials, but I guess most places they expect that the user already understands how events are generated. The tutorials mention that there is no more threads once twisted is used. My question is as follows: I have not understood how the callbacks are hit without (a) blocking the code or (b) having new threads. The usual example given is that of a program waiting for data coming through a socket. In the tutorials, it is mentioned that -in an event driven program, we schedule the code to hit when the remote server gets back to us - . Now, my question is - somebody has to still wait on that socket and check whether the data is received, and once all the data is received, call the appropriate callbacks. Is twisted creating a micro-thread which just waits on the socket and once the data is received, calls callFromThread for it to run on the main loop? If so, Even though data locking etc is not a problem, are we not still having threads? Will it not still cause scalability problems in high traffic? If not, could somebody let me know how it is done? -- http://mail.python.org/mailman/listinfo/python-list
