AFAIK threading is more for cpu intensive stuff In my case I'm opening a socket and sending the request(faster) and then babysitting the socket until it times out or receives the response(very slow) then parse the input(not nearly as much of this)
since this is I/O heavy, i saw asyncore recommended as the fastest option. i already artificially set the timeout limit very low for a socket to cut overhead. On Thu, Jan 10, 2013 at 10:22 AM, Scott Kragen <[email protected]> wrote: > Allison, > > Have you looked into threading for python? I have used this library in > several of projects because writing a thread pool from scratch started to > give me a headache. > > > http://code.activestate.com/recipes/577105-synchronization-decorator-for-class-methods/ > > The advantage of a thread pool is it can que based on the max amount of > threads you define. > > Scott > > On Thu, Jan 10, 2013 at 10:06 AM, allison nixon <[email protected]> wrote: > >> Say I'm writing a broadscanner for a pet project and I accidentally wrote >> the entire thing in Python. >> >> Using asyncore and sockets, the best I can get is 3 http requests per >> second, maybe 10 per second at the very max. For a scanner this is of >> course, very lame. >> >> My goal is to of course make as many http request as a desktop computer >> will handle, so perhaps 200 per second with whatever number of sockets >> waiting in the background for a response(which I check periodically for a >> response and parse when I get it) >> >> Is there any way I can acheive 200 requests per second without learning >> another programming language? >> >> _______________________________________________ >> Pauldotcom mailing list >> [email protected] >> http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom >> Main Web Site: http://pauldotcom.com >> > > > > -- > > ---------------------------------------------------------------------------------------------------------------------- > There is only one metric in security that can be truly measured and that > is failure! > > --- Jack Daniels > ---------------------------------------------------------------------------------------------------------------------- > > _______________________________________________ > Pauldotcom mailing list > [email protected] > http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom > Main Web Site: http://pauldotcom.com > -- _________________________________ Note to self: Pillage BEFORE burning.
_______________________________________________ Pauldotcom mailing list [email protected] http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom Main Web Site: http://pauldotcom.com
