> > Se we need to know a bit more about your 4.5-hour program before we can > determine whether threads can help. There is light at the end of the > tunnel, however, since even if threads don't work it's possible that the > multiprocessing module will (assuming you have multi-processor hardware > at your disposal).
What my program is doing is sending each line to a function that processes it via pycurl(with urllib fallback),mysqlDB(with _mysql fallback). It check the mysql database to see if this line exists. If it doesn't then it sends it either via mysql query or pycurl. Depending on the option set in the functions. Some sections of the function have time.sleep(6) in them. Otherwise things won't work. This considerably slows down performance. If I thread all lines then it will process more at the same time. So that means there will be like 10 or set amount threads running doing all steps in the functions. posting forms, performing queries and waiting for form postings to process on the server, etc... I hope this adds more light at the end of that tunnel. It currently works under my ubuntu install of python(2.5.x) and bt's python(2.4.3). Then reason why I added a fallback to MySQLdb and pycurl is then a person can install this on a server that is hosted elsewhere. Where you can't install python modules, due to permissions and such. I want it to work everywhere. There's alot more to this application, I'm not sure I can disclose at the moment. Seeing as it can be used for good or bad. I don't want it to get in the wrong hands if it's public. OTOH, I think I'll make it public. That's all up in the air at the moment. One thing it that it does make life easier for me. A lot easier. Although, I haven't made money with it. Yet. Plus, I want to make pyGTK frontend for it. Looking into that too. I wouldn't be against a private team assembling to create this though. As long as I can get money out of it somehow. Cuz I'm broke. and I live with my mom. Not sure how anyone can help me there. But I'll throw it up in the air for all to see. Maybe somethings comes out of it. This program is an idea I've been building inside my garage(my room) for about a year and a half. Built in PHP and python, now. Would something that uses pycurl,mysql be good for threading? It doesn't run on SMP but maybe one day. I also need to look into how to make a python package out of it. I researched some stuff awhile ago, but I didn't quite need it then. I just wanted to see what I'm getting into. Any other stuff about this would be appreciated to. Although of topic. Sorry. By the way, I wanted to really thank everyone for all your help. It means a lot to me. -Alex Goretoy http://www.alexgoretoy.com On Mon, Jan 5, 2009 at 9:17 PM, Steve Holden <st...@holdenweb.com> wrote: > I did, once upon a time, write code > that used several hundred threads to send emails, and gave a dramatic > speed-up (because of the network-bound nature of the task). Can I > presume that your original inquiry was a toy, and that your real problem > is also IO-bound? Otherwise I am unsure how you will benefit by > threading - if your line-processing tasks don't contain any IO then > using a threaded approach will not yield any speed-up at all. > > The example you quoted achieved its speed-up because a thread releases > the GIL while waiting for a network response, allowing other threads to > process. Thus it effectively ran all the pings in parallel. > > Se we need to know a bit more about your 4.5-hour program before we can > determine whether threads can help. There is light at the end of the > tunnel, however, since even if threads don't work it's possible that the > multiprocessing module will (assuming you have multi-processor hardware > at your disposal). >
-- http://mail.python.org/mailman/listinfo/python-list
