Hi,
: I am currently working on a project , which has lots of curl
: commands. I am using subprocess.Popen to run these curl commands.
: But somehow i see its bit slow. Is there a way/module by which i
: can improve the performance of the program.
Wow. Is curl a requirement or do you simply need to speak HTTP?
Some suggestions and questions come to my mind:
* Is your problem / cost the forking (i.e. how many times are you
calling curl)?
* Is the problem unresponsive or sluggish HTTP servers?
* Have you localized the slowness? (I use logging data and timing
information to try to figure that out.)
* How do you define 'slow'? How many curl commands is 'lots'?
Like 100 or 100,000?
Suggestions:
* Profile the code and see where the slowness is.
* Consider python-requests [0] to handle the HTTP fetches, instead
of calling subprocess.
* If after that, you discover that your bottleneck is mostly
network, consider a task parallelizing solution. One option
could be multiprocessing [1]. There are doubtless others.
-Martin
[0] http://docs.python-requests.org/en/latest/
[1] https://docs.python.org/2/library/multiprocessing.html
--
Martin A. Brown
http://linux-ip.net/
_______________________________________________
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
