Googled a nice example at
http://www.blog.pythonlibrary.org/2012/08/01/python-concurrency-an-example-of-a-queue/
though
it's python 2.x
import osimport Queueimport threadingimport urllib2
########################################################################class
Downloader(threading.Thread):
"""Threaded File Downloader"""
#----------------------------------------------------------------------
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue
#----------------------------------------------------------------------
def run(self):
while True:
# gets the url from the queue
url = self.queue.get()
# download the file
self.download_file(url)
# send a signal to the queue that the job is done
self.queue.task_done()
#----------------------------------------------------------------------
def download_file(self, url):
""""""
handle = urllib2.urlopen(url)
fname = os.path.basename(url)
with open(fname, "wb") as f:
while True:
chunk = handle.read(1024)
if not chunk: break
f.write(chunk)
#----------------------------------------------------------------------def
main(urls):
"""
Run the program
"""
queue = Queue.Queue()
# create a thread pool and give them a queue
for i in range(5):
t = Downloader(queue)
t.setDaemon(True)
t.start()
# give the queue some data
for url in urls:
queue.put(url)
# wait for the queue to finish
queue.join()
if __name__ == "__main__":
urls = ["http://www.irs.gov/pub/irs-pdf/f1040.pdf";,
"http://www.irs.gov/pub/irs-pdf/f1040a.pdf";,
"http://www.irs.gov/pub/irs-pdf/f1040ez.pdf";,
"http://www.irs.gov/pub/irs-pdf/f1040es.pdf";,
"http://www.irs.gov/pub/irs-pdf/f1040sb.pdf";]
main(urls)
On Sat, Jan 5, 2013 at 7:37 PM, Guido van Rossum <[email protected]> wrote:
> Yes. No (no time). You need to look for tutorials on threading in Python.
>
> On Sat, Jan 5, 2013 at 9:38 AM, xancorreu <[email protected]> wrote:
> > Do you refer on this:
> > http://docs.python.org/release/3.0.1/library/queue.html ?
> > Can you give me an inspiring code, thanks?
> >
> > Xan.
> >
> > Al 05/01/13 17:34, En/na Guido van Rossum ha escrit:
> >
> >> I recommend that you look at the Queue module. Think in terms of
> >> producers and consumers -- when the producer adds an item, it pushes
> >> it onto the queue. There may be multiple producers. The consumers are
> >> several threads that try to get items off the queue and process them.
> >> So your list becomes the queue.
> >>
> >> (BTW, this list is pretty inactive -- you're better off getting help
> >> on python-list.)
> >>
> >> On Sat, Jan 5, 2013 at 8:24 AM, xancorreu <[email protected]> wrote:
> >>>
> >>> Al 04/01/13 22:28, En/na Guido van Rossum ha escrit:
> >>>
> >>>> Hi Xan,
> >>>>
> >>>> You have simplified what you are trying to accomplish too far. Surely
> >>>> if it really was about popping from a list you shouldn't be using a
> >>>> thread pool. What is the real use case you are trying to model here?
> >>>>
> >>>> --Guido
> >>>
> >>>
> >>> Thanks, Guido, for answering. But why can't use threading for poping a
> >>> list?
> >>> My real case is a list of downloaded files that dynamically grows (as
> >>> user
> >>> add a file) and also it can lost elements (files which we downloaded).
> >>>
> >>> I was thinking to program concurrently, with futures, but before coding
> >>> the
> >>> biggest case, I select the essential (ans smaller) one.
> >>>
> >>> Any specific schema code for doing that?
> >>>
> >>> Thanks,
> >>> Xan.
> >>>
> >>>
> >>>> On Fri, Jan 4, 2013 at 11:59 AM, xancorreu <[email protected]>
> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I just want to translate this classic code:
> >>>>>
> >>>>> ml = [i for i in range(100)]
> >>>>>
> >>>>> while ml:
> >>>>> element = ml.pop()
> >>>>> print(element)
> >>>>>
> >>>>> to futures for running pop asynchronously. I started, but I don't see
> >>>>> how
> >>>>> to
> >>>>> do that. This [https://gist.github.com/4455376] is the first
> attempt,
> >>>>> but
> >>>>> it
> >>>>> fails when I put more elements on ml (that is ml is growing).
> >>>>>
> >>>>> Any hints?
> >>>>>
> >>>>> Thanks in advance,
> >>>>> Xan.
> >>>>> _______________________________________________
> >>>>> concurrency-sig mailing list
> >>>>> [email protected]
> >>>>> http://mail.python.org/mailman/listinfo/concurrency-sig
> >>>>
> >>>>
> >>>>
> >>> _______________________________________________
> >>> concurrency-sig mailing list
> >>> [email protected]
> >>> http://mail.python.org/mailman/listinfo/concurrency-sig
> >>
> >>
> >>
> >
> > _______________________________________________
> > concurrency-sig mailing list
> > [email protected]
> > http://mail.python.org/mailman/listinfo/concurrency-sig
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> concurrency-sig mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/concurrency-sig
>
_______________________________________________
concurrency-sig mailing list
[email protected]
http://mail.python.org/mailman/listinfo/concurrency-sig
