On Aug 9, 5:39 pm, MRAB <[EMAIL PROTECTED]> wrote: > On Aug 9, 7:25 pm, [EMAIL PROTECTED] wrote: > > > Hi all! I'm implementing one of my first multithreaded apps, and have > > gotten to a point where I think I'm going off track from a standard > > idiom. Wondering if anyone can point me in the right direction. > > > The script will run as a daemon and watch a given directory for new > > files. Once it determines that a file has finished moving into the > > watch folder, it will kick off a process on one of the files. Several > > of these could be running at any given time up to a max number of > > threads. > > > Here's how I have it designed so far. The main thread starts a > > Watch(threading.Thread) class that loops and searches a directory for > > files. It has been passed a Queue.Queue() object (watch_queue), and > > as it finds new files in the watch folder, it adds the file name to > > the queue. > > > The main thread then grabs an item off the watch_queue, and kicks off > > processing on that file using another class Worker(threading.thread). > > > My problem is with communicating between the threads as to which files > > are currently processing, or are already present in the watch_queue so > > that the Watch thread does not continuously add unneeded files to the > > watch_queue to be processed. For example...Watch() finds a file to be > > processed and adds it to the queue. The main thread sees the file on > > the queue and pops it off and begins processing. Now the file has > > been removed from the watch_queue, and Watch() thread has no way of > > knowing that the other Worker() thread is processing it, and shouldn't > > pick it up again. So it will see the file as new and add it to the > > queue again. PS.. The file is deleted from the watch folder after it > > has finished processing, so that's how i'll know which files to > > process in the long term. > > I would suggest something like the following in the watch thread: > > seen_files = {} > > while True: > # look for new files > for name in os.listdir(folder): > if name not in seen_files: > process_queue.add(name) > seen_files[name] = True > > # forget any missing files and mark the others as not seen, ready for > next time > seen_files = dict((name, False) for name, seen in seen_files.items() > if seen) > > time.sleep(1)
Hmm, this wouldn't work. It's not thread safe and the last line before you sleep doesn't make any sense. -- http://mail.python.org/mailman/listinfo/python-list