Re: multiprocessing problems

[Nils Ruettershoff]

Hi,

Adam Tauno Williams wrote:
[...]
Here's the guts of my latest incarnation.
def ProcessBatch(files):
    p = []
    for file in files:
        p.append(Process(target=ProcessFile,args=file))
    for x in p:
        x.start()
    for x in p:
        x.join()
    p = []
    return
Now, the function calling ProcessBatch looks like this:
def ReplaceIt(files):
    processFiles = []
    for replacefile in files:
        if(CheckSkipFile(replacefile)):
            processFiles.append(replacefile)
            if(len(processFiles) == 4):
                ProcessBatch(processFiles)
                processFiles = []
    #check for left over files once main loop is done and process them
    if(len(processFiles) > 0):
        ProcessBatch(processFiles)
    

According to this you will create files is sets of four, but an unknown
number of sets of four.

  
This is not correct, 'cause of the x.join(). This will block the parent 
process until all processes have been terminated. So as soon as the 
current set of processes have finished their job, a new set will be spawned.

Cheers,
Nils
-- 
http://mail.python.org/mailman/listinfo/python-list