The verb dir doesn't work on mac, I think it only works on windows, After
some modifications and remove the second MUTEX=: 10 T. 0
It seems like forever will all cpu full load.

I have only 1 short folder and about 1000 files.
My definition for flipPhotoByOrientation is 0:

I suspect your previous version was able to run by accident. Eg you didn't
protect every instance where FLS is accessed (read or set).


On Sun, 28 Aug 2022 at 1:46 AM Devon McCormick <devon...@gmail.com> wrote:

> My older, non-multithreaded, version works by dividing files into batches
> in advance but this means that there may be a 10-20% difference in task
> times because we do not know in advance how long it takes to process each
> file.  The advantage of the multithreaded version is that the variance
> between threads is minimal since all threads are pulling from the same
> stack.
> I should probably re-iterate that the code worked a few weeks ago and has
> only started failing lately.
>
> On Sat, Aug 27, 2022 at 8:47 AM bill lam <bbill....@gmail.com> wrote:
>
> > I am not sure why you used mutex. If you collect all file names in a
> > subdirectory and divide it into 15 batches (apparently you want 15 tasks)
> > and feed each batch to a different tasks then there won't be any
> > collisions.
> >
> >
> > On Sat, 27 Aug 2022 at 11:50 AM Devon McCormick <devon...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > So, here's some code I put together over the past few months to use
> > > multi-threading to parallelize a time-consuming process.
> > >
> > > We start with the top-level function:
> > >
> > > NB.* startFlipping: wrapper to set up for "flipN" by creating threads,
> > > setting up MUTEX,
> > > NB. moving to chip photo dir and flipping them on the chip.
> > > startFlipping=: 3 : 0
> > >    12 startFlipping y
> > > :
> > >    root=. y,':/DCIM/'         NB. y is drive letter of photo chip.
> > >    subds=. 0{"1 dir root,'*.'      NB. Process all subdirs' photos
> > >    svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0    NB. Fast
> mutex
> > >    for_sd. subds do. sd=. ;sd      NB. Process each sub-dir
> > >        1!:44 ] root,sd
> > >        if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1''
> > >            tms=. flipN x<.15       NB. Show time, run x threads, no
> more
> > > than 15.
> > >            tm=. tm,~6!:1''
> > >         end.
> > >     end.
> > >     1!:44 svdir
> > >     tm;tms
> > > NB.EG tms=. 12 startFlipping 'F'  NB. Run 12 threads for all .JPGs on
> F:
> > > )
> > >
> > > The thread-creation utility:
> > > createThreads=: 3 : '{{ 0 T. '''' }} ^:y]'''''
> > >
> > > Now we look at the cover for the core routine where we spin off the
> > threads
> > > in a for loop:
> > >
> > > flipN=: 3 : 0
> > >    tms=. i.0
> > >    while. y<#FLS do. pyx=. i.0
> > >        for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end.
> > >        tms=. tms,-/&>_2{.&.>pyx
> > >    end.
> > > NB.EG tms=. flipN 10   NB. Run on 10 threads
> > > )
> > >
> > > This uses a "for" rather than "each" because, in testing, "for" seemed
> > less
> > > likely to crash.
> > >
> > > Finally, the deepest function that concerns multi-threaders:
> > >
> > > NB.* flip1Photo: flip a single photo.
> > > flip1Photo=: 3 : 0
> > >    assert. *./nameExists &> 'MUTEX';'FLS'    NB. Need these 2 globals:
> > > FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0    NB. Fast mutex
> > >    tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1''      NB. Current thread
> > > number is "tn". "tm" is session start time.
> > >    if. 0<#FLS do.
> > >        myFl=. 'ThisFL',tntxt [ 11 T. MUTEX      NB. Lock mutex
> > >        (nms)=: split FLS [ nms=. myFl,' FLS'     NB. Take 1 filename
> from
> > > list
> > >        13 T. MUTEX                           NB. Unlock mutex
> > >        if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end.
> > >    end.
> > >    tn,(6!:1''),tm   NB. Thread #, end, start session time.
> > > NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1     NB. 2 threads
> > > )
> > >
> > > This concentrates all mutex-handling into one place and uses the thread
> > > number to try to avoid collisions.  If you wanted to try the code
> above,
> > > you could replace "flipPhotoByOrientation" with anything of your
> choice.
> > >
> > > Anyway, about a week or so ago, this code seemed to (mostly) work but
> > now I
> > > get this with only a second or so delay:
> > >    6!:2 'tms=. 12 startFlipping ''F'''  NB. Run 12 threads for all
> .JPGs
> > on
> > > F:
> > >
> > > Process shell<1> exited abnormally with code 5
> > >
> > > I'm afraid I may be on the bleeding edge here but does anyone have any
> > > ideas?  I've done a lot of testing and gotten a few things to work a
> > little
> > > but not for very long.  I used to be able to process hundreds of photos
> > > with this code.
> > >
> > > Any help would be appreciated.  The above code should be enough to
> > > re-produce the issue once "flipPhotoByOrientation" has been replaced by
> > > some target function, preferably one that takes a few seconds to run.
> > >
> > > Thanks,
> > >
> > > Devon
> > > --
> > >
> > > Devon McCormick, CFA
> > >
> > > Quantitative Consultant
> > > ----------------------------------------------------------------------
> > > For information about J forums see http://www.jsoftware.com/forums.htm
> > >
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
> >
>
>
> --
>
> Devon McCormick, CFA
>
> Quantitative Consultant
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm

Reply via email to