The verb dir doesn't work on mac, I think it only works on windows, After some modifications and remove the second MUTEX=: 10 T. 0 It seems like forever will all cpu full load.
I have only 1 short folder and about 1000 files. My definition for flipPhotoByOrientation is 0: I suspect your previous version was able to run by accident. Eg you didn't protect every instance where FLS is accessed (read or set). On Sun, 28 Aug 2022 at 1:46 AM Devon McCormick <devon...@gmail.com> wrote: > My older, non-multithreaded, version works by dividing files into batches > in advance but this means that there may be a 10-20% difference in task > times because we do not know in advance how long it takes to process each > file. The advantage of the multithreaded version is that the variance > between threads is minimal since all threads are pulling from the same > stack. > I should probably re-iterate that the code worked a few weeks ago and has > only started failing lately. > > On Sat, Aug 27, 2022 at 8:47 AM bill lam <bbill....@gmail.com> wrote: > > > I am not sure why you used mutex. If you collect all file names in a > > subdirectory and divide it into 15 batches (apparently you want 15 tasks) > > and feed each batch to a different tasks then there won't be any > > collisions. > > > > > > On Sat, 27 Aug 2022 at 11:50 AM Devon McCormick <devon...@gmail.com> > > wrote: > > > > > Hi, > > > So, here's some code I put together over the past few months to use > > > multi-threading to parallelize a time-consuming process. > > > > > > We start with the top-level function: > > > > > > NB.* startFlipping: wrapper to set up for "flipN" by creating threads, > > > setting up MUTEX, > > > NB. moving to chip photo dir and flipping them on the chip. > > > startFlipping=: 3 : 0 > > > 12 startFlipping y > > > : > > > root=. y,':/DCIM/' NB. y is drive letter of photo chip. > > > subds=. 0{"1 dir root,'*.' NB. Process all subdirs' photos > > > svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0 NB. Fast > mutex > > > for_sd. subds do. sd=. ;sd NB. Process each sub-dir > > > 1!:44 ] root,sd > > > if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1'' > > > tms=. flipN x<.15 NB. Show time, run x threads, no > more > > > than 15. > > > tm=. tm,~6!:1'' > > > end. > > > end. > > > 1!:44 svdir > > > tm;tms > > > NB.EG tms=. 12 startFlipping 'F' NB. Run 12 threads for all .JPGs on > F: > > > ) > > > > > > The thread-creation utility: > > > createThreads=: 3 : '{{ 0 T. '''' }} ^:y]''''' > > > > > > Now we look at the cover for the core routine where we spin off the > > threads > > > in a for loop: > > > > > > flipN=: 3 : 0 > > > tms=. i.0 > > > while. y<#FLS do. pyx=. i.0 > > > for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end. > > > tms=. tms,-/&>_2{.&.>pyx > > > end. > > > NB.EG tms=. flipN 10 NB. Run on 10 threads > > > ) > > > > > > This uses a "for" rather than "each" because, in testing, "for" seemed > > less > > > likely to crash. > > > > > > Finally, the deepest function that concerns multi-threaders: > > > > > > NB.* flip1Photo: flip a single photo. > > > flip1Photo=: 3 : 0 > > > assert. *./nameExists &> 'MUTEX';'FLS' NB. Need these 2 globals: > > > FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0 NB. Fast mutex > > > tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1'' NB. Current thread > > > number is "tn". "tm" is session start time. > > > if. 0<#FLS do. > > > myFl=. 'ThisFL',tntxt [ 11 T. MUTEX NB. Lock mutex > > > (nms)=: split FLS [ nms=. myFl,' FLS' NB. Take 1 filename > from > > > list > > > 13 T. MUTEX NB. Unlock mutex > > > if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end. > > > end. > > > tn,(6!:1''),tm NB. Thread #, end, start session time. > > > NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1 NB. 2 threads > > > ) > > > > > > This concentrates all mutex-handling into one place and uses the thread > > > number to try to avoid collisions. If you wanted to try the code > above, > > > you could replace "flipPhotoByOrientation" with anything of your > choice. > > > > > > Anyway, about a week or so ago, this code seemed to (mostly) work but > > now I > > > get this with only a second or so delay: > > > 6!:2 'tms=. 12 startFlipping ''F''' NB. Run 12 threads for all > .JPGs > > on > > > F: > > > > > > Process shell<1> exited abnormally with code 5 > > > > > > I'm afraid I may be on the bleeding edge here but does anyone have any > > > ideas? I've done a lot of testing and gotten a few things to work a > > little > > > but not for very long. I used to be able to process hundreds of photos > > > with this code. > > > > > > Any help would be appreciated. The above code should be enough to > > > re-produce the issue once "flipPhotoByOrientation" has been replaced by > > > some target function, preferably one that takes a few seconds to run. > > > > > > Thanks, > > > > > > Devon > > > -- > > > > > > Devon McCormick, CFA > > > > > > Quantitative Consultant > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > -- > > Devon McCormick, CFA > > Quantitative Consultant > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm