Now it seems to run fine, under the same circumstances (on the SD card),
with the code unchanged.
On Tue, Aug 30, 2022 at 10:20 PM Devon McCormick <devon...@gmail.com> wrote:
> Unfortunately it looks like it was the same release when it worked as when
> it didn't:
> Engine: j904/j64avx2/windows
> Beta-e: commercial/2022-07-16T19:21:14
> Library: 9.04.03
> Platform: Win 64
>
> My problem may have to do with peculiarities of running on an SD card as
> though it were a regular hard-drive, so we can say this problem is not
> reproducible until I come up with a cleaner example or it goes away as
> mysteriously as it arrived.
>
> I have another multithreading project I will explore to see if the same
> problem occurs there. If it does, it should be simple to reproduce.
>
> Until then, thanks for listening.
>
>
> On Sat, Aug 27, 2022 at 1:49 PM Henry Rich <henryhr...@gmail.com> wrote:
>
>> What release version did it work on?
>>
>> Henry Rich
>>
>> On 8/27/2022 1:46 PM, Devon McCormick wrote:
>> > My older, non-multithreaded, version works by dividing files into
>> batches
>> > in advance but this means that there may be a 10-20% difference in task
>> > times because we do not know in advance how long it takes to process
>> each
>> > file. The advantage of the multithreaded version is that the variance
>> > between threads is minimal since all threads are pulling from the same
>> > stack.
>> > I should probably re-iterate that the code worked a few weeks ago and
>> has
>> > only started failing lately.
>> >
>> > On Sat, Aug 27, 2022 at 8:47 AM bill lam <bbill....@gmail.com> wrote:
>> >
>> >> I am not sure why you used mutex. If you collect all file names in a
>> >> subdirectory and divide it into 15 batches (apparently you want 15
>> tasks)
>> >> and feed each batch to a different tasks then there won't be any
>> >> collisions.
>> >>
>> >>
>> >> On Sat, 27 Aug 2022 at 11:50 AM Devon McCormick <devon...@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi,
>> >>> So, here's some code I put together over the past few months to use
>> >>> multi-threading to parallelize a time-consuming process.
>> >>>
>> >>> We start with the top-level function:
>> >>>
>> >>> NB.* startFlipping: wrapper to set up for "flipN" by creating threads,
>> >>> setting up MUTEX,
>> >>> NB. moving to chip photo dir and flipping them on the chip.
>> >>> startFlipping=: 3 : 0
>> >>> 12 startFlipping y
>> >>> :
>> >>> root=. y,':/DCIM/' NB. y is drive letter of photo chip.
>> >>> subds=. 0{"1 dir root,'*.' NB. Process all subdirs' photos
>> >>> svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0 NB. Fast
>> mutex
>> >>> for_sd. subds do. sd=. ;sd NB. Process each sub-dir
>> >>> 1!:44 ] root,sd
>> >>> if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1''
>> >>> tms=. flipN x<.15 NB. Show time, run x threads, no
>> more
>> >>> than 15.
>> >>> tm=. tm,~6!:1''
>> >>> end.
>> >>> end.
>> >>> 1!:44 svdir
>> >>> tm;tms
>> >>> NB.EG tms=. 12 startFlipping 'F' NB. Run 12 threads for all .JPGs
>> on F:
>> >>> )
>> >>>
>> >>> The thread-creation utility:
>> >>> createThreads=: 3 : '{{ 0 T. '''' }} ^:y]'''''
>> >>>
>> >>> Now we look at the cover for the core routine where we spin off the
>> >> threads
>> >>> in a for loop:
>> >>>
>> >>> flipN=: 3 : 0
>> >>> tms=. i.0
>> >>> while. y<#FLS do. pyx=. i.0
>> >>> for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end.
>> >>> tms=. tms,-/&>_2{.&.>pyx
>> >>> end.
>> >>> NB.EG tms=. flipN 10 NB. Run on 10 threads
>> >>> )
>> >>>
>> >>> This uses a "for" rather than "each" because, in testing, "for" seemed
>> >> less
>> >>> likely to crash.
>> >>>
>> >>> Finally, the deepest function that concerns multi-threaders:
>> >>>
>> >>> NB.* flip1Photo: flip a single photo.
>> >>> flip1Photo=: 3 : 0
>> >>> assert. *./nameExists &> 'MUTEX';'FLS' NB. Need these 2
>> globals:
>> >>> FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0 NB. Fast mutex
>> >>> tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1'' NB. Current thread
>> >>> number is "tn". "tm" is session start time.
>> >>> if. 0<#FLS do.
>> >>> myFl=. 'ThisFL',tntxt [ 11 T. MUTEX NB. Lock mutex
>> >>> (nms)=: split FLS [ nms=. myFl,' FLS' NB. Take 1 filename
>> from
>> >>> list
>> >>> 13 T. MUTEX NB. Unlock mutex
>> >>> if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end.
>> >>> end.
>> >>> tn,(6!:1''),tm NB. Thread #, end, start session time.
>> >>> NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1 NB. 2 threads
>> >>> )
>> >>>
>> >>> This concentrates all mutex-handling into one place and uses the
>> thread
>> >>> number to try to avoid collisions. If you wanted to try the code
>> above,
>> >>> you could replace "flipPhotoByOrientation" with anything of your
>> choice.
>> >>>
>> >>> Anyway, about a week or so ago, this code seemed to (mostly) work but
>> >> now I
>> >>> get this with only a second or so delay:
>> >>> 6!:2 'tms=. 12 startFlipping ''F''' NB. Run 12 threads for all
>> .JPGs
>> >> on
>> >>> F:
>> >>>
>> >>> Process shell<1> exited abnormally with code 5
>> >>>
>> >>> I'm afraid I may be on the bleeding edge here but does anyone have any
>> >>> ideas? I've done a lot of testing and gotten a few things to work a
>> >> little
>> >>> but not for very long. I used to be able to process hundreds of
>> photos
>> >>> with this code.
>> >>>
>> >>> Any help would be appreciated. The above code should be enough to
>> >>> re-produce the issue once "flipPhotoByOrientation" has been replaced
>> by
>> >>> some target function, preferably one that takes a few seconds to run.
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Devon
>> >>> --
>> >>>
>> >>> Devon McCormick, CFA
>> >>>
>> >>> Quantitative Consultant
>> >>> ----------------------------------------------------------------------
>> >>> For information about J forums see
>> http://www.jsoftware.com/forums.htm
>> >>>
>> >> ----------------------------------------------------------------------
>> >> For information about J forums see http://www.jsoftware.com/forums.htm
>> >>
>> >
>>
>> ----------------------------------------------------------------------
>> For information about J forums see http://www.jsoftware.com/forums.htm
>>
>
>
> --
>
> Devon McCormick, CFA
>
> Quantitative Consultant
>
>
--
Devon McCormick, CFA
Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
