Re: [Jprogramming] Change in multi-threading?

[Henry Rich]

The 'runastask' option has been changed to 'worker'.

Henry Rich

On 8/27/2022 2:19 AM, Elijah Stone wrote:

Are you by any chance using wd to do the actual flipping?  As I recall, that is not thread-safe.  (When we make it threadsafe, it will most likely be by serialising everything, because qt is not threadsafe; so you wouldn't get a performance boost there, anyway.)  If not, what are you using?  It might be the culprit regardless.

I cannot reproduce any crashes using your code.  But I do see one issue: you are globally reassigning MUTEX in flip1Photo.  For me, that (correctly) causes errors to be thrown (when you attempt to unlock the not-already-locked mutex). But it might be the culprit, especially if you are not on the latest version (we used to use pthreads mutexes, which have ... interesting behaviour when misused.  Now using custom mutexes, which behave better and are faster). Does the crashing go away if you remove that assignment?

A note: this code is quite complicated by the manual partitioning of files to threads.  Have you seen u t.'runastask'?  It can automate away much of this work, and go faster too: just say something like flipPhotoByOrientation t.'runastask'&>photosList.  (If you are feeling particularly fancy, you can also sort the photos by size before starting, which will reduce starvation in cases of extreme asymmetry.)

 -E

On Fri, 26 Aug 2022, Devon McCormick wrote:

Hi,
So, here's some code I put together over the past few months to use
multi-threading to parallelize a time-consuming process.

We start with the top-level function:

NB.* startFlipping: wrapper to set up for "flipN" by creating threads,
setting up MUTEX,
NB. moving to chip photo dir and flipping them on the chip.
startFlipping=: 3 : 0
  12 startFlipping y
:
  root=. y,':/DCIM/'         NB. y is drive letter of photo chip.
  subds=. 0{"1 dir root,'*.'      NB. Process all subdirs' photos
  svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0    NB. Fast mutex
  for_sd. subds do. sd=. ;sd      NB. Process each sub-dir
      1!:44 ] root,sd
      if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1''
          tms=. flipN x<.15       NB. Show time, run x threads, no more
than 15.
          tm=. tm,~6!:1''
       end.
   end.
   1!:44 svdir
   tm;tms
NB.EG tms=. 12 startFlipping 'F'  NB. Run 12 threads for all .JPGs on F:
)

The thread-creation utility:
createThreads=: 3 : '{{ 0 T. '''' }} ^:y]'''''

Now we look at the cover for the core routine where we spin off the threads

in a for loop:

flipN=: 3 : 0
  tms=. i.0
  while. y<#FLS do. pyx=. i.0
      for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end.
      tms=. tms,-/&>_2{.&.>pyx
  end.
NB.EG tms=. flipN 10   NB. Run on 10 threads
)

This uses a "for" rather than "each" because, in testing, "for" seemed less

likely to crash.

Finally, the deepest function that concerns multi-threaders:

NB.* flip1Photo: flip a single photo.
flip1Photo=: 3 : 0
  assert. *./nameExists &> 'MUTEX';'FLS'    NB. Need these 2 globals:
FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0    NB. Fast mutex
  tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1''      NB. Current thread
number is "tn". "tm" is session start time.
  if. 0<#FLS do.
      myFl=. 'ThisFL',tntxt [ 11 T. MUTEX      NB. Lock mutex
      (nms)=: split FLS [ nms=. myFl,' FLS'     NB. Take 1 filename from
list
      13 T. MUTEX                           NB. Unlock mutex
      if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end.
  end.
  tn,(6!:1''),tm   NB. Thread #, end, start session time.
NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1     NB. 2 threads
)

This concentrates all mutex-handling into one place and uses the thread
number to try to avoid collisions.  If you wanted to try the code above,
you could replace "flipPhotoByOrientation" with anything of your choice.

Anyway, about a week or so ago, this code seemed to (mostly) work but now I

get this with only a second or so delay:

  6!:2 'tms=. 12 startFlipping ''F'''  NB. Run 12 threads for all .JPGs on

F:

Process shell<1> exited abnormally with code 5

I'm afraid I may be on the bleeding edge here but does anyone have any

ideas?  I've done a lot of testing and gotten a few things to work a little

but not for very long.  I used to be able to process hundreds of photos
with this code.

Any help would be appreciated.  The above code should be enough to
re-produce the issue once "flipPhotoByOrientation" has been replaced by
some target function, preferably one that takes a few seconds to run.

Thanks,

Devon
--

Devon McCormick, CFA

Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm

----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm

----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm