I have some old references with fairly detailed code here -
https://code.jsoftware.com/wiki/Community/Conference2012/Talks/ParallelSimulationInJ
- and here -
https://code.jsoftware.com/wiki/User:Devon_McCormick/ParallelizedJCodeExamples
. The latter one, though older, is more complete. I don't think I have
changed the code much since I wrote this. The only major change I can
think of is in the routine that monitors the multiple J processes: I use a
different underlying process monitor, either "pslist" or "tasklist" (both
for Windows). Also, I have played around with running multiple copies of
jconsole with distinct names, like J8Pll00.exe, J8Pll01.exe, and so on.
This is not necessary but I had the idea I might want to track the separate
processes, so giving them distinct names distinguishes them in a process
monitor.
The first link has some caveats about peculiarities I uncovered: when I
shell out multiple processes, they seem to be linked into the same process
space even though they show up as independent instances of jconsole on a
process monitor. This seems to be relevant mostly when there are errors as
one has to peel back the stack manually, one process at a time; they are
distinguishable by having different ARGV_z_ values.
The major change I have wanted to make is to use sockets so that a central
co-ordinator can parcel out the work in small pieces to a set of listening
processes. This would better even out the workload for what I'm doing
(flipping photos upright). The way it works now, the central process
pre-allocates the work to each process it spins off, so there is always one
that takes longer than all the others because it happened to get more
work. However, the difference between the fastest and slowest process is
not so great that I'm highly motivated to do this work, as interesting as
it would be, because the estimated gain is only on the order of a few
percent, 10% at the most.
One hack in this code is that I do not know how to dynamically figure out
how many cores I have available, so I have set up a table with my machine
names and how many cores each has available. I've found that it pays to
spin off one less process than there are cores so that the machine is still
usable while the routines are running. By cores, I mean "virtual" cores so
my current 10-core (Intel i9-10900F) machine can run 20 CPU-hungry
processes, or 19 if I want to be able to do anything else on the machine
while they are running.
What I've done is coarse-grained parallelism. Note that Marshall Lochbaum
presented a fine-grained approach at the 2012 J conference that is
complementary to this but I have not joined his work with my own.
Please feel free to play around with this and ask questions.
On Tue, Apr 6, 2021 at 8:36 PM Jose Mario Quintana <
jose.mario.quint...@gmail.com> wrote:
> Do you have handy a link to where can find your routines? It is probably
> close to what I have in mind given your description.
>
> On Tue, Apr 6, 2021 at 8:17 PM Devon McCormick <devon...@gmail.com> wrote:
>
> > My home-made parallelization routines spin off multiple copies of J, not
> > using fork explicitly, but it does give significant performance
> > improvement. It's very simple but works well enough on multi-cores that
> > I've never been motivated enough to try to improve it.
> >
> > On Tue, Apr 6, 2021 at 7:47 PM Jose Mario Quintana <
> > jose.mario.quint...@gmail.com> wrote:
> >
> > > > $ j -js "exit echo 2 [ (fork&cd bind '') '' [ load
> > > > 'data/jd/server/fork'"
> > > > 2
> > > > 2
> > >
> > > It seems to me that the above construction works for the UNIX family
> but
> > > not for Windows; at least, I managed to run a version of the above in a
> > > very basic BusyBox system but I could not figure out how to run a
> > > version of it in Windows 10. Am I wrong? (Admittedly, my knowledge
> > > regarding this matter is very limited.)
> > >
> > > > Practical, non-destructive use fork probably has a bunch of caveats,
> > > > but it does also in C programs.
> > >
> > > Imagine, for instance, that one wants to evaluate hundreds of times an
> > > expensive arbitrary verb (u), that takes minutes to produce a single
> > value,
> > > to plot the verb. In an ideal J world, u("_1) or u(&.>) could be used
> to
> > > run the evaluations in parallel in minutes as opposed to run them
> > serially
> > > in hundreds of minutes. Back to reality, I can find (I think) a
> > cumbersome
> > > way, using fork_jtask_, to save significant time when the computer has
> a
> > > multi-core processor running Windows or a UNIX family OS. However, I
> > > wonder how the experts would attack this kind of problem...
> > >
> > >
> > >
> > > On Thu, Apr 1, 2021 at 6:12 PM Julian Fondren <
> jfond...@minimaltype.com>
> > > wrote:
> > > >
> > > > A fork bomb is more suited to POSIX fork, which J can use:
> > > >
> > > > NB. you might have to reboot if you run this
> > > > load 'data/jd/server/fork'
> > > > [ F. (fork&cd bind '') ''
> > > >
> > > > Tested separately:
> > > >
> > > > $ j -js "exit echo 2 [ (fork&cd bind '') '' [ load
> > > > 'data/jd/server/fork'"
> > > > 2
> > > > 2
> > > >
> > > > echoing 2 twice, from the two J processes, before they both exit.
> > > >
> > > > [ F. (echo bind 2) ''
> > > >
> > > > echoing 2 until interrupted.
> > > >
> > > >
> > > > Practical, non-destructive use fork probably has a bunch of caveats,
> > > > but it does also in C programs.
> > > >
> > > > On 2021-04-01 16:08, Jose Mario Quintana wrote:
> > > > > Personally, I prefer the version where although the life of the
> > > > > individuals
> > > > > is ephemeral the species survives a lot longer, as it occurs in
> > nature.
> > > > > Either way, looking at the structure of the verb fork_jtask_ and
> its
> > > > > components, it seems to me that this is a kind of task far more
> > > > > suitable to
> > > > > C than J.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Mar 31, 2021 at 10:17 PM Raul Miller <
> rauldmil...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Sure, and here's a c program which will run into similar resource
> > > > >> limits:
> > > > >>
> > > > >> main() {
> > > > >> while (1) {
> > > > >> fork();
> > > > >> }
> > > > >> }
> > > > >>
> > > > >> This issue was probably one of the motivations for the ulimit
> > command
> > > > >> (which people almost never use, nowadays, because we have long
> since
> > > > >> learned to expect distributed programs to be well behaved).
> > > > >>
> > > > >> Take care,
> > > > >>
> > > > >> --
> > > > >> Raul
> > > > >>
> > > > >> On Wed, Mar 31, 2021 at 6:39 PM Jose Mario Quintana
> > > > >> <jose.mario.quint...@gmail.com> wrote:
> > > > >> >
> > > > >> > For some reason, probably the pandemic, recent posts regarding
> the
> > > verb
> > > > >> > fork_jtask_ evoked old memories. In the late '70s, while
> reading
> > a
> > > > >> passage
> > > > >> > in a book describing Von Newman's scheme for constructing
> > > > >> self-replicating
> > > > >> > machines, I realized I could design a self-replicating process
> > > capable of
> > > > >> > running in the computer environment at work. The computer was a
> > > > >> Burroughs
> > > > >> > B6700 and it had enabled the Inter Process Communication (IPC)
> > > facility
> > > > >> > which allowed a process to run another process. I wrote a tiny
> > > program
> > > > >> and
> > > > >> > showed it as a curiosity to a few of my colleagues telling them
> > that
> > > it
> > > > >> > would likely overwhelm the computer; but, for the same reason, I
> > > could
> > > > >> not
> > > > >> > test it.
> > > > >> >
> > > > >> > Shortly after I went to work for another institution and, in the
> > > early
> > > > >> > '80s, I moved from Mexico to England and I bought a little
> > > microcomputer
> > > > >> > called Sinclair QL. It had a multitasking OS called QDOS and a
> > > BASIC
> > > > >> > variant called SuperBASIC which was also the QDOS' command-line
> > > > >> > interpreter. So, I rewrote and ran a version of my tiny program
> > > and, as
> > > > >> > expected, the only way out was to, literally, pull-the-plug.
> > > > >> > (Incidentally, the machine which looked almost like a keyboard
> was
> > > also
> > > > >> > capable to run QL APL, which was a special version of MicroAPL's
> > > > >> APL.68000.)
> > > > >> >
> > > > >> > I had swamped not only j but also the OS a few times before, but
> > > never
> > > > >> > intentionally. So, this is a first for me, the following
> fleeting
> > > > >> > script (beware of line-wrapping) runs in an earlier custom
> version
> > > of
> > > > >> the j
> > > > >> > interpreter on Windows 10 but it should be able to run in the
> > latest
> > > and
> > > > >> > greatest public versions of j and also on other platforms
> > (changing
> > > what
> > > > >> > needs to be changed); however, my strong advice, unless one
> likes
> > to
> > > live
> > > > >> > dangerously, is:
> > > > >> >
> > > > >> > DO NOT RUN IT!
> > > > >> >
> > > > >> > NB. Saved as J:/temp/Virus.ijs
> > > > >> >
> > > > >> > (2!:55)@:_:@:(([fork_jtask_)^:2) '"J:/Program
> Files/J/bin/jqt.exe"
> > > > >> > "J:/temp/Virus.ijs"'
> > > > >> >
> > > > >> >
> > > > >> > PS. Many years later while visiting an old friend in New York,
> > who
> > > used
> > > > >> to
> > > > >> > be a member of the staff operating the B6700, he told me that
> one
> > of
> > > the
> > > > >> > most stressful times ever at work was when the B6700 suddenly
> kept
> > > > >> crashing
> > > > >> > and crashing for a few days, even missing a payroll deadline.
> The
> > > staff
> > > > >> > and the Burroughs technicians could not find anything wrong with
> > the
> > > > >> > hardware. The issue was that the system was too clever, after a
> > > crash it
> > > > >> > would automatically restart all the processes which were
> > > interrupted.
> > > > >> > Immediately after identifying the culprit, the sneaky tiny
> program
> > > which
> > > > >> > was very familiar to me, the general access to the IPC facility
> > was
> > > > >> > disabled...
> > > > >> >
> > > > >> > Long live the verb fork_jtask_! :)
> > > > >> >
> > > ----------------------------------------------------------------------
> > > > >> > For information about J forums see
> > > http://www.jsoftware.com/forums.htm
> > > > >>
> > ----------------------------------------------------------------------
> > > > >> For information about J forums see
> > > http://www.jsoftware.com/forums.htm
> > > > >>
> > > > >
> > ----------------------------------------------------------------------
> > > > > For information about J forums see
> > http://www.jsoftware.com/forums.htm
> > > >
> ----------------------------------------------------------------------
> > > > For information about J forums see
> http://www.jsoftware.com/forums.htm
> > > ----------------------------------------------------------------------
> > > For information about J forums see http://www.jsoftware.com/forums.htm
> > >
> >
> >
> > --
> >
> > Devon McCormick, CFA
> >
> > Quantitative Consultant
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
> >
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
--
Devon McCormick, CFA
Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
