Excellent! Thank you. On Tue, Apr 6, 2021 at 10:10 PM Devon McCormick <[email protected]> wrote:
> I have some old references with fairly detailed code here - > > https://code.jsoftware.com/wiki/Community/Conference2012/Talks/ParallelSimulationInJ > - and here - > > https://code.jsoftware.com/wiki/User:Devon_McCormick/ParallelizedJCodeExamples > . The latter one, though older, is more complete. I don't think I have > changed the code much since I wrote this. The only major change I can > think of is in the routine that monitors the multiple J processes: I use a > different underlying process monitor, either "pslist" or "tasklist" (both > for Windows). Also, I have played around with running multiple copies of > jconsole with distinct names, like J8Pll00.exe, J8Pll01.exe, and so on. > This is not necessary but I had the idea I might want to track the separate > processes, so giving them distinct names distinguishes them in a process > monitor. > > The first link has some caveats about peculiarities I uncovered: when I > shell out multiple processes, they seem to be linked into the same process > space even though they show up as independent instances of jconsole on a > process monitor. This seems to be relevant mostly when there are errors as > one has to peel back the stack manually, one process at a time; they are > distinguishable by having different ARGV_z_ values. > > The major change I have wanted to make is to use sockets so that a central > co-ordinator can parcel out the work in small pieces to a set of listening > processes. This would better even out the workload for what I'm doing > (flipping photos upright). The way it works now, the central process > pre-allocates the work to each process it spins off, so there is always one > that takes longer than all the others because it happened to get more > work. However, the difference between the fastest and slowest process is > not so great that I'm highly motivated to do this work, as interesting as > it would be, because the estimated gain is only on the order of a few > percent, 10% at the most. > > One hack in this code is that I do not know how to dynamically figure out > how many cores I have available, so I have set up a table with my machine > names and how many cores each has available. I've found that it pays to > spin off one less process than there are cores so that the machine is still > usable while the routines are running. By cores, I mean "virtual" cores so > my current 10-core (Intel i9-10900F) machine can run 20 CPU-hungry > processes, or 19 if I want to be able to do anything else on the machine > while they are running. > > What I've done is coarse-grained parallelism. Note that Marshall Lochbaum > presented a fine-grained approach at the 2012 J conference that is > complementary to this but I have not joined his work with my own. > > Please feel free to play around with this and ask questions. > > > On Tue, Apr 6, 2021 at 8:36 PM Jose Mario Quintana < > [email protected]> wrote: > > > Do you have handy a link to where can find your routines? It is probably > > close to what I have in mind given your description. > > > > On Tue, Apr 6, 2021 at 8:17 PM Devon McCormick <[email protected]> > wrote: > > > > > My home-made parallelization routines spin off multiple copies of J, > not > > > using fork explicitly, but it does give significant performance > > > improvement. It's very simple but works well enough on multi-cores > that > > > I've never been motivated enough to try to improve it. > > > > > > On Tue, Apr 6, 2021 at 7:47 PM Jose Mario Quintana < > > > [email protected]> wrote: > > > > > > > > $ j -js "exit echo 2 [ (fork&cd bind '') '' [ load > > > > > 'data/jd/server/fork'" > > > > > 2 > > > > > 2 > > > > > > > > It seems to me that the above construction works for the UNIX family > > but > > > > not for Windows; at least, I managed to run a version of the above > in a > > > > very basic BusyBox system but I could not figure out how to run a > > > > version of it in Windows 10. Am I wrong? (Admittedly, my knowledge > > > > regarding this matter is very limited.) > > > > > > > > > Practical, non-destructive use fork probably has a bunch of > caveats, > > > > > but it does also in C programs. > > > > > > > > Imagine, for instance, that one wants to evaluate hundreds of times > an > > > > expensive arbitrary verb (u), that takes minutes to produce a single > > > value, > > > > to plot the verb. In an ideal J world, u("_1) or u(&.>) could be > used > > to > > > > run the evaluations in parallel in minutes as opposed to run them > > > serially > > > > in hundreds of minutes. Back to reality, I can find (I think) a > > > cumbersome > > > > way, using fork_jtask_, to save significant time when the computer > has > > a > > > > multi-core processor running Windows or a UNIX family OS. However, I > > > > wonder how the experts would attack this kind of problem... > > > > > > > > > > > > > > > > On Thu, Apr 1, 2021 at 6:12 PM Julian Fondren < > > [email protected]> > > > > wrote: > > > > > > > > > > A fork bomb is more suited to POSIX fork, which J can use: > > > > > > > > > > NB. you might have to reboot if you run this > > > > > load 'data/jd/server/fork' > > > > > [ F. (fork&cd bind '') '' > > > > > > > > > > Tested separately: > > > > > > > > > > $ j -js "exit echo 2 [ (fork&cd bind '') '' [ load > > > > > 'data/jd/server/fork'" > > > > > 2 > > > > > 2 > > > > > > > > > > echoing 2 twice, from the two J processes, before they both exit. > > > > > > > > > > [ F. (echo bind 2) '' > > > > > > > > > > echoing 2 until interrupted. > > > > > > > > > > > > > > > Practical, non-destructive use fork probably has a bunch of > caveats, > > > > > but it does also in C programs. > > > > > > > > > > On 2021-04-01 16:08, Jose Mario Quintana wrote: > > > > > > Personally, I prefer the version where although the life of the > > > > > > individuals > > > > > > is ephemeral the species survives a lot longer, as it occurs in > > > nature. > > > > > > Either way, looking at the structure of the verb fork_jtask_ and > > its > > > > > > components, it seems to me that this is a kind of task far more > > > > > > suitable to > > > > > > C than J. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 31, 2021 at 10:17 PM Raul Miller < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > >> Sure, and here's a c program which will run into similar > resource > > > > > >> limits: > > > > > >> > > > > > >> main() { > > > > > >> while (1) { > > > > > >> fork(); > > > > > >> } > > > > > >> } > > > > > >> > > > > > >> This issue was probably one of the motivations for the ulimit > > > command > > > > > >> (which people almost never use, nowadays, because we have long > > since > > > > > >> learned to expect distributed programs to be well behaved). > > > > > >> > > > > > >> Take care, > > > > > >> > > > > > >> -- > > > > > >> Raul > > > > > >> > > > > > >> On Wed, Mar 31, 2021 at 6:39 PM Jose Mario Quintana > > > > > >> <[email protected]> wrote: > > > > > >> > > > > > > >> > For some reason, probably the pandemic, recent posts regarding > > the > > > > verb > > > > > >> > fork_jtask_ evoked old memories. In the late '70s, while > > reading > > > a > > > > > >> passage > > > > > >> > in a book describing Von Newman's scheme for constructing > > > > > >> self-replicating > > > > > >> > machines, I realized I could design a self-replicating process > > > > capable of > > > > > >> > running in the computer environment at work. The computer > was a > > > > > >> Burroughs > > > > > >> > B6700 and it had enabled the Inter Process Communication (IPC) > > > > facility > > > > > >> > which allowed a process to run another process. I wrote a > tiny > > > > program > > > > > >> and > > > > > >> > showed it as a curiosity to a few of my colleagues telling > them > > > that > > > > it > > > > > >> > would likely overwhelm the computer; but, for the same > reason, I > > > > could > > > > > >> not > > > > > >> > test it. > > > > > >> > > > > > > >> > Shortly after I went to work for another institution and, in > the > > > > early > > > > > >> > '80s, I moved from Mexico to England and I bought a little > > > > microcomputer > > > > > >> > called Sinclair QL. It had a multitasking OS called QDOS and > a > > > > BASIC > > > > > >> > variant called SuperBASIC which was also the QDOS' > command-line > > > > > >> > interpreter. So, I rewrote and ran a version of my tiny > program > > > > and, as > > > > > >> > expected, the only way out was to, literally, pull-the-plug. > > > > > >> > (Incidentally, the machine which looked almost like a keyboard > > was > > > > also > > > > > >> > capable to run QL APL, which was a special version of > MicroAPL's > > > > > >> APL.68000.) > > > > > >> > > > > > > >> > I had swamped not only j but also the OS a few times before, > but > > > > never > > > > > >> > intentionally. So, this is a first for me, the following > > fleeting > > > > > >> > script (beware of line-wrapping) runs in an earlier custom > > version > > > > of > > > > > >> the j > > > > > >> > interpreter on Windows 10 but it should be able to run in the > > > latest > > > > and > > > > > >> > greatest public versions of j and also on other platforms > > > (changing > > > > what > > > > > >> > needs to be changed); however, my strong advice, unless one > > likes > > > to > > > > live > > > > > >> > dangerously, is: > > > > > >> > > > > > > >> > DO NOT RUN IT! > > > > > >> > > > > > > >> > NB. Saved as J:/temp/Virus.ijs > > > > > >> > > > > > > >> > (2!:55)@:_:@:(([fork_jtask_)^:2) '"J:/Program > > Files/J/bin/jqt.exe" > > > > > >> > "J:/temp/Virus.ijs"' > > > > > >> > > > > > > >> > > > > > > >> > PS. Many years later while visiting an old friend in New > York, > > > who > > > > used > > > > > >> to > > > > > >> > be a member of the staff operating the B6700, he told me that > > one > > > of > > > > the > > > > > >> > most stressful times ever at work was when the B6700 suddenly > > kept > > > > > >> crashing > > > > > >> > and crashing for a few days, even missing a payroll deadline. > > The > > > > staff > > > > > >> > and the Burroughs technicians could not find anything wrong > with > > > the > > > > > >> > hardware. The issue was that the system was too clever, > after a > > > > crash it > > > > > >> > would automatically restart all the processes which were > > > > interrupted. > > > > > >> > Immediately after identifying the culprit, the sneaky tiny > > program > > > > which > > > > > >> > was very familiar to me, the general access to the IPC > facility > > > was > > > > > >> > disabled... > > > > > >> > > > > > > >> > Long live the verb fork_jtask_! :) > > > > > >> > > > > > > ---------------------------------------------------------------------- > > > > > >> > For information about J forums see > > > > http://www.jsoftware.com/forums.htm > > > > > >> > > > ---------------------------------------------------------------------- > > > > > >> For information about J forums see > > > > http://www.jsoftware.com/forums.htm > > > > > >> > > > > > > > > > ---------------------------------------------------------------------- > > > > > > For information about J forums see > > > http://www.jsoftware.com/forums.htm > > > > > > > ---------------------------------------------------------------------- > > > > > For information about J forums see > > http://www.jsoftware.com/forums.htm > > > > > ---------------------------------------------------------------------- > > > > For information about J forums see > http://www.jsoftware.com/forums.htm > > > > > > > > > > > > > -- > > > > > > Devon McCormick, CFA > > > > > > Quantitative Consultant > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > -- > > Devon McCormick, CFA > > Quantitative Consultant > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
