Yes, that AWS project was an interesting task. I had around 6 terabytes of IVR voice logs and grammars that needed to be parsed and categorized, so the audio could be used to train voice recognition systems. Raul did a great job in helping me deal with that huge amount of data by spinning up lots of AWS machines in parallel, and the project was a success.
Skip Skip Cave Cave Consulting LLC On Wed, Apr 7, 2021 at 10:33 AM Raul Miller <[email protected]> wrote: > Back a few years ago, working on a project with Skip Cave, we were > spinning up ec2 instances with each instance running a copy of J. > > Conceptually speaking the mechanism there was making a backup of the > entire machine, and restoring it from backup. Each ec2 instance being > a copy restored from backup (with a little script which runs when the > machine turns on which would start J). > > That approach definitely gives a speedup -- with 100 ec2 machines you > get a 100x speedup. But the overhead (of making a backup and of > retrieving and saving results) means that it's a perceptual slow-down > for small tasks. It's only a speedup for tasks that take a really long > time (that one took months, even with the speedup). > > That said, I should also mention that that particular example was not > "pure j" work -- there was also work that happened outside of J. But > that's pretty normal, in my experience. It's a big world out there. > > FYI, > > -- > Raul > > > > On Tue, Apr 6, 2021 at 8:17 PM Devon McCormick <[email protected]> wrote: > > > > My home-made parallelization routines spin off multiple copies of J, not > > using fork explicitly, but it does give significant performance > > improvement. It's very simple but works well enough on multi-cores that > > I've never been motivated enough to try to improve it. > > > > On Tue, Apr 6, 2021 at 7:47 PM Jose Mario Quintana < > > [email protected]> wrote: > > > > > > $ j -js "exit echo 2 [ (fork&cd bind '') '' [ load > > > > 'data/jd/server/fork'" > > > > 2 > > > > 2 > > > > > > It seems to me that the above construction works for the UNIX family > but > > > not for Windows; at least, I managed to run a version of the above in a > > > very basic BusyBox system but I could not figure out how to run a > > > version of it in Windows 10. Am I wrong? (Admittedly, my knowledge > > > regarding this matter is very limited.) > > > > > > > Practical, non-destructive use fork probably has a bunch of caveats, > > > > but it does also in C programs. > > > > > > Imagine, for instance, that one wants to evaluate hundreds of times an > > > expensive arbitrary verb (u), that takes minutes to produce a single > value, > > > to plot the verb. In an ideal J world, u("_1) or u(&.>) could be used > to > > > run the evaluations in parallel in minutes as opposed to run them > serially > > > in hundreds of minutes. Back to reality, I can find (I think) a > cumbersome > > > way, using fork_jtask_, to save significant time when the computer has > a > > > multi-core processor running Windows or a UNIX family OS. However, I > > > wonder how the experts would attack this kind of problem... > > > > > > > > > > > > On Thu, Apr 1, 2021 at 6:12 PM Julian Fondren < > [email protected]> > > > wrote: > > > > > > > > A fork bomb is more suited to POSIX fork, which J can use: > > > > > > > > NB. you might have to reboot if you run this > > > > load 'data/jd/server/fork' > > > > [ F. (fork&cd bind '') '' > > > > > > > > Tested separately: > > > > > > > > $ j -js "exit echo 2 [ (fork&cd bind '') '' [ load > > > > 'data/jd/server/fork'" > > > > 2 > > > > 2 > > > > > > > > echoing 2 twice, from the two J processes, before they both exit. > > > > > > > > [ F. (echo bind 2) '' > > > > > > > > echoing 2 until interrupted. > > > > > > > > > > > > Practical, non-destructive use fork probably has a bunch of caveats, > > > > but it does also in C programs. > > > > > > > > On 2021-04-01 16:08, Jose Mario Quintana wrote: > > > > > Personally, I prefer the version where although the life of the > > > > > individuals > > > > > is ephemeral the species survives a lot longer, as it occurs in > nature. > > > > > Either way, looking at the structure of the verb fork_jtask_ and > its > > > > > components, it seems to me that this is a kind of task far more > > > > > suitable to > > > > > C than J. > > > > > > > > > > > > > > > > > > > > On Wed, Mar 31, 2021 at 10:17 PM Raul Miller < > [email protected]> > > > > > wrote: > > > > > > > > > >> Sure, and here's a c program which will run into similar resource > > > > >> limits: > > > > >> > > > > >> main() { > > > > >> while (1) { > > > > >> fork(); > > > > >> } > > > > >> } > > > > >> > > > > >> This issue was probably one of the motivations for the ulimit > command > > > > >> (which people almost never use, nowadays, because we have long > since > > > > >> learned to expect distributed programs to be well behaved). > > > > >> > > > > >> Take care, > > > > >> > > > > >> -- > > > > >> Raul > > > > >> > > > > >> On Wed, Mar 31, 2021 at 6:39 PM Jose Mario Quintana > > > > >> <[email protected]> wrote: > > > > >> > > > > > >> > For some reason, probably the pandemic, recent posts regarding > the > > > verb > > > > >> > fork_jtask_ evoked old memories. In the late '70s, while > reading a > > > > >> passage > > > > >> > in a book describing Von Newman's scheme for constructing > > > > >> self-replicating > > > > >> > machines, I realized I could design a self-replicating process > > > capable of > > > > >> > running in the computer environment at work. The computer was a > > > > >> Burroughs > > > > >> > B6700 and it had enabled the Inter Process Communication (IPC) > > > facility > > > > >> > which allowed a process to run another process. I wrote a tiny > > > program > > > > >> and > > > > >> > showed it as a curiosity to a few of my colleagues telling them > that > > > it > > > > >> > would likely overwhelm the computer; but, for the same reason, I > > > could > > > > >> not > > > > >> > test it. > > > > >> > > > > > >> > Shortly after I went to work for another institution and, in the > > > early > > > > >> > '80s, I moved from Mexico to England and I bought a little > > > microcomputer > > > > >> > called Sinclair QL. It had a multitasking OS called QDOS and a > > > BASIC > > > > >> > variant called SuperBASIC which was also the QDOS' command-line > > > > >> > interpreter. So, I rewrote and ran a version of my tiny program > > > and, as > > > > >> > expected, the only way out was to, literally, pull-the-plug. > > > > >> > (Incidentally, the machine which looked almost like a keyboard > was > > > also > > > > >> > capable to run QL APL, which was a special version of MicroAPL's > > > > >> APL.68000.) > > > > >> > > > > > >> > I had swamped not only j but also the OS a few times before, but > > > never > > > > >> > intentionally. So, this is a first for me, the following > fleeting > > > > >> > script (beware of line-wrapping) runs in an earlier custom > version > > > of > > > > >> the j > > > > >> > interpreter on Windows 10 but it should be able to run in the > latest > > > and > > > > >> > greatest public versions of j and also on other platforms > (changing > > > what > > > > >> > needs to be changed); however, my strong advice, unless one > likes to > > > live > > > > >> > dangerously, is: > > > > >> > > > > > >> > DO NOT RUN IT! > > > > >> > > > > > >> > NB. Saved as J:/temp/Virus.ijs > > > > >> > > > > > >> > (2!:55)@:_:@:(([fork_jtask_)^:2) '"J:/Program > Files/J/bin/jqt.exe" > > > > >> > "J:/temp/Virus.ijs"' > > > > >> > > > > > >> > > > > > >> > PS. Many years later while visiting an old friend in New York, > who > > > used > > > > >> to > > > > >> > be a member of the staff operating the B6700, he told me that > one of > > > the > > > > >> > most stressful times ever at work was when the B6700 suddenly > kept > > > > >> crashing > > > > >> > and crashing for a few days, even missing a payroll deadline. > The > > > staff > > > > >> > and the Burroughs technicians could not find anything wrong > with the > > > > >> > hardware. The issue was that the system was too clever, after a > > > crash it > > > > >> > would automatically restart all the processes which were > > > interrupted. > > > > >> > Immediately after identifying the culprit, the sneaky tiny > program > > > which > > > > >> > was very familiar to me, the general access to the IPC facility > was > > > > >> > disabled... > > > > >> > > > > > >> > Long live the verb fork_jtask_! :) > > > > >> > > > > ---------------------------------------------------------------------- > > > > >> > For information about J forums see > > > http://www.jsoftware.com/forums.htm > > > > >> > ---------------------------------------------------------------------- > > > > >> For information about J forums see > > > http://www.jsoftware.com/forums.htm > > > > >> > > > > > > ---------------------------------------------------------------------- > > > > > For information about J forums see > http://www.jsoftware.com/forums.htm > > > > > ---------------------------------------------------------------------- > > > > For information about J forums see > http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > > > > > -- > > > > Devon McCormick, CFA > > > > Quantitative Consultant > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
