On Tue, Dec 22, 2015 at 3:20 AM, Mariano Martinez Peck <marianop...@gmail.com> wrote: > > > On Sun, Dec 20, 2015 at 5:21 PM, Thierry Goubier <thierry.goub...@gmail.com> > wrote: >> >> Hi Mariano, >> >> Le 20/12/2015 19:54, Mariano Martinez Peck a écrit : >>> >>> Hi Dimitri, Dave et all. >>> >>> First let me do a little disclaimer. Regarding the developments itself >>> of this sponsorship, we discussed about the topics I said in the first >>> email. However, those are not written in stone and that's why I also >>> asked for feedback. As you know, Pharo has relatively little resources >>> so those should be well spent. If we decide to take a spin into a >>> project X, it doesn't mean other projects are not worth. It simply means >>> that we think X will be the most useful for the time being. >>> >>> One of the final goals of the discusses topics, is to improve a bit the >>> infrastructure for using Pharo as scripting (imagine Coral). For such a >>> thing, we believe that improving FileSystem and OSProcess-like is a good >>> first step to improve the underlying infrastructure. >> >> >> Would we have access then to a command-line oriented minimal Pharo image? >> > > Well, just for putting perspective my sponsorship is about 3 full months of > work effort. It doesn't mean I won't work beyond that. But to give you an > idea. I think we all want minimal images , command line etc. And Pharo has > already done a huge effort in that direction.
>>> >>> That being said, I of course like the idea of distributing processing >>> across multiple images. No wonder, my PhD tool (Marea) and related >>> sub-projects (Fuel, Ghost proxies, etc) have something to do with the >>> mentioned topic. Dave already mentioned about RemoteTask, which was >>> something I was about to mention. There is also Seamless project (now >>> used by Remote Debugger stuff) and quite other tools. Finally, I think >>> this project may have a big percentage that could be considered as >>> "research". So having a PhD / master student working on this would also >>> be nice. In fact, maybe Denis Kudriashov might work on that topic??? (he >>> is now in RMOD team). >>> >>> Also, not that forking the image is not my main short-term goal. As >>> said, I will experiment first with a simple tool to execute OS commands >>> that would be FFI based, trying to make async operations, and that would >>> help in most cases. Maybe for those that are not supported, then >>> fallback in OSProcess. Note that Dave (OSProcess author) is very helpful >>> with me and we are discussing together about this. >> >> >> My guess is that, at least on Linux/Unix, that FFI will be exactly the >> same as the one used/expressed by the OSProcess plugin, that is >> popen/system. > > > No, wait. You are getting down to the details so I want to discuss :) > > OSProcess does not use popen()/system() but rather fork()+exec(). That means > that OSProcess has implementing the very very low building blocks that would > allow you do to almost everything. But then it has it's complicated matters. > fork() + exec() cannot be implemented via FFI (this was already discussed in > another thread). Can you link that thread? I'm curious to learn this. Is it multi-thread related? I read [a]... ..."The biggest problem with using fork+exec in this way is that you can't guarantee *nothing* happens between the fork call and the exec call. If, for example, the JVM decides to GC or move memory around, you can have a fatal crash at the JVM process level. Because of that, I don't recommend using fork + exec via FFI in JRuby, even though it's pretty cool. " ..."However, since this post I've learned of the "posix_spawn" [b] function available on most Unix variants. It's basically fork + exec in a single function, plus most of the typical security and IO tweaks you might do after forking and before execing. It's definitely my recommended alternative to fork+exec for JRuby." And posix_spawn also seems recommended for use with the JVM [d] and that Windows and POSIX might have similar usage, and even better match Windows CreateProcess than fork/exec [f] [g] [h] And since with 64-bit Spur we can expect very large images, spawn may be preferred over fork/exec to minimize memory usage for creating application subprocesses [i] Fork versus threads section of [c] or should we be making use of pthread_atfork [e] ? Overall I learnt something new about forking in a multi-threaded environment "When fork() is called, only the calling thread is duplicated in the child process." [a] http://blog.headius.com/2009/05/fork-and-exec-on-jvm-jruby-to-rescue.html [b] http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html [d] http://skife.org/java/2012/01/24/java_daemonization_with_gressil.html [c] http://repository.readscheme.org/ftp/papers/sw2002/gasbichler.pdf [e] http://stackoverflow.com/questions/4223553/c-does-exec-have-to-immediately-follow-fork-in-a-multi-threaded-process [f] http://stackoverflow.com/questions/33249253/what-is-the-difference-between-spawn-and-fork [g] http://stackoverflow.com/questions/5883462/linux-createprocess [h] http://www.linuxquestions.org/questions/programming-9/fork-exec-versus-posix_spawn-835385/ [i] http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html > However, there is a lot of other stuff that we believe it > could. So... what we wanted to see if it is possible is to have one only (or > very few) primitives (like forkAndExec) and all the rest via FFI. And then > all the features I listed in the first email of this thread. > > With that in mind, we believe that: > > 1) Before starting dealing with forkAndExec, we could try a first prototype > using popen(). That would allow us test a 100% FFI based approeach, deal > with writing and reading from pipes via stream, face the blocking issue, see > a prototype on how to read async from pipes etc etc etc. If we see this > works nicely, on a second step we can build another approach that instead of > popen() uses forkAndExec primitive. Is popen() being restricted to unidirectional part of this consideration [j] ? Also I read [j] that a problem with popen is that you do not get pid of child so you cannot wait on command to complete. Could posix_spawn be preferrable to popen? and just btw, mildly interesting is using sockets rather than pipes for interprocess communication [l] [j] http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipes-like-pipe-fork [k] http://stackoverflow.com/questions/6743771/popen-alternative [l] http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-functionality-i-o/25177958#25177958 > > 2) We believe that there is a large number of OSProcess users that use > OSProcess only for executing OS commands AND popen() approach would be more > than enough (we will do a survey about this). So, a very simply solution via > FFI for these users rather than having all OSProcess, Command Shell, the > plugins etc, may be worth. > > What do you think? > >> >> Now, given some of the difficulties seen when dealing with external >> processes, I'd be wary of the "async" side of things. Random failures >> seemingly linked to signal loss are certainly not something I'd like to >> debug. >> > > Indeed, it won't be easy I guess. But sometimes the process you execute may > take quite some time and blocking the VM for that period is not a good > choice. So... what would you prefer to solve that? Do you prefer image-based > pooling as OSProcess does when IOPlugin is not used? How much might your work overlap with the Threaded FFI project for Cog?... ..."The Cog VM is single-threaded, providing green threads to the Smalltalk level, as do most Smalltalk VMs. There is a prototype subclass of the CoInterpreter, CoInterpreterMT, which provides a Python-like multi-threaded VM where any number of threads can share the VM with only one running the VM at any one time. This uses an extremely simple low-level scheme for releasing and re-acquiring ownership of the VM designed by David Simmons. This results in any and all FFI calls being non-blocking since there is such a low overhead to making a non-blocking FFI call that all FFI calls can be non-blocking. Eliot has worked on this, implementing the prototype and the reentrant FFI plugin it requires. But this needs to be revived. It would be a really powerful enhancement." cheers -ben P.S. Just another article warning against mixing fork and multiple-threads (which it seems posix_spawn may avoid)... http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them
