Re: [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Dimitris Chloupis]

I mentions this because in OSProcess-Unix there is UnixProcess class, at
the class side there are several examples that launch a separate instance
of Squeak/Pharo and make the two communicate with each other. In the
instance of the class there are methods like forkHeadlessSqueak ,
forkHeadlessSqueakAndDo: etc.

So the functionality is there but since Mariano said he will be working to
improve OSProcess I thought that would a very useful area to work on. Since
not only will allows running on multiple cores but even to lift any memory
limits. Also I think this is a nice way to make the pharo enviroment which
I mean the running vm with an image and satelite files not only a
collection of objects but also a object itself  that can communicate with
other pharo enviroment instances (Pharo running processes) .

Fuel I assume enters here the equation as a data exchange format , the
problem I have with fuel is that its not backward compatible which for me
is a very big deal especially when we talk about threading and processes
which is a pretty fundamental feature. We need something that will be
stable but also code orientated that will remain backward compatible
because the last thing you want is initiating multiple processes that cant
understand each other because they happen to use diffirent image versions
that use diffirent verrsions of Fuel or what else.

On Sun, Dec 20, 2015 at 1:46 AM David T. Lewis  wrote:

> Mariano,
>
> I am really happy to see this and I look forword to helping you in
> any way I can.
>
> Dimitris,
>
> It is really interesting that you should suggest this. I think it may
> be a bit off topic for Mariano's initial goals, but Mariano is the
> driving force behind Fuel, which is the serializer that I use for
> RemoteTask in OSProcess/CommandShell. This attempts to do much of
> what you suggest, using Fuel to return the results of processing
> executed in a forked image (in the unix sense of "forked", i.e.
> extremely memory efficient).
>
> So maybe if I can help Mariano with his project (which of course I
> plan to do anyway), then he might be able to help me turn RemoteTask
> into something reliable enough for general purpose use.
>
> Another nice connection here is that Eliot's Spur image model will
> make #forkSqueak (the primitive behind RemoteTask) more efficient
> in cases where the cooperating images do a lot of processing, because
> memory changes are more localized in the Spur memory model. In principle
> it should be possible to run very large object memories in 64-bit Spur,
> while forking cooperating images that consume very little additional
> real memory.
>
> Dave
>
>
> On Sat, Dec 19, 2015 at 01:33:07PM +, Dimitris Chloupis wrote:
> > I am very excited for such a project.
> >
> > I hope you manage to fix the stdout problem on macos.
> >
> > One thing I would like to see is an expansion of the ability of OSProcess
> > to launch multiple instances of pharo and make them communicate with each
> > other. Thats a way to have pharo running on multiple cores. Maybe
> establish
> > a framework of communication of images that will be based on the existing
> > design of the fork mechanism in Pharo.
> >
> > Python is doing something similar with multiprocessing module  which
> > launches multiple processes of cpython executable and makes them
> > communication with each other through a similar api to its threading
> > module. This way python takes advantage of multiple cores. Python threads
> > like pharo are green threads and pharo like python has a GIL like
> > mechanism.
> >
> > Taking into account that current computers have at least 4 cores, its
> not a
> > bad idea to speed up pharo like this at least  400% ;)
> >
> > On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck <
> marianop...@gmail.com>
> > wrote:
> >
> > > Dear all,
> > >
> > > I am tremendously happy to announce you that Pharo Consortium will
> sponsor
> > > yet another development effort. In this particular case, it's my honor
> to
> > > carry on such an effort and I will be developing and improving a few
> things
> > > and ideas. All the contract and paperwork has already been done so it's
> > > time to start working.
> > >
> > > Regarding the developments itself, we discussed about these topics:
> > >
> > > 1) Experiment with a simpler yet more limited OSProcess alternative to
> > > execute OS commands.
> > > 2) Improving FileSystem in order to better deal with some POSIX stuff
> like
> > > symbolic links, unix file permissions, etc etc.
> > >
> > > I have already started with 1). OSProcess is super complete and it
> > > involves some packages (OSProcess / CommandShell) as well as some VM
> > > plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of
> features
> > > (like forking the running image) but we would like to focus only in
> > > executing OS commands. In addition, OSProcess dates from quite some
> years
> > > ago when many of the current infrastructure features did not 

Re: [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Pierce Ng]

On Sun, Dec 20, 2015 at 08:56:43AM +, Dimitris Chloupis wrote:
> Fuel I assume enters here the equation as a data exchange format , the
> problem I have with fuel is that its not backward compatible which for me

Once you have a set of OS processes running Pharo how they talk to each other
is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, JSONRPC,
msgpack, coordinate via Redis/SQLite/PostgreSQL, etc.

Pierce

Re: [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

Could you guys look at how to fix mouse clicks so the select first time 
and stay selected. I have noticed some delay and clicks don't do whjat I 
expect smoothly. I do not know if others know what I am talking about.


Snyways, it sounds like great fun, hopefully one happenss near me, 
either here or on the road.


Thanks!
robert

On 12/19/2015 08:23 AM, Mariano Martinez Peck wrote:

Dear all,

I am tremendously happy to announce you that Pharo Consortium will 
sponsor yet another development effort. In this particular case, it's 
my honor to carry on such an effort and I will be developing and 
improving a few things and ideas. All the contract and paperwork has 
already been done so it's time to start working.


Regarding the developments itself, we discussed about these topics:

1) Experiment with a simpler yet more limited OSProcess alternative to 
execute OS commands.
2) Improving FileSystem in order to better deal with some POSIX stuff 
like symbolic links, unix file permissions, etc etc.


I have already started with 1). OSProcess is super complete and it 
involves some packages (OSProcess / CommandShell) as well as some VM 
plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of 
features (like forking the running image) but we would like to focus 
only in executing OS commands. In addition, OSProcess dates from quite 
some years ago when many of the current infrastructure features did 
not exist yet.


So the idea is to think a simpler alternative to execute OS commands, 
using new features such as threaded FFI (for example, for reading 
async from pipes), pinned objects, etc. We want to use FFI as much as 
possible rather than VM plugins.


From the image side, we are thinking about an API which would look 
like a builder-like API (FileSystem, XStream, etc).


We have been discussing with David Lewis (OSProcess author) as well as 
with Eliot Miranda, Esteban Lorenzano, Damien Pollet, Stephane 
Ducasse, etc in order to agree in a project that would be worth doing 
(otherwise we would continue using existing tools). And they have all 
been very kind with me providing positive discussions.


I will soon do a survey to measure the most common use cases of 
OSProcess and related tools.


If you have any thought, question, feedback, or whatever, please let 
us know.


Finally, note that I will be doing this project together with another 
client's project (Quuve) and my life does not have much free time 
these days...so please be patient!!!


Best,

--
Mariano
http://marianopeck.wordpress.com


--
. .. .. ^,^ robert

Re: [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[David T. Lewis]

Mariano,

I am really happy to see this and I look forword to helping you in
any way I can.

Dimitris,

It is really interesting that you should suggest this. I think it may
be a bit off topic for Mariano's initial goals, but Mariano is the
driving force behind Fuel, which is the serializer that I use for
RemoteTask in OSProcess/CommandShell. This attempts to do much of
what you suggest, using Fuel to return the results of processing
executed in a forked image (in the unix sense of "forked", i.e.
extremely memory efficient).

So maybe if I can help Mariano with his project (which of course I
plan to do anyway), then he might be able to help me turn RemoteTask
into something reliable enough for general purpose use.

Another nice connection here is that Eliot's Spur image model will
make #forkSqueak (the primitive behind RemoteTask) more efficient
in cases where the cooperating images do a lot of processing, because
memory changes are more localized in the Spur memory model. In principle
it should be possible to run very large object memories in 64-bit Spur,
while forking cooperating images that consume very little additional
real memory.

Dave


On Sat, Dec 19, 2015 at 01:33:07PM +, Dimitris Chloupis wrote:
> I am very excited for such a project.
> 
> I hope you manage to fix the stdout problem on macos.
> 
> One thing I would like to see is an expansion of the ability of OSProcess
> to launch multiple instances of pharo and make them communicate with each
> other. Thats a way to have pharo running on multiple cores. Maybe establish
> a framework of communication of images that will be based on the existing
> design of the fork mechanism in Pharo.
> 
> Python is doing something similar with multiprocessing module  which
> launches multiple processes of cpython executable and makes them
> communication with each other through a similar api to its threading
> module. This way python takes advantage of multiple cores. Python threads
> like pharo are green threads and pharo like python has a GIL like
> mechanism.
> 
> Taking into account that current computers have at least 4 cores, its not a
> bad idea to speed up pharo like this at least  400% ;)
> 
> On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck 
> wrote:
> 
> > Dear all,
> >
> > I am tremendously happy to announce you that Pharo Consortium will sponsor
> > yet another development effort. In this particular case, it's my honor to
> > carry on such an effort and I will be developing and improving a few things
> > and ideas. All the contract and paperwork has already been done so it's
> > time to start working.
> >
> > Regarding the developments itself, we discussed about these topics:
> >
> > 1) Experiment with a simpler yet more limited OSProcess alternative to
> > execute OS commands.
> > 2) Improving FileSystem in order to better deal with some POSIX stuff like
> > symbolic links, unix file permissions, etc etc.
> >
> > I have already started with 1). OSProcess is super complete and it
> > involves some packages (OSProcess / CommandShell) as well as some VM
> > plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of features
> > (like forking the running image) but we would like to focus only in
> > executing OS commands. In addition, OSProcess dates from quite some years
> > ago when many of the current infrastructure features did not exist yet.
> >
> > So the idea is to think a simpler alternative to execute OS commands,
> > using new features such as threaded FFI (for example, for reading async
> > from pipes), pinned objects, etc. We want to use FFI as much as possible
> > rather than VM plugins.
> >
> > From the image side, we are thinking about an API which would look like a
> > builder-like API (FileSystem, XStream, etc).
> >
> > We have been discussing with David Lewis (OSProcess author) as well as
> > with Eliot Miranda, Esteban Lorenzano, Damien Pollet, Stephane Ducasse, etc
> > in order to agree in a project that would be worth doing (otherwise we
> > would continue using existing tools). And they have all been very kind with
> > me providing positive discussions.
> >
> > I will soon do a survey to measure the most common use cases of OSProcess
> > and related tools.
> >
> > If you have any thought, question, feedback, or whatever, please let us
> > know.
> >
> > Finally, note that I will be doing this project together with another
> > client's project (Quuve) and my life does not have much free time these
> > days...so please be patient!!!
> >
> > Best,
> >
> > --
> > Mariano
> > http://marianopeck.wordpress.com
> >

[Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Mariano Martinez Peck]

Dear all,

I am tremendously happy to announce you that Pharo Consortium will sponsor
yet another development effort. In this particular case, it's my honor to
carry on such an effort and I will be developing and improving a few things
and ideas. All the contract and paperwork has already been done so it's
time to start working.

Regarding the developments itself, we discussed about these topics:

1) Experiment with a simpler yet more limited OSProcess alternative to
execute OS commands.
2) Improving FileSystem in order to better deal with some POSIX stuff like
symbolic links, unix file permissions, etc etc.

I have already started with 1). OSProcess is super complete and it involves
some packages (OSProcess / CommandShell) as well as some VM plugins
(OSProcessPlugin, AIOPlugin). OSProcess provides lots of features (like
forking the running image) but we would like to focus only in executing OS
commands. In addition, OSProcess dates from quite some years ago when many
of the current infrastructure features did not exist yet.

So the idea is to think a simpler alternative to execute OS commands, using
new features such as threaded FFI (for example, for reading async from
pipes), pinned objects, etc. We want to use FFI as much as possible rather
than VM plugins.

>From the image side, we are thinking about an API which would look like a
builder-like API (FileSystem, XStream, etc).

We have been discussing with David Lewis (OSProcess author) as well as with
Eliot Miranda, Esteban Lorenzano, Damien Pollet, Stephane Ducasse, etc in
order to agree in a project that would be worth doing (otherwise we would
continue using existing tools). And they have all been very kind with me
providing positive discussions.

I will soon do a survey to measure the most common use cases of OSProcess
and related tools.

If you have any thought, question, feedback, or whatever, please let us
know.

Finally, note that I will be doing this project together with another
client's project (Quuve) and my life does not have much free time these
days...so please be patient!!!

Best,

-- 
Mariano
http://marianopeck.wordpress.com

Re: [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Dimitris Chloupis]

I am very excited for such a project.

I hope you manage to fix the stdout problem on macos.

One thing I would like to see is an expansion of the ability of OSProcess
to launch multiple instances of pharo and make them communicate with each
other. Thats a way to have pharo running on multiple cores. Maybe establish
a framework of communication of images that will be based on the existing
design of the fork mechanism in Pharo.

Python is doing something similar with multiprocessing module  which
launches multiple processes of cpython executable and makes them
communication with each other through a similar api to its threading
module. This way python takes advantage of multiple cores. Python threads
like pharo are green threads and pharo like python has a GIL like
mechanism.

Taking into account that current computers have at least 4 cores, its not a
bad idea to speed up pharo like this at least  400% ;)

On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck 
wrote:

> Dear all,
>
> I am tremendously happy to announce you that Pharo Consortium will sponsor
> yet another development effort. In this particular case, it's my honor to
> carry on such an effort and I will be developing and improving a few things
> and ideas. All the contract and paperwork has already been done so it's
> time to start working.
>
> Regarding the developments itself, we discussed about these topics:
>
> 1) Experiment with a simpler yet more limited OSProcess alternative to
> execute OS commands.
> 2) Improving FileSystem in order to better deal with some POSIX stuff like
> symbolic links, unix file permissions, etc etc.
>
> I have already started with 1). OSProcess is super complete and it
> involves some packages (OSProcess / CommandShell) as well as some VM
> plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of features
> (like forking the running image) but we would like to focus only in
> executing OS commands. In addition, OSProcess dates from quite some years
> ago when many of the current infrastructure features did not exist yet.
>
> So the idea is to think a simpler alternative to execute OS commands,
> using new features such as threaded FFI (for example, for reading async
> from pipes), pinned objects, etc. We want to use FFI as much as possible
> rather than VM plugins.
>
> From the image side, we are thinking about an API which would look like a
> builder-like API (FileSystem, XStream, etc).
>
> We have been discussing with David Lewis (OSProcess author) as well as
> with Eliot Miranda, Esteban Lorenzano, Damien Pollet, Stephane Ducasse, etc
> in order to agree in a project that would be worth doing (otherwise we
> would continue using existing tools). And they have all been very kind with
> me providing positive discussions.
>
> I will soon do a survey to measure the most common use cases of OSProcess
> and related tools.
>
> If you have any thought, question, feedback, or whatever, please let us
> know.
>
> Finally, note that I will be doing this project together with another
> client's project (Quuve) and my life does not have much free time these
> days...so please be patient!!!
>
> Best,
>
> --
> Mariano
> http://marianopeck.wordpress.com
>