Re: [Pharo-users] using UFFI in non-crashing way

[Ben Coman]

On Sun, May 7, 2017 at 7:45 PM,   wrote:
> On 07/04/17 18:28, Dimitris Chloupis wrote:
>> Do you guys received my CPU exceptions solution message ? Because the
>> issue you described is already resolved by hardware. Simple and elegant
>> solution too
>> On Fri, 7 Apr 2017 at 11:03, Denis Kudriashov > > wrote:
>>
>>
>> 2017-04-07 8:22 GMT+02:00 Esteban Lorenzano > >:
>>
>>> But then the problem is on "how to not lose your
>>> code", not in FFI...
>>>
>>>
>>> Restarting image is a problem. For any other "smalltalk"
>>> problems we got suitable debugger but not in that case.
>>>
>>>
>>> Also when image crashed we need to recover all objects state
>>> for the script which produce crash
>>
>> FFI is a border.
>> Once you start working with it, you are in C world.
>> And C world is nasty, hard and cumbersome.
>>
>>
>> Yes. And my remark was about possibility to catch C error to prevent
>> image crash, why it will make FFI less hard.
>>
>
>
> My experience with the UFFI so far is quite positive. Of course, a lot
> of care needs to go to the C part, but that's a fact of life with C's
> unsafe computational model.
>
> So, if the C code is well written and thoroughly tested, possibly
> outside Pharo, then integrating it by means of the UFFI is rather
> straightforward. I agree that small errors using the UFFI can lead to
> spurious or catastrophic outcomes, so unit (or even smaller scoped)
> testing really becomes vital.
>
> From what I can tell so far, the UFFI is fast, well performing and, with
> some care, not even so hard as it might seem at first sight.
>
>
> Greetings
> Raffaello
>

btw, I started a UFFI / Clang tutorial here (that I need to get back to)
http://blog.openinworld.com/2016/09/pharo-libclang-ffi-part-1-preamble/

cheers -ben

Re: [Pharo-users] using UFFI in non-crashing way

[raffaello . giulietti]

On 07/04/17 18:28, Dimitris Chloupis wrote:
> Do you guys received my CPU exceptions solution message ? Because the
> issue you described is already resolved by hardware. Simple and elegant
> solution too
> On Fri, 7 Apr 2017 at 11:03, Denis Kudriashov  > wrote:
> 
> 
> 2017-04-07 8:22 GMT+02:00 Esteban Lorenzano  >:
> 
>> But then the problem is on "how to not lose your
>> code", not in FFI... 
>>
>>
>> Restarting image is a problem. For any other "smalltalk"
>> problems we got suitable debugger but not in that case.
>>
>>
>> Also when image crashed we need to recover all objects state
>> for the script which produce crash
> 
> FFI is a border.
> Once you start working with it, you are in C world. 
> And C world is nasty, hard and cumbersome.
> 
> 
> Yes. And my remark was about possibility to catch C error to prevent
> image crash, why it will make FFI less hard.
> 


My experience with the UFFI so far is quite positive. Of course, a lot
of care needs to go to the C part, but that's a fact of life with C's
unsafe computational model.

So, if the C code is well written and thoroughly tested, possibly
outside Pharo, then integrating it by means of the UFFI is rather
straightforward. I agree that small errors using the UFFI can lead to
spurious or catastrophic outcomes, so unit (or even smaller scoped)
testing really becomes vital.

>From what I can tell so far, the UFFI is fast, well performing and, with
some care, not even so hard as it might seem at first sight.


Greetings
Raffaello

Re: [Pharo-users] using UFFI in non-crashing way

[Dimitris Chloupis]

Do you guys received my CPU exceptions solution message ? Because the issue
you described is already resolved by hardware. Simple and elegant solution
too
On Fri, 7 Apr 2017 at 11:03, Denis Kudriashov  wrote:

>
> 2017-04-07 8:22 GMT+02:00 Esteban Lorenzano :
>
> But then the problem is on "how to not lose your code", not in FFI...
>
>
> Restarting image is a problem. For any other "smalltalk" problems we got
> suitable debugger but not in that case.
>
>
> Also when image crashed we need to recover all objects state for the
> script which produce crash
>
>
> FFI is a border.
> Once you start working with it, you are in C world.
> And C world is nasty, hard and cumbersome.
>
>
> Yes. And my remark was about possibility to catch C error to prevent image
> crash, why it will make FFI less hard.
>

Re: [Pharo-users] using UFFI in non-crashing way

[Denis Kudriashov]

2017-04-07 8:22 GMT+02:00 Esteban Lorenzano :

> But then the problem is on "how to not lose your code", not in FFI...
>>
>>
>> Restarting image is a problem. For any other "smalltalk" problems we got
>> suitable debugger but not in that case.
>
>
> Also when image crashed we need to recover all objects state for the
> script which produce crash
>
>
> FFI is a border.
> Once you start working with it, you are in C world.
> And C world is nasty, hard and cumbersome.
>

Yes. And my remark was about possibility to catch C error to prevent image
crash, why it will make FFI less hard.

Re: [Pharo-users] using UFFI in non-crashing way

[Esteban Lorenzano]

> On 6 Apr 2017, at 10:39, Denis Kudriashov  wrote:
> 
> 
> 2017-04-06 10:37 GMT+02:00 Denis Kudriashov  >:
> But then the problem is on "how to not lose your code", not in FFI... 
> 
> Restarting image is a problem. For any other "smalltalk" problems we got 
> suitable debugger but not in that case.
> 
> Also when image crashed we need to recover all objects state for the script 
> which produce crash

FFI is a border.
Once you start working with it, you are in C world. 
And C world is nasty, hard and cumbersome.

In C world, pointers not correctly alloc’d produce a crash. 
In C world, pointers not correctly freed also produce a crash, eventually (and 
those are harder to fix). 
In C world everything pointing to wrong location will produce a crash (and you 
can point anything to anywhere). 

That’s why garbage collectors were invented, after all. 

So my only advice here is: 

- recheck all your allocs/free, rigorously
- save your image before test
- be sure epicea is working 

Finally, I would say the only way to prevent your image to crash is to not 
execute FFI calls inside it. To do this, you can always use ImageWorker 
(http://smalltalkhub.com/#!/~PharoExtras/ImageWorker) to actually execute your 
tests in a different execution. Yes is slow, but it will be safer. 

Esteban

Re: [Pharo-users] using UFFI in non-crashing way

[Denis Kudriashov]

2017-04-06 10:37 GMT+02:00 Denis Kudriashov :

> But then the problem is on "how to not lose your code", not in FFI...
>
>
> Restarting image is a problem. For any other "smalltalk" problems we got
> suitable debugger but not in that case.


Also when image crashed we need to recover all objects state for the script
which produce crash

Re: [Pharo-users] using UFFI in non-crashing way

[Denis Kudriashov]

2017-03-27 13:01 GMT+02:00 Guillermo Polito :

> But then the problem is on "how to not lose your code", not in FFI...


Restarting image is a problem. For any other "smalltalk" problems we got
suitable debugger but not in that case.

Re: [Pharo-users] using UFFI in non-crashing way

[p...@highoctane.be]

On Mon, Mar 27, 2017 at 1:01 PM, Guillermo Polito  wrote:

>
>
> On Mon, Mar 27, 2017 at 12:04 PM, Denis Kudriashov 
> wrote:
>
>>
>> 2017-03-27 11:34 GMT+02:00 Guillermo Polito :
>>
>>> Well, you can capture the signal, but the problem is not the signal
>>> itself but what caused the problem. The most common case of segfaut is
>>> memory corruption (and thus a program that tries to wrongly interpret wrong
>>> addresses). If you already corrupted the memory doing something wrong, how
>>> can you "fix it"?
>>
>>
>> I think important case here is creating wrappers for new C libraries.
>> Common mistake is passing wrong structures to foreign function. Then image
>> crashes and you need restart it and recover code - big waste of time and
>> losing smalltalk liveness.
>>
>
> But then the problem is on "how to not lose your code", not in FFI...
>

Am spending my time restarting my image due to this. Very frustrating to
recreate the setup etc.

Havign a way to detect go back one frame would be very handy.

Phil

Re: [Pharo-users] using UFFI in non-crashing way

[Guillermo Polito]

On Mon, Mar 27, 2017 at 12:04 PM, Denis Kudriashov 
wrote:

>
> 2017-03-27 11:34 GMT+02:00 Guillermo Polito :
>
>> Well, you can capture the signal, but the problem is not the signal
>> itself but what caused the problem. The most common case of segfaut is
>> memory corruption (and thus a program that tries to wrongly interpret wrong
>> addresses). If you already corrupted the memory doing something wrong, how
>> can you "fix it"?
>
>
> I think important case here is creating wrappers for new C libraries.
> Common mistake is passing wrong structures to foreign function. Then image
> crashes and you need restart it and recover code - big waste of time and
> losing smalltalk liveness.
>

But then the problem is on "how to not lose your code", not in FFI...

Re: [Pharo-users] using UFFI in non-crashing way

[Denis Kudriashov]

2017-03-27 11:34 GMT+02:00 Guillermo Polito :

> Well, you can capture the signal, but the problem is not the signal itself
> but what caused the problem. The most common case of segfaut is memory
> corruption (and thus a program that tries to wrongly interpret wrong
> addresses). If you already corrupted the memory doing something wrong, how
> can you "fix it"?


I think important case here is creating wrappers for new C libraries.
Common mistake is passing wrong structures to foreign function. Then image
crashes and you need restart it and recover code - big waste of time and
losing smalltalk liveness.

Re: [Pharo-users] using UFFI in non-crashing way

[Guillermo Polito]

On Sun, Mar 26, 2017 at 7:21 PM, Peter Uhnak  wrote:

> On Sun, Mar 26, 2017 at 05:45:04PM +0200, Esteban Lorenzano wrote:
> >
> > > On 26 Mar 2017, at 15:40, Peter Uhnak  wrote:
> > >
> > > Hi,
> > >
> > > is it possible to use UFFI and avoid crashing the image when the
> called code segfaults?
> > >
> > > In other words, can I somehow wrap the call and throw in-image
> Exception/Error on failure instead of the whole thing crashing?
> >
> > that’s not possible, once you segfault, you segfault.
>
> Ah, ok.
>

Well, you can capture the signal, but the problem is not the signal itself
but what caused the problem. The most common case of segfaut is memory
corruption (and thus a program that tries to wrongly interpret wrong
addresses). If you already corrupted the memory doing something wrong, how
can you "fix it"?


>
> >
> > > Or is the only way to do that have the called thing running as a
> separate app (e.g. via OSSubProcess) and communicate via pipes or sockets?
> >
> > that’s not done.
> > no idea how much effort is required to implement, but I imagine is not
> trivial.
>
> Nono, I didn't mean making UFFI calls via socket (although that would be
> interesting), but rather making a C app that would make the calls, and that
> would provide a socket interface. (Which is what I typically do with other
> langs).
>
> Thanks,
> Peter
>
>

Re: [Pharo-users] using UFFI in non-crashing way

[Thierry Goubier]

Le 26/03/2017 à 19:21, Peter Uhnak a écrit :

On Sun, Mar 26, 2017 at 05:45:04PM +0200, Esteban Lorenzano wrote:



On 26 Mar 2017, at 15:40, Peter Uhnak  wrote:

Hi,

is it possible to use UFFI and avoid crashing the image when the
called code segfaults?

In other words, can I somehow wrap the call and throw in-image
Exception/Error on failure instead of the whole thing crashing?


that’s not possible, once you segfault, you segfault.


Ah, ok.




Or is the only way to do that have the called thing running as a
separate app (e.g. via OSSubProcess) and communicate via pipes or
sockets?


that’s not done. no idea how much effort is required to implement,
but I imagine is not trivial.


Nono, I didn't mean making UFFI calls via socket (although that would
be interesting), but rather making a C app that would make the calls,
and that would provide a socket interface. (Which is what I typically
do with other langs).


With UFFI and a standard way to marshall types over a socket (ASN.1 or 
even xml-rpc), this could be generated automatically.


Of course, you may have a few performance issues, but you could have a 
long-lived child to avoid paying the startup time, and mmap / shared 
memory to lower the cost of moving memory back and forth.


Thierry


Thanks, Peter

Re: [Pharo-users] using UFFI in non-crashing way

[Peter Uhnak]

On Sun, Mar 26, 2017 at 05:45:04PM +0200, Esteban Lorenzano wrote:
> 
> > On 26 Mar 2017, at 15:40, Peter Uhnak  wrote:
> > 
> > Hi,
> > 
> > is it possible to use UFFI and avoid crashing the image when the called 
> > code segfaults?
> > 
> > In other words, can I somehow wrap the call and throw in-image 
> > Exception/Error on failure instead of the whole thing crashing?
> 
> that’s not possible, once you segfault, you segfault.

Ah, ok.

> 
> > Or is the only way to do that have the called thing running as a separate 
> > app (e.g. via OSSubProcess) and communicate via pipes or sockets?
> 
> that’s not done. 
> no idea how much effort is required to implement, but I imagine is not 
> trivial. 

Nono, I didn't mean making UFFI calls via socket (although that would be 
interesting), but rather making a C app that would make the calls, and that 
would provide a socket interface. (Which is what I typically do with other 
langs).

Thanks,
Peter

Re: [Pharo-users] using UFFI in non-crashing way

[David T. Lewis]

On Sun, Mar 26, 2017 at 05:45:04PM +0200, Esteban Lorenzano wrote:
> 
> > On 26 Mar 2017, at 15:40, Peter Uhnak  wrote:
> > 
> > Hi,
> > 
> > is it possible to use UFFI and avoid crashing the image when the called 
> > code segfaults?
> > 
> > In other words, can I somehow wrap the call and throw in-image 
> > Exception/Error on failure instead of the whole thing crashing?
> 
> that???s not possible, once you segfault, you segfault.
> 
> > Or is the only way to do that have the called thing running as a separate 
> > app (e.g. via OSSubProcess) and communicate via pipes or sockets?
> 
> that???s not done. 
> no idea how much effort is required to implement, but I imagine is not 
> trivial. 
>

I have not tried it with FFI calls, but maybe this will do what you want:

  RemoteTask do: [ 2 + 2 ] ==> 4

Instead of [ 2 + 2 ] use some block containing your FFI calls.

It works in Pharo. To get RemoteTask, load OSProcess and CommandShell.
You also need Fuel, but I think that is in the image by default.

An explanation of RemoteTask is at http://wiki.squeak.org/squeak/6176.
You are right, it was not trivial to implement ;-)

HTH,
Dave
 

Re: [Pharo-users] using UFFI in non-crashing way

[Dimitris Chloupis]

> Hi,
>
> is it possible to use UFFI and avoid crashing the image when the called
> code segfaults?
>
>
Yes its possible. When I was exploring the means to live code with C++ I
came to specific implementations of live coding for C++. One of them
explored a feature called "Hardware exceptions" .Hardware exceptions are
very similar to regular exception with the big difference that are handled
by the CPU and not the language or the OS.

Crashes is a standard behavior that is imposed by the OS kernel in the case
that an app tries to access memory is not allowed to access or access it in
the way it is not allowed to access it. An exception will capture this
behavior it wont allow the application to crash.

Hardware exceptions are a standard feature for most CPUs and obviously they
are not implemented at language level.

For example you can find documentation about Hardware exceptions on Windows
at

https://msdn.microsoft.com/en-us/library/windows/desktop/ms680657(v=vs.85).aspx

This means the C code you will be calling with UFFI will have to be wrapped
inside a Hardware exception.

When a crash is suppose to happen inside code that is wrapped with a
hardware exception, the error (seffault in your example) will be displayed
in the console but the application wont crash it will continue to execute
depending on how you handle the exception.

Generally hardware exception are to be avoided because they can hide bugs
because the OS not always reports and can crash an app without an error in
that you will be oblivious to what went wrong, when and how as the crash
will not happen.



> In other words, can I somehow wrap the call and throw in-image
> Exception/Error on failure instead of the whole thing crashing?
>
>
Its possible if you make a C function that contains a hardware exception
take as argument a pointer to a specific function and then calls it inside
the exception.


> Or is the only way to do that have the called thing running as a separate
> app (e.g. via OSSubProcess) and communicate via pipes or sockets?
>

It wont make a diffirence if the crash is caused by UFFI it will still
crash Pharo that applies even if you use hardware exceptions , so you need
always to be careful.

Plus Pharo VM at any point can crash for its own reason, that is completely
unrelated to what you are doing

Thanks,
> Peter
>

So yes its possible, but you still need to be careful.

Re: [Pharo-users] using UFFI in non-crashing way

[Esteban Lorenzano]

> On 26 Mar 2017, at 15:40, Peter Uhnak  wrote:
> 
> Hi,
> 
> is it possible to use UFFI and avoid crashing the image when the called code 
> segfaults?
> 
> In other words, can I somehow wrap the call and throw in-image 
> Exception/Error on failure instead of the whole thing crashing?

that’s not possible, once you segfault, you segfault.

> Or is the only way to do that have the called thing running as a separate app 
> (e.g. via OSSubProcess) and communicate via pipes or sockets?

that’s not done. 
no idea how much effort is required to implement, but I imagine is not trivial. 

Esteban

> 
> Thanks,
> Peter
> 

[Pharo-users] using UFFI in non-crashing way

[Peter Uhnak]

Hi,

is it possible to use UFFI and avoid crashing the image when the called code 
segfaults?

In other words, can I somehow wrap the call and throw in-image Exception/Error 
on failure instead of the whole thing crashing?

Or is the only way to do that have the called thing running as a separate app 
(e.g. via OSSubProcess) and communicate via pipes or sockets?

Thanks,
Peter