Re: Proposed change to ForeignPtr

[Alastair Reid]

> BTW, having two languages with separated heaps interact is a big
> mess as soon as you can have cycles, which you usually cannot
> exclude.  Alastair already pointed that out and Martin Odersky also
> has nice stories to tell about this.

Hmmm, way back in '94, my thought was that the only thing to do in the
presence of these cycles was to run the two GCs at once with a rather
intimate communication between them where one says 'I can reach X' the
other says 'ok, now I can reach Y', and eventually they both run out
of objects to trace and they can discard unreached objects.

The problem with this is that it runs into the same problems Malcolm
and I are so keen to avoid: the other language has to be able to
trigger GCs at more or less arbitrary times.

Since then, I've read a lot about non-stop concurrent GC which doesn't
need the two GCs to run simultaneously.  Just thinking aloud, I wonder
if it would be any easier to implement?  The communication would be
much the same ('I'm starting', 'I can reach X' and 'I'm done') but 
there'd be no need to synchronize the GCs.


-- 
Alastair Reid [EMAIL PROTECTED]  
Reid Consulting (UK) Limited  http://www.reid-consulting-uk.ltd.uk/alastair/

___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Manuel M T Chakravarty wrote:
[snip]
> BTW, having two languages with separated heaps interact is a
> big mess as soon as you can have cycles, which you usually
> cannot exclude.  Alastair already pointed that out and
> Martin Odersky also has nice stories to tell about this.
[snip]
Yeah yeah I know, indeed I think I pointed it out in the message which
started this whole discussion.  In the example application I am thinking of,
I think cycles can be excluded in the first instance.  In general I think
there are various rather complicated things one might do in the finalizers 
to deal with cycles, and frankly I'd rather be able to write the code for
this in Haskell rather than in C.
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Manuel M T Chakravarty]

George Russell <[EMAIL PROTECTED]> wrote,

> Manuel wrote (snipped)
> 
> > I have changed this in the spec now.  I attach the wording
> > used in the spec.
> 
> > \item[newForeignPtr ::\ Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a)]
> >   Turn a plain memory reference into a foreign object by associating a
> >   finalizer with the reference.  The finalizer is represented by a pointer to
> >   an external function, which will be executed after the last reference to the
> >   foreign object is dropped.  On invocation, the finalizer receives a pointer
> >   to the associated foreign object as an argument.  Note that there is no
> >   guarantee on how soon the finalizer is executed after the last reference was
> >   dropped; this depends on the details of the Haskell storage manager. The
> >   only guarantee is that the finalizer runs before the program terminates.
> > 
> >   Whether a finaliser may call back into the Haskell system is system
> >   dependent.  Portable code may not rely on such call backs.
> >   
> > \item[addForeignPtrFinalizer ::\ ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO
> >   ()] Add another finalizer to the given foreign object. No guarantees are
> >   made on the order in which multiple finalizers for a single object are run.
> 
> I think this is all a rather murky area.  Consider two systems, let's call them
> Haskell and Foogle, which both operate heaps and do their own storage allocation,
> but also communicate over similar FFIs.  We might very reasonably
> have situations where fairly complex  inter-language pointers exist, so for example 
>Haskell holds a 
> ForeignPtr to something in the Foogle heap; the pointed-to Foogle object in turn 
>references
> a Haskell object (presumably provided via StablePtr).  Now suppose Haskell wants to
> drop the ForeignPtr.  Then the logical thing for the finalizer to do is to tell 
>Foogle 
> that Haskell is no longer interested in the Foogle object.  This then gives Foogle
> the chance on its own garbage collection to in turn drop the Haskell StablePtr.  In 
>turn
> this means somehow running StablePtr.freeStablePtr.  However this scheme I don't 
>know if
> that's legal, because the Haskell finalizer you need to run "freeStablePtr" is 
>indirectly
> provoked by the initial Haskell finalizer.  
> 
> This is a pity, because this might actually be a fairly good way of managing garbage 
>collection 
> between Foogle and Haskell.  Of course you would need at least reference counters 
>(if you can
> guarantee there are no cycles containing both languages) or something more powerful 
>otherwise,
> but reference counters at least can be provided.  Furthermore I do actually have a 
>real case
> in mind where I might use something like this, though I'd rather not go into details 
>at this
> time.
> 
> I'm afraid I haven't been following this thread lately, so I don't know what the big 
>problem
> is about calling Haskell from a finalizer; I suppose it's something to do with us 
>being in the
> middle of garbage collection.  However wouldn't it be better to allow finalizers to 
>somehow provide
> an action which may call Haskell, but (a) may be performed at some later date (such 
>as when the GC is
> over); (b) consequently, may not assume that the pointer finalized still points to 
>anything?

There are good reasons why the 1.0 version of the FFI
Addendum restricts itself to Haskell <-> C interaction and
not more.  The scenario that you sketch is one such reason.
In other words, the kind of functionality that you want is
currently not being supported by the standard FFI.
(Nevertheless, system-specific extensions to the standard in
GHC allow you to implement what you want).

BTW, having two languages with separated heaps interact is a
big mess as soon as you can have cycles, which you usually
cannot exclude.  Alastair already pointed that out and
Martin Odersky also has nice stories to tell about this.

Cheers,
Manuel
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

RE: Proposed change to ForeignPtr

[Simon Marlow]

I'm afraid George's questions have also rekindled my curiosity about
whether implementing Haskell finalizers is really as hard as it sounds.
Much has been written, but I still don't think we've got to the nub of
the issue.

On the face of it, if you can implement 'foreign import ccall safe',
then you have a re-entrant runtime system.  The times at which the
program can make one of these foreign calls are limited, i.e. in the IO
monad only - but I believe there's nothing particularly special about IO
computations in the evaluation models used by nhc98 and Hugs (correct me
if I'm wrong).

Alastair writes:

> The way GHC implements preemption is an optimized form of: set a bit
> when preemption is needed; and make sure that generated code will test
> that bit whenever it is in a position to perform a context switch.
> 
> What you're asking Hugs and NHC to do is: add a function to a list
> whenever you have a finalizer to run; make sure the interpreter will
> test that bit whenever it is in a position to perform a context
> switch.  
> 
> It's basically the same.  We don't have to mess around with signals to
> provide a regular timer interrupt but that's the easy bit of the code.

Ok so far.

> We can probably avoid messing around with multiple C stacks.  That's a
> significant saving but, it's the complexity of that is fairly
> self-contained - we could probably steal some code from some other
> language implementation.
> 
> The cost is going over all data structures in the system making sure
> that operations on them are suitably atomic.  One of the issues I
> remember from old versions of GHC was that some of the primops would
> do some work, allocate some memory, then finish the job.  The classic
> error to make in that code was for the second half of the code to
> assume almost anything about what happened in the first half of the
> code: how much space is on the stack, does a table have free space in
> it, is this pointer into the middle of an object ok?

You certainly can't keep local variables live across a heap check,
everything has to be saved on the stack.  Hugs is different because it
has a conservative GC, so doesn't need to save everything on the stack
for a GC.  But how does it implement a safe foreign call?  Presumably it
must save away state on the stack in a way that the computation can be
resumed safely, and that's all you need in order to be able to run a
Haskell finalizer.

> The problem is the scope: every single data structure and every bit of
> code that accesses it has to be vetted and we have to keep it in mind
> as we maintain the code.  It's a high price to pay and I don't think
> it's necessary (because all you really need is for the runtime systems
> to talk to each other in very limited ways).

I don't quite understand this: could you give a concrete example of some
extra invariant that has to be maintained?  In GHC, context switches
happen at very precise points (i.e. heap checks) so I don't think we
have these kind of problems; certainly I don't remember vetting every
single data structure.  Can't Hugs use a similar approach?

Cheers,
Simon
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Simon Peyton-Jones wrote:
[snip]
> 3.  If the finalizer, in turn, needs to call some kind of foreign 'free'
> routine (e.g. the finaliser for a Haskell object encapsulating a Foogle
> object calls the Foogle free code; or the finalizer for a Foogle object
> encapsulating a Haskell value calls the Haskell free routine) it should
> be allowed to do so at any time.   BUT DOING SO DOES NOT trigger an
> immediate GC in the other system.
[snip]
I think the problem is that you may want it to trigger arbitrary computation
at some time.  For example if Foogle wants to let go of a Haskell StablePtr,
you may want to decrement (in Haskell) some reference counter associated with
the StablePtr.  Or indeed do some more complicated operation if you are worried
about circular datastructures which cross languages.  This is why I don't think
Alastair's suggestion of providing C with access to StablePtr.freeStablePtr
is enough.  However this computation doesn't have to be "immediate"; the Haskell 
RTS could reasonably delay it until the GC is over.


> Several people have explained why it would be bad to trigger immediate
> GC. All the 'free' routine does is record that Foogle no longer holds
> this pointer (or vice versa).  That information (as Alastair puts it)
> tweaks local data structures (yes, there may need to be a lock if there
> are separate OS threads involved), and that info is exploited at the
> next GC.
> 
> 4.  None of this says anything at all about asynchronous calls into
> (say) Haskell from Foogle.  I don't understand what the issues are
> exactly.
Suppose Hugs and GHC are running in the same OS thread, calling each other.
GHC is running various threads which call Hugs functions.  At the same time
Hugs has finalizers which somehow call GHC.  If I understand the situation
correctly, during the called GHC functions the GHC RTS functions normally,
meaning the other GHC threads get run, meaning the Hugs functions get called,
meaning (presumably) horrible chaos.  If it seems pointless linking Haskell with
Haskell, substitute Java for GHC.
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

RE: Proposed change to ForeignPtr

[Simon Peyton-Jones]

I confess that I have not followed the twists and turns of this
discussion, but it seems to have gotten more complicated than necessary.
There are several separate issues.

1.  Can a finaliser for a Haskell value be an arbitrary Haskell
computation?  For GHC, yes.  For Hugs, no (and for good reasons).So
Hugs can only allow a C procedure as a finaliser. 

2.  If a garbage collector (whether the Haskell gc or the Foogle gc)
finds that it is dropping a pointer to an object, when may it run the
finalizer?  Answer: it should be allowed to do so at any time: when the
pointer is dropped (for ref-count gc) or during gc (as Hugs does) or
after gc (as GHC does).

3.  If the finalizer, in turn, needs to call some kind of foreign 'free'
routine (e.g. the finaliser for a Haskell object encapsulating a Foogle
object calls the Foogle free code; or the finalizer for a Foogle object
encapsulating a Haskell value calls the Haskell free routine) it should
be allowed to do so at any time.   BUT DOING SO DOES NOT trigger an
immediate GC in the other system.  

Several people have explained why it would be bad to trigger immediate
GC. All the 'free' routine does is record that Foogle no longer holds
this pointer (or vice versa).  That information (as Alastair puts it)
tweaks local data structures (yes, there may need to be a lock if there
are separate OS threads involved), and that info is exploited at the
next GC.

4.  None of this says anything at all about asynchronous calls into
(say) Haskell from Foogle.  I don't understand what the issues are
exactly.

Simon

___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Alastair Reid wrote:
[snip]
> What you're asking Hugs and NHC to do is: add a function to a list
> whenever you have a finalizer to run; make sure the interpreter will
> test that bit whenever it is in a position to perform a context
> switch.
Am I really asking that much?  In that paper you wrote you already propose
that the Haskell system implement functions which tell C when it enters GC and
when it leaves GC.  So what it seems you need is for this to set a flag so that
(1) When the (Foogle) finalizer runs, it runs normally if this flag is unset (no GC is
running, so presumably safe).  Otherwise add the action to a queue.
(2) When Haskell leaves GC it checks the queue and runs any pending actions,
backing up if it's necessary to do so.

The flag and queue need to be atomically accessed, but since (for Hugs and NHC)
you are assuming only one OS thread, that should be enough.  You don't for example 
have to
worry about exported functions being called absolutely everywhere.
[snip]
> The cost is going over all data structures in the system making sure
> that operations on them are suitably atomic.
[snip]
If something like that would work, the cost during normal operation would only be
the cost of setting and clearing the flag at the start and end of GC, and checking the
queue at the end of GHC. 

I'm sorry, it's frightfully arrogant of me to argue over details of Haskell compilers 
with
their implementors, but what else can I do here?  Anyway the point is a general one; 
can we implement
FFI without needing the whole machinery of concurrency?  
> > If it's really impossible for NHC or Hugs to implement this, I think
> > I would still rather it was left to the NHC and Hugs documentation
> > to admit that exported Haskell functions basically don't work in
> > some circumstances, rather than to the GHC documentation to say that
> > actually they do.
> 
> It's a matter of taste how you do these things.

If we take it that there is no way for a finalizer to call functions exported from NHC 
or Hugs,
my personal preference would be to regard this as an undesirable implementation glitch 
which
should be documented in the NHC and Hugs documentation.  This would then be something 
users of
the FFI with NHC and Hugs have to worry about, rather than users of the FFI in 
general.  I don't
think it makes the FFI worthless for NHC and Hugs, any more than the recent discovery 
that IO is
not really a monad on most implementations makes the Haskell standard worthless.
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Alastair Reid]

> [snip] No, you do not really need separate threads for this problem
> to occur.  All you need is, say, Hugs to call a GHC-exported
> function as a finalizer, in the same OS thread, GHC to run a garbage
> collection during this function, and the garbage collection in turn
> to want to run a Hugs finalizer, before the finalizer called from
> Hugs has finished. 

The Hugs and GHC runtimes can talk to each other just fine (or, if
they can't it's a simple oversight and well fix it).

There's no problem with GHC and Hugs each telling each other that some
object they own has one less pointer to it.  Next time it is
convenient for the runtime, it can run a GC, perhaps recognize that
there's a few GHC objects it can release and it tells the GHC runtime
that it can release them.

The reason we can do this is because it has limited scope: just a few
data structures have to be tweaked to avoid GHC coming in when Hugs'
data structures are in an inconsistent state.

It's quite a different matter to allow arbitrary Haskell code to be
run - that means the entire runtime system and libraries have to be
made reentrant.

> Of course one wouldn't normally want to link GHC
> from Hugs, but if even these two cannot be made to meet, I don't
> know how you expect Haskell to call anything else with a reasonably
> flexible GC system; it puts the kybosh on Java for example, which I
> am fairly sure makes plenty of use of both callbacks and finalizers.

That's fine, they can have all the finalizers they want.
The finalizers can fiddle with things in the runtime system
to tell the GC whatever they want.

> In any case it seems to me just as dangerous to assume that the
> implementation does not use OS threads, as to assume it does.

The internal structure of your apps is up to you - use locks to avoid
using single-threaded code in multithreaded manner.

> You are effectively writing on top of the FFI document "If your
> program does this perfectly reasonable combination of finalizers, it
> will fall over in an undefined way should the implementation use OS
> threads; furthermore there is no way around this".  Basically the
> fact that there is only one OS thread is an implementation detail,
> not something that the user should have to think about.

Programmers are used to dealing with code which is single threaded or
not reentrant.  It's quite common.

> Is it really the case that neither NHC nor Hugs can implement a list
> of actions to be taken at the first convenient point after GC has
> finished without implementing the whole machinery of preemptive
> concurrency?  I take Malcolm Wallace's word for it that it isn't
> trivial, but why do you need for example asynchronous interruption
> of Haskell threads, wait queues, or time slices?  Surely what you
> need is some way of backing up the state upon return from GC in such
> a way that you can run the queued IO actions, which may be hard but
> is a long way off preemptive concurrency.

The way GHC implements preemption is an optimized form of: set a bit
when preemption is needed; and make sure that generated code will test
that bit whenever it is in a position to perform a context switch.

What you're asking Hugs and NHC to do is: add a function to a list
whenever you have a finalizer to run; make sure the interpreter will
test that bit whenever it is in a position to perform a context
switch.  

It's basically the same.  We don't have to mess around with signals to
provide a regular timer interrupt but that's the easy bit of the code.

We can probably avoid messing around with multiple C stacks.  That's a
significant saving but, it's the complexity of that is fairly
self-contained - we could probably steal some code from some other
language implementation.

The cost is going over all data structures in the system making sure
that operations on them are suitably atomic.  One of the issues I
remember from old versions of GHC was that some of the primops would
do some work, allocate some memory, then finish the job.  The classic
error to make in that code was for the second half of the code to
assume almost anything about what happened in the first half of the
code: how much space is on the stack, does a table have free space in
it, is this pointer into the middle of an object ok?

The problem is the scope: every single data structure and every bit of
code that accesses it has to be vetted and we have to keep it in mind
as we maintain the code.  It's a high price to pay and I don't think
it's necessary (because all you really need is for the runtime systems
to talk to each other in very limited ways).

> If it's really impossible for NHC or Hugs to implement this, I think
> I would still rather it was left to the NHC and Hugs documentation
> to admit that exported Haskell functions basically don't work in
> some circumstances, rather than to the GHC documentation to say that
> actually they do.

It's a matter of taste how you do these things.

--
Alastair Reid [EMAIL

Re: Proposed change to ForeignPtr

[George Russell]

Alastair Reid wrote:
[snip]
> You ask how garbage collection can work if you can't invoke it at
> arbitrary times?  You ask what use foreign export is if you can't call
> the exported function at arbitrary times?
> [I hope I'm paraphrasing your questions appropriately]
> 
> Both questions start by assuming that you have Haskell and Foosh [or
> whatever your other language is called] running in separate threads
> (otherwise things wouldn't happen at arbitrary times).  Put them in a
> single thread and there's no problem: you're not doing things at
> arbitrary times, you're doing them at quite specific times: Foosh had
> control and it called into Haskell or Haskell had control and it
> called into Foosh.
[snip]
No, you do not really need separate threads for this problem to occur.
All you need is, say, Hugs to call a GHC-exported function as a finalizer,
in the same OS thread, GHC to run a garbage collection during this function,
and the garbage collection in turn to want to run a Hugs finalizer, before
the finalizer called from Hugs has finished.  Of course one wouldn't normally
want to link GHC from Hugs, but if even these two cannot be made to meet, I
don't know how you expect Haskell to call anything else with a reasonably
flexible GC system; it puts the kybosh on Java for example, which I am fairly
sure makes plenty of use of both callbacks and finalizers.

In any case it seems to me just as dangerous to assume that the implementation
does not use OS threads, as to assume it does.  You are effectively writing on
top of the FFI document "If your program does this perfectly reasonable combination
of finalizers, it will fall over in an undefined way should the implementation
use OS threads; furthermore there is no way around this".  Basically the fact
that there is only one OS thread is an implementation detail, not something that
the user should have to think about.

Is it really the case that neither NHC nor Hugs can implement a list of actions to
be taken at the first convenient point after GC has finished without implementing the
whole machinery of preemptive concurrency?  I take Malcolm Wallace's word for it
that it isn't trivial, but why do you need for example asynchronous interruption of 
Haskell threads, wait queues, or time slices?  Surely what you need is some way of
backing up the state upon return from GC in such a way that you can run the queued
IO actions, which may be hard but is a long way off preemptive concurrency.

If it's really impossible for NHC or Hugs to implement this, I think I would still
rather it was left to the NHC and Hugs documentation to admit that exported Haskell
functions basically don't work in some circumstances, rather than to the GHC 
documentation
to say that actually they do.
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Alastair Reid]

> I appreciate it's not Malcolm Wallace's fault that NHC can't
> implement a form of dynamic wrapper which can be used at all times.
> However since I do not think this problem is going to go away, would
> it not be better to put it in the specification but admit that NHC
> can't do it?  Otherwise the FFI specification will be very good (as
> it is) if you want to call languages like C, but will be inadequate
> if you want to communicate with other languages which also know
> about things like concurrency and garbage-collection, such as
> Haskell or even Java.

I think the two of you are getting caught up in implementation details
(which always seem fixable) instead of looking at fundamentals (which
tend not to be).

The issue is that allowing finalizers to be Haskell code or to call
Haskell code forces us to add preemptive concurrency to every Haskell
implementation.  Hugs has cooperative concurrency but that isn't
enough, it has to be preemptive.

To see why, suppose Hugs is busy calculating sum [1..100] and a
garbage collection finds it has to run a Haskell function, it can't
run it straight away because that would preempt the original
execution.  

Likewise, it can't run it immediately that the GC ends - that's still
preemption.

Hugs is probably busy evaluating a + or (:) or the like when the GC
happens, what if we wait until that finishes and then run the
finalizers?  Maybe.  The world is in a cleaner state at this point but
it's still tricky.  And, again, you've just added most of the
machinery required for implementing preemptive concurrency.

Hugs does allow context switches in the IO monad.  Could we allow it
there?  Maybe, we could.  It wouldn't do you much good though because
Haskell can spend arbitrarily long running pure code so your
finalizers might not get run for a long time or maybe never.


So what's the objection to adding preemptive concurrency?

1) The goal was to be able to call foreign code and have it call us.
   It seems like there's something wrong if doing that forces you to
   add concurrency.

2) Preemptive concurrency is one of those things that you can't just
   paint on later - you really have to rewrite the whole thing from
   scratch because it's just too hard to find all the pieces that
   make the assumption that they are single-threaded.

3) Adding preemptive concurrency guarantees a steady source of hard to
   repeat bugs - people aren't very good at writing concurrent code.

Cooperative concurrency is much easier to handle because context
switches happen at times when both tasks are ready for it: the sender
because it knows it is passing control over, the receiver because it
put itself in a clean state before it handed over control.


You ask how garbage collection can work if you can't invoke it at
arbitrary times?  You ask what use foreign export is if you can't call
the exported function at arbitrary times?
[I hope I'm paraphrasing your questions appropriately]

Both questions start by assuming that you have Haskell and Foosh [or
whatever your other language is called] running in separate threads
(otherwise things wouldn't happen at arbitrary times).  Put them in a
single thread and there's no problem: you're not doing things at
arbitrary times, you're doing them at quite specific times: Foosh had
control and it called into Haskell or Haskell had control and it
called into Foosh.  

In other words, the problems you raise come from wanting preemptive
concurrency so it's no wonder that the ffi extension doesn't address
them.  We could start working on a concurrency extension but that's a
different beast.

That said, we probably do need a few hooks to make Haskell talk to
other GCs better.  See the paper I pointed at this afternoon for
a sketch of what you need if you're to avoid space leaks.

--
Alastair Reid [EMAIL PROTECTED]  
Reid Consulting (UK) Limited  http://www.reid-consulting-uk.ltd.uk/alastair/





___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Malcolm Wallace wrote
[snip]
> When sequential control returns to the Foogle world, that is when
> the Haskell GC is guaranteed to be complete.
[snip]
I don't think I recognise this notion of "sequential control".  It seems to be
assuming there is only one "real" OS thread in existence, and it is impossible
for both Foogle and Haskell to have "sequential control" over this thread at any
given moment.  When they do not have sequential control, they are not allowed to
perform garbage collection.  This model no doubt fits NHC well enough, but it seems
to be assuming an awful lot about Foogle or about the system you are running on.
To take a concrete example, suppose Haskell == NHC and Foogle = GHC, then it seems
to me the model breaks down because I imagine GHC is quite happy to do GCs and
invoke finalizers in the middle of exported functions.  Even worse, GHC at least is
now capable thanks to Sigbjorn of running in several OS threads at once (on Windows) 
so 
it may be doing GCs at any time whatever. 

I appreciate it's not Malcolm Wallace's fault that NHC can't implement a form of 
dynamic wrapper which can be used at all times.  However since I do not think this
problem is going to go away, would it not be better to put it in the specification but
admit that NHC can't do it?  Otherwise the FFI specification will be very good (as it 
is) 
if you want to call languages like C, but will be inadequate if you want to 
communicate 
with other languages which also know about things like concurrency and 
garbage-collection, 
such as Haskell or even Java.
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Malcolm Wallace]

George Russell <[EMAIL PROTECTED]> writes:

> Unfortunately some sort of cascade is exactly what we want and need when
> the Haskell finaliser indicates that Haskell is no longer interested in
> some Foogle object, which means Foogle can run a GC which indicates
> Foogle is no longer interested in some Haskell object and so on . . .

Yes.  The key thing is just to delay the steps of the cascade so
that you are not trying to call Haskell/Foogle code whilst there is
no Haskell/Foogle heap available to run it.

> > After that, it doesn't matter when the Foogle finaliser decides to run.
>
> But surely Foogle has no way of knowing when the Haskell GC is over?

When sequential control returns to the Foogle world, that is when
the Haskell GC is guaranteed to be complete.

> Suppose Haskell does
> 
> [enter GC]
> ...
> [run finaliser 1]
> ...
> [run finaliser 2]
> ...
> [leave GC]
> 
> Then you want Foogle to delay any Haskell calls consequent on finaliser
> 1 until [leave GC], don't you?  How can it?

Because the Haskell calls don't happen until the Foogle GC invokes them.

Ok, you have a Haskell ForeignPtr which is really a Foogle object.
It becomes garbage.  At the next Haskell GC, its finaliser is run.
The finaliser is not Haskell code.  The finaliser is Foogle code.
This finaliser runs and a Haskell StablePtr contained within the
Foogle object becomes garbage.  Although the Haskell object is
regarded as a StablePtr in Haskell land, it is a Foogle ForeignPtr.
Hence, its finaliser does not run yet.  The Haskell GC finishes.
Computation proceeds.  At some later moment, Foogle exhausts its heap
and starts a GC.  The finaliser for the garbage object is now run.
This is not Foogle code, it is Haskell code.  In fact, it is the Haskell
routine freeStablePtr.  This is ok, because we have sufficient heap to
run the finaliser.  The StablePtr is released.  The Foogle GC finishes.
Computation proceeds.

> Is it really so difficult to create some queue of delayed functions
> which can be appended to from C and which nhc checks every time it does
> "leave GC"?

I did implement this, once.  The problem is that we don't know when to
run the delayed finaliser.  The moment Haskell GC finishes is *not*
a good time.  Some arbitrary reduction step is in mid-progress,
and running another process at this moment corrupts the context.
What you really need is some kind of scheduler that can decide when
it is safe to run an independent thread.

Regards,
Malcolm
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

George Russell wrote:
[snip]
> Is it really so difficult to create some queue of delayed functions which can be 
>appended
> to from C and which nhc checks every time it does "leave GHC"?
[snip]
Sorry, that was rather impolite of me.  Now I've looked back through the archives I 
see that
Malcolm has tried this and found it difficult for good reason.  All the same I don't
see how it can be avoided.  In Foogle-land they can't know anything about when Haskell 
is
running a garbage collection or not, and of course Foogle might be running various 
concurrent
threads (just like Haskell), so if you are going to say that Foogle is never to call a
Haskell function during a Haskell garbage collection, I don't see how Foogle is to 
call Haskell
at all.
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Malcolm Wallace wrote
[snip]
> Quite simply, no finaliser (whether in Foogle or Haskell) should
> be capable of triggering a garbage collection within its call.
> This condition is absolutely necessary to prevent a cascade effect
> of cross-language garbage collections, where a finaliser in Haskell
> could trigger a GC in Foogle which triggers another (nested) GC in
> Haskell etc.
[snip]
Unfortunately some sort of cascade is exactly what we want and need when
the Haskell finaliser indicates that Haskell is no longer interested in some
Foogle object, which means Foogle can run a GC which indicates Foogle is no
longer interested in some Haskell object and so on . . .

> Thus, if Haskell.freeSomething calls Foogle.freeSomething, and
> Foogle.freeSomething cannot cause a Foogle GC, then no Foogle
> finalisers are run yet, and so Foogle *cannot* call the Haskell world
> until the Haskell GC is complete.  After that, it doesn't matter
> when the Foogle finaliser decides to run.
[snip]
But surely Foogle has no way of knowing when the Haskell GC is over?
Suppose Haskell does

[enter GC]
...
[run finaliser 1]
...
[run finaliser 2]
...
[leave GC]

Then you want Foogle to delay any Haskell calls consequent on finaliser 1 until [leave 
GC],
don't you?  How can it?

Is it really so difficult to create some queue of delayed functions which can be 
appended
to from C and which nhc checks every time it does "leave GHC"?
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Malcolm Wallace]

George Russell <[EMAIL PROTECTED]> writes:

> Well, if the nhc98 RTS can't work out a time when it will be possible to
> run the delayed finalizer, how in creation do you expect the Foogle
> finalizer or whatever Foogle GC process it invokes to do so?

Quite simply, no finaliser (whether in Foogle or Haskell) should
be capable of triggering a garbage collection within its call.
This condition is absolutely necessary to prevent a cascade effect
of cross-language garbage collections, where a finaliser in Haskell
could trigger a GC in Foogle which triggers another (nested) GC in
Haskell etc.

Thus, if Haskell.freeSomething calls Foogle.freeSomething, and
Foogle.freeSomething cannot cause a Foogle GC, then no Foogle
finalisers are run yet, and so Foogle *cannot* call the Haskell world
until the Haskell GC is complete.  After that, it doesn't matter
when the Foogle finaliser decides to run.

Note that all of this assumes a sequential processing model, where
control is always with one of Foogle or Haskell, never both.  The FFI
spec explicitly avoids introducing any requirement for concurrency,
since it aims to be a minimal extension to H'98.

Regards,
Malcolm
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Malcolm Wallace wrote (about delayed finalizers):
> This was the originally proposed solution for re-entrant finalisers
> in nhc98 and Hugs.  The problem is that, certainly for nhc98, it is
> nigh impossible for the RTS to identify a "convenient" later moment
> at which to run the delayed finaliser.
Well, if the nhc98 RTS can't work out a time when it will be possible to
run the delayed finalizer, how in creation do you expect the Foogle finalizer
or whatever Foogle GC process it invokes to do so?
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Malcolm Wallace]

George Russell <[EMAIL PROTECTED]> writes:

> However in the case I cited it isn't particularly important to execute
> the callback at once.  If calling Haskell at arbitrary times is
> unimplementable, could there instead be some way of some way like
> "wrapper" of exporting a Haskell function[args] -> IO () so that the
> resulting C function, when executed, merely registers the Haskell action
> to be executed as soon as the Haskell RTS finds it convenient, rather
> than at once?

This was the originally proposed solution for re-entrant finalisers
in nhc98 and Hugs.  The problem is that, certainly for nhc98, it is
nigh impossible for the RTS to identify a "convenient" later moment
at which to run the delayed finaliser.

Regards,
Malcolm
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Alastair Reid wrote
[snip]
> We should provide a C function hs_freeStablePtr and explicitly
> say that it is safe to call this from inside a finalizer.
[snip]
This would be the simplest solution, but would not permit you to do
anything more sophisticated at the Haskell side, such as reference counting
or more complicated strategies to check for possible cycles.  You might end
up having to do all your inter-language GC code in C, ugh.
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

RE: Proposed change to ForeignPtr

[George Russell]

Simon Marlow wrote
> George: the problem with allowing callbacks from a finalizer is that in
> order to implement this your system essentially has to be able to make a
> pre-emptive context switch from the currently running Haskell
> computation to the finalizer; this turned out to have a fairly high
> implementation cost for nhc98 and Hugs, so we decided to amend the spec
> instead.  You'll still be able to do this in GHC, of course, but it
> won't be portable.

However in the case I cited it isn't particularly important to execute the callback
at once.  If calling Haskell at arbitrary times is unimplementable,
could there instead be some way of some way like "wrapper" of exporting a Haskell 
function
[args] -> IO () so that the resulting C function, when executed, merely registers the
Haskell action to be executed as soon as the Haskell RTS finds it convenient, rather 
than
at once?
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Malcolm Wallace wrote
[snip]
> I don't see the problem.  The Foogle garbage collector runs separately
> and asynchronously to the Haskell GC.  A Foogle object is released
> by the Haskell collector, then at a later moment in time, a Haskell
> object is released by the Foogle collector.  Where is the conflict?
[snip]
Does "at a later moment in time" mean that it is late enough that we can
be sure calling Haskell will be OK?

Look, suppose for simplicity that Foogle implements an identical FFI to
Haskell.  So we have

(Haskell ForeignPtr A) == (Foogle StablePtr A)
(Foogle StablePtr A) points to (Foogle ForeignPtr B)
(Foogle ForeignPtr B) == (Haskell StablePtr B)

Haskell frees (Haskell ForeignPtr A) thereby causing the finalizer.
This presumably does the Foogle equivalent of freeStablePtr on A.
This *may* trigger an immediate Foogle garbage collection (I think Foogle's
RTS is within its rights if it does), so that Foogle now wants to finalize
Foogle ForeignPtr B.  The Foogle finalizer action for foreignPtr B involves
calling back to Haskell so that (Haskell StablePtr B) can have freeStablePtr
applied to it.

Now is the FFI specification going to guarantee that however quickly the
Foogle garbage collector executes the Haskell finalizer, things will work.
It seems to me the wording suggested does not guarantee this.  All it says
that the finaliser cannot portably "call back into the Haskell system".  But
when the finaliser provokes an immediate GC which calls back into the
Haskell system?

I'm altogether rather puzzled by this notion of FunPtr's which are allowed to
call Haskell back at some times and not others.  Nor do I understand how it's
supposed to work, say, on truly parallel Haskell implementations.

[snip]
> Why do you suggest a need for reference counts?  In the absence
> of cycles, surely the existing two garbage collectors (howsoever
> implemented) are sufficient.
[snip]
Garbage collectors need roots.  If I understand the situation correctly,
a StablePtr is itself a root, until explicitly freed.  If you are handing out 
the same object via the same StablePtr to several
different Foogle things, it seems to me you might attach a reference counter
to the StablePtr, so that when Foogle says "I am no longer interested in this
(Haskell) StablePtr", you decrement the reference counter by 1, and deallocate
if it reaches zero.  Alternatively of course you could create a fresh StablePtr
every time something is passed to Foogle.

I think reference counters are sometimes used in things like CORBA, where 
a similar problem arises.  Of course they cannot cope with interlanguage cycles.
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Alastair Reid]

> I think this is all a rather murky area.  Consider two systems,
> let's call them Haskell and Foogle, which both operate heaps and do
> their own storage allocation, but also communicate over similar
> FFIs.

This is indeed a murky area.

Especially tricky when you have two GC's (even if one is nothing more
than reference counting) is the issue of cycles that span the two
systems: neither side is willing to release its pointer into the other
world until the other side releases its pointer.  (See section 5.2 of
http://www.reid-consulting-uk.ltd.uk/alastair/publications/FP94.ps.gz
for more on the problem and a sketch of a fix.)

> We might very reasonably have situations where fairly complex
> inter-language pointers exist, so for example Haskell holds a
> ForeignPtr to something in the Foogle heap; the pointed-to Foogle
> object in turn references a Haskell object (presumably provided via
> StablePtr).  Now suppose Haskell wants to drop the ForeignPtr.  Then
> the logical thing for the finalizer to do is to tell Foogle that
> Haskell is no longer interested in the Foogle object.  This then
> gives Foogle the chance on its own garbage collection to in turn
> drop the Haskell StablePtr.  In turn this means somehow running
> StablePtr.freeStablePtr.  However this scheme I don't know if that's
> legal, because the Haskell finalizer you need to run "freeStablePtr"
> is indirectly provoked by the initial Haskell finalizer.

We should provide a C function hs_freeStablePtr and explicitly
say that it is safe to call this from inside a finalizer.

> Of course you would need at least reference counters (if you can
> guarantee there are no cycles containing both languages) or
> something more powerful otherwise, but reference counters at least
> can be provided. 

At least half the times I've used ForeignPtrs (aka ForeignObjs aka
MallocPtrs), I've had to implement my own reference counting so, yes,
it does seem like it'd be good to include reference counting in the
implementation somehow.  

I don't remember ever finding a good way to do it though.  The design
seems obvious enough but it seems there's always some odd little
wrinkle like the object already having its own reference counting
implementation or the other world wanting to talk about foo_t* (i.e.,
the type the external library provides) instead of hs_stablePtr_t.  In
particular, if I put a wrapper around an object so that I can attach a
reference counter, you somehow always find yourself faced with the
problem that a C library hands you a pointer to an unwrapped object
and you have to try to track down the reference counter for it.

Any ideas welcome.

--
Alastair Reid [EMAIL PROTECTED]  
Reid Consulting (UK) Limited  http://www.reid-consulting-uk.ltd.uk/alastair/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

RE: Proposed change to ForeignPtr

[Simon Marlow]

Malcolm says:
> I don't see the problem.  The Foogle garbage collector runs separately
> and asynchronously to the Haskell GC.  A Foogle object is released
> by the Haskell collector, then at a later moment in time, a Haskell
> object is released by the Foogle collector.  Where is the conflict?

Foogle might be doing reference counting instead of GC, so it might want
to run the finalizer synchronously.

George: the problem with allowing callbacks from a finalizer is that in
order to implement this your system essentially has to be able to make a
pre-emptive context switch from the currently running Haskell
computation to the finalizer; this turned out to have a fairly high
implementation cost for nhc98 and Hugs, so we decided to amend the spec
instead.  You'll still be able to do this in GHC, of course, but it
won't be portable.

Cheers,
Simon
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Malcolm Wallace]

George Russell <[EMAIL PROTECTED]> writes:

> We might
> very reasonably have situations where fairly complex  inter-language
> pointers exist, so for example Haskell holds a ForeignPtr to something
> in the Foogle heap; the pointed-to Foogle object in turn references a
> Haskell object (presumably provided via StablePtr).  Now suppose Haskell
> wants to drop the ForeignPtr.  Then the logical thing for the finalizer
> to do is to tell Foogle that Haskell is no longer interested in the
> Foogle object.  This then gives Foogle the chance on its own garbage
> collection to in turn drop the Haskell StablePtr.  In turn this means
> somehow running StablePtr.freeStablePtr.  However this scheme I don't
> know if that's legal, because the Haskell finalizer you need to run
> "freeStablePtr" is indirectly provoked by the initial Haskell finalizer.

I don't see the problem.  The Foogle garbage collector runs separately
and asynchronously to the Haskell GC.  A Foogle object is released
by the Haskell collector, then at a later moment in time, a Haskell
object is released by the Foogle collector.  Where is the conflict?

> Of course you
> would need at least reference counters (if you can guarantee there are
> no cycles containing both languages) or something more powerful
> otherwise, but reference counters at least can be provided.

Why do you suggest a need for reference counts?  In the absence
of cycles, surely the existing two garbage collectors (howsoever
implemented) are sufficient.

> I'm afraid I haven't been following this thread lately, so I don't know
> what the big problem is about calling Haskell from a finalizer; I
> suppose it's something to do with us being in the middle of garbage
> collection.

There is only a problem if Haskell is called directly by a finaliser
since, as you noted, there is no heap available to run the finaliser.
An indirect Haskell call triggered at some indeterminate point in
the future, but in any case after the Haskell GC has completed,
is no problem at all.

Regards,
Malcolm
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Manuel wrote (snipped)

> I have changed this in the spec now.  I attach the wording
> used in the spec.

> \item[newForeignPtr ::\ Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a)]
>   Turn a plain memory reference into a foreign object by associating a
>   finalizer with the reference.  The finalizer is represented by a pointer to
>   an external function, which will be executed after the last reference to the
>   foreign object is dropped.  On invocation, the finalizer receives a pointer
>   to the associated foreign object as an argument.  Note that there is no
>   guarantee on how soon the finalizer is executed after the last reference was
>   dropped; this depends on the details of the Haskell storage manager. The
>   only guarantee is that the finalizer runs before the program terminates.
> 
>   Whether a finaliser may call back into the Haskell system is system
>   dependent.  Portable code may not rely on such call backs.
>   
> \item[addForeignPtrFinalizer ::\ ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO
>   ()] Add another finalizer to the given foreign object. No guarantees are
>   made on the order in which multiple finalizers for a single object are run.

I think this is all a rather murky area.  Consider two systems, let's call them
Haskell and Foogle, which both operate heaps and do their own storage allocation,
but also communicate over similar FFIs.  We might very reasonably
have situations where fairly complex  inter-language pointers exist, so for example 
Haskell holds a 
ForeignPtr to something in the Foogle heap; the pointed-to Foogle object in turn 
references
a Haskell object (presumably provided via StablePtr).  Now suppose Haskell wants to
drop the ForeignPtr.  Then the logical thing for the finalizer to do is to tell Foogle 
that Haskell is no longer interested in the Foogle object.  This then gives Foogle
the chance on its own garbage collection to in turn drop the Haskell StablePtr.  In 
turn
this means somehow running StablePtr.freeStablePtr.  However this scheme I don't know 
if
that's legal, because the Haskell finalizer you need to run "freeStablePtr" is 
indirectly
provoked by the initial Haskell finalizer.  

This is a pity, because this might actually be a fairly good way of managing garbage 
collection 
between Foogle and Haskell.  Of course you would need at least reference counters (if 
you can
guarantee there are no cycles containing both languages) or something more powerful 
otherwise,
but reference counters at least can be provided.  Furthermore I do actually have a 
real case
in mind where I might use something like this, though I'd rather not go into details 
at this
time.

I'm afraid I haven't been following this thread lately, so I don't know what the big 
problem
is about calling Haskell from a finalizer; I suppose it's something to do with us 
being in the
middle of garbage collection.  However wouldn't it be better to allow finalizers to 
somehow provide
an action which may call Haskell, but (a) may be performed at some later date (such as 
when the GC is
over); (b) consequently, may not assume that the pointer finalized still points to 
anything?
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Alastair Reid]

> I have changed this in the spec now.  I attach the wording used in
> the spec.

Looks good.

A
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Manuel M T Chakravarty]

Manuel M T Chakravarty <[EMAIL PROTECTED]> wrote,

> We seem to have a consensus on this one.  We change the type
> of the existing functions to
> 
>   newForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a)
>   addForeignPtrFinalizer :: ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO ()
> 
> For GHC, I propose to put the closure-based versions into an
> extra module (that's easy enough with the hierarchical
> libraries).  This makes changing over old code easier, as it
> merely requires to alter the import and not all occurences
> of the functions.
> 
> Any objections?

I have changed this in the spec now.  I attach the wording
used in the spec.

Manuel

-=-

\item[newForeignPtr ::\ Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a)]
  Turn a plain memory reference into a foreign object by associating a
  finalizer with the reference.  The finalizer is represented by a pointer to
  an external function, which will be executed after the last reference to the
  foreign object is dropped.  On invocation, the finalizer receives a pointer
  to the associated foreign object as an argument.  Note that there is no
  guarantee on how soon the finalizer is executed after the last reference was
  dropped; this depends on the details of the Haskell storage manager. The
  only guarantee is that the finalizer runs before the program terminates.

  Whether a finaliser may call back into the Haskell system is system
  dependent.  Portable code may not rely on such call backs.
  
\item[addForeignPtrFinalizer ::\ ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO
  ()] Add another finalizer to the given foreign object. No guarantees are
  made on the order in which multiple finalizers for a single object are run.

___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Alastair Reid]

> I thought we have settled this already.  See my appended email from
> three weeks ago.  The only reason I haven't changed it in the spec
> yet is that I wanted to give people time to object.  Nobody has.
> So, I would deem this accepted (reusing the old function names btw).

Sorry, yes you had.
I'd failed to mark it as done in my email.

I'll tweak Hugs to conform.

A

___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Manuel M T Chakravarty]

Alastair Reid <[EMAIL PROTECTED]> wrote,

> [Now that we've gotten the library specification issue out of the way,
> I'd like to revive discussion of this topic.  Last time we discussed
> it, there seemed to be concensus that we would make a change but I
> didn't get much response when I made a concrete proposal.  I'd like to
> resolve this promptly so that the impending Hugs release can match
> what the spec says (or vice-versa...).]
> 
> Since requiring ForeignPtr.newForeignPtr would require preemptive
> concurrency (see previous discussion), I propose the following changes:
[..]

I thought we have settled this already.  See my appended
email from three weeks ago.  The only reason I haven't
changed it in the spec yet is that I wanted to give people
time to object.  Nobody has.  So, I would deem this
accepted (reusing the old function names btw).

Manuel

-=-

Subject: Re: Proposed change to ForeignPtr
From: Manuel M T Chakravarty <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Date: Mon, 12 Aug 2002 23:00:22 +1000 (EST)
Reply-To: [EMAIL PROTECTED]
X-Mailer: Mew version 2.2 on XEmacs 21.4.6 (Common Lisp)

We seem to have a consensus on this one.  We change the type
of the existing functions to

  newForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a)
  addForeignPtrFinalizer :: ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO ()

For GHC, I propose to put the closure-based versions into an
extra module (that's easy enough with the hierarchical
libraries).  This makes changing over old code easier, as it
merely requires to alter the import and not all occurences
of the functions.

Any objections?

Cheers,
Manuel
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

RE: Proposed change to ForeignPtr

[Simon Marlow]

> 1) Add these functions:
> 
>  makeForeignPtr
>:: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a)
>  attachForeignPtrFinalizer 
>:: ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO ()
> 
>It is implementation defined whether the free functions are allowed
>to call Haskell functions.
> 
> 2) Remove newForeignPtr and addForeignPtrFinalizer
>[GHC can go ahead and list them as non-standard extensions]
> 
> 
> There's a minor issue about whether the old function names should be
> reused (leaving GHC to come up with its own names) or not.  I have
> ceased to care either way.

I have a slight preference for re-using the old names, at least for
newForeignPtr.  The reason is that it follows the naming conventions in 

http://www.haskell.org/~simonmar/libraries/conventions.html

Cheers,
Simon
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Manuel M T Chakravarty]

We seem to have a consensus on this one.  We change the type
of the existing functions to

  newForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a)
  addForeignPtrFinalizer :: ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO ()

For GHC, I propose to put the closure-based versions into an
extra module (that's easy enough with the hierarchical
libraries).  This makes changing over old code easier, as it
merely requires to alter the import and not all occurences
of the functions.

Any objections?

Cheers,
Manuel
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Manuel M T Chakravarty]

"Simon Marlow" <[EMAIL PROTECTED]> wrote,
> > > That's a tricky one.  From the standards point of view, I am
> > > actually *very* reluctant to introduce new names.  On the other
> > > hand, reusing the old names will lead to a couple of unhappy emails
> > > from people using the old interface again.
> > 
> > But only a couple I conjecture.
> 
> Heh, I distinctly remember several complaints from *you* in the past
> when things have changed in GHC!

I don't see a contradiction in these two statements.  After
all Alastair can only account for one complaint per change
...and he won't complain this time; so, there will be less.
;-)

Manuel
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

RE: Proposed change to ForeignPtr

[Simon Marlow]

> > What do you expect to happen if the finaliser calls a foreign
> > exported function?
> 
> Good question.
> 
> I do not expect that to work on any platform that has difficulty
> implementing newForeignPtr (because you could use it to implement
> newForeignPtr).
> 
> I don't know if it would be likely to work on GHC.
> 
> I think the spec should say that it is an error or undefined
> depending on whether GHC supports reentrant finalizers or not.

Yes, it will work in GHC.  makeForeignPtr is easily implemented in terms
of newForeignPtr, using a foreign import "dynamic".

> > That's a tricky one.  From the standards point of view, I am
> > actually *very* reluctant to introduce new names.  On the other
> > hand, reusing the old names will lead to a couple of unhappy emails
> > from people using the old interface again.
> 
> But only a couple I conjecture.

Heh, I distinctly remember several complaints from *you* in the past
when things have changed in GHC!

Cheers,
Simon
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Malcolm Wallace]

Alastair Reid <[EMAIL PROTECTED]> writes:

> I'm not sure which position you're preferring here.  I lean a bit
> towards using the old names for the new functions (the ones with free
> functions) and finding new names for the old functions (the ones with
> closure arguments).

That would be my preference as well.  Keep the existing standard names,
but change their type signatures to reflect what Hugs and nhc98 can
actually implement.  Add new names with the old signatures for what
ghc can additionally implement via its concurrency model.

Regards,
Malcolm
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Alastair Reid]

>> > That's a tricky one.  From the standards point of view, I am >
>> actually *very* reluctant to introduce new names.  On the other >
>> hand, reusing the old names will lead to a couple of unhappy emails
>> > from people using the old interface again.
>> 
>> But only a couple I conjecture.

> I read this as you would also (= like me) be in favour of keeping
> the old names.  Right?

I'm not sure which position you're preferring here.  I lean a bit
towards using the old names for the new functions (the ones with free
functions) and finding new names for the old functions (the ones with
closure arguments).


-- 
Alastair Reid [EMAIL PROTECTED]  
Reid Consulting (UK) Limited  http://www.reid-consulting-uk.ltd.uk/alastair/

___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Manuel M T Chakravarty]

Alastair Reid <[EMAIL PROTECTED]> wrote,

> > What do you expect to happen if the finaliser calls a foreign
> > exported function?
> 
> Good question.
> 
> I do not expect that to work on any platform that has difficulty
> implementing newForeignPtr (because you could use it to implement
> newForeignPtr).
> 
> I don't know if it would be likely to work on GHC.

SimonM, what do you think?

> I think the spec should say that it is an error or undefined
> depending on whether GHC supports reentrant finalizers or not.

IMHO, it's a nice feature to have.  I understand that the
spec can't require it, as systems without preemptive threads
can't implement it.  However, it would be a pity if the new
interfaces would mean that even systems that feature
preemptive threads can't have it.

> > That's a tricky one.  From the standards point of view, I am
> > actually *very* reluctant to introduce new names.  On the other
> > hand, reusing the old names will lead to a couple of unhappy emails
> > from people using the old interface again.
> 
> But only a couple I conjecture.

I read this as you would also (= like me) be in favour of
keeping the old names.  Right?

Other opinions?

Cheers,
Manuel
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Alastair Reid]

Alastair wrote [snip]
> makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO ForeignObj

> [snip] I don't understand this proposal.  What is a ForeignObj?

Sorry, that was a typo.  The result type should be 

  IO (ForeignPtr a)

> I call a C function, which gives me a cString :: Ptr CChar, and it's
> my responsibility to free it when I've finished with it.  So I
> convert it to a ForeignPtr:

>foreignPtr <- mkForeignPtr cString

> and then always refer to the pointer via foreignPtr.  When
> foreignPtr is garbage collected the space is freed.  So how do I do
> this with your proposal?

With the existing spec, you would write:

> foreign import free :: Ptr CChar -> IO ()
>
> foo = do
>   ...
>   foreignPtr <- newForeignPtr cString (free cString)
>   ...

With my proposal, you would write:

> foreign import "&" free :: FunPtr (Ptr CChar -> IO ())
>
> foo = do
>   ...
>   foreignPtr <- newForeignPtr cString free
>   ...

All the rest of your code to manipulate ForeignPtrs remains the same.
(Well, there's a corresponding change in addForeignFinalizer if you
happen to use that.)


-- 
Alastair Reid [EMAIL PROTECTED]  
Reid Consulting (UK) Limited  http://www.reid-consulting-uk.ltd.uk/alastair/

___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[George Russell]

Alastair wrote
[snip]
> I propose the following changes:
> 
> 1) Add these functions:
> 
>   makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO ForeignObj
>   attachForeignPtrFinalizer :: ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO ()
> 
> 2) Remove newForeignPtr and addForeignPtrFinalizer
[snip]
I don't understand this proposal.  What is a ForeignObj?

I call a C function, which gives me a cString :: Ptr CChar, and
it's my responsibility to free it when I've finished with it.  So I convert it
to a ForeignPtr:

   foreignPtr <- mkForeignPtr cString

and then always refer to the pointer via foreignPtr.  When foreignPtr is garbage 
collected
the space is freed.  So how do I do this with your proposal?
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Alastair Reid]

> I assume you meant
>   makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a)

Oops, yes.

> What do you expect to happen if the finaliser calls a foreign
> exported function?

Good question.

I do not expect that to work on any platform that has difficulty
implementing newForeignPtr (because you could use it to implement
newForeignPtr).

I don't know if it would be likely to work on GHC.

I think the spec should say that it is an error or undefined
depending on whether GHC supports reentrant finalizers or not.

> That's a tricky one.  From the standards point of view, I am
> actually *very* reluctant to introduce new names.  On the other
> hand, reusing the old names will lead to a couple of unhappy emails
> from people using the old interface again.

But only a couple I conjecture.

--
Alastair Reid [EMAIL PROTECTED]  
Reid Consulting (UK) Limited  http://www.reid-consulting-uk.ltd.uk/alastair/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi

Re: Proposed change to ForeignPtr

[Manuel M T Chakravarty]

Alastair Reid <[EMAIL PROTECTED]> wrote,

> Since requiring ForeignPtr.newForeignPtr would require preemptive
> concurrency (see previosu discussion), I propose the following changes:
> 
> 1) Add these functions:
> 
>   makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO ForeignObj

I assume you meant

  makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a)

>   attachForeignPtrFinalizer :: ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO ()

What do you expect to happen if the finaliser calls a
foreign exported function?  (I am sorry, but I still haven't
been able to closely look at the entire ForeignPtr
discussion, so maybe this question was already answered
there.)

> 2) Remove newForeignPtr and addForeignPtrFinalizer
> 
> Two variants on this proposal:
> 
> 1) Reuse the old function names instead of inventing new ones.
> 
>I don't care much either way.  

That's a tricky one.  From the standards point of view, I am
actually *very* reluctant to introduce new names.  On the
other hand, reusing the old names will lead to a couple of
unhappy emails from people using the old interface again.

Maybe we should have an -package old-ffi for compatibility.

> 2) Instead of removing the old functions, rename them and mark them
>as an optional extra.  
> 
>I think this would be a mistake - the GHC documentation is the
>right place to mention extensions which are not available on all
>ffi implementations.

We have avoided optional extras so far and I don't see a
reason to deviate from that policy now.  Moreover, it is
easy to define the old interface in terms of the new one
that you propose (given you can handle callbacks in
finalisers). 

Cheers,
Manuel
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi