Re: Mark procedures and LilyPond

[Mark H Weaver]

Andy Wingo  writes:

> On Thu 05 Nov 2015 11:29, l...@gnu.org (Ludovic Courtès) writes:
>
>> What we need above all is to address LilyPond’s use case.  I proposed a
>> solution at  but
>> never understood whether/why it was considered unfit.
>
> I agree with you that the patch there looks reasonable to me too, though
> AFAIU the original code should work just fine too.
>
> There area few things at play.
>
>  (1) A bug related to SMOB finalization and marking that affects
>  LilyPond
>
>  (2) The utility of mark procedures in general
>
>  (3) The suitability of mark procedures for future uses
>
>  (4) Whether we can get by without mark procedures, and if so, how.
>
> For (1) it seems to me that we just have a bug.  A SMOB mark function
> was called on an object after the finalizer.  Note that having
> the finalizer called doesn't mean that the GC object was collected -- it
> just means it was collectable, perhaps in a clique of objects.
> Finalization being asynchronous with marking it's possible that a clique
> of objects was only half-finalized when a new mark procedure runs.  The
> mark procedure saw an object on which free() was already called -- this
> is possible.

Yes, exactly.

> We should fix Guile so to "null out" the SMOB typecode when the SMOB
> finalizer is called.  If our mark procedure sees a SMOB that has already
> been finalized, it just returns.

Unfortunately, I doubt this will be sufficient for LilyPond.  The small
example case in , which is apparently
representative of how things are typically done in LilyPond, has
structures like this:

   ____
  Objects in  |  |  |  |
  GC-managed  |  SMOB 1  |  |  SMOB 2  |
 heap |__|  |__|
 |   ^ |   ^
.|...|.|...|..
   __v___|___  _ __v___|___
  Objects in  |  ||   STL   |   |  |
  normal heap |C++ object|--->|container|-->|C++ object|
  (not scanned|__||_|   |__|
 by GC)


The SMOB finalizers free the associated C++ objects below them.  Now,
suppose that none of the objects above are reachable, so both SMOBs are
queued for finalization.  Now suppose that SMOB 2 is finalized first,
thus freeing the C++ object below it.  Suppose further that we null out
the SMOB 2 typecode.

Now another GC occurs and the marker is called on SMOB 1.  It's typecode
has not yet been nulled, so the user-specified mark procedure is called.
The mark procedure for SMOB 1 iterates over the STL container, and for
each C++ object within, marks the corresponding SMOB via the upward
pointer, in this case SMOB 2.  Although SMOB 2's typecode has been
zeroed out, the discovery of the fact is too late, because the (freed)
C++ object below it has been referenced.

As far as I can tell, this way of doing things depends on the old 1.8 GC
finalization semantics, where all of the finalizers are called before
the mutator resumes execution.

  Mark

Re: Mark procedures and LilyPond

[Ludovic Courtès]

Mark H Weaver  skribis:

> Unfortunately, I doubt this will be sufficient for LilyPond.  The small
> example case in , which is apparently
> representative of how things are typically done in LilyPond, has
> structures like this:
>
>____
>   Objects in  |  |  |  |
>   GC-managed  |  SMOB 1  |  |  SMOB 2  |
>  heap |__|  |__|
>  |   ^ |   ^
> .|...|.|...|..
>__v___|___  _ __v___|___
>   Objects in  |  ||   STL   |   |  |
>   normal heap |C++ object|--->|container|-->|C++ object|
>   (not scanned|__||_|   |__|
>  by GC)

Thanks for the picture, that’s very helpful!

I still think we should be able to get rid of GC mark procedures in this
case, sidestepping the bug you describe.

This pattern is very common when writing bindings.  I can’t imagine that
there's something insurmountable here.  Am I missing something?

Ludo’.

Re: Mark procedures and LilyPond

[Stefan Monnier]

>____
>   Objects in  |  |  |  |
>   GC-managed  |  SMOB 1  |  |  SMOB 2  |
>  heap |__|  |__|
>  |   ^ |   ^
> .|...|.|...|..
>__v___|___  _ __v___|___
>   Objects in  |  ||   STL   |   |  |
>   normal heap |C++ object|--->|container|-->|C++ object|
>   (not scanned|__||_|   |__|
>  by GC)


> The SMOB finalizers free the associated C++ objects below them.  Now,
> suppose that none of the objects above are reachable, so both SMOBs are
> queued for finalization.  Now suppose that SMOB 2 is finalized first,
> thus freeing the C++ object below it.

It's clearly wrong for SMOB2's finalizer to free its C++ object here
since that object is still reachable from C++ objects.


Stefan

Re: Mark procedures and LilyPond

[Hans Åberg]

> On 6 Nov 2015, at 13:32, Mark H Weaver  wrote:

>> We should fix Guile so to "null out" the SMOB typecode when the SMOB
>> finalizer is called.  If our mark procedure sees a SMOB that has already
>> been finalized, it just returns.
> 
> Unfortunately, I doubt this will be sufficient for LilyPond.  The small
> example case in , which is apparently
> representative of how things are typically done in LilyPond, has
> structures like this:
> 
>   ____
>  Objects in  |  |  |  |
>  GC-managed  |  SMOB 1  |  |  SMOB 2  |
> heap |__|  |__|
> |   ^ |   ^
> .|...|.|...|..
>   __v___|___  _ __v___|___
>  Objects in  |  ||   STL   |   |  |
>  normal heap |C++ object|--->|container|-->|C++ object|
>  (not scanned|__||_|   |__|
> by GC)
> 
> 
> The SMOB finalizers free the associated C++ objects below them.  Now,
> suppose that none of the objects above are reachable, so both SMOBs are
> queued for finalization.  Now suppose that SMOB 2 is finalized first,
> thus freeing the C++ object below it.  Suppose further that we null out
> the SMOB 2 typecode.
> 
> Now another GC occurs and the marker is called on SMOB 1.  It's typecode
> has not yet been nulled, so the user-specified mark procedure is called.
> The mark procedure for SMOB 1 iterates over the STL container, and for
> each C++ object within, marks the corresponding SMOB via the upward
> pointer, in this case SMOB 2.  Although SMOB 2's typecode has been
> zeroed out, the discovery of the fact is too late, because the (freed)
> C++ object below it has been referenced.
> 
> As far as I can tell, this way of doing things depends on the old 1.8 GC
> finalization semantics, where all of the finalizers are called before
> the mutator resumes execution.

Is Guile 2 doing its own finalization, not synced withe the Boehm GC?