On Monday 24 August 2009 22:44:59 Bill Hart wrote:
> On 23 Aug, 21:42, Nils Bruin <nbr...@sfu.ca> wrote:
> > The following problem came up while trying to use ecl as a library
> > inside sage:
> >
> > Both sage and ecl use GMP for their multi-precision arithmetic, and
> > both call mp_set_memory_functions to register their memory managers.
> > This obviously doesn't work, since GMP only keeps track of one set of
> > memory management routines. Ecl needs to use its memory manager, since
> > it relies on Boehm for garbage collection and sage needs to use its
> > own because python relies on reference counting.
> >
> > I have left the discussion specific to ECL, but the problem arises in
> > general: GMP has global state, so using the same GMP from several
> > places can lead to conflicts.
> >
> > Juanjo, the lead developer of ecl has suggested several work-arounds
> > (Thank you, Juanjo. Your expertise is a great help). I hope some
> > people with more knowledge of sage memory management, mpir and build
> > processes can comment on the feasibility and maintainability of the
> > different options:
> >
> > 1. Build ecl with its private gmp library.
> > (a) Can that be done on link level? i.e., take ecl.so and gmp.so
> > and produce ecl-gmp.so from the two, with all references of ecl.so to
> > gmp.so resolved and made private? (i.e., make an ecl.so with gmp
> > "statically linked in").
> > (b) Can that be done compile-time for ecl? i.e., when you compile
> > ecl to produce ecl.so, make the compiler/linker resolve any references
> > to gmp and prevent any of the gmp symbols from leaking out as public
> > in ecl.so?
> > (c) Is there an economical and maintainable way of pushing gmp-for-
> > ecl in its own namespace?
> >
To me 1) sounds like the best option, if ecl links with it own static version
of gmp , then as long as we aren't leaking any symbols everything should just
work . We are leaking some symbols though , some by accident , and some on
purpose for testing/tuning, would have to have a closer look to see how many
and what they exactly are , and how easy it is to "switch them off" for a
local ecl static build.
> > 2. Make the mp_set_memory_functions state thread-local and run ecl in
> > its own thread.
> > (a) Does that involve changing the gmp/mpir library?
>
> It would involve changing mpir. In general it is a bad idea to use
> thread local for such things. One problem is that thread local storage
> is not available on all the platforms we support. Secondly it is
> generally a bad idea to use thread local storage to manage memory
> management issues, though admittedly this tends to be about the only
> option remaining after trying everything else. But in general it leads
> to subtle annoying bugs with very low reproducibility (due to bugs
> elsewhere).
>
> > 3. Make sage use Boehm as well :-). That may be an option when
> > reference counting is discovered to be an insufficient memory
> > management strategy for sage in general.
> >
> > 4. Change ECL so that its uses of GMP are braced by
> > mp_set_memory_function
> > (a) That wouldn't be thread-safe, would it? Or are GMP instructions
> > "atomic" with respect to threads? Could we make them that way?
>
> No it wouldn't be threadsafe.
>
> > (b) In that case, it might be helpful to have a complete list of GMP
> > instructions that could possibly lead to calls to alloc,realloc,free.
> > Only those would need treatment.
>
> There are hundreds.
>
> > (c) Could this be done by properly mangling gmp.h?
>
> You couldn't possibly mangle it any more than it already is.
>
> Anyhow, here is the relevant code from mp_set_fns.c in mpir:
>
> void
> mp_set_memory_functions (void *(*alloc_func) (size_t),
> void *(*realloc_func) (void *, size_t, size_t),
> void (*free_func) (void *, size_t))
> {
> if (alloc_func == 0)
> alloc_func = __gmp_default_allocate;
> if (realloc_func == 0)
> realloc_func = __gmp_default_reallocate;
> if (free_func == 0)
> free_func = __gmp_default_free;
>
> __gmp_allocate_func = alloc_func;
> __gmp_reallocate_func = realloc_func;
> __gmp_free_func = free_func;
> }
>
> Not very extensive huh. I actually don't see a solution to this
> problem that involves mpir. In fact I think this is a security issue
> and I don't think this capability should even exist.
>
> > Thanks to Juanjo for helping this project progress. Other possibly
> > relevant remarks:
> >
> > 5. This problem isn't unique for ecl/sage. Any setting where gmp is
> > used by several libraries at once potentially causes this problem
> >
This problem is not specific to gmp , so I expect it has been "solved"
elsewhere!!!
> > 6. This problem does not arise in general when you try to extend
> > python with ecl, because normal python does not use gmp. So building a
> > python/lisp hybrid is quite feasible.
> >
> > 7. The main drive for using ecl-as-a-library in sage is to get a
> > faster interface with Maxima. If all options above turn out to involve
> > a lot of work or are expensive to maintain, perhaps we need to find
> > another way of speeding up the interface. As a side effect, we already
> > have maxima-as-a-library in ecl now. Processes can communicate
> > efficiently in various ways. Shared memory? Send data info through a
> > pipe? With a little programming on both the sage and the ecl side, we
> > could make a binary communication protocol. Definitely not as
> > satisfying a solution as a library interface, but perhaps good enough.
>
>
--~--~---------~--~----~------------~-------~--~----~
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel-unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://www.sagemath.org
-~----------~----~----~----~------~----~------~--~---
