On 23 Aug, 21:42, Nils Bruin <nbr...@sfu.ca> wrote:
> The following problem came up while trying to use ecl as a library
> inside sage:
>
> Both sage and ecl use GMP for their multi-precision arithmetic, and
> both call mp_set_memory_functions to register their memory managers.
> This obviously doesn't work, since GMP only keeps track of one set of
> memory management routines. Ecl needs to use its memory manager, since
> it relies on Boehm for garbage collection and sage needs to use its
> own because python relies on reference counting.
>
> I have left the discussion specific to ECL, but the problem arises in
> general: GMP has global state, so using the same GMP from several
> places can lead to conflicts.
>
> Juanjo, the lead developer of ecl has suggested several work-arounds
> (Thank you, Juanjo. Your expertise is a great help). I hope some
> people with more knowledge of sage memory management, mpir and build
> processes can comment on the feasibility and maintainability of the
> different options:
>
> 1. Build ecl with its private gmp library.
> (a) Can that be done on link level? i.e., take ecl.so and gmp.so
> and produce ecl-gmp.so from the two, with all references of ecl.so to
> gmp.so resolved and made private? (i.e., make an ecl.so with gmp
> "statically linked in").
> (b) Can that be done compile-time for ecl? i.e., when you compile
> ecl to produce ecl.so, make the compiler/linker resolve any references
> to gmp and prevent any of the gmp symbols from leaking out as public
> in ecl.so?
> (c) Is there an economical and maintainable way of pushing gmp-for-
> ecl in its own namespace?
>
> 2. Make the mp_set_memory_functions state thread-local and run ecl in
> its own thread.
> (a) Does that involve changing the gmp/mpir library?
It would involve changing mpir. In general it is a bad idea to use
thread local for such things. One problem is that thread local storage
is not available on all the platforms we support. Secondly it is
generally a bad idea to use thread local storage to manage memory
management issues, though admittedly this tends to be about the only
option remaining after trying everything else. But in general it leads
to subtle annoying bugs with very low reproducibility (due to bugs
elsewhere).
>
> 3. Make sage use Boehm as well :-). That may be an option when
> reference counting is discovered to be an insufficient memory
> management strategy for sage in general.
>
> 4. Change ECL so that its uses of GMP are braced by
> mp_set_memory_function
> (a) That wouldn't be thread-safe, would it? Or are GMP instructions
> "atomic" with respect to threads? Could we make them that way?
No it wouldn't be threadsafe.
> (b) In that case, it might be helpful to have a complete list of GMP
> instructions that could possibly lead to calls to alloc,realloc,free.
> Only those would need treatment.
There are hundreds.
> (c) Could this be done by properly mangling gmp.h?
You couldn't possibly mangle it any more than it already is.
Anyhow, here is the relevant code from mp_set_fns.c in mpir:
void
mp_set_memory_functions (void *(*alloc_func) (size_t),
void *(*realloc_func) (void *, size_t, size_t),
void (*free_func) (void *, size_t))
{
if (alloc_func == 0)
alloc_func = __gmp_default_allocate;
if (realloc_func == 0)
realloc_func = __gmp_default_reallocate;
if (free_func == 0)
free_func = __gmp_default_free;
__gmp_allocate_func = alloc_func;
__gmp_reallocate_func = realloc_func;
__gmp_free_func = free_func;
}
Not very extensive huh. I actually don't see a solution to this
problem that involves mpir. In fact I think this is a security issue
and I don't think this capability should even exist.
>
> Thanks to Juanjo for helping this project progress. Other possibly
> relevant remarks:
>
> 5. This problem isn't unique for ecl/sage. Any setting where gmp is
> used by several libraries at once potentially causes this problem
>
> 6. This problem does not arise in general when you try to extend
> python with ecl, because normal python does not use gmp. So building a
> python/lisp hybrid is quite feasible.
>
> 7. The main drive for using ecl-as-a-library in sage is to get a
> faster interface with Maxima. If all options above turn out to involve
> a lot of work or are expensive to maintain, perhaps we need to find
> another way of speeding up the interface. As a side effect, we already
> have maxima-as-a-library in ecl now. Processes can communicate
> efficiently in various ways. Shared memory? Send data info through a
> pipe? With a little programming on both the sage and the ecl side, we
> could make a binary communication protocol. Definitely not as
> satisfying a solution as a library interface, but perhaps good enough.
--~--~---------~--~----~------------~-------~--~----~
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel-unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://www.sagemath.org
-~----------~----~----~----~------~----~------~--~---
