On Mon, 24 Aug 2009, Bill Hart wrote:
> On 23 Aug, 21:42, Nils Bruin <nbr...@sfu.ca> wrote:
>> The following problem came up while trying to use ecl as a library
>> inside sage:
>>
>> Both sage and ecl use GMP for their multi-precision arithmetic, and
>> both call mp_set_memory_functions to register their memory managers.
>> This obviously doesn't work, since GMP only keeps track of one set of
>> memory management routines. Ecl needs to use its memory manager, since
>> it relies on Boehm for garbage collection and sage needs to use its
>> own because python relies on reference counting.
>>
>> I have left the discussion specific to ECL, but the problem arises in
>> general: GMP has global state, so using the same GMP from several
>> places can lead to conflicts.
>>
>> Juanjo, the lead developer of ecl has suggested several work-arounds
>> (Thank you, Juanjo. Your expertise is a great help). I hope some
>> people with more knowledge of sage memory management, mpir and build
>> processes can comment on the feasibility and maintainability of the
>> different options:
>>
>> 1. Build ecl with its private gmp library.
>> (a) Can that be done on link level? i.e., take ecl.so and gmp.so
>> and produce ecl-gmp.so from the two, with all references of ecl.so to
>> gmp.so resolved and made private? (i.e., make an ecl.so with gmp
>> "statically linked in").
>> (b) Can that be done compile-time for ecl? i.e., when you compile
>> ecl to produce ecl.so, make the compiler/linker resolve any references
>> to gmp and prevent any of the gmp symbols from leaking out as public
>> in ecl.so?
>> (c) Is there an economical and maintainable way of pushing gmp-for-
>> ecl in its own namespace?
This seems like the cleanest and easiest solution. It is possible to
statically link in some libraries and dynamically link in others, at least
with gcc.
>> 2. Make the mp_set_memory_functions state thread-local and run ecl in
>> its own thread.
>> (a) Does that involve changing the gmp/mpir library?
>
> It would involve changing mpir. In general it is a bad idea to use
> thread local for such things. One problem is that thread local storage
> is not available on all the platforms we support. Secondly it is
> generally a bad idea to use thread local storage to manage memory
> management issues, though admittedly this tends to be about the only
> option remaining after trying everything else. But in general it leads
> to subtle annoying bugs with very low reproducibility (due to bugs
> elsewhere).
Sounds like we're asking for bugs, and trying to use threads to separate
something entirely unrelated.
>>
>> 3. Make sage use Boehm as well :-). That may be an option when
>> reference counting is discovered to be an insufficient memory
>> management strategy for sage in general.
This will happen when either Sage switches away from Python or Python
starts using Boehm...neither of which I see happening any time soon. The
advantage of Python's memory manager (for our use) is that it's optimized
for allocating and deallocating lots of small chunks of memory, which
matches our use-case for GMP integers well.
>> 4. Change ECL so that its uses of GMP are braced by
>> mp_set_memory_function
>> (a) That wouldn't be thread-safe, would it? Or are GMP instructions
>> "atomic" with respect to threads? Could we make them that way?
>
> No it wouldn't be threadsafe.
>
>> (b) In that case, it might be helpful to have a complete list of GMP
>> instructions that could possibly lead to calls to alloc,realloc,free.
>> Only those would need treatment.
>
> There are hundreds.
>
>> (c) Could this be done by properly mangling gmp.h?
>
> You couldn't possibly mangle it any more than it already is.
>
> Anyhow, here is the relevant code from mp_set_fns.c in mpir:
>
> void
> mp_set_memory_functions (void *(*alloc_func) (size_t),
> void *(*realloc_func) (void *, size_t, size_t),
> void (*free_func) (void *, size_t))
> {
> if (alloc_func == 0)
> alloc_func = __gmp_default_allocate;
> if (realloc_func == 0)
> realloc_func = __gmp_default_reallocate;
> if (free_func == 0)
> free_func = __gmp_default_free;
>
> __gmp_allocate_func = alloc_func;
> __gmp_reallocate_func = realloc_func;
> __gmp_free_func = free_func;
> }
>
> Not very extensive huh. I actually don't see a solution to this
> problem that involves mpir. In fact I think this is a security issue
> and I don't think this capability should even exist.
>
>>
>> Thanks to Juanjo for helping this project progress. Other possibly
>> relevant remarks:
>>
>> 5. This problem isn't unique for ecl/sage. Any setting where gmp is
>> used by several libraries at once potentially causes this problem
>>
>> 6. This problem does not arise in general when you try to extend
>> python with ecl, because normal python does not use gmp. So building a
>> python/lisp hybrid is quite feasible.
>>
>> 7. The main drive for using ecl-as-a-library in sage is to get a
>> faster interface with Maxima. If all options above turn out to involve
>> a lot of work or are expensive to maintain, perhaps we need to find
>> another way of speeding up the interface. As a side effect, we already
>> have maxima-as-a-library in ecl now. Processes can communicate
>> efficiently in various ways. Shared memory? Send data info through a
>> pipe? With a little programming on both the sage and the ecl side, we
>> could make a binary communication protocol. Definitely not as
>> satisfying a solution as a library interface, but perhaps good enough.
Essentially, you're suggesting writing a more Sage-friendly front end for
Maxima (as the current front-end was targeted at an actual human at the
console). It would probably be a step forward, but more complicated than
need be.
- Robert
--~--~---------~--~----~------------~-------~--~----~
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel-unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://www.sagemath.org
-~----------~----~----~----~------~----~------~--~---
