[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: Closing in favor of #41324, which adds just the most important functions. -- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> Add a minimal decimal capsule API ___ Python tracker

[issue22194] access to cdecimal / libmpdec API

Change by Paul Ganssle : -- nosy: +p-ganssle ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: libmpdec has always been a separate project, predating the integration into Python-3.3. Before the Python-3.3 release, Jim Jewett suggested a cleaner library/module separation (and he was right, it made the code much more readable). Then distributors wanted

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: Of course Windows is a problem. I do not know how to implement --with-system-libmpdec on Windows. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22194 ___

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: For the record, we have realized that pure C code brought less than a 2x speedup compared to straightforward Python code, so cdecimal support has become a low-priority concern for Numba. For the curious, the C code is from libmpdec's own benchmark (Web site

[issue22194] access to cdecimal / libmpdec API

Martin v. Löwis added the comment: I'd like to focus this issue a bit. Antoine originally proposed that non-Python code might want to access libmpdec. However, given that this is now a separate project (as it seems), I don't think it's Python's task to make the API available. If it is a

[issue22194] access to cdecimal / libmpdec API

Changes by Alexander Belopolsky alexander.belopol...@gmail.com: -- nosy: +belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22194 ___ ___

[issue22194] access to cdecimal / libmpdec API

Larry Hastings added the comment: I think this is a duplicate of #15237. -- nosy: +larry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22194 ___

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: Well, we have two issues now: 1) Make the _decimal API available via capsule. 2) Make the libmpdec symbols public (i.e. remove GCC visibility push(hidden) from Modules/_decimal/libmpdec/mpdecimal.h. The question here is now whether 2) is safe. Note

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: large parts of the cdecimal on PyPI (which uses the incompatible libmpdec-2.3) would need to be rewritten. Ah, so it has an incompatible ABI? That will complicate things a bit :-) -- ___ Python tracker

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: Note that active symbol hiding has always only worked for gcc (but I think on Windows and AIX the symbols are hidden by default anyway). Does it mean a separate Windows and AIX solution should be found? I think if we can't make the mpd symbols available in a

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: Antoine Pitrou rep...@bugs.python.org wrote: large parts of the cdecimal on PyPI (which uses the incompatible libmpdec-2.3) would need to be rewritten. Ah, so it has an incompatible ABI? That will complicate things a bit :-) Yes, cdecimal on PyPI is slower,

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: Le 03/09/2014 18:33, Stefan Krah a écrit : Yes, cdecimal on PyPI is slower, has a different ABI, uses libmpdec-2.3, has subtle differences in the context handling, cannot subclass the context, ... ;) I think a common C API for 2.7 is only possible with

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: Sure, if there are people who write python3-only C modules (I can't think of one right now). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22194 ___

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: If there aren't today, there will be in a few years time, and they'll be glad they can support Python 3.5. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22194

[issue22194] access to cdecimal / libmpdec API

Larry Hastings added the comment: Are there any other modules where the capsule API works in both CPython and PyPy? I thought capsule APIs were decidedly implementation-specific. Not that I'm not for it in theory. But this is some crazy uncharted hyper-compatibility territory we're talking

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: The compatibility discussion was for the cdecimal-2.3 package that's hosted on PyPI and used for Python 2.7. IOW, will people use a capsule API that's Python-3 only? Compatibility with pypy would be esoteric indeed. :) --

[issue22194] access to cdecimal / libmpdec API

Changes by Stefan Krah stefan-use...@bytereef.org: Removed file: http://bugs.python.org/file36446/module_get_symbol.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22194 ___

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: Thanks, Stefan. So everyone agrees that Capsule is the right way for the API. Then this issue is about making the libmpdec symbols public. I've tried to produce a collision with duplicate symbols as outlined in msg176486, but I haven't been successful (on

[issue22194] access to cdecimal / libmpdec API

Stefan Behnel added the comment: Stefan (Behnel), could you comment on the strategy that you had in mind? Is it similar to module_get_symbol.diff or entirely different? I agree with Antoine that a Capsule would work better here (and yes, the performance problem of capsules is only with cases

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: Antoine Pitrou rep...@bugs.python.org wrote: Who are those people? #16745 was opened by you :-) MvL, in #4555 (msg176486). Platform specific maybe, but no hack: I was thinking about storing the DSO handle in the PyModuleObject struct and add functions to

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: Le 24/08/2014 04:56, Stefan Krah a écrit : I'm not sure about OS X, but I would be surprised if it did not work. To my limited knowledge, Capsules are slow, see also here (the penultimate paragraph): They are slow if you have to lookup and unwrap a capsule

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: Ah yes, the array of function pointers is directly accessible. I did not look close enough -- reading the word spam 100x in the docs always makes me skim the text. ;) -- ___ Python tracker rep...@bugs.python.org

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: MvL, in #4555 (msg176486). Ok, I'm cc'ing Martin then :-) Note RTLD_LOCAL seems to be the default with dlopen(). Now I don't know how that behaves when you have a chained library loading, e.g.: Apache -- dlopen(Python dll) -- dlopen(_decimal dll) _decimal

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: Antoine Pitrou rep...@bugs.python.org wrote: (which means that, perhaps, the answer is to make the mpd_ prefix configurable with a #define?) I don't know 100% what you have in mind, but Debian and Arch already ship --with-system-libmpdec, so only the mpd_*

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: Le 24/08/2014 11:11, Stefan Krah a écrit : Antoine Pitrou rep...@bugs.python.org wrote: (which means that, perhaps, the answer is to make the mpd_ prefix configurable with a #define?) I don't know 100% what you have in mind, but Debian and Arch already

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: That's what I meant. The issue here is that Python's libmpdec is not exposed to third-party code at all. Also there should probably be a (thin?) API to get at the underlying mpdec object from a cdecimal PyObject (apologies for the poor wording, I'm actually

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: Le 23/08/2014 16:42, Stefan Krah a écrit : That's what I meant. The issue here is that Python's libmpdec is not exposed to third-party code at all. Also there should probably be a (thin?) API to get at the underlying mpdec object from a cdecimal PyObject

[issue22194] access to cdecimal / libmpdec API

Stefan Behnel added the comment: (for the record, the context is that we would like to support decimal objects efficiently in Numba) Same for Cython, although I guess we wouldn't do more than shipping the necessary declarations and (likely) also enable auto-coercion between the libmpdec

[issue22194] access to cdecimal / libmpdec API

Changes by Antoine Pitrou pit...@free.fr: -- nosy: +lemburg ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22194 ___ ___ Python-bugs-list mailing

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: 3) It's not clear whether users would not be served better by using functions from libmpdec directly (much faster, probably less error handling). That's what I meant. The issue here is that Python's libmpdec is not exposed to third-party code at

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: Note that you could expose a C API even if decimal didn't become built-in, see Include/datetime.h for example. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22194

[issue22194] access to cdecimal / libmpdec API

Antoine Pitrou added the comment: Relatedly, is the libmpdec ABI stable? That is, if I build a separate libmpdec, can I expect it to handle cdecimal's innards fine, or will there be problems? -- ___ Python tracker rep...@bugs.python.org

[issue22194] access to cdecimal / libmpdec API

Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file36378/api-demo[1].c ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22194 ___

[issue22194] access to cdecimal / libmpdec API

Stefan Krah added the comment: I'm a little unsure what to do with the API, see also #15237: 1) I'm not too fond of the Capsule method, especially because it *seems* possible to get at the symbols directly on Linux and Windows (provided that they aren't static of course). 2) I

[issue22194] access to cdecimal / libmpdec API

New submission from Antoine Pitrou: Currently cdecimal exports no C API that I know of, and it makes sure the libmpdec symbols are kept private in the .so file. It would be nice for external C code (or, in general, non-Python code) to be able to access cdecimal objects, and make operations on