Re: [Cython] Should we start using the internal CPython APIs?
da-woods schrieb am 04.11.23 um 14:45: I'm a bit late in replying to this but here are some unordered thoughts. * I'm fairly relaxed about using `Py_BUILD_CORE` if useful - I think we mostly do have good fallback paths for most things so can adapt quickly when stuff changes. I'm not entirely relaxed about it, but I agree that the fallbacks should usually make it easy to keep things working also after larger changes in CPython. * CYTHON_USE_CPYTHON_CORE_DETAILS sounds reasonable, but it's yet another variation to test. True. * I wonder if fixing up the limited API implementation should be higher priority than creating a third level been "full" and "limited API". I think there's potential for all three. Basically modes "aggressively fast", "highly compatible" and "version independent". The latter is what the Stable ABI together with the Limited API should give you. * I recall we were planning to ditch c89 as a strict requirement after 3.0? Incompatibility with C++ might be more of an issue though. Yes. C++ is not an issue for CPython, so their internal header files are not tested with C++ at all. That's the highest potential for breakage, if we accept to generate C99 from Cython 3.1 onwards. We should make sure that we use "-std=c89" in at least one Cython 3.0 test setup, BTW. * Even so, if there's a good way of turning it off then we could say: "if you want strict c89 support then you can't use CYTHON_USE_CPYTHON_CORE_DETAILS" and people would always have options. That could be part of it, yes. * Waiting and seeing may be a good option for now. I agree. This still seems best for now, especially given the amount of recent changes in the C-API. Let's wait for those to settle down, at least. Thanks everyone for your opinions and comments! Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Should we start using the internal CPython APIs?
I'm a bit late in replying to this but here are some unordered thoughts. * I'm fairly relaxed about using `Py_BUILD_CORE` if useful - I think we mostly do have good fallback paths for most things so can adapt quickly when stuff changes. * CYTHON_USE_CPYTHON_CORE_DETAILS sounds reasonable, but it's yet another variation to test. * I wonder if fixing up the limited API implementation should be higher priority than creating a third level been "full" and "limited API". * I recall we were planning to ditch c89 as a strict requirement after 3.0? Incompatibility with C++ might be more of an issue though. * Even so, if there's a good way of turning it off then we could say: "if you want strict c89 support then you can't use CYTHON_USE_CPYTHON_CORE_DETAILS" and people would always have options. * Waiting and seeing may be a good option for now. David On 30/10/2023 20:42, Stefan Behnel wrote: Thank you for your comments so far. Stefan Behnel schrieb am 29.10.23 um 22:06: I seriously start wondering if we shouldn't just define "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro guard that triggers its #define) and include the internal "pycore_*.h" CPython header files from here: https://github.com/python/cpython/tree/main/Include/internal I just remembered that there's a one major technical issue with this. CPython now requires C99 for its own code base (Py3.13 actually uses "-std=c11" on my side). While they care about keeping public header files compatible with C89 and C++, their internal header files may not always have that quality, and won't be tested for it. So, governance is one argument, but technical reasons can also make this appear less appealing overall. I'll let things settle some more and see in what direction Py3.13 will eventually be moving. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Should we start using the internal CPython APIs?
Thank you for your comments so far. Stefan Behnel schrieb am 29.10.23 um 22:06: I seriously start wondering if we shouldn't just define "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro guard that triggers its #define) and include the internal "pycore_*.h" CPython header files from here: https://github.com/python/cpython/tree/main/Include/internal I just remembered that there's a one major technical issue with this. CPython now requires C99 for its own code base (Py3.13 actually uses "-std=c11" on my side). While they care about keeping public header files compatible with C89 and C++, their internal header files may not always have that quality, and won't be tested for it. So, governance is one argument, but technical reasons can also make this appear less appealing overall. I'll let things settle some more and see in what direction Py3.13 will eventually be moving. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Should we start using the internal CPython APIs?
Hi All,
>
>> I seriously start wondering if we shouldn't just define
>> "Py_BUILD_CORE"
>
>
> This sounds just too much, and it's like a war declaration on what they
> are trying to do.
>
I have the same opinion. I think this will negate all effort that the C/API
workgroup is trying to do - to limit leaking internals of cpython. I would
do this if there is no other way to move forward.
I would suggest convincing cpython core devs to move API sharing cpython
internals to unstable API tier [1] and to use it. Reasons:
1. This tier is intended for projects like cython ("Tools requiring access
to CPython internals (e.g. advanced debuggers and JIT compilers)" [2])
hence I think there won't be huge opposition to provide needed low-level
API for Cython.
2. Unstable tier has guaranteed stability for each major release so we will
avoid issues with changes in minor releases which can happen for internal
API
Please keep in mind that I am not involved in python API in any way so this
is just my personal opinion with very limited knowledge.
Matus
[1] https://docs.python.org/3/c-api/stable.html#unstable-c-api
[2] https://peps.python.org/pep-0689/#unstable-c-api-tier
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Should we start using the internal CPython APIs?
On Mon, 30 Oct 2023 at 00:12, Stefan Behnel wrote: > Hi all, > > given the latest blow against exposing implementation details of CPython > in > their C-API (see https://github.com/cython/cython/pull/5767 for the > endless > story), In this new world order of political correctness, they will not say it directly, but the fact that Cython is probably the biggest consumer of these private APIs, this is a way to hard-press Cython to stop doing that once and for all. I remember some exchanges in GitHub where Python core devs manifested quite a bit of dislike/disdain for what Cython is doing. > I seriously start wondering if we shouldn't just define > "Py_BUILD_CORE" This sounds just too much, and it's like a war declaration on what they are trying to do. > (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro > guard that triggers its #define) and include the internal "pycore_*.h" > CPython header files from here: > > https://github.com/python/cpython/tree/main/Include/internal > > and this is totally reasonable, as it is under Cython's control. However, how much would this be different from CYTHON_LIMITED_API, or how the new define would interact with CYTHON_LIMITED_API? > > There's a risk, clearly, that these internals change even during point > releases. Maybe not a big risk, but not impossible either. We'd have to > deal with that and so would our users. > And that would be the biggest blow, actually. If this is truly a possibility, then I will have to turn off the use of CPython internals in my project. I don't want my already released packages to stop building just after a CPython patch release. There is no way I'm willing to deal with this mess, I already have too much maintenance work going on. > OTOH, having a single macro switch would make it easy for users to adapt > if > something breaks on their side, and also easy to benchmark if it makes a > difference for their code. > Definitely > > We could also leave it off by default and simply allow users with high > performance needs to enable it manually. Or start by leaving it off until > a > new CPython X.Y release has stabilised and its (used-by-us) internals have > proven not to change, and then switch it on for that release series. In > any > case, having a single switch for this feels like it could be easy to > handle. > Given that there is no guarantee of stability across point releases, I would argue that the macro should be off by default. Power users and packagers (that is, people that usually know how things work) can easily turn it on with "CFLAGS=-DFOO=1". What you do not want is novices and occasional users with little or no knowledge of Cython or even Python to flood your issue trackers with issues titled "I cannot install your thing" after a pip install failure. What about adding a Cython compiler directive to set the default value of the new macro? That way, it is up to the project's authors/maintainers to decide on how to handle this, they are in a better position to weigh the performance vs. portability trade-off. You still have to decide on the default value if the compiler directive is not set, but in this case I call for Stefan to pick it according to his preference/convenience, he certainly deserves the right to do so. -- Lisandro Dalcin Senior Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Should we start using the internal CPython APIs?
Hi all, given the latest blow against exposing implementation details of CPython in their C-API (see https://github.com/cython/cython/pull/5767 for the endless story), I seriously start wondering if we shouldn't just define "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro guard that triggers its #define) and include the internal "pycore_*.h" CPython header files from here: https://github.com/python/cpython/tree/main/Include/internal This would give us greater freedom in accessing all the implementation details, so that we could directly integrate with those. We'd obviously still need one or more fallback implementations for "stable CPython", Limited API, PyPy and friends. There's a risk, clearly, that these internals change even during point releases. Maybe not a big risk, but not impossible either. We'd have to deal with that and so would our users. OTOH, having a single macro switch would make it easy for users to adapt if something breaks on their side, and also easy to benchmark if it makes a difference for their code. We could also leave it off by default and simply allow users with high performance needs to enable it manually. Or start by leaving it off until a new CPython X.Y release has stabilised and its (used-by-us) internals have proven not to change, and then switch it on for that release series. In any case, having a single switch for this feels like it could be easy to handle. What do you think? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
