Re: [Cython] Ready for Cython 3.1 ?
Stefan Behnel schrieb am 05.11.23 um 23:06: I'd like to ease our feature development by using more modern Python features in our code base and by targeting less Python versions in Cython 3.1 compared to the "all things supported" Cython 3.0. I created a 3.0.x maintenance branch and removed the Py<3.7 test jobs from the master branch. That should make the CI response visibly faster. Happy code cleaning :) Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Ready for Cython 3.1 ?
Lisandro Dalcin schrieb am 06.11.23 um 09:05: On Mon, 6 Nov 2023 at 01:19, Stefan Behnel wrote: it looks like Cython 3.0.6 is going to be a "most things fixed" kind of release for the 3.0.x release series. I'm having issues using CYTHON_LIMITED_API with some Python versions (<=3.9). If you are not in a rush to release 3.0.6, I would like to have some time to properly investigate what's going on. I'd rather postpone these things to 3.1. They are not critical for 3.0, and as I wrote, I think it's actually helpful for users to target 3.1 rather than 3.0. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Ready for Cython 3.1 ?
da-woods schrieb am 06.11.23 um 08:48: > I also consider Cython 3.1 a prime target for better Limited API support. Yes - but I wouldn't treat complete support as a blocker (I don't think this is what you meant though). It's experimental in 3.0 and I don't expect it to "fully" work in 3.1. There's a separate question about what we consider the minimum viable Limited API version we want to support. I imagine that'll ultimately be decided by "what we can make work", but I don't think it'll be less that 3.4 (when PyType_GetSlot) was added. It's probably something to decide later. That's another thing that moving the support to 3.1 would solve. If we can target Py3.7/3.8+ instead of older versions, then also the Limited API will be more usable. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Should we start using the internal CPython APIs?
da-woods schrieb am 04.11.23 um 14:45: I'm a bit late in replying to this but here are some unordered thoughts. * I'm fairly relaxed about using `Py_BUILD_CORE` if useful - I think we mostly do have good fallback paths for most things so can adapt quickly when stuff changes. I'm not entirely relaxed about it, but I agree that the fallbacks should usually make it easy to keep things working also after larger changes in CPython. * CYTHON_USE_CPYTHON_CORE_DETAILS sounds reasonable, but it's yet another variation to test. True. * I wonder if fixing up the limited API implementation should be higher priority than creating a third level been "full" and "limited API". I think there's potential for all three. Basically modes "aggressively fast", "highly compatible" and "version independent". The latter is what the Stable ABI together with the Limited API should give you. * I recall we were planning to ditch c89 as a strict requirement after 3.0? Incompatibility with C++ might be more of an issue though. Yes. C++ is not an issue for CPython, so their internal header files are not tested with C++ at all. That's the highest potential for breakage, if we accept to generate C99 from Cython 3.1 onwards. We should make sure that we use "-std=c89" in at least one Cython 3.0 test setup, BTW. * Even so, if there's a good way of turning it off then we could say: "if you want strict c89 support then you can't use CYTHON_USE_CPYTHON_CORE_DETAILS" and people would always have options. That could be part of it, yes. * Waiting and seeing may be a good option for now. I agree. This still seems best for now, especially given the amount of recent changes in the C-API. Let's wait for those to settle down, at least. Thanks everyone for your opinions and comments! Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Ready for Cython 3.1 ?
Hi all, it looks like Cython 3.0.6 is going to be a "most things fixed" kind of release for the 3.0.x release series. Given the work that lies ahead of us for Cython 3.1, I think we're at a point to get started on that, making the future 3.0.x releases stable and "boring". As a reminder, Cython 3.1 will remove support for Python 2.7 and Python 3.[567], i.e. all Python versions that are now EOL. Python 3.8 will continue to receive security fixes for another year. Python 3.7 is EOL but still up for debate since it's probably not hard to support and still maintained in some Linux distributions for another couple of years. But I'm fine with considering it legacy. We'll probably notice if it gets in the way while preparing Cython 3.0, and can leave support in until there's a reason to remove it. https://github.com/cython/cython/issues/2800 I'd like to ease our feature development by using more modern Python features in our code base and by targeting less Python versions in Cython 3.1 compared to the "all things supported" Cython 3.0. I also consider Cython 3.1 a prime target for better Limited API support. Users probably won't care both for that and for outdated Python versions at the same time. Or, they can use Cython 3.0.x for continued legacy support. Since Cython 3.1 is mostly about ripping out old code, we can try to keep the development cycle short, so that new features don't have to wait that long. Certainly not as long as for Cython 3.0… Is everyone and everything ready to start working on Cython 3.1? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Should we start using the internal CPython APIs?
Thank you for your comments so far. Stefan Behnel schrieb am 29.10.23 um 22:06: I seriously start wondering if we shouldn't just define "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro guard that triggers its #define) and include the internal "pycore_*.h" CPython header files from here: https://github.com/python/cpython/tree/main/Include/internal I just remembered that there's a one major technical issue with this. CPython now requires C99 for its own code base (Py3.13 actually uses "-std=c11" on my side). While they care about keeping public header files compatible with C89 and C++, their internal header files may not always have that quality, and won't be tested for it. So, governance is one argument, but technical reasons can also make this appear less appealing overall. I'll let things settle some more and see in what direction Py3.13 will eventually be moving. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Should we start using the internal CPython APIs?
Hi all, given the latest blow against exposing implementation details of CPython in their C-API (see https://github.com/cython/cython/pull/5767 for the endless story), I seriously start wondering if we shouldn't just define "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro guard that triggers its #define) and include the internal "pycore_*.h" CPython header files from here: https://github.com/python/cpython/tree/main/Include/internal This would give us greater freedom in accessing all the implementation details, so that we could directly integrate with those. We'd obviously still need one or more fallback implementations for "stable CPython", Limited API, PyPy and friends. There's a risk, clearly, that these internals change even during point releases. Maybe not a big risk, but not impossible either. We'd have to deal with that and so would our users. OTOH, having a single macro switch would make it easy for users to adapt if something breaks on their side, and also easy to benchmark if it makes a difference for their code. We could also leave it off by default and simply allow users with high performance needs to enable it manually. Or start by leaving it off until a new CPython X.Y release has stabilised and its (used-by-us) internals have proven not to change, and then switch it on for that release series. In any case, having a single switch for this feels like it could be easy to handle. What do you think? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Can we remove the FastGIL implementation?
da-woods schrieb am 19.09.23 um 21:38: I think the detail that was missing is you need to add the `#cython: fast_gil = True` to enable it. [...] So my conclusion is that from 3.11 onwards Python sped up their own GIL handling to about the same as we used to have, and fastgil has turned into a pessimization. I tried the benchmark with the master branch on my side again, this time with correct configuration. :) Turns out that enabling the FastGIL feature makes it much slower for me (on Ubuntu Linux 20.04) in both Py3.8 and 3.10: """ * Python 3.10 (-DCYTHON_FAST_GIL=0) Running the test (already held)... took 1.2482502460479736 Running the test (released)... took 6.444956541061401 Running the test (already held)... took 1.2358744144439697 Running the test (released)... took 6.4064109325408936 * Python 3.10 (-DCYTHON_FAST_GIL=1) Running the test (already held)... took 2.243091583251953 Running the test (released)... took 7.32707667350769 Running the test (already held)... took 2.4065449237823486 Running the test (released)... took 7.50264573097229 """ I also tried it with PGO enabled and got more or less the same result. The Python installations that I tried it with were both PGO builds. It's probably mixed across platforms, different configurations and C compilers. I looked through the "What's new" document for Py3.10 and 3.11 but couldn't find mentions of GIL improvements. Just that some other things have become faster. So – disable the feature in Python 3.11 and later? (Currently it's disabled in 3.12+.) Py3.11+ would suggest that we keep the code in Cython 3.1, since that will support older Python versions that still seem to benefit from it. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Can we remove the FastGIL implementation?
Hi, I've seen reports that Cython's "FastGIL" implementation (which basically keeps the GIL state in a thread-local variable) is no longer faster than CPython's plain GIL implementation in recent Python 3.x versions. Potentially even slower. See the report in https://github.com/cython/cython/issues/5703 It would be helpful to get user feedback on this. If you have GIL-heavy Cython code, especially with nested with-nogil/with-gil sections across functions, and a benchmark that exercises it, could you please run the benchmark with and without the feature enabled and report the results? You can add "-DCYTHON_FAST_GIL=0" to your CFLAGS to disabled it (and "=1" to enable it explicitly). It's enabled by default in CPython 3.6-3.11 (but disabled in Cython 0.29.x on Python 3.11). Thanks, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0.2 released
Hi all, Cython 3.0.2 is released. It fixes two major regressions in 3.0.1, so please upgrade if that failed for you. https://cython.readthedocs.io/en/latest/src/changes.html Have fun, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0 final released
Hi all, after close to five long years, I'm proud to announce the release of Cython 3.0. It's done. It's out. Finally! The full list of improvements compared to the 0.29.x release series is entirely incredible. https://cython.readthedocs.io/en/latest/src/changes.html Cython 3.0 is better than any other Cython release before, in all aspects. It's much more Python, integrates better with C and C++, supports more Python implementations and configurations, provides many great new language features – it's faster, safer and easier to use. It's simply better. New language features include: - Python 3 syntax and semantics by default - Cython type annotations in plain Python code - automatic NumPy ufunc generation - fast @dataclass and @total_ordering extension types - safe exception propagation in C functions by default - Unicode identifiers in Cython code All of this wouldn't have been possible without the help of the many, many people who contributed code and documentation, tested features, found and described bugs, helped debugging problems. Those who started using Cython in new environments, new build systems, new use cases, and helped to get it working there. Who proposed new features or found mismatches and gaps in the existing set of features. Thank you all, you helped making Cython 3.0 an awesome language! Along the way, we added two people to the list of Cython developers. * David Woods has contributed a tremendous list of features and fixes to this release. It would honestly not have been possible without his efforts. * Matúš Valo has put a lot of work into the documentation and the pure Python mode. He found many issues that make Cython now easier and more consistent to use from Python code. Thank you both for your contributions. I'm happy to work together with you. Everyone, have fun using Cython 3.0, and whatever good comes after it. Best, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0 RC 2 released
Hi all, after close to five long years, we're almost there – I've pushed a release candidate for Cython 3.0 with a long list of bug fixes (followed by a second one with one important fix). https://cython.readthedocs.io/en/latest/src/changes.html Please give it some final testing. Unless we find something really serious in the RC2 release, the changes for the final release will be very limited and safe, hopefully none at all. The RC is just in time for this week's US-SciPy, and I'll make sure we have a final release for next week's EuroPython in Praha. Have fun, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Current CI crashes in Py3.12
Hi, just a note that the current CI crashes in Py3.12b1 are due to https://github.com/python/cpython/issues/104614 They fixed it and Py3.12b2 will hopefully support multiple inheritance of extension types again. It's expected next week (June 6th). Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] cython 3 migration update and next releases
Dima Pasechnik schrieb am 21.05.23 um 11:38: On Sun, 21 May 2023, 10:21 Stefane Fermigier, wrote: IFAIK, 15k lines of Cython makes it among one of the largest Cython projects I'm aware of (I did some research a couple of years ago): https://github.com/sfermigier/awesome-cython#some-projects-with-more-that-10-000-lines-of-cython-code > SageMath has 700K Cython lines, yet not mentioned. Certainly worth mentioning, yes. Looking at the numbers, I also noticed that lxml is listed in the 5-10k lines range. It actually has about 18k lines of Cython code (.pyx/.pxi files) and another 1.5k lines in compiled Python (.py) files, according to pygount [1]. I tried sloccount first, but that doesn't seem to have Cython support. Might be worth redoing that count for the other projects as well. Stefan [1] https://pypi.org/project/pygount/ ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] cython 3 migration update and next releases
matus valo schrieb am 16.05.23 um 21:09: I would like to inform you about recent porting of projects to Cython 3. Recently, I participated in migration of 3 bigger projects to Cython 3: Thanks a lot for doing this, Matúš. It helps Cython as much as it helps these projects. When migrating to Cython 3, I was able to find out several issues in the Cython, all of them are merged in master now. Hence, I would like to ask about next steps. It would help greatly to release Cython 3 beta3. This will allow me to pin scipy CI to real pre-release instead of master branch. I'll try to get beta 3 released soon, but need to find a bit of consecutive time to get it out. There are still a couple of PRs that I'd like to look through. Moreover, I would like to ask whether we can do the final Cython 3 release after beta 3. The rationale is that the projects won't start really using Cython 3 until we do the final release. Now, we have 3 big users of Cython migrated, hence I think we have some confidence that Cython 3 is ready. What do you think? It's probably a good time to have a final call for merges. Promoting and voting for PRs is welcome. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0 beta 2 is released
Hi everyone, we received a lot of feedback for our first beta release (thanks you, everyone!) and were able to (hopefully) resolve all blockers that prevented some of you from making good use of it. Let's hear what you think about the second beta. It's up on PyPI. https://cython.readthedocs.io/en/latest/src/changes.html#beta-2-2023-03-26 Have fun, Stefan Stefan Behnel schrieb am 26.02.23 um 11:31: Hi all, Cython 3.0 has left the alpha status – the first beta release is available from PyPI. The changes in this release are huge – and the full list of improvements compared to the 0.29.x release series is entirely incredible. Cython 3.0 is better than any other Cython release before, in all aspects. It's much more Python, integrates better with C++, supports more Python implementations and configurations, provides many great new language features – it's faster, safer and easier to use. It's simply better. https://cython.readthedocs.io/en/latest/src/changes.html#beta-1-2023-02-25 The development of the Cython 3.0 release series started all the way back in 2018, with the first branch commit happening on October 27, 2018. https://github.com/cython/cython/commit/c2de8efb67f80bff59975641aac387d652324e4e List of Milestones along the way, and a long list of contributors: https://github.com/cython/cython/issues/4022#issuecomment-1404305257 Thank you to everyone who contributed. Especially to David Woods, who contributed a tremendous amount of changes, both fixes and new features. Thank you, David! A couple of people have also joined in an effort to make the documentation reflect what this great new Cython has to offer. Thank you all, our users will love you for your help. https://github.com/cython/cython/issues/4187 https://cython.readthedocs.io/en/latest/ Now, go and give it a try. We've taken great care to make the transition from Cython 0.29.x as smooth as possible, which was not easy given the large amount of changes, including some well-motivated breaking changes. We wanted to let all users benefit from this new release. Let us know how it works for you, and tell others about it. :) Have fun, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0 beta 1 is released
Hi all, Cython 3.0 has left the alpha status – the first beta release is available from PyPI. The changes in this release are huge – and the full list of improvements compared to the 0.29.x release series is entirely incredible. Cython 3.0 is better than any other Cython release before, in all aspects. It's much more Python, integrates better with C++, supports more Python implementations and configurations, provides many great new language features – it's faster, safer and easier to use. It's simply better. https://cython.readthedocs.io/en/latest/src/changes.html#beta-1-2023-02-25 The development of the Cython 3.0 release series started all the way back in 2018, with the first branch commit happening on October 27, 2018. https://github.com/cython/cython/commit/c2de8efb67f80bff59975641aac387d652324e4e List of Milestones along the way, and a long list of contributors: https://github.com/cython/cython/issues/4022#issuecomment-1404305257 Thank you to everyone who contributed. Especially to David Woods, who contributed a tremendous amount of changes, both fixes and new features. Thank you, David! A couple of people have also joined in an effort to make the documentation reflect what this great new Cython has to offer. Thank you all, our users will love you for your help. https://github.com/cython/cython/issues/4187 https://cython.readthedocs.io/en/latest/ Now, go and give it a try. We've taken great care to make the transition from Cython 0.29.x as smooth as possible, which was not easy given the large amount of changes, including some well-motivated breaking changes. We wanted to let all users benefit from this new release. Let us know how it works for you, and tell others about it. :) Have fun, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0 planning - Re: [cython-users] Re: Anything missing for 0.29.33 ?
Hi Matúš, Matúš Valo schrieb am 06.12.22 um 15:58: I have a thought about Cython 3.0. Based on discussion in [1] we should be done with all breaking changes. There is also [2] but PR is already there [3] (I am not sure what is the state of the PR though). Is it possible to make final release (In case we postpone [2] to Cython 3.1 or 3.0.X)? Or, at least, can we move closer to final release and release beta or RC version? I think it would be great to communicate to the community how far we are from final release (not in time but e.g. this is beta/RC release and will be followed by final release). Additional reason for release of Cython 3.0 is that in near future (Python 3.12) two important components used by Cython will be removed: imp module and distutils. In my opinion, Cython 3.0 should be released early to ensure users transition period so we can avoid back-porting this changes to 0.29.X releases. Any thoughts? [1] https://github.com/cython/cython/issues/4022 [2] https://github.com/cython/cython/issues/4936 [3] https://github.com/cython/cython/pull/5016 Let's see that we get [3] merged to close [2], I think then we're ready for a new release, once 0.29.33 is out. As you wrote, we're through with the breaking changes for 3.0 then, so yes, a first beta release might be appropriate. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Welcome David Woods as a Cython core developer
Hi everyone, with the release of the first 3.0 alpha that supports Python 3.11 (aptly named "alpha 11"), I'm happy to announce that David Woods has been promoted to a Cython core developer. David has shown an extraordinary commitment and dedication over the last years. His first merged commits were already back in 2015, mostly related to the C++ support. But within the last two years, he voluntarily took over more and more responsibility for bugs and issues and developed several major new features for the project. This includes the Walrus operator (PEP 572), cdef dataclasses (modelled after PEP 557), internal "std::move()" usage in C++ mode or support for Unicode identifiers and module names, all of which form a major part of the 3.0 feature set. David has more than deserved a place in the circle of present and prior core devs. David, thank you for your impressive work on Cython, and welcome to the core team! Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Nested prange loops - (was: [cython-users] Converting to Python objects with nogil (inside prange for loop))
Hi, nested prange loops seem to be a common gotcha for users. I can't say if there is ever a reason to do this, but at least I can't think of any. For me, this sounds like we should turn it into a compile time error – unless someone can think of a use case? Even in that case, I'd still emit a warning since it seems so unlikely to be intended. Please reply to the cython-users list to facilitate user feedback. Stefan Forwarded Message Subject: Re: [cython-users] Converting to Python objects with nogil (inside prange for loop) Date: Fri, 15 Jul 2022 07:43:26 +0100 with nogil, parallel(): for i in prange(N): for j in prange(km.BatchSize): You usually only want one loop in a set of nested loops to be prange. Typically the outer loop, but in this case it might be easier to parallelize the inner loop. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] The Cython compiler is 20 years old today !
Dear Cython community, it's now 20 years since Greg Ewing posted his first announcement of Pyrex, the tool that is now known and used under the name Cython. https://mail.python.org/pipermail/python-list/2002-April/126661.html It was a long way, and I've written up some of it in a blog post: http://blog.behnel.de/posts/cython-is-20/ Today, if you're working on any kind of larger application in Python, you're likely to have some piece of code downloaded into your venv that was built with Cython. Or many of them. I'm proud of what we have achieved. And I'm happy to see and talk to the many, many users out there whom we could help to help their users get their work done. Happy anniversary, Cython! Stefan PS: The list of Cython implemented packages on PyPI is certainly incomplete, so please add the classifier to yours if it's missing. With almost 3000 dependent packages on Github (and almost 100,000 related repos), I'm sure we can crack the number of 1000 Cython built packages on PyPI as a birthday present. (No Spam, please, just honest classifiers.) https://pypi.org/search/?q==-created=Programming+Language+%3A%3A+Cython https://github.com/cython/cython/network/dependents?dependent_type=PACKAGE ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] question on submitting a possibly massive bug report
website.reader via cython-devel schrieb am 25.01.22 um 01:09: I am not familiar with Cython, but have spent a few weeks looking at compiler warnings posted when the mathematical package called "sage v9.4" is compiled, which takes several hours to build, since hundreds of code units are invovled in this massive build project. I logged 341 errors during the cythonizing part of the compile run, and found 110 code units (C packages) which I was able to fix so that the recompile would have no warnings. The warnings were legitimate. There are 4 categories of these warnings. 1. Using an unitialized variable with an unknown value 2. Comparing signed and unsigned variables 3. Discarding a const specifier to a variable upon use elsewhere in the code 4. Coercing a pointer to a variable of the wrong type (or vice versa) I did speak to one knowledgable person about this, but my question is this a) do I submit 341 bug reports covering all the warnings? b) since 110 code units were affected do I file 110 bug reports for each code unit? b) do I submit just one bug report for each of the 4 categories above, thus just 4 bug reports? c) do I just list all the warning messages obtained from the massive build run so everyone can get some idea of the problems being faced? I did look at the C code and the pyx code generating it and definitely cython is the origination here of these issues. Since I am NOT yet familiar with cython from scratch, at the moment I am at a loss to write litte tiny programs illustrating the problem. Cython is a code generator, so there probably are only a few places where a larger bunch of issues originate from. You already grouped them by type (1-4), and those likely belong to one cause (or a few related causes). Just open one issue for each of the four. Then please list a few source code examples in each, together with the C code that Cython generated for them, and the warning that the C compiler gave you. If we later find that not all warnings can be resolved this way, we'll see what we can do about the rest. Please make sure to provide the Cython version that you are using. The latest release is 3.0.0a10 (and the main development goes there), although there is a legacy stable version series 0.29.x that most projects are still using and where we will continue to fix bugs for another while. But new reports should best target 3.0 in order to avoid chasing zombies. Thanks, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Preparing the next 3.0 alpha
Hi, I just released 0.29.25 and would like to let the next 3.0 alpha follow it soon. Please ping me on any PRs that you think are ready enough to include and short enough to merge quickly. Thanks, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Annotations and Cython 3
Am 26. Oktober 2021 07:54:10 MESZ schrieb Lisandro Dalcin: >In any case, if you know how and can implement the __future__ import, >that >would be great. Otherwise, valid Python3 code using that import >currently fails to compile in Cython. Probably just an oversight. A no-op future import is trivial to add and ignore. >In addition, perhaps you should consider adding a compiler directive There is one (annotation_typing) that controls whether Cython reads annotations as types or not (which isn't what you're asking for). Apart from that, since there is no future unimport, there should probably be a way to get either behaviour, regardless of the default. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Remove Py3.4 support in Cython 3.0?
This should also have gone to cython-users (not just cython-devel), as it has the larger audience. Stefan Behnel wrote: Hi, I looked up where Python 3.4 is still being used. Ubuntu 14.04 – EOS 2019, EOL April 2022 Debian 8 – EOL Centos 6 – EOL Later distribution versions have newer CPythons: Ubuntu 16.04: Py3.5 (EOS April 2021, EOL 2024) Debian 9: Py3.5 (EOL June 2022) Centos 7: Py3.6 (EOL June 2024) I think we can safely drop Py3.4 support already for Cython 3.0. The advantage isn't big, since we still support Py2.7, but we can at least stop testing it, and start relying on the Py3.5 C-API for Py3. Given that Py3.5 is also going out of support in major Linux distributions within a year's time, I'd suggest we drop support for it in Cython 3.1, together with Py2 support, making the minimum version Py3.6. That would mean that we can start using f-strings, dict ordering, and what not. Any objections? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0 alpha 7 released
Hi all, I just released the next alpha version of Cython 3.0. It's been a while since the last release, almost 10 months, it seems, so there is a fairly long list of changes. And I haven't even merged all PRs yet that are still waiting to make it into the final release. https://github.com/cython/cython/blob/c6571f327/CHANGES.rst As always, feedback and code contributions are welcome. Even though this is an alpha release, please start using it instead of 0.29 if you can – it is so much better! Once it has generated the C/C++ code from your sources, you can test them thoroughly and then distribute them or do with them whatever you like. You won't even have to tell your users that you had an alpha version generate them for you. :) Enjoy! Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Anything to include in the next 3.0 alpha release?
Hi Lisandro! :) Lisandro Dalcin schrieb am 24.03.21 um 21:27: > On Wed, 24 Mar 2021 at 23:08, Stefan Behnel wrote: >> I'll prepare a new 3.0 alpha release during the next days. If you have or >> know anything that is ready for inclusion that I should take a look at, >> please send me a quick note. >> > > Years, ago, I contributed some bits of what today is the autodoc transform > (#cython: embedsignature=True). > I have a reimplementation of this transform for use within mpi4py > https://github.com/mpi4py/mpi4py/blob/master/conf/cyautodoc.py > The main feature of this reimplementation is to render types using > annotation typing. > The purpose is twofold: the signatures in docstrings look much better, and > they are also good enough to generate typing stuff by simply copying over > the signature Cython generates. > So, I have this ad hoc script that extracts signatures from the first > docstring line and generates *.pyi stubs for mpi4py: > https://github.com/mpi4py/mpi4py/blob/master/conf/mpistubgen.py > The ah hoc stubgen script generates the following MPI.pyi stub file: > https://github.com/mpi4py/mpi4py/blob/master/src/mpi4py/MPI.pyi > I insist: all the signatures you see in that MPI.pyi stub file is a > verbatim copy of the first line of docstrigns Cython generates with the > reimplemented transform. > > Do you think it would be worth pushing this reimplementation to Cython > to have it available in the upcoming 3.0? > The main and perhaps only drawback is that the output is not backward > compatible with the previous output. > But IMHO the new output looks way better and it is totally in line with the > new trend of using annotations and typing. > > PS: I should have raised this proposal earlier, but I've been quite busy, > sorry about that. Probably not something for the next alpha, but yes, that would be nice. And 3.0 is the right time to make this change. Requires a proper PR, though, and a bunch of tests that make sure that what we generate here is reasonable and correct, especially in the light of Cython's pure Python type annotations. Also thinking of C++ types here… Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Anything to include in the next 3.0 alpha release?
Hi, I'll prepare a new 3.0 alpha release during the next days. If you have or know anything that is ready for inclusion that I should take a look at, please send me a quick note. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Infer types and array.pxd
Am 2. August 2020 17:02:39 MESZ schrieb da-woods: > I'm not convinced that compiler directives like "infer_types" should > propagate into the cimported files at all. > > It seems reasonable to me that these should be controlled by the author > of the pxd file, not by the eventual cimporter. I agree. It might not be as strikt as that, there could still be directives that should be inherited, but definitely not all of them. Can't look right now, but there might already be a ticket about this. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Infer types and array.pxd
Oliver Haas schrieb am 02.08.20 um 13:44: > The array.pxd ( > https://github.com/cython/cython/blob/master/Cython/Includes/cpython/array.pxd) > requires infer_types = True when cimported. Is there a reason not to > explicitly type the variable op as array in the lines 134 and 141? The return type of the function that creates the object is known, so the normal (safe) type inference should work here without setting "infer_types=True". Do you want to set "infer_types=False", or why is this failing for you? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fedora failures with Cython 3.0a5
Hi Marcel, thanks for testing this, and for triaging the issues! Marcel Plch schrieb am 27.05.20 um 10:27: > In Fedora, we rebuilt 152 packages against Cython 3.0a5 we have found > some breakages that might be relevant to you. We are not entirely sure > whether these issues are caused by the individual projects' wrong usage > of Cython, or if Cython broke them unintentionally. Do any of the > failures look like Cython bugs to you? We can provide moredetail if one > looks interesting. Some expected problems, like python2 > syntax/semantics, were taken into account and noted below as issues of > the individual projects. > > The linked logs will be garbage-collected in about 2 weeks; let us know > if this happens and we can rebuild the packages. > > > # Missing .pxd files > Four packages were broken by being unable to locate .pxd files to compile. > Packages broken by this: > - mlpack > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01409078-mlpack/builder-live.log.gz > - MDAnalysis > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01409419-python-MDAnalysis/builder-live.log.gz > - pplpy > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01411510-python-pplpy/builder-live.log.gz > - bintrees > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01409724-python-bintrees/builder-live.log.gz > > Can this be caused by different import semantics of Python 3? Yes, that's most likely the reason. Could just be a "from . cimport …" missing. You can also build them with the directive "py2_import=True" to get back Py2 import semantics (i.e. trying a relative import first). > # GIL issues > In three packages, problems with the GIL occurred. Some operations > requiring the gil were executed when gil wasn't being held and vice > versa. > Packages broken by this: > - tables > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01408091-python-tables/builder-live.log.gz The GIL errors here seem due to the missing .pxd imports. There's a failing star-import somewhere which probably messes things up a lot. > - scikit-image > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01412046-python-scikit-image/builder-live.log.gz See Gregory Lee's email. > - ssh2-python > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01412227-python-ssh2-python/builder-live.log.gz Could also be due to failing cimports. > # AttributeError in Cython > One package might have revealed a bug inside of Cython. It triggers a > scope inside of ExprNodes.py that seems to have a missing attribute that > should be there. See the log for more info. > Package broken by this: > - giacpy > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01410600-python-giacpy/builder-live.log.gz Seems a genuine bug in Cython. I created a ticket: https://github.com/cython/cython/issues/3651 > # Some imports fail to be executed > This might be related to some of the previous problems, or a bug in > cimport. Two packages failed tests on importing a module from the the > same package. In fisx, the import was done from a Cython generated > file. In cytoolz, a cythonized module was partially initialized, > possibly due to cyclic import. > Package broken by this: > - fisx > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01410439-python-fisx/builder-live.log.gz This could be trying to import a package-local module without a relative import or qualified module name, i.e. a Py2-import. > - cytoolz > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01410135-python-cytoolz/builder-live.log.gz Yes, could be a cyclic import. This now shows due to PEP-489. Needs some investigation to see what a good fix would be, but these things are usually best resolved on user side. > # Assignment to a read-only property > One package broke on error stating attempts to assign to a read only > properties. > Package broken by this: Pandas > https://download.copr.fedorainfracloud.org/results/dormouse/py38cy30a5/fedora-rawhide-x86_64/01408020-python-pandas/builder-live.log.gz The pandas team made us aware of one such issue, and it's probably the same. This is due to the old NumPy C-API, which was deprecated several years ago (2013?) and is now being phased out. They'll probably resolve it on their side (or might have already). Not directly Cython related, we just follow NumPy here now. > Some problems were expected/unrelated: > - 9 packages use python 2 syntax and don't set the default Might be easiest to set the directive "language_level=2" for these. > - 2 packages were affected by the -fno-common default in GCC 10 Thanks for
[Cython] Cython 3.0 alpha 5 released
Hi all, here's the 5th alpha for 3.0. It goes together with the equally fresh 0.29.18 release that contains a long list of bug fixes, as well as faster memory view slicing. Download: https://pypi.org/project/Cython/3.0a5/ Changelog: https://github.com/cython/cython/blob/master/CHANGES.rst Let's see where fate takes these releases. Stefan Stefan Behnel schrieb am 05.05.20 um 19:32: > Hi all, > > here's yet another alpha release for you, with some great new features and > even more bug fixes this time, and now hopefully generating proper C code > for nogil code again. > > Download: > https://pypi.org/project/Cython/3.0a4/ > > Changelog: > https://github.com/cython/cython/blob/master/CHANGES.rst > > I'll wait a day or two for reports on bugs and regressions and then also > prepare the corresponding 0.29.18 release with the relevant backported bug > fixes. > > Have fun, > Stefan > > > > Stefan Behnel schrieb am 27.04.20 um 18:36: >> … and a third alpha is out, with several fixes (thanks!) for reported >> regressions and issues (thanks!), as well as streamlined GIL handling in >> nogil functions that contain "with gil" sections. >> >> Download: >> https://pypi.org/project/Cython/3.0a3/ >> >> Changelog: >> https://github.com/cython/cython/blob/master/CHANGES.rst >> >> This release was brought to you with the invaluable help of Yusuf Cat >> Stevens and Deutsche Bahn. Thanks to everyone out there who's doing their >> job keeping others alive/healthy/undepressed/fed/mobile/connected/working/… >> in these (for many people) difficult days. >> >> Stefan >> >> >> >> Stefan Behnel schrieb am 23.04.20 um 17:39: >>> Hi all, >>> >>> here's the second alpha. I hope we got all major regressions fixed for this >>> release that were reported since alpha 1. >>> >>> Download: >>> https://pypi.org/project/Cython/3.0a2/ >>> >>> Changelog: >>> https://github.com/cython/cython/blob/master/CHANGES.rst >>> >>> Have fun, >>> Stefan >>> >>> >>> >>> Stefan Behnel schrieb am 12.04.20 um 12:24: >>>> Dear Cython users and devs, >>>> >>>> today, I'm happy to announce the first alpha release of Cython 3.0. >>>> >>>> https://pypi.org/project/Cython/3.0a1/ >>>> >>>> It took us a while to get to this point, well more than a year's time, but >>>> we received a lot of help along the way, most notably from David Woods, >>>> Jeroen Demeyer and Matti Picus. Thanks a lot, and also to the many other >>>> contributors! For this release, we already have 182 closed issues and >>>> merged PRs, including 109 PRs contributed by non-core devs! >>>> >>>> >>>> ** What is Cython 3.0? >>>> >>>> Cython 3.0 is our effort to bring Cython up to date with modern Python 3, >>>> after an 18 year long development history. According to the Python mailing >>>> list archive [1] and Greg's download directory [2], Cython's predecessor >>>> Pyrex 0.1 was announced and released to the public on April 4th, 2002. >>>> >>>> Cython is finally coming of age. :) >>>> >>>> >>>> ** So, what's new? >>>> >>>> Too much. Way too much for this announcement. Cython 3.0 comes with a very >>>> long list of new features, bug fixes and modernisations, a few of which are >>>> backwards incompatible, but in a good way. See the latest changelog: >>>> >>>> https://github.com/cython/cython/blob/master/CHANGES.rst >>>> >>>> Here's a short teaser list anyway: >>>> - Python 3 semantics by default, legacy Python 2 semantics via directive. >>>> - No more deprecated NumPy C-API usage. >>>> - Unicode module names, imports, and identifiers (PEP-3131, PEP-489). >>>> - Support for the fast vectorcall protocol (PEP-590). >>>> - Inlined properties on external cdef classes. >>>> - Faster dispatch for fused functions. >>>> - First steps towards supporting CPython's stable ABI (PEP-384). >>>> >>>> >>>> ** How alpha is it? >>>> >>>> Well. It's not complete yet [3]. It still has some known issues. There are >>>> still some unmerged PRs waiting. Support for the limited API is … limited. >>>> >>>> But, it's in a good shape and probably ready enough for you to make use of >>>> all thos
Re: [Cython] Cython 3.0 alpha 4 released
Jakub Wilk schrieb am 06.05.20 um 20:31: > Hi Stefan! > > Could you push the 3.0a4 tag to GitHub? Done, thanks for the reminder. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0 alpha 4 released
Hi all, here's yet another alpha release for you, with some great new features and even more bug fixes this time, and now hopefully generating proper C code for nogil code again. Download: https://pypi.org/project/Cython/3.0a4/ Changelog: https://github.com/cython/cython/blob/master/CHANGES.rst I'll wait a day or two for reports on bugs and regressions and then also prepare the corresponding 0.29.18 release with the relevant beckported bug fixes. Have fun, Stefan Stefan Behnel schrieb am 27.04.20 um 18:36: > … and a third alpha is out, with several fixes (thanks!) for reported > regressions and issues (thanks!), as well as streamlined GIL handling in > nogil functions that contain "with gil" sections. > > Download: > https://pypi.org/project/Cython/3.0a3/ > > Changelog: > https://github.com/cython/cython/blob/master/CHANGES.rst > > This release was brought to you with the invaluable help of Yusuf Cat > Stevens and Deutsche Bahn. Thanks to everyone out there who's doing their > job keeping others alive/healthy/undepressed/fed/mobile/connected/working/… > in these (for many people) difficult days. > > Stefan > > > > Stefan Behnel schrieb am 23.04.20 um 17:39: >> Hi all, >> >> here's the second alpha. I hope we got all major regressions fixed for this >> release that were reported since alpha 1. >> >> Download: >> https://pypi.org/project/Cython/3.0a2/ >> >> Changelog: >> https://github.com/cython/cython/blob/master/CHANGES.rst >> >> Have fun, >> Stefan >> >> >> >> Stefan Behnel schrieb am 12.04.20 um 12:24: >>> Dear Cython users and devs, >>> >>> today, I'm happy to announce the first alpha release of Cython 3.0. >>> >>> https://pypi.org/project/Cython/3.0a1/ >>> >>> It took us a while to get to this point, well more than a year's time, but >>> we received a lot of help along the way, most notably from David Woods, >>> Jeroen Demeyer and Matti Picus. Thanks a lot, and also to the many other >>> contributors! For this release, we already have 182 closed issues and >>> merged PRs, including 109 PRs contributed by non-core devs! >>> >>> >>> ** What is Cython 3.0? >>> >>> Cython 3.0 is our effort to bring Cython up to date with modern Python 3, >>> after an 18 year long development history. According to the Python mailing >>> list archive [1] and Greg's download directory [2], Cython's predecessor >>> Pyrex 0.1 was announced and released to the public on April 4th, 2002. >>> >>> Cython is finally coming of age. :) >>> >>> >>> ** So, what's new? >>> >>> Too much. Way too much for this announcement. Cython 3.0 comes with a very >>> long list of new features, bug fixes and modernisations, a few of which are >>> backwards incompatible, but in a good way. See the latest changelog: >>> >>> https://github.com/cython/cython/blob/master/CHANGES.rst >>> >>> Here's a short teaser list anyway: >>> - Python 3 semantics by default, legacy Python 2 semantics via directive. >>> - No more deprecated NumPy C-API usage. >>> - Unicode module names, imports, and identifiers (PEP-3131, PEP-489). >>> - Support for the fast vectorcall protocol (PEP-590). >>> - Inlined properties on external cdef classes. >>> - Faster dispatch for fused functions. >>> - First steps towards supporting CPython's stable ABI (PEP-384). >>> >>> >>> ** How alpha is it? >>> >>> Well. It's not complete yet [3]. It still has some known issues. There are >>> still some unmerged PRs waiting. Support for the limited API is … limited. >>> >>> But, it's in a good shape and probably ready enough for you to make use of >>> all those cool new features. Please give it a try and report back if you >>> find issues that we can still improve on in the upcoming pre-releases. PRs >>> very welcome! >>> >>> As always, if it works for you, use it. Generate your C code locally with >>> it, test it, then ship it to your users. Once the code is compiled, they >>> don't even have to know that you've used an alpha version. :) >>> >>> >>> Have fun, >>> >>> Stefan >>> >>> >>> >>> [1] https://mail.python.org/pipermail/python-list/2002-April/126661.html >>> [2] https://www.csse.canterbury.ac.nz/greg.ewing/python/Pyrex/oldtar/ >>> [3] https://github.com/cython/cython/milestone/58 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython 3.0 alpha 3 released
… and a third alpha is out, with several fixes (thanks!) for reported regressions and issues (thanks!), as well as streamlined GIL handling in nogil functions that contain "with gil" sections. Download: https://pypi.org/project/Cython/3.0a3/ Changelog: https://github.com/cython/cython/blob/master/CHANGES.rst This release was brought to you with the invaluable help of Yusuf Cat Stevens and Deutsche Bahn. Thanks to everyone out there who's doing their job keeping others alive/healthy/undepressed/fed/mobile/connected/working/… in these (for many people) difficult days. Stefan Stefan Behnel schrieb am 23.04.20 um 17:39: > Hi all, > > here's the second alpha. I hope we got all major regressions fixed for this > release that were reported since alpha 1. > > Download: > https://pypi.org/project/Cython/3.0a2/ > > Changelog: > https://github.com/cython/cython/blob/master/CHANGES.rst > > Have fun, > Stefan > > > > Stefan Behnel schrieb am 12.04.20 um 12:24: >> Dear Cython users and devs, >> >> today, I'm happy to announce the first alpha release of Cython 3.0. >> >> https://pypi.org/project/Cython/3.0a1/ >> >> It took us a while to get to this point, well more than a year's time, but >> we received a lot of help along the way, most notably from David Woods, >> Jeroen Demeyer and Matti Picus. Thanks a lot, and also to the many other >> contributors! For this release, we already have 182 closed issues and >> merged PRs, including 109 PRs contributed by non-core devs! >> >> >> ** What is Cython 3.0? >> >> Cython 3.0 is our effort to bring Cython up to date with modern Python 3, >> after an 18 year long development history. According to the Python mailing >> list archive [1] and Greg's download directory [2], Cython's predecessor >> Pyrex 0.1 was announced and released to the public on April 4th, 2002. >> >> Cython is finally coming of age. :) >> >> >> ** So, what's new? >> >> Too much. Way too much for this announcement. Cython 3.0 comes with a very >> long list of new features, bug fixes and modernisations, a few of which are >> backwards incompatible, but in a good way. See the latest changelog: >> >> https://github.com/cython/cython/blob/master/CHANGES.rst >> >> Here's a short teaser list anyway: >> - Python 3 semantics by default, legacy Python 2 semantics via directive. >> - No more deprecated NumPy C-API usage. >> - Unicode module names, imports, and identifiers (PEP-3131, PEP-489). >> - Support for the fast vectorcall protocol (PEP-590). >> - Inlined properties on external cdef classes. >> - Faster dispatch for fused functions. >> - First steps towards supporting CPython's stable ABI (PEP-384). >> >> >> ** How alpha is it? >> >> Well. It's not complete yet [3]. It still has some known issues. There are >> still some unmerged PRs waiting. Support for the limited API is … limited. >> >> But, it's in a good shape and probably ready enough for you to make use of >> all those cool new features. Please give it a try and report back if you >> find issues that we can still improve on in the upcoming pre-releases. PRs >> very welcome! >> >> As always, if it works for you, use it. Generate your C code locally with >> it, test it, then ship it to your users. Once the code is compiled, they >> don't even have to know that you've used an alpha version. :) >> >> >> Have fun, >> >> Stefan >> >> >> >> [1] https://mail.python.org/pipermail/python-list/2002-April/126661.html >> [2] https://www.csse.canterbury.ac.nz/greg.ewing/python/Pyrex/oldtar/ >> [3] https://github.com/cython/cython/milestone/58 > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Problems with "private" functions/methods in Cython 3.0a2
da-woods schrieb am 23.04.20 um 22:54: > On 23/04/2020 21:32, Stefan Behnel wrote: >> The question is: how bad is it for existing code, and how can we either >> keep things working or make it easy for users to adapt? Should we implement >> some kind of fallback lookup that still finds unmangled global names if >> they exist? >> >> I mean, double-underscore names are not entirely unheard of in C code and C >> libraries. The Python behaviour redners their usage from within classes >> more difficult. >> > I think this could actually be a real problem for people trying to interact > with C libraries. I'd favour a fallback that finds unmangled cdef/c names > only (and maybe gives a helpful warning but no match if a Python name is > found). That gives 100% compatibility on the Python side, but a sensible > fallback on the C side. Ok, I think this is worth a ticket by now. Let's discuss this over there. https://github.com/cython/cython/issues/3544 Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Problems with "private" functions/methods in Cython 3.0a2
da-woods schrieb am 23.04.20 um 20:47: > On closer inspection I believe both are following Python behaviour (and > thus behaving correctly). In Python any > `__name_preceded_by_double_underscores` used within a class ends up looking > up `_Classname__name_preceded_by_double_underscores`. Therefore in case 1 > it's looking up `_Bar__foo` and not finding it. Hmm, right. Py3.6 gives me this: >>> class T: ... def f(self): return __xyz ... >>> __xyz = 2 >>> T().f() Traceback (most recent call last): File "", line 1, in File "", line 2, in f NameError: name '_T__xyz' is not defined > Therefore, I think it is the "correct" thing to do. > > This proved a pain with the `__pyx_unpickle` functions that Cython > generates. In the end I just disabled it for `__pyx_` names, which I guess > could be called hypocrisy . Yeah, it seems correct then. Meaning, new code will benefit from the lack of differences with normal Python behaviour. The question is: how bad is it for existing code, and how can we either keep things working or make it easy for users to adapt? Should we implement some kind of fallback lookup that still finds unmangled global names if they exist? I mean, double-underscore names are not entirely unheard of in C code and C libraries. The Python behaviour redners their usage from within classes more difficult. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Problems with "private" functions/methods in Cython 3.0a2
Hi,
thanks for the very detailed report.
I'm CC-ing cython-devel, but let's keep the discussion for this on the
cython-users list.
The background here is Github issue 1382 and PR 3123
https://github.com/cython/cython/issues/1382
https://github.com/cython/cython/pull/3123
This was newly merged for alpha 2, and I consider it somewhat provisional.
It makes "private" Python names follow Python semantics, but if that poses
excessive breakage for existing code, then we'll have to find a pragmatic
way to deal with it.
'Toni Barth' via cython-users schrieb am 23.04.20 um 19:07:
> I just updated Cython from 3.0a1 to 3.0a2 in one of my projects and
> noticed that it seems to introduce issues with private
> functions/methods, means methods with leading __.
>
> I noticed two cases which now fail in my project. I'm however unsure if
> thats supposed to happen or if its a problem with the new Cython build.
>
>
> Case 1: private function not declared
>
>
> crash.pxd:
>
>
> cpdef __foo()
>
> cdef class Bar:
> cpdef run(Bar self)
>
>
> crash.pyx:
>
>
> cpdef __foo():
> print("foo")
>
> cdef class Bar:
> cpdef run(Bar self):
> __foo()
>
> print("bar")
>
>
> Running this with the following command:
>
> py -3 -c "import pyximport;pyximport.install();import
> crash;crash.Bar().run()"
>
>
> Result:
>
> crash.c
> C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python36_64\lib\site-packages\Cython\Compiler\Main.py:344:
> FutureWarning: Cython directive 'language_level' not set, using '3str'
> for now (Py3). This has changed from earlier releases! File:
> D:\crash\crash.pxd
> tree = Parsing.p_module(s, pxd, full_module_name)
>
> Error compiling Cython file:
>
> ...
> cpdef __foo():
> print("foo")
>
> cdef class Bar:
> cpdef run(Bar self):
> __foo()
> ^
>
>
> crash.pyx:6:4: undeclared name not builtin: __foo
At least this looks like a bug to me. Global module names should not be
impacted by this change.
> Error compiling Cython file:
>
> ...
> cpdef __foo():
> print("foo")
>
> cdef class Bar:
> cpdef run(Bar self):
> __foo()
> ^
>
>
> crash.pyx:6:4: '__foo' redeclared
>
> Error compiling Cython file:
>
> ...
> cpdef __foo()
> ^
>
>
> crash.pxd:1:12: Previous declaration is here
Lovely :)
> Another problem, probably related, occurs when trying to override a
> private method in a derived class:
>
>
> Case 2: overriding private class methods
>
>
> base.pxd:
>
>
> cdef class Base:
> cdef void __foo(Base self)
> cpdef run(Base self)
>
>
> base.pyx:
>
>
> cdef class Base:
>
> cdef void __foo(Base self):
> print("foo!")
>
> cpdef run(Base self):
> self.__foo()
>
>
> extend.pxd:
>
>
> from base cimport Base
>
> cdef class Extend(Base):
> pass
>
>
> extend.pyx:
>
>
> cdef class Extend(Base):
>
> cdef void __foo(Extend self):
> print("bar")
>
> Running with the following command:
>
> py -3 -c "import pyximport;pyximport.install();import
> extend;extend.Extend().run()"
>
>
> Results in the following output:
>
>
> extend.c
> C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python36_64\lib\site-packages\Cython\Compiler\Main.py:344:
> FutureWarning: Cython directive 'language_level' not set, using '3str'
> for now (Py3). This has changed from earlier releases! File:
> D:\crash\extend.pxd
> tree = Parsing.p_module(s, pxd, full_module_name)
>
> Error compiling Cython file:
>
> ...
> cdef class Extend(Base):
>
> cdef void __foo(Extend self):
> ^
>
>
> extend.pyx:3:7: C method '_Extend__foo' not previously declared in
> definition part of extension type 'Extend'
This, OTOH, is intended. Private methods are not meant to be inherited and
overridden. That's what makes them "private" :) That's consistent with
Python semantics.
I think it's somewhat up for debate if this should apply in the same way to
cdef classes as for Python classes. But from a language design point of
view, having this kind of difference between the two seems a bad idea.
Do you feel like argueing against this? :)
Stefan
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0 alpha 2 released
Hi all, here's the second alpha. I hope we got all major regressions fixed for this release that were reported since alpha 1. Download: https://pypi.org/project/Cython/3.0a2/ Changelog: https://github.com/cython/cython/blob/master/CHANGES.rst Have fun, Stefan Stefan Behnel schrieb am 12.04.20 um 12:24: > Dear Cython users and devs, > > today, I'm happy to announce the first alpha release of Cython 3.0. > > https://pypi.org/project/Cython/3.0a1/ > > It took us a while to get to this point, well more than a year's time, but > we received a lot of help along the way, most notably from David Woods, > Jeroen Demeyer and Matti Picus. Thanks a lot, and also to the many other > contributors! For this release, we already have 182 closed issues and > merged PRs, including 109 PRs contributed by non-core devs! > > > ** What is Cython 3.0? > > Cython 3.0 is our effort to bring Cython up to date with modern Python 3, > after an 18 year long development history. According to the Python mailing > list archive [1] and Greg's download directory [2], Cython's predecessor > Pyrex 0.1 was announced and released to the public on April 4th, 2002. > > Cython is finally coming of age. :) > > > ** So, what's new? > > Too much. Way too much for this announcement. Cython 3.0 comes with a very > long list of new features, bug fixes and modernisations, a few of which are > backwards incompatible, but in a good way. See the latest changelog: > > https://github.com/cython/cython/blob/master/CHANGES.rst > > Here's a short teaser list anyway: > - Python 3 semantics by default, legacy Python 2 semantics via directive. > - No more deprecated NumPy C-API usage. > - Unicode module names, imports, and identifiers (PEP-3131, PEP-489). > - Support for the fast vectorcall protocol (PEP-590). > - Inlined properties on external cdef classes. > - Faster dispatch for fused functions. > - First steps towards supporting CPython's stable ABI (PEP-384). > > > ** How alpha is it? > > Well. It's not complete yet [3]. It still has some known issues. There are > still some unmerged PRs waiting. Support for the limited API is … limited. > > But, it's in a good shape and probably ready enough for you to make use of > all those cool new features. Please give it a try and report back if you > find issues that we can still improve on in the upcoming pre-releases. PRs > very welcome! > > As always, if it works for you, use it. Generate your C code locally with > it, test it, then ship it to your users. Once the code is compiled, they > don't even have to know that you've used an alpha version. :) > > > Have fun, > > Stefan > > > > [1] https://mail.python.org/pipermail/python-list/2002-April/126661.html > [2] https://www.csse.canterbury.ac.nz/greg.ewing/python/Pyrex/oldtar/ > [3] https://github.com/cython/cython/milestone/58 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Help with cygdb bug
Hi Volker! Volker Weißmann schrieb am 22.04.20 um 20:44: > cy exec print(localvar) > > currently does not work. And I need your help to fix it. Do you mean "cy exec print(x)" or "cy print x" here? (The latter is what you used in your example at the end.) > If I'm not mistaken, the function "_fill_locals_dict" in libcython.py is > supposed to find and set all local variables. > > The "for name, cyvar in iterator:" correctly iterates over all local > variables, but cyvar.type is not PythonObject but CObject. > > The cython line "localvar= 49" gets translated to the C-line > "__pyx_v_localvar = 49;". Notice how the type of__pyx_v_localvar is long > and not a static PyObject* . If an equivalent "static PyObject*" would > exist in the C-code, I would know how to make 'cy exec print(localvar)' > work (the reason why global variables work, is because they are of type > static PyObject*). I think I could live with the Python code "print(localvar)" not working, as long as I have something like "p localvar" and "pyo localvar". I guess the issue here is the name mangling, which should happen automatically. Can't we map "cy print x" to "p __pyx_v_x" for C types and "pyo __pyx_v_x" for Python object types? (And, isn't that how it works right now?) > I see two ways to make `cy exec print(localvar)` work: > > 1. Let cygdb generate C-code to convert a long (or any other type) to an > equivalent static PyObject* (For long's it would be > __Pyx_PyInt_From_long). Then let cygdb execute this code by running > gdb.parse_and_eval(code). Then read the resulting variable with > something like > > code = ''' > (PyObject *) PyDict_SetItem( > (PyObject *) %d, > (PyObject *) %d, > (PyObject *) %s) > ''' % (local_dict_pointer, pystringp, cname) > > gdb.parse_and_eval(code) I'd rather not go take that route. If it's a C value, gdb should be able to print it. We should better not go through Python here. > 2. Compile the argument of cy exec (in this case "print(localvar)") to > C-code (while getting the local variables right) and run it using > gdb.parse_and_eval(code). > > > Ideally, cy exec var=123 should work like cy set var=123. That seems a cool feature – but also a lot of implementation work. My gut feeling is that there may be tons of special cases where code needs to be adapted somehow before or after the code generation in order to make it do what users would expect it to do. I'd rather like to avoid complexity where we can. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 3.0 alpha 1 released
Dear Cython users and devs, today, I'm happy to announce the first alpha release of Cython 3.0. https://pypi.org/project/Cython/3.0a1/ It took us a while to get to this point, well more than a year's time, but we received a lot of help along the way, most notably from David Woods, Jeroen Demeyer and Matti Picus. Thanks a lot, and also to the many other contributors! For this release, we already have 182 closed issues and merged PRs, including 109 PRs contributed by non-core devs! ** What is Cython 3.0? Cython 3.0 is our effort to bring Cython up to date with modern Python 3, after an 18 year long development history. According to the Python mailing list archive [1] and Greg's download directory [2], Cython's predecessor Pyrex 0.1 was announced and released to the public on April 4th, 2002. Cython is finally coming of age. :) ** So, what's new? Too much. Way too much for this announcement. Cython 3.0 comes with a very long list of new features, bug fixes and modernisations, a few of which are backwards incompatible, but in a good way. See the latest changelog: https://github.com/cython/cython/blob/master/CHANGES.rst Here's a short teaser list anyway: - Python 3 semantics by default, legacy Python 2 semantics via directive. - No more deprecated NumPy C-API usage. - Unicode module names, imports, and identifiers (PEP-3131, PEP-489). - Support for the fast vectorcall protocol (PEP-590). - Inlined properties on external cdef classes. - Faster dispatch for fused functions. - First steps towards supporting CPython's stable ABI (PEP-384). ** How alpha is it? Well. It's not complete yet [3]. It still has some known issues. There are still some unmerged PRs waiting. Support for the limited API is … limited. But, it's in a good shape and probably ready enough for you to make use of all those cool new features. Please give it a try and report back if you find issues that we can still improve on in the upcoming pre-releases. PRs very welcome! As always, if it works for you, use it. Generate your C code locally with it, test it, then ship it to your users. Once the code is compiled, they don't even have to know that you've used an alpha version. :) Have fun, Stefan [1] https://mail.python.org/pipermail/python-list/2002-April/126661.html [2] https://www.csse.canterbury.ac.nz/greg.ewing/python/Pyrex/oldtar/ [3] https://github.com/cython/cython/milestone/58 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Preparing the first alpha release of Cython 3.0
Hi all, I'm curently preparing a first alpha release for Cython 3.0 (alpha 1). Please ping me on any open pull requests that you think are ready and should be included. Note that this is not the end of the journey – anything that's not included right now can still come in later, before the first beta. (That's later this year, before you ask.) There will definitely be at least one more alpha release before we get even there. So, please concentrate on changes for this release that have a potentially disruptive user impact, so that our users can start preparing their code and/or complain early in order to keep us from making The One Big Mistake of breaking their code. Thanks, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Auto-generation of wrapper types
da-woods schrieb am 12.03.20 um 16:11: > The process of wrapping a C struct or C++ class in an extension type often > has the user doing a pretty mechanical duplication of attributes/functions > that Cython already knows about. I'm looking at doing: > > cdef struct S: > int a > # etc. > > then `cython.autowrap[S]` would create an extension type that wraps S by > value. Sounds like a good idea in general. Questions: What makes C structs and C++ classes special enough to make this a separate language feature? That should be part of a motivation somewhere (probably in a ticket). How does this relate to the support for @dataclass in GH-2903? Is the only difference that you would write "x.member" instead of "x.structattr.member"? https://github.com/cython/cython/issues/2903 Why "autowrap[S]" and not "autowrap(S)"? I'm asking also because there should probably be a way for this to work in Python files, for which a function call seems more natural. It also feels like a @decorator would somehow fit quite well here. Even when defined in a .pxd file, the struct would still only be exported in the corresponding module (with the same name). Something like this: @cclass cdef struct S: int x Not sure if reusing @cclass is a good idea, but it doesn't seem wrong. Defining this in a .pxd file could then even allow cimporting modules to understand the extension type wrapper as well, by simply reconstructing the type representation on their side. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Status
John Skaller2 schrieb am 01.02.20 um 15:38: > On 1 Feb 2020, at 20:00, Greg Ewing wrote: >> On 1/02/20 3:29 pm, John Skaller2 wrote: >>> But the all hell breaks loose for pointers. Your hack only >>> works for rvalues. >> >> Yes, using these it's possible for Cython to accept something >> that will later be rejected by the C compiler. It's not perfect, >> but it gets the job done most of the time. > > My concern is that the C compiler wont reject it, the program > will corrupt data or crash: > > int32_t *x=..; *x = 42; > int64_t *x=..; *x = 42; > > The C compiler will overwrite 4 or 8 bytes. Which one matters. As stated before, the C compiler will see the correct definition of the types in the header files. And the generated C code will use the same type names that the user had in their source code. "int64_t" will not magically become "int" there. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Size of output
John Skaller2 schrieb am 01.02.20 um 15:32: > My problem is trying to read the generated code. You might be looking for "cython -a". Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Size of output
Robert Bradshaw schrieb am 01.02.20 um 00:49: > Taking a quick glance at an auto-generated file for an empty .pyx, we have > [...] > ~300 lines module setup code. Even for trivial modules, we still > declare and call functions for creating globals, preparing types, etc. > even if we don't have any globals, types, etc. :) > I agree there's some fat that could be trimmed there, but not sure > it'd be worth the effort. +1 In any non-trivial module where the size really starts to matter, the portions of unused code that we generate will probably make no difference at all. I'll also add that it can sometimes be very difficult to avoid generating unused code due to target (CPython x.y, PyPy, …) specific utility code that comes with its own dependencies, which then may or may not be used on a given target. Removing the final 0.5% of code that is "unused in some cases" is really not something I would want to invest days into, each time we make a change in Cython. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Status
John Skaller2 schrieb am 30.01.20 um 03:47: > Ok, I have processed most of the Cython/Include/python/*.pxd files. > > Seems these were mechanically derived from docs? They were manually copied over time. > Because some of the bugs in the docs are also in these files. Not sure what > you > would like to do so I’ll ask here before doing github issues. > > 1. Its not all there. I couldn’t find the API for generators for example. > If the API is extracted from the docs the script could be run again. > > 2. pyport is plain wrong. It contains conflicting C typedefs. PRs welcome. > 3. Some of the files describe extension modules (datetime, array) > rather than the core API. That's fine. > 4. ../cython/Cython/Includes/cpython/weakref.pxd:bint > PyWeakref_CheckProxy(ob) > > ob should be PyObject* No, the declaration looks correct to me. The input is an object. It does not seem helpful to me to require users to do a cast. > there’s a couple more (maybe in ref as well). PRs welcome. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hello
John Skaller2 schrieb am 28.01.20 um 10:57: > What’s a “descr” when its at home? A descriptor, a special protocol in Python. https://docs.python.org/3/howto/descriptor.html > There are some special words in pxd files like “list” which mean > PyObject that happens to be a list. Is there a list of these somewhere? See Builtin.py. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hello
John Skaller2 schrieb am 27.01.20 um 06:56: > Hi! I have just built Cython but haven’t used it yet. > > I want to check I understand what it does whilst examing the sources in the > repository. > Please let me know if I have it wrong! > > Given a Python file, Cython parses it, and translates it to the equivalent C, > which can > then be compiled to binary, thereby bypassing the overhead of interpreting > bytecode, > but still executing the same accesses to the CPython run time API that the > interpreter would. Not the same, just equivalent operations. Cython often generates code that bypasses the "usual" C-API calls. > So there may be a small speed improvement, or maybe not (because it misses > optimisations > the interpreter might be able to spot). Yes, it goes both ways. Cython generates many fast-paths that the interpreter doesn't have or cannot easily provide, so there is a high chance that compiled code is faster than interpreted code. The speed improvement is usually somewhere around 30% without type annotations, but highly dependant on your actual code. OTOH, recent CPython versions added some internal infrastructure that speeds up important operations, which isn't always easy to use from external tools like Cython. So there are a few cases where the runtime has an inherent advantage, although it doesn't always show much in comparison. And we keep fighting back. :) > The binary will typically be a C extension module > that can be loaded and operate the same way as the original Python. > > Now, Cython is an extension of Python which allows some extra stuff, including > type annotations, and other directives related to integration with C. These > can be > used to facilitate integration with external C libraries directly, and > mapping into > Python, as if the code were written in C, only we’re using a Python like > language > representing a subset of C instead of C. It's really mostly about data types (and Cython's mixed Python/C type system). Python allows you to do a lot of seemingly different things in the same syntactic constructs, and Cython does the same when it generates the C code by adapting it to the data types that the source operates on. > Additionally, the compiler recognises the type annotations, and can reduce or > eliminate run time type checks, improving performance, or even replacing > common constructions in Python which much faster ones that do the same job > “closer to the metal”. Yes. > To make this work, the CPython API itself is represented in a set of *.pxd > files > found in the repository in Includes/cpython No. :) These files are only for end users to allow importing pre-declared parts of the CPython C-API for their own use (in case they feel like it). Cython itself does not use or need them, but if they help you… > splitting the logic of the compiler > roughly into two parts: the front and back end. The front end groks Python > and Cython code whilst the back end generates the actual C. Back to a Yes. > Just FYI, I’m the developer of a programming language, Felix, which is C++ > code > generator. You can think of it as a meta-programming language for C++ with a > proper type system. Felix binds C/C++ code with statements like: > > type PyObject = “PyObject*”; > fun add: PyObject * PyObject -> PyObject = “Py_AddLong($1)”; > > and can use the bindings like: > > var a : PyObject = …. > var b: PyObject = ... > var sum = add (a,b); > > so in some ways its doing the same kind of job as Cython, except it isn’t > specialised to bind to Python, it can bind to anything written in C or C++. > Including the Python API as illustrated. As Greg's question hinted, Cython knows a lot about the object reference counting that CPython uses for garbage collection. That's one of the main reasons why people prefer it over using the CPython C-API directly. Writing correct code in the latter is quite difficult and requires a lot of discipline. Cython gives you that for free. > One possible future goal is to replace NumPy with something much better. You're not the first. :) Look at other projects like Pythran, Numba, Theano, numexpr, … Pythran actually integrates with Cython's type system to generate C++ code from NumPy expressions. > I also have code written in Python that I might translate to C using Cython. > It would be kind of interesting to use Cython to generate C, and then create > bindings to that C in Felix, so instead of calling the Python C API, we call > the Cython generated API instead, allowing people to write libraries for > Felix in Cython instead of C or Felix. However that’s a more major > integration task. > You’d want Cython to generate the Felix bindings, or at least output meta-data > that would allow them to be generated easily .. such as .. a *.pxd file !! I would encourage you to generate Cython code instead of C/C++ directly, if you want to interact with Python. There's no need to create
Re: [Cython] Adding GPU support to cython
Hi Frank, sorry for leaving this unresponded for a while. I'm far from an expert in this, but it looks like no-one else jumped in, so here's my response. Schlimbach, Frank schrieb am 06.01.20 um 12:09: > I would like to work on extending cython with a way to offload cython code to > a GPU. I found to related CEPs > (https://github.com/cython/cython/wiki/enhancements-opencl and > https://github.com/cython/cython/wiki/enchancements-metadefintions). So, just for a bit of context, these CEPs were written a decade ago, and Cython's code base, feature set, and the general Python ecosystem have evolved a lot since then. For example, "cython.parallel" and "prange()" weren't even implemented back then. > My current thinking is that a solution along the OpenCL CEP is most > effective, it does require many code changes and seems to provide a good > tradeoff between usability and efficiency. > > I would like to suggest a few modifications to this approach, like > > * using SYCL instead of OpenCL to closely follow existing parallel/prange > semantics more easily > * selecting the device (CPU, GPU) per region rather than per file > * maybe allowing calling appropriately annotated and written external > functions > > I would be very grateful for any thoughts about this topic in general and for > any advice on how to approach this so that a solution is found that is most > broadly useful and most cythonic. It would definitely be cool to generate GPU support from the existing Cython patterns, in addition to the OpenMP code that we already generate. If that can be done, then users could enable GPU support by adding a C compiler define to their CFLAGS (rather than rerunning Cython), or even select between the two versions at runtime. If the GPU support is per region, then how is the code section shipped to the GPU? Is the infrastructure for this provided by the OpenCL framework or does the user or the module need to set something up in addition? Finally, generally speaking: - PR welcome - simple approach preferred (at least to get this started and prototyped) - discussion welcome on this mailing list - GitHub feature ticket seems to be missing, with a link to the ML thread https://mail.python.org/pipermail/cython-devel/2020-January/005262.html Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: .gitignore file in github repo
omár sweidán schrieb am 08.01.20 um 22:47: > Is there a way for me to contribute to the C files in the Cython folder? > When I changed them, it does not show in the changed files list (e.g. in > Git GUI). Why are the C files put in the gitignore file? We exclude all files that are generated by Cython. There is no use in changing them, and they will be overwritten by a rebuild. Change the Python files instead that they are generated from. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Progress towards Cython 3.0
Hi, tl;dr: help with the remaining work would be appreciated. The "3.0" milestone on github currently lists 24 open tickets, so there's still quite a bit to do before the release. Some of them are already PRs that need more work in one way or another. https://github.com/cython/cython/milestone/58 The upside is that we've already closed twice as many issues as are still open. https://github.com/cython/cython/issues?utf8=✓=is%3Aclosed+type%3Aissue+milestone%3A3.0 I went through the open issues and marked everything as "blocker" that represents behavioural changes. We should make sure to include those in this release, or postpone them to the next major release. https://github.com/cython/cython/issues?utf8=%E2%9C%93=is%3Aopen+label%3A%22P%3A+blocker%22+milestone%3A3.0 Some tickets seem easy to do (marked as "good first issue"). But also the "binding" PR is a big one that will need some more fiddling with the details, some of which can be investigated separately. https://github.com/cython/cython/pull/2864 Any help with the open issues would be greatly appreciated. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Backport language_level=2 fix?
Jeroen Demeyer schrieb am 27.06.19 um 12:44: > Since Cython 3.0 doesn't seem close to being released, is there any chance > to backport https://github.com/cython/cython/pull/2792 > > It's a fairly simple and reasonable bugfix. Done: https://github.com/cython/cython/commit/d019d9f3e057cae5f873280bf78bce639f2a81da Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Empty defines of Py_TPFLAGS_METHOD_DESCRIPTOR?
Jeroen Demeyer schrieb am 18.06.19 um 13:47: > What's the purpose of commit > https://github.com/cython/cython/commit/8ac1a6a55fc44ff858cc367af556250a91f71d16 > > I can't think of any reasonable scenario where Py_TPFLAGS_METHOD_DESCRIPTOR > would be defined differently than in CPython (i.e. with value 1<<17). > > This is an honest question, I don't know what I'm missing. We define a couple of CPython flag names for older versions, and some of them are defined as 0 instead of their bit value, because we are not using them ourselves and it gives '&' tests an obvious 0 result. We don't define this one specifically, but I wanted to allow this kind of empty definition for the flag, that's all. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Reason for __Pyx_PyObject_Call2Args, __Pyx_PyObject_Call3Args
Jeroen Demeyer schrieb am 13.06.19 um 15:14: > Is there any particular reason why Cython has specific code like > __Pyx_PyObject_Call2Args, __Pyx_PyObject_Call3Args, ... > > And then there is even more complicated Python code in PyMethodCallNode > generating something like __Pyx_PyObject_Call2Args for an arbitrary number > of arguments. > > This could be simplified a lot by just writing code once to deal with n > arguments and then letting the C compiler do the clever stuff: if you write > a loop where the number of iterations is a small compile-time constant, the > compiler should unroll it. > > I checked that GCC-6.4.0 is able to do this and the much simpler code with > a for loop is as efficient as the current code. But maybe there are other > compilers that don't do this (or there used to be such compilers when that > code was written)? > > I stumbled on this while working on > https://github.com/cython/cython/issues/2996 It was originally written that way because there were slight differences, and only 0/1 args were special cased initially. The method optimisation then showed that it's worth special casing the self argument, which gave rise to the other helpers. I've seen your PR, which cleans this up very nicely. Thanks! Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Performance comparison with CPython for attribute/item access
Stefan Behnel schrieb am 16.02.19 um 22:39: > - The list append/pop optimisations seem to slow down non-lists > unproportionally, for deques by 3x compared to CPython. That seems worth > improving. > > CPython 3.8 (63fa1cfece) > > Stack (or queue) operations: > 61.6 ns list_append_pop > 53.9 ns deque_append_pop > > Cython 3.0a0 (f1eaa9c1f) > > Stack (or queue) operations: > 20.7 ns list_append_pop > 155.4 ns deque_append_pop > > Cython 3.0a0 (no-opt) > = > Stack (or queue) operations: > 66.4 ns list_append_pop > 75.1 ns deque_append_pop Fixed for Py3.7+: 20.0 ns list_append_pop 45.5 ns deque_append_pop https://github.com/cython/cython/issues/2850 Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Performance comparison with CPython for attribute/item access
Hi, Raymond Hettinger wrote a micro benchmark script for comparing the performance of basic attribute and item access patterns across Python versions and build configurations, so I tested the initially committed version with Cython. https://github.com/python/cpython/blob/master/Tools/scripts/var_access_benchmark.py Results are below, comparing Cython (master) to CPython 3.8 (master), and also disabling all C-time optimisations via the "CYTHON_*" macros (no-opt). Some things to note: - Most operations in Cython are around 30-50% faster. - C-level things like local variables are not measurable in Cython. - Setting class variables is very slow in both CPython and Cython, probably for the same (unknown) reason, maybe the method cache or so. https://bugs.python.org/issue36012 - The dict version check for Python globals is worth it. Disabling it in Cython with "-DCYTHON_USE_DICT_VERSIONS=0" slows down the lookup by 5x (2.2ns -> 10ns). - Disabling PyList optimisations with "-DCYTHON_USE_PYLIST_INTERNALS=0" slows down the "list_append_pop" benchmark by 5x (21ns -> 102ns). - The list append/pop optimisations seem to slow down non-lists unproportionally, for deques by 3x compared to CPython. That seems worth improving. Stefan CPython 3.8 (63fa1cfece) Variable and attribute read access: 5.4 ns read_local 6.0 ns read_nonlocal 15.7 ns read_global 23.5 ns read_builtin 23.1 ns read_classvar_from_class 20.4 ns read_classvar_from_instance 31.5 ns read_instancevar 25.4 ns read_instancevar_slots 23.8 ns read_namedtuple 34.5 ns read_boundmethod Variable and attribute write access: 6.2 ns write_local 6.7 ns write_nonlocal 19.1 ns write_global 113.2 ns write_classvar 44.6 ns write_instancevar 33.0 ns write_instancevar_slots Data structure read access: 23.5 ns read_list 24.0 ns read_deque 25.6 ns read_dict Data structure write access: 26.0 ns write_list 27.1 ns write_deque 32.0 ns write_dict Stack (or queue) operations: 61.6 ns list_append_pop 53.9 ns deque_append_pop Timing loop overhead: 0.4 ns loop_overhead Cython 3.0a0 (f1eaa9c1f) Variable and attribute read access: 0.2 ns read_local 0.2 ns read_nonlocal 2.2 ns read_global 0.2 ns read_builtin 13.8 ns read_classvar_from_class 11.1 ns read_classvar_from_instance 21.3 ns read_instancevar 15.5 ns read_instancevar_slots 13.6 ns read_namedtuple 21.5 ns read_boundmethod Variable and attribute write access: 0.2 ns write_local 0.1 ns write_nonlocal 13.0 ns write_global 92.9 ns write_classvar 29.6 ns write_instancevar 16.1 ns write_instancevar_slots Data structure read access: 4.0 ns read_list 4.3 ns read_deque 16.5 ns read_dict Data structure write access: 4.3 ns write_list 6.4 ns write_deque 21.4 ns write_dict Stack (or queue) operations: 20.7 ns list_append_pop 155.4 ns deque_append_pop Timing loop overhead: 0.1 ns loop_overhead Cython 3.0a0 (no-opt) = Variable and attribute read access: 0.2 ns read_local 0.2 ns read_nonlocal 15.6 ns read_global 0.2 ns read_builtin 16.1 ns read_classvar_from_class 12.1 ns read_classvar_from_instance 21.9 ns read_instancevar 16.3 ns read_instancevar_slots 14.5 ns read_namedtuple 23.8 ns read_boundmethod Variable and attribute write access: 0.2 ns write_local 0.2 ns write_nonlocal 14.2 ns write_global 99.4 ns write_classvar 35.0 ns write_instancevar 22.4 ns write_instancevar_slots Data structure read access: 5.7 ns read_list 6.1 ns read_deque 21.1 ns read_dict Data structure write access: 8.4 ns write_list 8.4 ns write_deque 24.0 ns write_dict Stack (or queue) operations: 66.4 ns list_append_pop 75.1 ns deque_append_pop Timing loop overhead: 0.2 ns loop_overhead ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Should Cython 3.0 still have Python 2.x support?
Robert Bradshaw schrieb am 11.02.19 um 16:54: > On Fri, Feb 8, 2019 at 12:45 PM Stefan Behnel wrote: >> Robert Bradshaw schrieb am 06.02.19 um 10:37: >>> Realistically, I think that if we don't support 2.7 in the 3.0 >>> release, we're signing up for maintaining a long-lived 2.9 branch, >>> which I'd like to avoid. >> >> Although the same would probably apply to Cy3.0 if we cut Py2.7 support in >> Cy3.1. Some Linux enterprise distributions will support Py2.7 for another >> 6-10 years or so. It doesn't feel like switching now or in a couple of >> months makes a big difference on that front. > > I think the changes in Cy2.9 and Cy3.0 are bigger than those between > Cy3.0 and Cy3.1, and in particular there are user-facing changes (e.g. > the language level default) that make it harder for users of Cython to > support both from a single codebase (though for that one I suppose one > could manually set a directive). Ok, lets's keep Py2 support for Cy3.0. Why have one reason to celebrate when you can have two? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.29.5 fixes a crash for cpdef functions
Hi everyone, Cython 0.29.5 was released with an important bug fix for extension types that have "cpdef" methods. If they get subclassed by Python classes that do not define additional attributes (or anything that would require them to have an instance `__dict__`), then calling the cpdef-methods repeatedly triggers a crash in the optimised code in 0.29.x that checks for subclass overrides. The issue was reported here: https://github.com/cython/cython/issues/2823 It's a new issue in the 0.29.x series and only appears under Python 3.6 and later, so older versions were not affected. Everyone using 0.29.x with cpdef methods is encouraged to upgrade. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Minimum Py3 version (3.4) – was: [cython-users] Should Cython 3.0 still have Python 2.x support?
Stefan Behnel schrieb am 02.02.19 um 12:22: > We currently support Py3.3+ for Py3. Py3.4 is expected to reach its EOL > in March, but still seems worth supporting for another while. Not sure > about 3.3, probably an "as long as it doesn't hurt" case. Actually, I just checked – we do not test Py3.3 compatibility anymore, so it's probably broken already. I had to disable the CI tests for it some time after its support ended in late 2017, because neither travis nor appveyor were still providing it in a usable way. So, the minimum Py3 version becomes 3.4 then, and AFAIR, that's in some Linux distros with long term support, so we should keep it as long as we can easily support it. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Should Cython 3.0 still have Python 2.x support?
[CC-ing cython-devel, but please keep the discussion on cython-users] Hi all, we were recently asked on the bug tracker [1] when Cython is planning to end its support for Python 2.x, so I'd like to get some more user opinions on this. We are planning to release Cython 3.0 this year [2], with some well selected backwards incompatible changes and a tiny bit of feature reduction, so … wouldn't that be a good time to cut off even more of the old cruft? Should we really have a release that supports the CPython versions 3.8 (~October) all the way back to 2.7? Or would Python 2.7-3.7 be enough in 0.29, and only Python 3.[345]-3.8+ for Cython 3.0? (*) Remember, removing legacy support as part of a new release doesn't change anything about the old releases, so they will still keep working for everyone who wants Python 2.x support, even long after January 1st, 2020. As I wrote in the ticket, I would not object to anyone backporting fixes to 0.29 that we implement for newer releases, and continuing with point releases for it whenever there is something to release in the legacy maintenance branch. (And as long as there aren't any breaking changes in Py3.8, 0.29.x might also work there.) What do you think? Stefan [1] https://github.com/cython/cython/issues/2800 [2] https://github.com/cython/cython/milestone/58 (*) We currently support Py3.3+ for Py3. Py3.4 is expected to reach its EOL in March, but still seems worth supporting for another while. Not sure about 3.3, probably an "as long as it doesn't hurt" case. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.29.4 fixes exception handling for division by zero
Hi all, I just released Cython 0.29.4 with an important bug fix(*) for division operations that involve constant numerators. If the denominator turns out to be zero at runtime, it could happen that no ZeroDivisionError gets raised, and instead an arbitrary result returned. https://github.com/cython/cython/issues/2820 Upgrading is recommended to restore correctness. Stefan (*) You could argue that support for division by zero is a rather oustanding feature, but … I think the surprise factor outweighs the coolness here. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] More syntax warnings – a perfect task for a new contributor
Hi all, there is currently a discussion on python-dev about letting the compiler generate more SyntaxWarnings for "obviously unintended" code. https://mail.python.org/pipermail/python-dev/2019-January/156113.html Serhiy Storchaka has written a proof-of-concept patch for CPython here: https://bugs.python.org/issue15248 I think Cython could also be improved in that regard. While we reject a few more things at compile time than CPython's compiler, we tend to be very relaxed about things that seem wrong at compile time and would most likely fail at runtime (but might not get executed). Cython is a much smarter compiler than the one in CPython, and could base such warnings on its much greater knowledge about code and types. This would be an excellent task for someone who wants to start contributing to Cython, because the infrastructure is mostly there and each little warning should be relatively easy to add together with a unit test and is a little, self-contained task all by itself. I created a ticket for it: https://github.com/cython/cython/issues/2814 Anyone interested? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hooking tp_clear()
Jeroen Demeyer schrieb am 09.01.19 um 15:43: > (reviving this thread after I thought about it some more...) > > As I mentioned in the original post, I want something like __dealloc__ but > with access to a particular cdef attribute (representing a Python object). > Since Python attributes of cdef classes may have been cleared by tp_clear, > they cannot be accessed in __dealloc__. > > I suggested hooking tp_clear to do the cleanup but maybe there is a simpler > solution: do that cleanup in __dealloc__ anyway but make sure that this > specific attribute is not cleared in tp_clear. > > So really I want something like @cython.no_gc_clear (just found out about > that now) but for a specific attribute only. > > There could be new syntax like > > cdef class X: > cdef no_gc_clear object attr > > This is a better solution than a blanket @cython.no_gc_clear: in my code, > I'm sure that there are no reference cycles involving only that attribute, > but there may be reference cycles involving other attributes. > > This should be fairly easy to implement. Any comments before I submit a PR? I would like to avoid adding Cython specific features that are better served by the general finalisation mechanism of PEP-442. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Debugging Cython programs with PDB
Hi! Nice idea. Prakhar Goel schrieb am 29.12.18 um 15:02: > I wanted to test the waters on this idea. > > The idea is to allow debugging Cython programs with PDB. This relies > on making call-backs to trigger the sys.trace functionality every now > and then. It is very similar to how profiling is handled so I figured > that a bunch of the infra for this is already present. It involves a > lot of overhead so it would only be enabled for a special debug mode > but could be very useful in that capacity since it would allow using > identical tools for both the Python and Cython modules. Additionally > other people have built debugging tools on top of sys.settrace so some > of the more advanced debugging facilities would also become at least > partially available for Cython modules. > > It would be a fair bit of work. I'm not asking you to do this work of > course! Just looking for some feedback here. The rough pieces as far > as I can tell: > >> Adding in a flag for a special debug mode that calls Python's trace >> functions appropriately. > >> code-gen for all the trace calls. Most of this should already be in place. I wouldn't want a special flag for it, just extend the existing tracing support (which works for coverage analysis, mostly). >> Ideally we want some kind of wrapped call-frame that exports the current set >> of variables. Cython knows quite a bit about the variables currently in >> scope thanks to the cdef declarations. Not only those, it actually knows all defined names in the current scope (except for the global scope, which is extensible at runtime). >> Ideally we should export these over to the Python side (with appropriate >> wrappers perhaps for the raw-c structs?). This is probably the most work but >> I'm hoping that getting a MVP here wouldn't be too hard... There is support for locals(), which does most of what you need here. Regarding frames, Cython's tracing code uses frames already. I think the important steps would be: 1) Make sure there is only one frame per function execution. Currently, there are cases where we use one per source code line, which is an ugly hack in lack of proper Cython line number reporting support. This can be achieved by finalising the lnotab support in PR-93, and then cleaning up the way we create code objects and frames for tracing. https://github.com/cython/cython/pull/93 2) Extend the meta-data that the code objects (which we already create, just mostly empty) provide for each of the functions, since that is where PDB gets its introspection details from. 3) Make the frame refer to the current "locals", Preferably in a way that only generates the dict on request, not for all frames. In the worst case, that could be something to enable with a C compile time define, as we do for tracing in general. 4) Trial and error fixing. :) So, yeah, there is a bit of work involved, but it seems doable and worth doing. Are you interested in giving this a try? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Python subinterpreters support problem in v0.29
Ricardo Dias schrieb am 11.12.18 um 23:16: > On 11/12/18 19:39, Stefan Behnel wrote: >> Ricardo Dias schrieb am 10.12.18 um 14:42: >>> In the recent Cython 0.29 version was introduced a commit [1] that >>> hinders the usage of python subinterpreters. >>> >>> I discovered this the hard way when suddenly a component I was working >>> on started to crash. The component in question is the ceph-mgr daemon >>> from the Ceph project [2]. >>> >>> Python subinterpreters are the basic building block for the >>> plugin/module architecture of ceph-mgr. Each "manager module" runs in >>> its own python subinterpreter. Furthermore, all python bindings for the >>> client libraries of Ceph, such as librados, librbd, libcephfs, and >>> librgw, are implemented as Cython modules, and in the particular case of >>> librados, all ceph-mgr plugin modules import the rados Cython module >>> upon initialization. >>> >>> In practice, with Cython 0.29 we can only load one module, because the >>> following modules will refuse to load. >>> >>> After discovering this issue, we "temporarily" prevent the issue by >>> restricting the version of Cython as a dependency [3]. But we don't want >>> to keep this restriction indefinitely and would prefer a fix from the >>> Cython side. >>> >>> Do you think it's feasible to implement a flag to disable the safe guard >>> introduced in [1]? That way we could re-enable subinterpreters at our >>> own risk. >>> >>> [1] >>> https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50 >>> [2] https://github.com/ceph/ceph >>> [3] https://github.com/ceph/ceph/pull/25328 >> >> My guess is that your modules just silently leaked object references and >> memory with the previous Cython versions. That is why we now inserted a >> guard that detects cases where the module init function is executed >> multiple times, which would overwrite the state of the previous run. The >> shared library of an extension module is only loaded once, so any global C >> state is shared for the entire process, regardless of how often CPython >> calls the module init function. > > I assume that the problem with subinterpreters occurs when a cython > module declares some static/global variables, which might cause > undesirable side-effects upon module loading in several subinterpreters. > > I believe the cython modules that we develop in Ceph do not declared any > global state, and therefore the modules have been working good when > loaded by several subinterpreters. This question already came up recently in the cython-users mailing list (where I'd say it belongs). Since I couldn't find a web version of my reply anywhere outside of google-groups, I'll copy my reply below: """ > We use a lot of scratch > interpreters to keep independent tasks isolated from one another which is > when I ran into the error. In theory, PEP-489 would allow this. https://www.python.org/dev/peps/pep-0489/ In practice, it's not that easy, because avoiding global state requires a lot of work and makes some things slower, especially access to module globals. It also cannot be done in normal C in all cases, because global cdef functions simply do not have access to non-static module globals, since you cannot pass an additional context into them without changing their signature. Imagine a (statically defined global) C callback function that tries to do a type check against a (module/runtime instance specific) extension type. Which is the right type to check against in that case? Depending on how such a function gets called, there might not even be a thread-local to recover its global module context from. These things could still be done by generating module specific C functions at runtime, but then you're really leaving the platform independent sector of C. These problems are not specific to Cython. Most CPython extension modules are not prepared to work with multiple interpreters, mostly for the same reasons. These issues can be worked around in some cases by carefully crafting the global state in a way that allows it to be shared across interpreters, but this is such a special case that Cython rather assumes that the module init code is not safe to be re-executed. And the code that Cython generates internally is also far from PEP-489 clean. I started doing some work towards getting rid of globals here: https://github.com/cython/cython/pull/1919 And, what a surprise, it's not easy. It turned out that PEP-489 isn't really enough here, so PEP-573 was written to investigate the details and resolve the remaining issues. https://www.pyt
Re: [Cython] Python subinterpreters support problem in v0.29
Ricardo Dias schrieb am 10.12.18 um 14:42: > In the recent Cython 0.29 version was introduced a commit [1] that > hinders the usage of python subinterpreters. > > I discovered this the hard way when suddenly a component I was working > on started to crash. The component in question is the ceph-mgr daemon > from the Ceph project [2]. > > Python subinterpreters are the basic building block for the > plugin/module architecture of ceph-mgr. Each "manager module" runs in > its own python subinterpreter. Furthermore, all python bindings for the > client libraries of Ceph, such as librados, librbd, libcephfs, and > librgw, are implemented as Cython modules, and in the particular case of > librados, all ceph-mgr plugin modules import the rados Cython module > upon initialization. > > In practice, with Cython 0.29 we can only load one module, because the > following modules will refuse to load. > > After discovering this issue, we "temporarily" prevent the issue by > restricting the version of Cython as a dependency [3]. But we don't want > to keep this restriction indefinitely and would prefer a fix from the > Cython side. > > Do you think it's feasible to implement a flag to disable the safe guard > introduced in [1]? That way we could re-enable subinterpreters at our > own risk. > > [1] > https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50 > [2] https://github.com/ceph/ceph > [3] https://github.com/ceph/ceph/pull/25328 My guess is that your modules just silently leaked object references and memory with the previous Cython versions. That is why we now inserted a guard that detects cases where the module init function is executed multiple times, which would overwrite the state of the previous run. The shared library of an extension module is only loaded once, so any global C state is shared for the entire process, regardless of how often CPython calls the module init function. I am surprised that your setup didn't crash in any way. Could you explain a bit more how you are using this feature? Are the different subinterpreters running in parallel or sequentially? The ceph repo looks huge. Any pointers where I should start looking? I actually wonder if we could at least support sequential usages through the module cleanup mechanism. Once a module is cleaned up and all global objects freed, calling the module init function again should be ok. Apart from that, here is the feature ticket for module specific global state: https://github.com/cython/cython/issues/2343 Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] longintrepr.h issues with MinGW
Robert Bradshaw schrieb am 24.10.2018 um 01:44: > On Tue, Oct 23, 2018 at 10:22 PM Stefan Behnel wrote: >> My guess is that MinGW is just fairly rarely used to build CPython >> extensions overall. >> >>> but perhaps we should even consider releasing bugfixes for previous >>> Cython versions. >> >> You mean a 0.28.6, or even older than that? > > I wish there were a good way to see what users are actually using in the > wild... At least from what I hear from time to time, many people tend to stick to whatever was there when they started using Cython, and then manually upgrade in irregular intervals whenever they feel like they need something from a newer version. Especially in-house, something like, say, 0.23 might still be in use. But then, we don't really need to care about in-house use of this, because people control their environment and their C compiler there. If they didn't experience this bug yet, then it's probably not a problem for them at all. Regarding more widely used libraries that are used across different environments, I'd say that they'd probably have added support for CPython 3.7 by now. Anything before Cython 0.28 didn't even compile there, and 0.29 is recommended for it, so I think we can somewhat safely assume that this bug is most pressing for those two versions. Thanks to Mark Shannon for raising the baseline here. :) >> As I said, MinGW does not seem to be used very widely… > > Yes, but it is used, and this is (IMHO) a really severe bug. Agreed. Let's release a 0.28.6 with this fix and move on to 0.29.1. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] longintrepr.h issues with MinGW
Robert Bradshaw schrieb am 22.10.2018 um 11:03: > Given that https://bugs.python.org/issue4709 results in extension > modules that seem to work, but silently produce completely incorrect > answers, I'm thinking we should either disable our long-unpacking code > on these platforms, or at the very least give a runtime error if we > detect issues like sizeof(void*) != SIZEOF_VOID_P (or both). We could try to detect MinGW-64 and set the "MS_WIN64" macro before importing the CPython headers. That seems to be the right fix. Can't say if we also need to define the "MS_WIN32" macro, but would probably be better to have it. CPython defines both for MSVC. There is a "__MINGW64__" macro to detect MinGW-64, it seems. > I'm not sure what has made this issue pop up more lately (Python 3), My guess is that MinGW is just fairly rarely used to build CPython extensions overall. > but perhaps we should even consider releasing bugfixes for previous > Cython versions. You mean a 0.28.6, or even older than that? As I said, MinGW does not seem to be used very widely… Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Failing test reimport_from_subinterpreter
Jeroen Demeyer schrieb am 21.10.2018 um 12:47: > On 2018-10-16 20:30, Stefan Behnel wrote: >> Difficult to say why this would fail to find the package. Could it be an >> import path problem? Current directory missing from the PYTHONPATH or >> something like that? > > The problem is that the error depends on the proverbial "phase of the > moon". When running the full testsuite, I always get the error. But any > attempt at debugging it makes the error go away. So it must be something > subtle, like a race condition or bad interaction with another test. Maybe print "sys.path" and "sys.meta_path" in a couple of places? Before and while running the test suite? There are pyximport tests, for example, and "pyxim..." < "reimp...". It does sound like something in the import system is leaking from previous tests. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Failing test reimport_from_subinterpreter
Jeroen Demeyer schrieb am 16.10.2018 um 12:42: > I'm getting various reports of the test reimport_from_subinterpreter > failing (within Sage, Cython 0.29, Python 2.7.15). > > Annoyingly, the problem can only be reproduced when running the full Cython > testsuite, not when running the test in isolation. I'll continue to > investigate, but I'm already posting it here in case somebody has an idea. > The error is not very enlightening (is there a way to get the exception > from the subinterpreter?): > > End-to-end reimport_from_subinterpreter ... > /home/jdemeyer/sage-test/local/bin/python2 setup.py build_ext --inplace > Compiling package/subtest.pyx because it changed. > Compiling subtest.pyx because it changed. > [1/2] Cythonizing package/subtest.pyx > [2/2] Cythonizing subtest.pyx > running build_ext > building 'package.subtest' extension > creating build > creating build/temp.linux-ppc64le-2.7 > creating build/temp.linux-ppc64le-2.7/package > gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused > -fPIC -I/home/jdemeyer/sage-test/local/include/python2.7 -c > package/subtest.c -o build/temp.linux-ppc64le-2.7/package/subtest.o > gcc -pthread -shared -L/home/jdemeyer/sage-test/local/lib > -Wl,-rpath,/home/jdemeyer/sage-test/local/lib > -L/home/jdemeyer/sage-test/local/lib > -Wl,-rpath,/home/jdemeyer/sage-test/local/lib > build/temp.linux-ppc64le-2.7/package/subtest.o > -L/home/jdemeyer/sage-test/local/lib -lpython2.7 -o > /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/run/reimport_from_subinterpreter/package/subtest.so > > building 'subtest' extension > gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused > -fPIC -I/home/jdemeyer/sage-test/local/include/python2.7 -c subtest.c -o > build/temp.linux-ppc64le-2.7/subtest.o > gcc -pthread -shared -L/home/jdemeyer/sage-test/local/lib > -Wl,-rpath,/home/jdemeyer/sage-test/local/lib > -L/home/jdemeyer/sage-test/local/lib > -Wl,-rpath,/home/jdemeyer/sage-test/local/lib > build/temp.linux-ppc64le-2.7/subtest.o -L/home/jdemeyer/sage-test/local/lib > -lpython2.7 -o > /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/run/reimport_from_subinterpreter/subtest.so > > > /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/Cython/Compiler/Main.py:367: > FutureWarning: Cython directive 'language_level' not set, using 2 for now > (Py2). This will change in a later release! File: > /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/run/reimport_from_subinterpreter/package/subtest.pyx > > tree = Parsing.p_module(s, pxd, full_module_name) > /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/Cython/Compiler/Main.py:367: > FutureWarning: Cython directive 'language_level' not set, using 2 for now > (Py2). This will change in a later release! File: > /home/jdemeyer/sage-test/local/var/tmp/sage/build/cython-0.29/src/TEST_TMP/run/reimport_from_subinterpreter/subtest.pyx > > tree = Parsing.p_module(s, pxd, full_module_name) > > > /home/jdemeyer/sage-test/local/bin/python2 -c "import subtest; > subtest.run_main()" > Module loaded: package.subtest > > > > /home/jdemeyer/sage-test/local/bin/python2 -c "import subtest; > subtest.run_sub()" > > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named package The error message is right above. ^^^ > Traceback (most recent call last): > File "", line 1, in > File "subtest.pyx", line 44, in subtest.run_sub > assert 0 == run_in_subinterpreter(b'import package') > AssertionError Difficult to say why this would fail to find the package. Could it be an import path problem? Current directory missing from the PYTHONPATH or something like that? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.29 final released
Dear Cythonistas, after half a year of development, many community pull requests and a lot of feedback and good ideas in online discussions and at conferences, I'm proud to release Cython 0.29. This is a major feature release that comes with many great improvements and several important bug fixes. See the long list of changes below. Download: https://pypi.org/project/Cython/0.29/ https://github.com/cython/cython/releases/tag/0.29 Changelog: https://github.com/cython/cython/blob/0.29/CHANGES.rst Foresight: Given that Cython has been in critical production use all over the world for several years, but never found the perfect time for a 1.0 version bump, we designate this to become the last 0.x release of the project and decided to skip the 1.0 release which the 0.x series has long represented anyway. Planning has already started [1] for the next major release, titled "3.0". It will finally switch the default Cython language level from Py2 to Py3, to match what users expect from a Python compiler these days without additional options or configuration. Cython 2 code will continue to be supported as before with the directive "language_level=2", although the future default (called "3str", Py3 with str literals) is already available in the 0.29 release, so users might prefer switching to that. We're happy to hear your feedback on the proposed changes. [1] https://github.com/cython/cython/milestone/58 Have fun, Stefan 0.29 (2018-10-14) = Features added -- * PEP-489 multi-phase module initialisation has been enabled again. Module reloads in other subinterpreters raise an exception to prevent corruption of the static module state. * A set of ``mypy`` compatible PEP-484 declarations were added for Cython's C data types to integrate with static analysers in typed Python code. They are available in the ``Cython/Shadow.pyi`` module and describe the types in the special ``cython`` module that can be used for typing in Python code. Original patch by Julian Gethmann. (Github issue #1965) * Memoryviews are supported in PEP-484/526 style type declarations. (Github issue #2529) * ``@cython.nogil`` is supported as a C-function decorator in Python code. (Github issue #2557) * Raising exceptions from nogil code will automatically acquire the GIL, instead of requiring an explicit ``with gil`` block. * C++ functions can now be declared as potentially raising both C++ and Python exceptions, so that Cython can handle both correctly. (Github issue #2615) * ``cython.inline()`` supports a direct ``language_level`` keyword argument that was previously only available via a directive. * A new language level name ``3str`` was added that mostly corresponds to language level 3, but keeps unprefixed string literals as type 'str' in both Py2 and Py3, and the builtin 'str' type unchanged. This will become the default in the next Cython release and is meant to help user code a) transition more easily to this new default and b) migrate to Python 3 source code semantics without making support for Python 2.x difficult. * In CPython 3.6 and later, looking up globals in the module dict is almost as fast as looking up C globals. (Github issue #2313) * For a Python subclass of an extension type, repeated method calls to non-overridden cpdef methods can avoid the attribute lookup in Py3.6+, which makes them 4x faster. (Github issue #2313) * (In-)equality comparisons of objects to integer literals are faster. (Github issue #2188) * Some internal and 1-argument method calls are faster. * Modules that cimport many external extension types from other Cython modules execute less import requests during module initialisation. * Constant tuples and slices are deduplicated and only created once per module. (Github issue #2292) * The coverage plugin considers more C file extensions such as ``.cc`` and ``.cxx``. (Github issue #2266) * The ``cythonize`` command accepts compile time variable values (as set by ``DEF``) through the new ``-E`` option. Patch by Jerome Kieffer. (Github issue #2315) * ``pyximport`` can import from namespace packages. Patch by Prakhar Goel. (Github issue #2294) * Some missing numpy and CPython C-API declarations were added. Patch by John Kirkham. (Github issues #2523, #2520, #2537) * Declarations for the ``pylifecycle`` C-API functions were added in a new .pxd file ``cpython.pylifecycle``. * The Pythran support was updated to work with the latest Pythran 0.8.7. Original patch by Adrien Guinet. (Github issue #2600) * ``%a`` is included in the string formatting types that are optimised into f-strings. In this case, it is also automatically mapped to ``%r`` in Python 2.x. * New C macro ``CYTHON_HEX_VERSION`` to access Cython's version in the same style as ``PY_HEX_VERSION``. * Constants in ``libc.math`` are now declared as ``const`` to simplify their handling. * An additional ``check_size`` clause
[Cython] Cython 0.29 final release candidate
Hi again! Here's a second and final release candidate. https://github.com/cython/cython/releases/tag/0.29rc2 We extended the "language_level" directive with a new "3str" option and removed the previously added "str_is_str" directive, which had the same intention but lead mostly to confusion in the last RC. This RC also contains a fix for another NumPy related warning about mismatching object sizes. I'll collect more feedback for the next two weeks, and then push the final release. Download: https://github.com/cython/cython/archive/0.29rc2.zip https://github.com/cython/cython/archive/0.29rc2.tar.gz Updated changelog: https://github.com/cython/cython/blob/0.29rc2/CHANGES.rst Stefan Stefan Behnel schrieb am 16.09.2018 um 17:48: > Hi all, > > after half a year of development, many community pull requests and a lot of > feedback and good ideas in online discussions and at conferences, I'm proud > to release the first beta of Cython 0.29. This is a major feature release > that comes with many great improvements and several important bug fixes. > See the long list of changes below. > > Please give it some testing to help us quickly advance to the final release. > > Download: > https://github.com/cython/cython/releases/tag/0.29b1 > > Changelog: > https://github.com/cython/cython/blob/0dcb5d1930e573caa8494fe838c4c2cd4e2041f2/CHANGES.rst > > > Foresight: > Given that Cython has been in critical production use all over the world > for several years, but never found the perfect time for a 1.0 version bump, > we designate this to become the last 0.x release of the project and decided > to skip the 1.0 release which the 0.x series has long represented anyway. > > Planning has already started [1] for the next major release, titled "3.0". > It will finally switch the default Cython language level from Py2 to Py3, > to match what users expect from a Python compiler these days without > additional options or configuration. Cython 2 code will continue to be > supported as before with the directive "language_level=2", although there > are ideas how to help with the modernisation. We're happy to hear your > feedback. > > [1] https://github.com/cython/cython/milestone/58 > > > Have fun, > > Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Preparing the language level change
Stefan Behnel schrieb am 21.09.2018 um 09:38: > There are two parts of > information here, so maybe we should actually split them internally (in > "Main.Context.set_language_level() ?) and keep the language_level = 3 but > just avoid the "unicode_literals" part. I thought about this some more. The real question is: what should the world be like once Cython 3.0 is out? What we want to do for Cy3 is to change the default behaviour, which mostly impacts Py2/3 ported code. That's why we now warn about a missing "language_level" switch. When people respond to that and set the language level explicitly, that makes them opt out of the default change. Perfect so far. Do we then still want to have a separate option floating around that says "but I want str" ? I don't think so, because that will be the default in Cy3 anyway. Thus, what I think we want is that people either specify the language level explicitly, or get the new default. And to get the new default *now*, in a future proof way, I think the best option is to set an explicit language level, not a separate directive. Thus, I now agree with Jeroen's early intuition that a new language level switch is the right interface. I'll change the implementation to do what I wrote in the quoted paragraph above and push a new RC. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Preparing the language level change
John Ehresman schrieb am 25.09.2018 um 16:27: > On 9/25/18 1:24 AM, Stefan Behnel wrote: >> I started off with "unicode_literals=False", and then renamed it because >> this name didn't cover the change of "str" to "unicode" (i.e. renaming the >> usages of the builtin type internally, so that "str(x)" actually calls >> "unicode(x)"). >> >> Looks like this bikeshed needs painting, so let's have a quick(!) >> discussion or vote. >> >> - should this feature touch the builtin type at all? >> >> - more opinions on the name? > > I think the name str_is_str will be confusing to developers who begin with > Python 3 because they'll probably assume str is unicode. It is unicode for them, at least in Py3. In Py2, it's str. Thus the name "str_is_str", it's "str" in both versions. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Preparing the language level change
Daniele Nicolodi schrieb am 25.09.2018 um 00:28: > On 24-09-2018 14:05, Stefan Behnel wrote: >> I added a new directive "str_is_str=True" which can be combined with >> "language_level=3" to get the desired behaviour. It keeps the 'str' builtin >> type as it is (it would otherwise become 'unicode' with level 3) and keeps >> unprefixed string literals as type 'str' in Py2 and Py3. Everything else >> should depend solely on the language_level switch. > > For consistency with the CPython > > from __future__ import unicode_literals > > wouldn't it be better to call this directive "str_literals"? > > I realize there isn't 100% overlap in the functionality of the two, but > I find the "str_is_str" name not very descriptive. I started off with "unicode_literals=False", and then renamed it because this name didn't cover the change of "str" to "unicode" (i.e. renaming the usages of the builtin type internally, so that "str(x)" actually calls "unicode(x)"). Looks like this bikeshed needs painting, so let's have a quick(!) discussion or vote. - should this feature touch the builtin type at all? - more opinions on the name? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.29 rc 1 - Re: Cython 0.29 beta 1 released
Hi again! Here's a release candidate. Please give it some final testing. https://github.com/cython/cython/tree/0.29rc1 Download: https://github.com/cython/cython/archive/cf7b60ff6814ce8faf7fab2990c423452faa2f0a.zip Updated changelog: https://github.com/cython/cython/blob/0.29rc1/CHANGES.rst Stefan Stefan Behnel schrieb am 16.09.2018 um 17:48: > Hi all, > > after half a year of development, many community pull requests and a lot of > feedback and good ideas in online discussions and at conferences, I'm proud > to release the first beta of Cython 0.29. This is a major feature release > that comes with many great improvements and several important bug fixes. > See the long list of changes below. > > Please give it some testing to help us quickly advance to the final release. > > Download: > https://github.com/cython/cython/releases/tag/0.29b1 > > Changelog: > https://github.com/cython/cython/blob/0dcb5d1930e573caa8494fe838c4c2cd4e2041f2/CHANGES.rst > > > Foresight: > Given that Cython has been in critical production use all over the world > for several years, but never found the perfect time for a 1.0 version bump, > we designate this to become the last 0.x release of the project and decided > to skip the 1.0 release which the 0.x series has long represented anyway. > > Planning has already started [1] for the next major release, titled "3.0". > It will finally switch the default Cython language level from Py2 to Py3, > to match what users expect from a Python compiler these days without > additional options or configuration. Cython 2 code will continue to be > supported as before with the directive "language_level=2", although there > are ideas how to help with the modernisation. We're happy to hear your > feedback. > > [1] https://github.com/cython/cython/milestone/58 > > > Have fun, > > Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Preparing the language level change - Re: [cython-users] Cython 0.29 beta 1 released
Stefan Behnel schrieb am 21.09.2018 um 17:46: > Robert Bradshaw schrieb am 21.09.2018 um 17:30: >> I agree that this doesn't really feel like a language-level thing. >> >> There seem three desired behaviors here: >> >> language_level=2 where currently "abc" is always a bytes object >> langauge_level=3 where currently "abc" is always a unicode object >> >> and a third option, where "abc" is a str object (depending on the >> runtime). We should support all three of these modes. > > Correction: with "language_level=2", unprefixed string literals are "str", > i.e. "bytes" in Py2 and "unicode" in Py3. "language_level=3" always makes > them unicode strings, and that's what Jeroen was referring to. > > It's really difficult to guess what the most common use case is here. The > only reason why the type changing unprefixed "str" literals are relevant is > that Py2 cannot handle Unicode in some cases (and Unicode strings are > memory hogs in Py2), but Py3 requires Unicode in most cases (and handles it > efficiently). So, the problem here is really Py2, and the problem will go > away as soon as we dump support for it. Until then, however, we're trying > to find a solution to make the language level switch bearable and easy the > transition. I added a new directive "str_is_str=True" which can be combined with "language_level=3" to get the desired behaviour. It keeps the 'str' builtin type as it is (it would otherwise become 'unicode' with level 3) and keeps unprefixed string literals as type 'str' in Py2 and Py3. Everything else should depend solely on the language_level switch. https://github.com/cython/cython/commit/cea42915c5e9ea1da9187aa3c55f3f16d04ba1e3 I think we're now set for the release. I'll prepare a candidate. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Preparing the language level change - Re: [cython-users] Cython 0.29 beta 1 released
Jeroen Demeyer schrieb am 20.09.2018 um 10:59: > On 2018-09-20 10:15, Jeroen Demeyer wrote: >> On 2018-09-20 07:55, Stefan Behnel wrote: >>> I can see that this would be helpful. It's unfortunate, though, that this >>> would introduce a temporary option that complicates the current integer >>> "language_level". >> >> Does it really have to be an integer? We can make it a string >> internally, but allow it to be set as integer too. That's just a matter >> of putting "language_level = str(language_level)" in the appropriate places. That's pretty much what I meant with "complicates". :) It's very easy to compare "language_level < 3" now. With strings, not so much. In fact, maybe that's not even the way to go. There are two parts of information here, so maybe we should actually split them internally (in "Main.Context.set_language_level() ?) and keep the language_level = 3 but just avoid the "unicode_literals" part. >> Once it's a string, we can add an additional language level "3str". I'm >> willing to make a PR if you agree with this strategy. I think a PR that takes care of splitting the two parts would be worth looking at. Then, make sure we do the right checks for either of them in the right places. The language level isn't always the right indication for specific behaviour. > Further brainstorming: you could also re-purpose language_level as a set of > flags describing various features, like the __future__ flags. I faintly remember proposals for a "from __past__ import …". One more use case right here. But the problem is that there are really different things to support. One is setting the behaviour globally, e.g. via setup.py. The other is setting it on a file-by-file basis. I expect projects to settle on one way to write code, so a global setting is definitely required, and that's the language level. But then, there are probably cases where this is worth overriding for a single file or two. And the language level doesn't seem right for that. Explicitly opting out of "unicode_literals" would be cleaner. So, maybe just allow "cython: unicode_literals=False" ? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Preparing the language level change - Re: [cython-users] Cython 0.29 beta 1 released
Jeroen Demeyer schrieb am 18.09.2018 um 10:23: > On 2018-09-16 17:48, Stefan Behnel wrote: >> * Cython now emits a warning when no ``language_level`` (2 or 3) is set >> explicitly > > Currently, language_level=3 breaks a lot of code due to unicode_literals. > As I mentioned a few times, it would be good to have a way to specify > "language_level=3 except for unicode_literals". Ideally, this option should > exist before 0.29 is released. I can see that this would be helpful. It's unfortunate, though, that this would introduce a temporary option that complicates the current integer "language_level". Meaning, we'd introduce something now that we then can't get rid of for…ever. And it seems to me like a separate directive wouldn't solve this either because it's not really orthogonal to the language level. Any idea? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Safer exception handling
Robert Bradshaw schrieb am 17.09.2018 um 15:44: > One of the pain points in Cython is that one must explicitly annotate > non-object returning functions with except clauses. Would it be worth > trying to change the default here, making exception-suppressing opt-in > rather than opt-out? > > There are a couple of open questions, e.g. > * What would the syntax be? > * What about extern functions? > * What would the performance impact be? Could it be mitigated? One more point: what about nogil functions? They can, in theory, raise exceptions, but they almost never do in practice. Given that the default for return type annotations is exception propagation (i.e. Python semantics) now, mixing that into a nogil function means that the caller would almost always check for exceptions needlessly. And we do not currently have a way to say "no exception return value" to get rid of that check. That's something we'd need, especially if we change the default. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Safer exception handling
Robert Bradshaw schrieb am 17.09.2018 um 15:44: > One of the pain points in Cython is that one must explicitly annotate > non-object returning functions with except clauses. Would it be worth > trying to change the default here, making exception-suppressing opt-in > rather than opt-out? > > There are a couple of open questions, e.g. > * What would the syntax be? > * What about extern functions? > * What would the performance impact be? Could it be mitigated? Probably a good idea, and worth a 3.0 ticket. Note that the default for return type annotations is to automatically propagate exceptions, in order to keep the drop from Python semantics gentle. @cython.cfunc def func() -> cython.int: ... is read as "except? -1", unless declared otherwise. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.29 beta 1 released
Hi all, after half a year of development, many community pull requests and a lot of feedback and good ideas in online discussions and at conferences, I'm proud to release the first beta of Cython 0.29. This is a major feature release that comes with many great improvements and several important bug fixes. See the long list of changes below. Please give it some testing to help us quickly advance to the final release. Download: https://github.com/cython/cython/releases/tag/0.29b1 Changelog: https://github.com/cython/cython/blob/0dcb5d1930e573caa8494fe838c4c2cd4e2041f2/CHANGES.rst Foresight: Given that Cython has been in critical production use all over the world for several years, but never found the perfect time for a 1.0 version bump, we designate this to become the last 0.x release of the project and decided to skip the 1.0 release which the 0.x series has long represented anyway. Planning has already started [1] for the next major release, titled "3.0". It will finally switch the default Cython language level from Py2 to Py3, to match what users expect from a Python compiler these days without additional options or configuration. Cython 2 code will continue to be supported as before with the directive "language_level=2", although there are ideas how to help with the modernisation. We're happy to hear your feedback. [1] https://github.com/cython/cython/milestone/58 Have fun, Stefan 0.29 beta 1 (2018-09-16) Features added -- * PEP-489 multi-phase module initialisation has been enabled again. Module reloads raise an exception to prevent corruption of the static module state. * A set of ``mypy`` compatible PEP-484 declarations were added for Cython's C data types to integrate with static analysers in typed Python code. They are available in the ``Cython/Shadow.pyi`` module and describe the types in the special ``cython`` module that can be used for typing in Python code. Original patch by Julian Gethmann. (Github issue #1965) * Memoryviews are supported in PEP-489 style type declarations. (Github issue #2529) * Raising exceptions from nogil code will automatically acquire the GIL, instead of requiring an explicit ``with gil`` block. * ``@cython.nogil`` is supported as a C-function decorator in Python code. (Github issue #2557) * ``cython.inline()`` supports a direct ``language_level`` keyword argument that was previously only available via a directive. * In CPython 3.6 and later, looking up globals in the module dict is almost as fast as looking up C globals. (Github issue #2313) * For a Python subclass of an extension type, repeated method calls to non-overridden cpdef methods can avoid the attribute lookup in Py3.6+, which makes them 4x faster. (Github issue #2313) * (In-)equality comparisons of objects to integer literals are faster. (Github issue #2188) * Some internal and 1-argument method calls are faster. * Modules that cimport many external extension types from other Cython modules execute less import requests during module initialisation. * The coverage plugin considers more C file extensions such as ``.cc`` and ``.cxx``. (Github issue #2266) * The ``cythonize`` command accepts compile time variable values (as set by ``DEF``) through the new ``-E`` option. Patch by Jerome Kieffer. (Github issue #2315) * ``pyximport`` can import from namespace packages. Patch by Prakhar Goel. (Github issue #2294) * Some missing numpy and CPython C-API declarations were added. Patch by John Kirkham. (Github issues #2523, #2520, #2537) * Declarations for the ``pylifecycle`` C-API functions were added in a new .pxd file ``cpython.pylifecycle``. * The Pythran support was updated to work with the latest Pythran 0.8.7. Original patch by Adrien Guinet. (Github issue #2600) * ``%a`` is included in the string formatting types that are optimised into f-strings. In this case, it is also automatically mapped to ``%r`` in Python 2.x. * New C macro ``CYTHON_HEX_VERSION`` to access Cython's version in the same style as ``PY_HEX_VERSION``. Bugs fixed -- * The exception handling in generators and coroutines under CPython 3.7 was adapted to the newly introduced exception stack. Users of Cython 0.28 who want to support Python 3.7 are encouraged to upgrade to 0.29 to avoid potentially incorrect error reporting and tracebacks. * Crash when importing a module under Stackless Python that was built for CPython. Patch by Anselm Kruis. (Github issue #2534) * 2-value slicing of typed sequences failed if the start or stop index was None. Patch by Christian Gibson. (Github issue #2508) * Multiplied string literals lost their factor when they are part of another constant expression (e.g. 'x' * 10 + 'y' => 'xy'). * String formatting with the '%' operator didn't call the special ``__rmod__()`` method if the right side is a string subclass that implements it. (Python issue 28598) *
Re: [Cython] Cython 3.0 and "unicode_literals"
Jeroen Demeyer schrieb am 22.08.2018 um 13:10: > On 2018-08-19 08:26, Stefan Behnel wrote: >> Should we make that a new directive rather than a language level? Like >> "py2_str=str"? That would allow its use together with language_level=3 >> already in the next release. > > With a new new directive, you also run into compatibility problems. What > should the default be? py2_str=str or py2_str=unicode? The former breaks > code assuming that it's unicode and the latter doesn't really solve > anything: stuff will still break when language_level=3 becomes the default. > > My proposal is a new setting language_level=3str (meaning: everything that > language_level=3 does, except unicode_literals) and make that the default. > That way, you keep full compatibility with code already setting the > language_level. You also have reasonably good chances that code that > currently uses the implicit language_level=2 will continue to work with > language_level=3str. I thought about this some more and I like the idea. There's still the true-division issue, but strings are certainly the biggest blocker in migrations. As long as people want to support Python 2.x, "3str" is a way to help them with it, and for Py3-only users, it won't make a difference. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hooking tp_clear()
Jeroen Demeyer schrieb am 07.09.2018 um 10:14:
> On 2018-09-07 06:35, Stefan Behnel wrote:
>> Maybe you actually want "tp_finalize"?
>>
>> https://www.python.org/dev/peps/pep-0442/
>>
>> Cython moves "__del__" methods there in Py3.4+.
>
> First of all, are you sure? I tried to cythonize
>
> cdef class X:
> def __del__(self):
> pass
>
> but the generated C code has
>
> static PyTypeObject __pyx_type_6cytest_X = {
> /* ... */
> #if PY_VERSION_HEX >= 0x030400a1
> 0, /*tp_finalize*/
> #endif
> };
Interesting. I thought it did, but apparently, it doesn't. "__del__"
doesn't currently have any special meaning for extension types. Then let's
change that and give it the same meaning that PEP-442 gives it for Python
classes. Want to write a PR?
> In any case, I want something which works with Python 2.7 too. Could we
> somehow call __del__ from tp_clear() in Python 2.7?
That would be problematic, because tp_clear() is not guaranteed to be
called. Quite the contrary, it's only called if the object is so unlucky to
participate in a reference cycle that gets cleaned up. In that case, we
would have to call it twice (in tp_clear() and tp_dealloc()) in order to
make sure that it always gets called. In Py3.4+, it always gets called
separately before tp_clear() or tp_dealloc().
Can't you just wait a year or two … ? ;)
Or, since it's a new feature, we could just document that it gets called
twice in Py2 and that people should take care of writing their code
accordingly.
OTOH, the way PEP 442 guarantees that it only gets called once is by
setting a flag in the GC header. We could see if we can steal an unused
flag there to do the same thing in Py2.
Stefan
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hooking tp_clear()
Jeroen Demeyer schrieb am 06.09.2018 um 22:54: > Cython's __dealloc__ special method is meant to deal with cleaning up > instances of cdef classes. However, this hooks tp_dealloc() and does not > have meaningful access to Python attributes, since those might have been > cleared by tp_clear(). > > I have a concrete use case where I want something like __dealloc__ but > *before* Python attributes are cleared. So this really belongs in tp_clear(). > > Using a PyObject* attribute in the cdef class with manual reference > counting is not a solution since this attribute could genuinely occur in a > reference cycle. > > So I would suggest to support a __clear__ special method, which would then > be called both by tp_clear() and tp_dealloc(). It's important to note that > this should be idempotent: it will be called at least once before Python > attributes are cleared but it may also be called later. Maybe you actually want "tp_finalize"? https://www.python.org/dev/peps/pep-0442/ Cython moves "__del__" methods there in Py3.4+. > PS: I never really understood the technical difference between tp_clear() > and tp_dealloc(). It seems to me that these serve a very similar purpose: > why can't the garbage collector just call tp_dealloc()? The problem are reference cycles, in which there definitely is a life reference to the object *somewhere* else. Thus, the GC cannot simply deallocate the object, it must try to delete the references instead. This is what "tp_clear" is used for, it clears all references that an object inside of a reference cycle has towards other objects (or at least those that can participate in that cycle). This will (hopefully) trigger a cascade of deallocations along the cycle. If that isn't enough, and there is still a cycle, then the clearing needs to be repeated until all references to the last object in the cycle are cleared. AFAIR, tp_clear() is *only* called by the cyclic garbage collector and not during normal refcounting deallocation. The GC process is: tp_visit() to detect cycles, tp_clear() to break them. tp_dealloc() is then only called indirectly by the normal refcounting cleanup, not directly by the GC. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython 0.29 – or 29.0 ?
Am 18. August 2018 21:11:14 MESZ schrieb Jeroen Demeyer:
>>> Basically I am asking for
>>>
>>> type("foo") is str
>>>
>>> both on Python 2 and Python 3.
>>
>> That's another breaking change
>
>I'm not following here... what I'm asking is how Cython behaves
>currently (with language_level=2).
But only for one specific feature. The combination of features that you want
does not exist yet. And enabling it for language_level=3 would break existing
Py3 code. Meaning, we'd need a new setting here to make this non-breaking, but
in any case, we'd create a new language subset that code would have to be
adapted to.
You removed the part where I said that I consider it a reasonable request, and
I think this would help users to make a migration step towards Py3, while
continuing to support Py2 as long as necessary. Should we make that a new
directive rather than a language level? Like "py2_str=str"? That would allow
its use together with language_level=3 already in the next release.
Stefan
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython 0.29 – or 29.0 ?
Am 18. August 2018 12:06:15 MESZ schrieb Jeroen Demeyer:
>On 2018-08-18 10:52, Robert Bradshaw wrote:
>> Do the existing c_string_type and c_string_encoding directives not
>> cover this usecase?
>
>Those seem to refer to C strings, I am talking about the Python type of
>string literals.
>
>> I suppose what you're asking for is str being the
>> str of the runtime, even if language_level=3 is set.
>
>Basically I am asking for
>
>type("foo") is str
>
>both on Python 2 and Python 3.
That's another breaking change, and might suggest some other adaptations, e.g.
for the type of f-strings. But it does sound like a reasonable request. And,
fingers crossed, it won't be relevant forever.
Stefan
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Preparing Cython 3.0 with breaking changes
Hi all, following the versioning discussion, I created a milestone to collect (breaking) changes that should go into the future Cython 3.0 release. https://github.com/cython/cython/milestone/58 While a major version change is a good time to fix things that, in retrospect, have led us too far in the wrong direction, we are not planning to remove the support for existing code any time soon that relies on the current semantics. For now, it looks like the major breaking changes will be to directive defaults, which means that setting them explicitly to their current configuration (e.g. in setup.py) should keep code working across Cython releases. Feedback welcome. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython 0.29 – or 29.0 ?
Yury V. Zaytsev schrieb am 17.08.2018 um 10:59: > On Fri, 17 Aug 2018, James C. McPherson wrote: >> 1.0 runs the risk of hitting "never install a 1.0 release" habits. > > How about 2.9.0 ;-) ? Sold! :) Proposal: we'll release a 2.9.0 next, which gives a warning that users should explicitly enable "language_level=2" in their setup.py when compiling Py2 code, and then release 3.0 as the next feature release, which requires this option for Py2 code and defaults to compiling with Py3 syntax and semantics. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython 0.29 – or 29.0 ?
Robert Bradshaw schrieb am 16.08.2018 um 00:48: > If we're going to ditch the 0.x, I'd go for 1.0 as well. I'm a huge fan of > semantic versioning. I'm ok with it, just fear that it might become excessive for Cython (ok, I'm the one who proposed jumping to 29.0, but...). Basically, any time we add something to Includes/ or anything to the language, we'd have to increase the minor version, and any time we fix something that might break some person's code, we'd have to increase the major version? Some changes are simply not important enough to merit shouting out the breakage that almost no-one would otherwise notice. And unless we're adding new syntax that was an error before, almost any change in the compiler bears a tiny risk of breaking something for someone. > The primary reasons we kept the 0.x scheme were that > > * We wanted full compatibility with CPython (we're nearly there, or at > least it's safe to say the differences are on par with those between > different versions and implementations of Python), and I've half-heartedly considered it done for years. The rest are bugs, and we have enough of those, also enough that are worse than incomplete Python semantics. The list of Python related tickets in github is short: https://github.com/cython/cython/issues?q=is%3Aissue+is%3Aopen+label%3A%22Python+Semantics%22 and the most relevant one is certainly #1159, which hasn't seen much attention in years: https://github.com/cython/cython/issues/1159 > * We wanted the flexibility to possibly jettison archaic features of the > language or otherwise clean things up. > > It's worth revisiting to see if either of these are still relevant (*and* > likely to be worked on in the near future). It's true that Cython often offers more than one way to do something, also in terms of syntax. There is a certain level of perpetual request for static typing in PEP-489 notation (which is supported), but I don't see it completely replace Cython's own efficient syntax any time soon, it's not a clear champion. We can think about dropping some of the redundant pure Python syntax when we switch to Py3-by-default, maybe next year. But even then, we'd have to keep the Py2-mode around in order to support existing code bases. Hej, that gives us an alternative for the versioning switch. We could release Cython 3.0 when we change the default language level (and require users to select "language_level=2" for legacy code). Definitely a breaking change that merits inceasing the major version, and 3.0 seems very suitable. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.29 – or 29.0 ?
Hello everyone, I had a request during my talk at EuroPython this year to help the aspiring Cython users out there convince their managers to accept Cython in their tech stack. Cython has been around for more than a decade now, has been in heavy production use pretty much all of that time, has proven to have everything it needs as a programming language, and still resorts to a 0.x versioning scheme. I think it's time to leave the 0.x behind us and make it clear that it's a reliable programming language that won't easily change in a major code breaking way anymore. Which version should we choose next? Some projects have adopted year based release schemes, e.g. pip recently jumped from 10.x to 18.0. Browsers follow the marketing path and boldly increase their version number by a whole 1 on every feature release. I'd be happy to follow them and release a Cython 29.0 next, mostly to make it clear that we're not really changing anything from the previous release cycles, but really just move the versioning scheme one place to the left to get rid of the uninformative "0." prefix. What do you think? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython compilation speed vs. installed binary module sizes
Hi all, when building Cython, it compiles a couple of its own modules. I was interested in what this brings for the separate parts of the compiler, so I used the compilation of Cython's compiled modules as benchmark and compiled the separate parts incrementally, using the latest CPython 3.7.0. Baseline, uncompiled Python run: real0m14.972s user0m14.565s sys 0m0.165s Compiling the parser: Cython/Compiler/Scanning.py Cython/Plex/Scanners.py Cython/Plex/Actions.py Cython/Compiler/Parsing.py real0m13.614s user0m13.320s sys 0m0.136s Additionally: Cython/Compiler/Visitor.py Cython/Compiler/FlowControl.py real0m7.857s user0m7.778s sys 0m0.074s Additionally: Cython/Compiler/Code.py Cython/StringIOTree.py real0m7.356s user0m7.068s sys 0m0.102s Only compiling the tree visitor and the control flow analysis: Cython/Compiler/Visitor.py Cython/Compiler/FlowControl.py real0m9.029s user0m8.899s sys 0m0.082s These are the current shared library sizes on my machine: 57040 Cython/Plex/Actions.cpython-37m-x86_64-linux-gnu.so 72592 Cython/StringIOTree.cpython-37m-x86_64-linux-gnu.so 78880 Cython/Runtime/refnanny.cpython-37m-x86_64-linux-gnu.so 86384 Cython/Plex/Scanners.cpython-37m-x86_64-linux-gnu.so 127952 Cython/Compiler/Pythran.cpython-37m-x86_64-linux-gnu.so 227480 Cython/Compiler/Scanning.cpython-37m-x86_64-linux-gnu.so 322136 Cython/Compiler/Visitor.cpython-37m-x86_64-linux-gnu.so 559864 Cython/Tempita/_tempita.cpython-37m-x86_64-linux-gnu.so 598712 Cython/Compiler/FlowControl.cpython-37m-x86_64-linux-gnu.so 902904 Cython/Compiler/Parsing.cpython-37m-x86_64-linux-gnu.so 1133176 Cython/Compiler/Code.cpython-37m-x86_64-linux-gnu.so 4167120 insgesamt This suggests that by only compiling the 6 modules of the scanner, parser, tree visitor and CFA, we get about half the binary sizes, but almost the same speed. One caveat: Being pure Python, Cython does not use all that many of its own language features, so other code might suffer more or less. I also tried lxml's main module, and it compiles about 7% slower when removing Code.so. Given that it's the biggest of the modules, that feels acceptable for me, so I removed Code, Pythran, Lexicon and StringIOTree from the list of compiled modules for now. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Should we start requiring "language_level=2" for .pyx files?
Hi, Python 3 is clearly taking over the world these days, so it starts feeling arcane to require Py2 syntax in .pyx files. Increasingly, it means that people cannot just rename .py files anymore to start optimising them, because the .py file has a high chance of being written in Py3 syntax. Eventually, we will have to switch to Py3 syntax by default in order to follow what most people are (or will be) used to. As a transition, I think we could start warning about cases where the language level is not set explicitly. If people start marking their code as being "Cython 2.x code", either with an in-file directive or from their setup.py, we will have less of a problem in the future to change the default. What do you think? Any other ideas, comments, objections? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.28.1 released
Hi all, thanks to your excellent bug reports, here is a bug fix release for the recent 0.28. https://pypi.python.org/pypi/Cython/0.28.1 Changelog: https://github.com/cython/cython/blob/0.28.1/CHANGES.rst Bugs fixed for this release: * ``PyFrozenSet_New()`` was accidentally used in PyPy where it is missing from the C-API. * Assignment between some C++ templated types were incorrectly rejected when the templates mix ``const`` with ``ctypedef``. (Github issue #2148) * Undeclared C++ no-args constructors in subclasses could make the compilation fail if the base class constructor was declared without ``nogil``. (Github issue #2157) * Bytes %-formatting inferred ``basestring`` (bytes or unicode) as result type in some cases where ``bytes`` would have been safe to infer. (Github issue #2153) * ``None`` was accidentally disallowed as typed return value of ``dict.pop()``. (Github issue #2152) Have fun, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.28 released
Hi everyone, I'm pleased to announce the immediate availability of Cython 0.28, after almost half a year of development. https://pypi.python.org/pypi/Cython/0.28 The major new features include: * Cdef classes can now multiply inherit from ordinary Python classes, as long as (exactly) one base class is an extension type. * The "const" modifier can be applied to memoryview declarations to allow read-only buffers as input. * C code in the docstring of a "cdef extern" block is copied verbatimly into the generated file. The changelog lists the various other big and small features, optimisations and bug fixes that went into this release: https://github.com/cython/cython/blob/0.28/CHANGES.rst Have fun, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.28 release candidate is available
... and one more iteration, since in order to fix an error reporting bug, I had to change a setting for internally generated code, which should not impact user code, but might, especially in C++ mode. So here's the release candidate: https://github.com/cython/cython/releases/tag/0.28rc1 Stefan Stefan Behnel schrieb am 09.03.2018 um 13:28: > Hi again, > > thanks to your feedback, I was able to solve a couple of issues and > regressions in the first beta. Here's the second one. > > Release: > https://github.com/cython/cython/releases/tag/0.28b2 > > Differences: > https://github.com/cython/cython/compare/0.28b1...0.28b2 > > Overall changelog for 0.28: > https://github.com/cython/cython/blob/0.28b2/CHANGES.rst > > Looks like the final release is close. If nothing major comes up, I'll > prepare the final release in a couple of days. > > Stefan > > > > Stefan Behnel schrieb am 04.03.2018 um 13:15: >> Hi all, >> >> after almost half a year of development and a late couple of weeks of bug >> hunting and feature cleanup, I'm happy to announce the availability of the >> first beta of Cython 0.28. >> >> https://github.com/cython/cython/releases/tag/0.28b1 >> >> This is a rather big feature release, please give it some testing. I'll not >> even start listing the major improvements here, but it comes with some >> really long-requested language features and lots of new optimisations, so >> please take a look at the changelog: >> >> https://github.com/cython/cython/blob/0.28b1/CHANGES.rst >> >> Have fun, >> >> Stefan > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.28 beta 2 is available
Hi again, thanks to your feedback, I was able to solve a couple of issues and regressions in the first beta. Here's the second one. Release: https://github.com/cython/cython/releases/tag/0.28b2 Differences: https://github.com/cython/cython/compare/0.28b1...0.28b2 Overall changelog for 0.28: https://github.com/cython/cython/blob/0.28b2/CHANGES.rst Looks like the final release is close. If nothing major comes up, I'll prepare the final release in a couple of days. Stefan Stefan Behnel schrieb am 04.03.2018 um 13:15: > Hi all, > > after almost half a year of development and a late couple of weeks of bug > hunting and feature cleanup, I'm happy to announce the availability of the > first beta of Cython 0.28. > > https://github.com/cython/cython/releases/tag/0.28b1 > > This is a rather big feature release, please give it some testing. I'll not > even start listing the major improvements here, but it comes with some > really long-requested language features and lots of new optimisations, so > please take a look at the changelog: > > https://github.com/cython/cython/blob/0.28b1/CHANGES.rst > > Have fun, > > Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython 0.28 beta 1 is available!
Hi all, after almost half a year of development and a late couple of weeks of bug hunting and feature cleanup, I'm happy to announce the availability of the first beta of Cython 0.28. https://github.com/cython/cython/releases/tag/0.28b1 This is a rather big feature release, please give it some testing. I'll not even start listing the major improvements here, but it comes with some really long-requested language features and lots of new optimisations, so please take a look at the changelog: https://github.com/cython/cython/blob/0.28b1/CHANGES.rst Have fun, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Segfault with large cdef'd list
Robert Bradshaw schrieb am 07.01.2018 um 09:48: > On Sat, Jan 6, 2018 at 10:57 PM, Dan Stromberg wrote: >> I'm getting a weird segfault from a tiny function (SSCCE) using cython >> with python 2.7. I'm seeing something similar with cython and python >> 3.5, though I did not create an SSCCE for 3.5. >> >> This same code used to work with slightly older cythons and pythons, >> and a slightly older version of Linux Mint. >> >> The code is at http://stromberg.dnsalias.org/svn/why-is-python-slow/trunk >> (more specifically at >> http://stromberg.dnsalias.org/svn/why-is-python-slow/trunk/tst.pyx ) >> >> In short, cdef'ing a list of doubles with about a million elements, >> and using only the 0th element once, segfaults - but cdef'ing a >> slightly smaller array does not segfault under otherwise identical >> conditions. >> >> Any suggestions? Does Cython have a limit on the max size of a stack frame? >> > Cython itself doesn't impose any limits, but it does inherit whatever > limit exists in the C complier and runtime. The variance may be due to > whatever else happens to be placed on the stack. Let me add that I wouldn't consider it a good idea to allocate large chunks of memory on the stack. If it's meant to hold substantial amounts of data (which also suggests that there is a substantial amount of processing and/or copying involved), it's probably also worth a [PyMem_]malloc() call. Heap allocation allows you to respond to allocation failures with a MemoryError rather than a crash, as you get now. How much stack space you have left is user controlled through call depth and recursion, which makes it a somewhat easy target. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
