[Python-ideas] wrapping up 4 years of open source python > 12 pep ideas
Dear python enthusiasts, --(you can skip this part if you're in a hurry)-- 4 years ago I was discovering this great language. Already an experienced software developer and data scientist at that time, accustomed with both low-level (C++), Object-Oriented (Java), Query (T-SQL), and Script (MATLAB) languages, I still have to admit that I was blown away by the capabilities and expressiveness of python ! Our team leveraged python during this period to develop our second cloud-based Analytics as a Service platform. As I found out, even if the quantity of available open source python libraries was already huge, many elements were missing in order to cover all of our needs in an elegant, "pythonic" way. So I first developed some common libraries for internal use, and eventually made some of them open source to make them more mature and to get community feedback. At first it was difficult to create elegant libs, but after a dozen I think that I began to "get it right". Now that we are reaching the 4th year anniversary (and I am about to take some rest for a second paternity leave), I thought that it would be a great time to do a summary of the main issues that I came across with python, and share this with you. Most of these issues were so blocking/frustrating that I had to create a library to fix them, when I could. Still, I believe that some deserve a better fix, directly in the core python stdlib or, for one or two of them maybe, in the language itself. --- The result is a list of 12 PEP topics/ideas that can be found and discussed here: https://github.com/smarie/python-pep-ideas/issues . For reference, the list of open source libraries that were at the origin of the discovery or the fix of these issues is here: https://github.com/smarie/ALL_OF_THE_ABOVE#overview . I tried to use git labels to relate ideas with their related lib. I hope that this will serve the python community and maybe make python even greater in the future! Kind regards - stay safe -- Sylvain ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T3DSXIKWX64HFMEVAYVBQ7NKOPYWW2QF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Getting the version number for a package or module, reliably
Thanks for the feedback Xavier Indeed, this “last resort” strategy only works if your project source is a git repository. I rely on the excellent setuptools_scm to handle that special case. By the way if someone knows an equivalent for other popular scm systems, please let me know (or better suggest it here: https://github.com/smarie/python-getversion/issues ) I forgot to add something in my previous email: the original reason for developing this library was to provide a version-aware persistency layer based on json. Basically when you deserialize an object, your code would receive the associated version, which allows for legacy-aware deserialization (typically storing a machine learning model for 6 months and needing to upgrade your code while still being able to deserialize it). I guess that this use case could also apply to pickle – from what I remember, pickle does not like it too much when the object to deserialize does not correspond to the same versions of the classes used. Best Sylvain De : Xavier Combelle Envoyé : samedi 6 juillet 2019 11:34 À : Sylvain MARIE Objet : Re: [Python-ideas] Getting the version number for a package or module, reliably [External email: Use caution with links and attachments] I'm certainly wrong, but version of a development version of a typical library is probably not reliable, as typically the version number is bumped when a new version is shipped so the code can be very different of the version given. Le sam. 6 juil. 2019 11:13, Sylvain MARIE via Python-ideas mailto:python-ideas@python.org>> a écrit : Dear python enthusiasts, For some industrial project a few years ago we needed a reliable way to get the version number for a package or module. I found out that there was actually no method working in all edge cases, including: * Built-in modules * Unzipped wheels and eggs added to the python patch * Non-installed project under development with version control information available * Packages both installed and added to the python path (typically a developer working on a new version) So I created one, and finally found the time to publish it. No rocket science here, but you may find this new package useful: https://smarie.github.io/python-getversion/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.github.io%2Fpython-getversion%2F=02%7C01%7Csylvain.marie%40se.com%7C20dc338bee554edb628108d701f519e8%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C1%7C636980024523465904=tw6DivTCfYf5xFmEOUCkVjSdAK9ZB%2BVjjpfHwij%2Ba8s%3D=0> It works with any imported module for now, including submodules. Along with the version you get details about why a given version number is returned (is it because of the __version__ attribute that was found, or because of the Version metadata in the installed package, etc.) Also, if one edge case is missing, it is fairly easy to add it. If I missed something in the stdlib (I acme across the importlib.metadata initiative but as of now it does not seem to cover all of the above cases), please let me know so that I can cite it in the documentation and even redirect to it if it happens to already cover all the cases. Happy summer to all ! -- Sylvain ___ Python-ideas mailing list -- python-ideas@python.org<mailto:python-ideas@python.org> To unsubscribe send an email to python-ideas-le...@python.org<mailto:python-ideas-le...@python.org> https://mail.python.org/mailman3/lists/python-ideas.python.org/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman3%2Flists%2Fpython-ideas.python.org%2F=02%7C01%7Csylvain.marie%40se.com%7C20dc338bee554edb628108d701f519e8%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C1%7C636980024523465904=Qwl176QO8d1S8%2FumncRb7YA9Nvo0G5eASUvYyYxvZhs%3D=0> Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/OEXCPM45R2YBE5GBVRXK54CZGZFSBNDI/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Farchives%2Flist%2Fpython-ideas%40python.org%2Fmessage%2FOEXCPM45R2YBE5GBVRXK54CZGZFSBNDI%2F=02%7C01%7Csylvain.marie%40se.com%7C20dc338bee554edb628108d701f519e8%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C1%7C636980024523475900=YyOpqRKVEIlHkxU73p%2B6eTRm0mOW%2Fssll%2B963z8GowU%3D=0> Code of Conduct: http://python.org/psf/codeofconduct/<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2F=02%7C01%7Csylvain.marie%40se.com%7C20dc338bee554edb628108d701f519e8%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C1%7C636980024523475900=FAeFYR6fjm%2FWGWnowP7boQC7kSgjXJVqRGoaEhdYk40%3D=0> __ This email has been scanned by the Symantec Email Security.cloud service. __ __
Re: [Python-ideas] Problems (and solutions?) in writing decorators
All of your answers are true, > (Chris) > "my_decorator(x=foo)" is not going to look like "@my_decorator \n def foo". That’s one of the many ways that `decopatch` uses to perform the disambiguation, indeed. But that’s already the user helping the lib, not the other way round > (Christopher) > @something applied on def fun is exactly the same as something(fun) True. However applying decorator manually is already for advanced users, so users of a decopatche-d decorator will not mind calling something()(fun). In fact it is consistent with when they use it with arguments : something(arg)(fun). Another criterion is : how easy would it be to implement an inspect.is_decorator_call(frame) method returning True if and only if frame is the @ statement ? If that’s fairly easy, well, I’m pretty sure that this is good stuff. From a naïve user, not accustomed with the long history of this language, is very strange that an operator such as @ (the decorator one, not the other one) is completely not detectable by code, while there are so many hooks available in python for all other operators (+, -, etc.). Eventually that’s obviously your call, I’m just there to give feedback from what I see of the python libs development community. -- Sylvain De : Python-ideas De la part de Christopher Barker Envoyé : mercredi 20 mars 2019 06:50 À : Greg Ewing Cc : python-ideas Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators [External email: Use caution with links and attachments] Also: @something def fun(): ... Is exactly the same as: def fun() ... fun = something(fun) So you can’t make a distinction based whether a given usage is as a decoration. -CHB On Tue, Mar 19, 2019 at 12:26 PM Greg Ewing mailto:greg.ew...@canterbury.ac.nz>> wrote: Sylvain MARIE via Python-ideas wrote: > `my_decorator(foo)` when foo is a callable will always look like > `@my_decorator` applied to function foo, because that's how the language is > designed. I don't think it's worth doing anything to change that. Everywhere else in the language, there's a very clear distinction between 'foo' and 'foo()', and you confuse them at your peril. I don't see why decorators should be any different. -- Greg ___ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideas=02%7C01%7Csylvain.marie%40se.com%7C2e9308c9904c40f3702408d6acf7fc3a%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636886578423739625=%2BSHOm4PYwt6p%2F6Of69z0o19sH%2FimE4YsUFFtaEMlYCc%3D=0> Code of Conduct: http://python.org/psf/codeofconduct/<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2F=02%7C01%7Csylvain.marie%40se.com%7C2e9308c9904c40f3702408d6acf7fc3a%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636886578423739625=uO%2Bq8AahHYhCVObE%2Fq3Tn%2BODC0vsdjWfX2niuuNgemE%3D=0> -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython __ This email has been scanned by the Symantec Email Security.cloud service. __ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Problems (and solutions?) in writing decorators
Stephen > If the answer is "maybe", IMO PyPI is the right solution for distribution. Very wise words, I understand this point. However as of today it is *not* possible to write such a library in a complete way, without an additional tool from the language itself. Trust me, I tried very hard :). Indeed `my_decorator(foo)` when foo is a callable will always look like `@my_decorator` applied to function foo, because that's how the language is designed. So there is always one remaining ambiguous case to cover, and I have to rely on some ugly tricks such as asking users for a custom disambiguator. So if the decision is to let community-provided libraries like `decopatch` solve this problem, would you please consider providing us with the missing information? For example a new method in the `inspect` package could be `inspect.is_decorator_call(frame)`, that would return True if and only if the given frame is a decorator usage call as in `@my_decorator`. That piece would be enough - I would gladly take care of the rest in `decopatch`. Thanks for the feedback ! Sylvain -Message d'origine- De : Stephen J. Turnbull Envoyé : vendredi 15 mars 2019 04:56 À : Sylvain MARIE Cc : David Mertz ; python-ideas Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators [External email: Use caution with links and attachments] ________ Sylvain MARIE via Python-ideas writes: > I totally understand your point of view. However on the other hand, > many > very popular open source projects out there have the opposite > point of > view and provide decorators that can seamlessly be used > with and without > arguments (pytest, attrs, click, etc.). So after a > while users get used > to this behavior and expect it from all > libraries. Making it easy to > implement is therefore something quite > important for developers not to > spend time on this useless > “feature”. That doesn't follow. You can also take it that "educating users to know the difference between a decorator and a decorator factory is therefore something quite important for developers not to spend time on this useless 'feature'." I'm not a fan of either position. I don't see why developers of libraries who want to provide this to their users shouldn't have "an easy way to do it", but I also don't see a good reason to encourage syntactic ambiguity by providing it in the standard library. I think this is a feature that belongs in the area of "you *could* do it, but *should* you?" If the answer is "maybe", IMO PyPI is the right solution for distribution. Steve -- Associate Professor Division of Policy and Planning Science https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fturnbull.sk.tsukuba.ac.jp%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C1c5b73f96ee240ef288608d6a8fa2030%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636882189569411152sdata=EcCuG%2Bad0oJoT1Fupd7086wkxHUPapCEKN2x3zDvCw0%3Dreserved=0 Faculty of Systems and Information Email: turnb...@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN __ This email has been scanned by the Symantec Email Security.cloud service. __ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Problems (and solutions?) in writing decorators
%7C01%7Csylvain.marie%40se.com%7Ceba5365e816845b8258808d6a716b054%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636880113229754958=KqOxLc6%2FpStoc4sfIV1BqAAutbrMxQXfqmCb3yznlY8%3D=0> this is not the case because with functools.wrapt: - the wrapper code will execute even when the provided arguments are invalid. - the wrapper code cannot easily access an argument using its name, from the received *args, **kwargs. Indeed one would have to handle all cases (positional, keyword, default) and therefore to use something like Signature.bind(). For this reason I proposed a replacement in `makefun`: https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.github.io%2Fpython-makefun%2F%23signature-preserving-function-wrappers=02%7C01%7Csylvain.marie%40se.com%7Ceba5365e816845b8258808d6a716b054%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636880113229754958=5yNDNnhCjpTctqX951So2cnR2YqzxTiqp%2F%2BBj7XbibE%3D=0> -- Now bridging the gap. Of course a very interesting use cases for decorators is to create decorators that create a signature-preserving wrapper. It is possible to combine decopatch and makefun for this: https://smarie.github.io/python-decopatch/#3-creating-function-wrappers<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.github.io%2Fpython-decopatch%2F%233-creating-function-wrappers=02%7C01%7Csylvain.marie%40se.com%7Ceba5365e816845b8258808d6a716b054%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636880113229764963=4n3fuJLLekEOwdmddmMjP00XocHei4dWqV5%2FJosoTR0%3D=0> . Decopatch even proposes a "double-flat" development style where you directly write the wrapper body, as explained in the doc. Did I answer your questions ? Thanks again for the quick feedback ! Best, Sylvain -Message d'origine- De : Python-ideas mailto:se@python.org>> De la part de Steven D'Aprano Envoyé : mardi 12 mars 2019 12:30 À : python-ideas@python.org<mailto:python-ideas@python.org> Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators [External email: Use caution with links and attachments] On Tue, Mar 12, 2019 at 09:36:41AM +, Sylvain MARIE via Python-ideas wrote: > I therefore proposed > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma > rie.github.io<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frie.github.io=02%7C01%7Csylvain.marie%40se.com%7Ceba5365e816845b8258808d6a716b054%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636880113229774972=AaBx8DaTf0SJZT6msZo8RlHnkxGHV0JaSBJ2DNXa3IA%3D=0>%2Fpython-makefun%2Fdata=02%7C01%7Csylvain.marie%40s > e.com<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fe.com=02%7C01%7Csylvain.marie%40se.com%7Ceba5365e816845b8258808d6a716b054%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636880113229774972=o4qqJGFIUrLp3433N%2FIiUethE9Rtd91GiRZXOvYfxHQ%3D=0>%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae > 68fef%7C0%7C0%7C636879872385158085sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2 > Fekk35bnYGvmEFJyCXaLDyLm9I%3Dreserved=0 . In particular it > provides an equivalent of `@functools.wraps` that is truly > signature-preserving Tell us more about that please. I'm very interested in getting decorators preserve the original signature. -- Steven ___ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideasdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=XcYfEginmDF7kIpGGA0XxDZKpUn9e4p2zPFk7UAruYg%3Dreserved=0<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideas=02%7C01%7Csylvain.marie%40se.com%7Ceba5365e816845b8258808d6a716b054%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636880113229784972=o9KkGrtGOIkL%2BhMYU169rlUoTeQRrFvetVm6dF2ejys%3D=0> Code of Conduct: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=20ZrtVQZbpQ54c96veSXIOfEK7rKy0ggj0omTZg3ri8%3Dreserved=0<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2F=02%7C01%7Csylvain.marie%40se.com%7Ceba5365e816845b8258808d6a716b054%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636880113229784972=PSas3%2FPwjJetvYbwe8vfo4PdpXHV74Nx6VTQHBZF91I%3D=0> __ This email has been scanned by the Symantec Email Security.cl
Re: [Python-ideas] Problems (and solutions?) in writing decorators
perceive some win over what wrapt does. On Tue, Mar 12, 2019, 9:52 AM Sylvain MARIE mailto:sylvain.ma...@se.com>> wrote: David, Steven, Thanks for your interest ! As you probably know, decorators and function wrappers are *completely different concepts*. A decorator can directly return the decorated function (or class), it does not have to return a wrapper. Even more, it can entirely replace the decorated item with something else (not even a function or class!). Try it: it is possible to write a decorator to replace a function with an integer, even though it is probably not quite useful :) `decopatch` helps you write decorators, whatever they are. It "just" solves the annoying issue of having to handle the no-parenthesis and with-parenthesis calls. In addition as a 'goodie', it proposes two development styles: *nested* (you have to return a function) and *flat* (you directly write what will happen when the decorator is applied to something). -- Now about creating signature-preserving function wrappers (in a decorator, or outside a decorator - again, that's not related). That use case is supposed to be covered by functools.wrapt. Unfortunately as explained here https://stackoverflow.com/questions/308999/what-does-functools-wraps-do/55102697#55102697<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F308999%2Fwhat-does-functools-wraps-do%2F55102697%2355102697=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186393926=Dmkd0Ld1HsuCJtJDCS6GDUgwH3GO66zwYMpZ9Ow%2B18k%3D=0> this is not the case because with functools.wrapt: - the wrapper code will execute even when the provided arguments are invalid. - the wrapper code cannot easily access an argument using its name, from the received *args, **kwargs. Indeed one would have to handle all cases (positional, keyword, default) and therefore to use something like Signature.bind(). For this reason I proposed a replacement in `makefun`: https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.github.io%2Fpython-makefun%2F%23signature-preserving-function-wrappers=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186403936=SO319cmnKXiUK6Zek%2FSioHIhFfnQCZRpAF3kim84mv8%3D=0> -- Now bridging the gap. Of course a very interesting use cases for decorators is to create decorators that create a signature-preserving wrapper. It is possible to combine decopatch and makefun for this: https://smarie.github.io/python-decopatch/#3-creating-function-wrappers<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.github.io%2Fpython-decopatch%2F%233-creating-function-wrappers=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186403936=FpCGLYwJHFJMWmStJhI%2F7NzXxmJPwC3hf9afg2zfXik%3D=0> . Decopatch even proposes a "double-flat" development style where you directly write the wrapper body, as explained in the doc. Did I answer your questions ? Thanks again for the quick feedback ! Best, Sylvain -Message d'origine- De : Python-ideas mailto:se@python.org>> De la part de Steven D'Aprano Envoyé : mardi 12 mars 2019 12:30 À : python-ideas@python.org<mailto:python-ideas@python.org> Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators [External email: Use caution with links and attachments] ____________ On Tue, Mar 12, 2019 at 09:36:41AM +, Sylvain MARIE via Python-ideas wrote: > I therefore proposed > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma > rie.github.io<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frie.github.io=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186413941=8QNKZapq0Z9VQtREcqt4f8EcKbTteHWqiAhD1kVnDyc%3D=0>%2Fpython-makefun%2Fdata=02%7C01%7Csylvain.marie%40s > e.com<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fe.com=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186413941=73ekIlXJ7nwxcsHYz91JotQ%2F0eGH7h6YC2e3U5pqXow%3D=0>%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae > 68fef%7C0%7C0%7C636879872385158085sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2 > Fekk35bnYGvmEFJyCXaLDyLm9I%3Dreserved=0 . In particular it > provides an equivalent of `@functools.wraps` that is truly > signature-preserving Tell us more about that please. I'm very interested in getting decorators preserve the original signature. -- Steven ___ Python-ideas mailing list Python-ideas@pytho
Re: [Python-ideas] Problems (and solutions?) in writing decorators
David I realize that you were pointing at the 'wrapt' library not the functools.wrapt library. >From what I have tried with that library, it does not solve the issue solved >with decopatch. Sylvain -Message d'origine- De : Sylvain MARIE Envoyé : mardi 12 mars 2019 14:53 À : Steven D'Aprano ; python-ideas@python.org; David Mertz Objet : RE: [Python-ideas] Problems (and solutions?) in writing decorators David, Steven, Thanks for your interest ! As you probably know, decorators and function wrappers are *completely different concepts*. A decorator can directly return the decorated function (or class), it does not have to return a wrapper. Even more, it can entirely replace the decorated item with something else (not even a function or class!). Try it: it is possible to write a decorator to replace a function with an integer, even though it is probably not quite useful :) `decopatch` helps you write decorators, whatever they are. It "just" solves the annoying issue of having to handle the no-parenthesis and with-parenthesis calls. In addition as a 'goodie', it proposes two development styles: *nested* (you have to return a function) and *flat* (you directly write what will happen when the decorator is applied to something). -- Now about creating signature-preserving function wrappers (in a decorator, or outside a decorator - again, that's not related). That use case is supposed to be covered by functools.wrapt. Unfortunately as explained here https://stackoverflow.com/questions/308999/what-does-functools-wraps-do/55102697#55102697 this is not the case because with functools.wrapt: - the wrapper code will execute even when the provided arguments are invalid. - the wrapper code cannot easily access an argument using its name, from the received *args, **kwargs. Indeed one would have to handle all cases (positional, keyword, default) and therefore to use something like Signature.bind(). For this reason I proposed a replacement in `makefun`: https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers -- Now bridging the gap. Of course a very interesting use cases for decorators is to create decorators that create a signature-preserving wrapper. It is possible to combine decopatch and makefun for this: https://smarie.github.io/python-decopatch/#3-creating-function-wrappers . Decopatch even proposes a "double-flat" development style where you directly write the wrapper body, as explained in the doc. Did I answer your questions ? Thanks again for the quick feedback ! Best, Sylvain -Message d'origine- De : Python-ideas De la part de Steven D'Aprano Envoyé : mardi 12 mars 2019 12:30 À : python-ideas@python.org Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators [External email: Use caution with links and attachments] On Tue, Mar 12, 2019 at 09:36:41AM +, Sylvain MARIE via Python-ideas wrote: > I therefore proposed > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma > rie.github.io%2Fpython-makefun%2Fdata=02%7C01%7Csylvain.marie%40s > e.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae > 68fef%7C0%7C0%7C636879872385158085sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2 > Fekk35bnYGvmEFJyCXaLDyLm9I%3Dreserved=0 . In particular it > provides an equivalent of `@functools.wraps` that is truly > signature-preserving Tell us more about that please. I'm very interested in getting decorators preserve the original signature. -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideasdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=XcYfEginmDF7kIpGGA0XxDZKpUn9e4p2zPFk7UAruYg%3Dreserved=0 Code of Conduct: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=20ZrtVQZbpQ54c96veSXIOfEK7rKy0ggj0omTZg3ri8%3Dreserved=0 __ This email has been scanned by the Symantec Email Security.cloud service. __ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Problems (and solutions?) in writing decorators
David, Steven, Thanks for your interest ! As you probably know, decorators and function wrappers are *completely different concepts*. A decorator can directly return the decorated function (or class), it does not have to return a wrapper. Even more, it can entirely replace the decorated item with something else (not even a function or class!). Try it: it is possible to write a decorator to replace a function with an integer, even though it is probably not quite useful :) `decopatch` helps you write decorators, whatever they are. It "just" solves the annoying issue of having to handle the no-parenthesis and with-parenthesis calls. In addition as a 'goodie', it proposes two development styles: *nested* (you have to return a function) and *flat* (you directly write what will happen when the decorator is applied to something). -- Now about creating signature-preserving function wrappers (in a decorator, or outside a decorator - again, that's not related). That use case is supposed to be covered by functools.wrapt. Unfortunately as explained here https://stackoverflow.com/questions/308999/what-does-functools-wraps-do/55102697#55102697 this is not the case because with functools.wrapt: - the wrapper code will execute even when the provided arguments are invalid. - the wrapper code cannot easily access an argument using its name, from the received *args, **kwargs. Indeed one would have to handle all cases (positional, keyword, default) and therefore to use something like Signature.bind(). For this reason I proposed a replacement in `makefun`: https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers -- Now bridging the gap. Of course a very interesting use cases for decorators is to create decorators that create a signature-preserving wrapper. It is possible to combine decopatch and makefun for this: https://smarie.github.io/python-decopatch/#3-creating-function-wrappers . Decopatch even proposes a "double-flat" development style where you directly write the wrapper body, as explained in the doc. Did I answer your questions ? Thanks again for the quick feedback ! Best, Sylvain -Message d'origine- De : Python-ideas De la part de Steven D'Aprano Envoyé : mardi 12 mars 2019 12:30 À : python-ideas@python.org Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators [External email: Use caution with links and attachments] On Tue, Mar 12, 2019 at 09:36:41AM +0000, Sylvain MARIE via Python-ideas wrote: > I therefore proposed > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma > rie.github.io%2Fpython-makefun%2Fdata=02%7C01%7Csylvain.marie%40s > e.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae > 68fef%7C0%7C0%7C636879872385158085sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2 > Fekk35bnYGvmEFJyCXaLDyLm9I%3Dreserved=0 . In particular it > provides an equivalent of `@functools.wraps` that is truly > signature-preserving Tell us more about that please. I'm very interested in getting decorators preserve the original signature. -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideasdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=XcYfEginmDF7kIpGGA0XxDZKpUn9e4p2zPFk7UAruYg%3Dreserved=0 Code of Conduct: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=20ZrtVQZbpQ54c96veSXIOfEK7rKy0ggj0omTZg3ri8%3Dreserved=0 __ This email has been scanned by the Symantec Email Security.cloud service. __ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Problems (and solutions?) in writing decorators
Dear python enthusiasts, Writing python decorators is indeed quite a tideous process, in particular when you wish to add arguments, and in particular in two cases : all optional arguments, and one mandatory argument. Indeed in these two cases there is a need to disambiguate between no-parenthesis and with-parenthesis usage. After having struggled with this pattern for two years in various open source and industrial projects, I ended up writing a library to hopefully solve this once and for all: https://smarie.github.io/python-decopatch/ . It is extensively tested (203 tests) against many combinations of signature/calls. I would gladly appreciate any feedback ! Please note that there is a "PEP proposal draft" in the project page because I belive that the best a library can do will always be a poor workaround, where the interpreter or stdlib could really fix it properly. Sorry for not providing too much implementation details in that page, my knowledge of the python interpreter is unfortunately quite limited. -- Finally there is an additional topic around decorators : people tend to believe that decorators and function wrappers are the same, which is absolutely not the case. I used the famous `decorator` lib in many projects but I was not satisfied because it was solving both issues at the same time, maintaining the confusion. I therefore proposed https://smarie.github.io/python-makefun/ . In particular it provides an equivalent of `@functools.wraps` that is truly signature-preserving (based on the same recipe than `decorator`). Once again, any feedback would be gladly appreciated ! Kind regards Sylvain -Message d'origine- De : Python-ideas De la part de Greg Ewing Envoyé : vendredi 26 octobre 2018 00:04 À : python-ideas Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators [External email: Use caution with links and attachments] Jonathan Fine wrote: > I also find writing decorators a bit > hard. It seems to be something I have to learn anew each time I do it. > Particularly for the pattern > > @deco(arg1, arg2) def fn(arg3, arg4): > # function body > > Perhaps doing something with partial might help here. Anyone here > interested in exploring this? > I can't think of a way that partial would help. But would you find it easier if you could do something like this? class deco(Decorator): def __init__(self, arg1, arg2): self.arg1 = arg1 self.arg2 = arg2 def invoke(self, func, arg3, arg4): # function body Implementation: class Decorator: def __call__(self, func): self._wrapped_function = func return self._wrapper def _wrapper(self, *args, **kwds): return self.invoke(self._wrapped_function, *args, **kwds) -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideasdata=02%7C01%7Csylvain.marie%40se.com%7C295cecd58fc3461f42c108d63ac5e03e%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C63676101605889sdata=Z8OET1CZZWnmN5czi0rZ1X57%2FDd4a9IDbbSujsDNWzk%3Dreserved=0 Code of Conduct: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C295cecd58fc3461f42c108d63ac5e03e%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C63676101615894sdata=u0hvM5cR%2BR1Vni%2BV48WoNF%2FpriCaOG5%2BFAXayaTGsYY%3Dreserved=0 __ This email has been scanned by the Symantec Email Security.cloud service. __ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
