[Python-Dev] Re: PEP 654 except* formatting
Throwing in another +1 for `except group`. It's explicit, doesn't introduce new punctuation and avoids confusion with unpacking. Regards, Ammar On Mon, Oct 4, 2021, 3:31 AM Antoine Pitrou wrote: > On Sun, 3 Oct 2021 19:42:29 +0200 > Łukasz Langa wrote: > > > > -1 > > > > If I could read the vertical line as a pipe character, the expression > would read "except or E as e". > > But I can't read it that way anyway. Instead, all I see is a lowercase > EXCEPTL. > > > > My idea is this: > > > > try: > > ... > > except group E as e: > > ... > > except group E1, T2 as e: > > ... > > > > Should be doable given the magical match-case contextual keywords > precedent. This looks nice and is explicit, since you will always get an > ExceptionGroup instance under `e`. > > +1. This is much more helpful to the reader than the cryptic > asterisk. > > Regards > > Antoine. > > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/INK6TSOGGODA4NZ3CI5MOXIAI4Z4CZ53/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4ZUBUDQ4CGXYJAIYKMJMJBGUGGTODECF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Is anyone relying on new-bugs-announce/python-bugs-list/bugs.python.org summaries
Hey everyone, As part of PEP 588, migrating bugs.python.org issues to Github, there are two current mailing list related items that need a replacement or need to be turned down. 1. Weekly summary emails with bug counts and issues from the week, example: https://mail.python.org/archives/list/python-dev@python.org/thread/JRFJ4QH7TR35HFRQWOYPPCGOYRFAXK24/ 2. Emails sent to the new-bugs-announce and python-bugs-list for new issues and comments to existing issues. I wanted to quickly gauge if anyone is relying on these mailing lists/emails or if it's a convenient part of your workflow. Feel free to chime in here or on the migration issues directly with your use-case: 1. https://github.com/psf/gh-migration/issues/6 2. https://github.com/psf/gh-migration/issues/7 Thanks, Ammar ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LEQ5G64OVOFGY63563XDPF52R5XAVAIY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Critique of PEP 657
Hi Terry, > IDLE currently uses traceback.extract_tb and traceback.print_list Awesome, that should work out perfectly then. Our current proof-of-concept implementation augments the traceback.FrameSummary class to include `end_lineno`, `colno` and `end_colno` attributes. We will make sure to add you as a reviewer on that change so you can give it an early shot at integrating with IDLE. Regards, Ammar ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/J622Y35DCXIBHEF72VENBJ5DX6F5ZFPJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Critique of PEP 657
Hi Mark, Thank you for the feedback. Let me address/elaborate some of the points that Pablo touched on. > PEP 657 proposes that locations for exceptions be treated as ranges, > whereas tracing, profiling and debugging currently treat locations as > points. I don't think we're making strong claims that the full `(line, end_line, column, end_column)` should be the canonical representation for exception locations. The only concrete place we suggest their usage is in the printing of tracebacks. The information is not exposed in the exception or traceback objects intentionally as part of this. The place we make a reference to non-traceback tooling being able to use this information is coverage tools being able to perform expression-level granularity in coverage. As a quick example consider: x = True or f() might be marked as covered by a line coverage tool but this PEP potentially exposes extra information that might help show that the function call is not covered. > Consider this example: > https://github.com/python/cpython/blob/main/Lib/test/test_compile.py#L861 And this example continues to work just as it does right now. There is no change to the tracing apis or the existing co_lines method. I think your concern is if tracing tools switched to using PEP 657 (which they are not obligated to), but even that case works out: co_lines() returns (6, 8, 4) for the CALL_METHOD. co_positions() from PEP 657 returns (4, 6, 5, 6) corresponding to (line, end_line, column, end_column) for the CALL_METHOD. > For example, how would one set a breakpoint on line 4 above? Just as they would right now, they don't need to change how they set breakpoints. > PEP 657 claims it is fully backwards compatible, but it cannot be both > backwards compatible and consistent. I think there's a misunderstanding on what backwards compatibility means between us, can you concretely explain how this PEP or its implementation would break existing tools? I understand your concerns about having two potentially conflicting APIs for source locations but that is not a backwards compatibility problem. > 1. Clarify, in detail, the impact on line-based tools like profilers, > coverage.py and debuggers. This should include help on how to use the > new APIs and where using the old APIs might result in behavioral changes. As mentioned, we don't have an expectation for line-based tools to switch to the new API. Its primary consumer is meant to be the traceback mechanism. Usage of the old APIs will and must not lead to any behavioral changes. > 2. Change the C API to a single function: > int PyCode_Addr2Location(PyCodeObject *co, int addr, int *startline, int > *startcolumn, int *endline, int *endcolumn) Thank you, this is a great suggestion. > 3. Drop the opt-out option. > If the extra information is optional, then the compression scheme must > allow for that; making the code more complex and potentially less > efficient. Does opting out use the start of the range, or the old line, > as the location? In the future if a fantastic compression scheme is figured out that requires both the end lines and column information, I think it would be acceptable to make the opt-out only suppress the printing of the carets in the traceback while still maintaining the data in the code objects. This would still be backwards compatible. > 4. Drop the limitation on column offsets. > The data needs to be compressed anyway, so allowing arbitrary column > offsets is effectively free. Sure, I think there were plenty of good ideas thrown around compression and lazy-loading in the last PEP 657 thread so this is more of just a soft-limitation of the current implementation of the PEP. This limit can be increased in the future without any changes in the API or hurting backwards compatibility. > 6. Store all location information in a single table (this applies more > to the implementation than the PEP) > Using four separate objects to hold the location info adds a lot of > overhead for most functions. I would just like to cap off and address this point together. The PEP is meant primarily to aid with debugging and improve tracebacks. The API is designed to be very limited and allow for a lot of room for expansion and optimization. It does not make strong prescriptive claims that tools must switch to the new richer information as your mail seems to suggest. The other smaller aspects like the internal storage formats, not storing data when opted out are concerns that can be addressed without making any breaking changes. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VWTCMNX5J6K7FMX4VPYHBU5DU34AMERI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: IRC #python-dev channel is now on Libera Chat (bye bye Freenode)
This section on wikipedia https://en.wikipedia.org/wiki/Freenode#Ownership_change_and_conflict and this article have a good summary: https://hackaday.com/2021/05/20/freenode-debacle-prompts-staff-exodus-new-network/ Essentially what happened is that freenode was bought out a while ago by a company, there was some dispute over access between the "rightful owners" and the self-organized freenode staff. This caused the freenode staff to resign and start a new network called libera.chat. After that a bunch of channels moved over to libera chat and most recently if your topic mentioned libera.chat, the new freenode owners will take it over, ban anyone from chatting in it and change the topic (as happened with #python-fr). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OVTBKL344XIAVP4W3PR4MXIPSGAFZUQ4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks
> The cost I'm concerned about is the runtime cost of worse code, because > the compiler can't perform some optimizations due the constraints of > providing the extended debug information. Aah thanks for clarifying, I see what you mean now. In cases like this where the compiler is making optimizations, I think it is perfectly fine to just elide the column information. While it would be nice to maintain accurate columns wherever possible, you shouldn't constrain improvements and optimizations based on it. The traceback machinery will simply not print out the carets in that case and everything should just work smoothly. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EB24LA7L5C35QHQTFLB6QZX26E77O6QM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks
> While nicer locations for errors is great, it won't be popular if it has > a negative impact on performance. > Locations need to tracked through the compiler. In performance sensitive contexts won't most code be pre-compiled into pyc files anyway? I feel like the performance cost of accurate column tracking in the compiler isn't too big of a concern unless I'm missing something. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JDRA26FG5TXD3D7VMLE2UNKQQ4WIHLEF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Release of a responsive python-docs-theme 2021.5
> I wasn’t able to find examples - is this what is up and running on > docs.python.org, or is that a future plan? I believe currently only the pre-release (3.10) documentation https://docs.python.org/3.10/ and in-development (3.11) documentation https://docs.python.org/3.11/ is using the theme. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XI5N2TB3ARAFLV4U2JMU7NBTEK4VZ3ZO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks
I really like this idea Nathaniel. We already have a section considering lazy-loading column information in the draft PEP but obviously it has a pretty high implementation complexity since it necessitates a change in the pyc format and the importlib machinery. Long term this might be the way to go for column information and line information to alleviate the memory burden. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XS435QQOBWWQNU2FY6RVLA4YUXJCN7XF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Doc tests failing for many PRs on GitHub
Filed https://github.com/pypa/setuptools/issues/2362 against setuptools, not sure what we should do locally to fix this in the meantime. On Mon, Aug 31, 2020 at 1:11 PM Ammar Askar wrote: > > Karthik, your analysis is correct. You just have to import importlib > first to get the error: > > >>> import importlib > >>> import importlib.util > >>> importlib.util.abc.Loader > Traceback (most recent call last): > File "", line 1, in > AttributeError: module 'importlib.util' has no attribute 'abc' > > On Mon, Aug 31, 2020 at 12:09 PM Karthikeyan wrote: > > > > The last successful build is > > https://travis-ci.com/github/python/cpython/jobs/379161961 > > Recent failure build is > > https://travis-ci.com/github/python/cpython/jobs/379329436 > > > > I can see setuptools being upgraded from 49.6.0 to 50.0.0 as a difference. > > I can see the below command to print sphinx-build version error out in the > > makefile but unfortunately the error is being redirected to /dev/null in > > the makefile and I needed to run it explicitly to see the output. There is > > a recent change that moves Loader to _abc and imports Loader from _abc > > inside abc is done with below commit. The same code is present in 49.6.0 > > too at > > https://github.com/pypa/setuptools/blob/04e3df22df840c6bb244e9b27bc56750c44b7c85/_distutils_hack/__init__.py#L79 > > . So I am not sure why this causes error. > > > > Pinning the dependency of setuptools to 49.6.0 in Doc/makefile runs the > > tests fine without error. Can you please try pinning it and see if it helps? > > > > I am not sure if the below commit is the cause that is picked up in > > setuptools but someone can correct me if I am wrong about the importlib > > machinery error. > > > > commit 9e09849d20987c131b28bcdd252e53440d4cd1b3 > > Author: Victor Stinner > > Date: Wed Jun 17 23:15:59 2020 +0200 > > > > bpo-41006: importlib.util no longer imports typing (GH-20938) > > > > Create importlib._abc submodule to avoid importing typing when > > importlib.util is imported. Move Loader ABC into importlib._abc. > > > > ./python > > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > > [GCC 7.5.0] on linux > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import importlib.util > > >>> importlib.util.abc.Loader > > Traceback (most recent call last): > > File "", line 1, in > > AttributeError: module 'importlib.util' has no attribute 'abc' > > >>> > > ➜ cpython git:(master) ✗ git checkout > > 9e09849d20987c131b28bcdd252e53440d4cd1b3~1 Lib/importlib/util.py > > ➜ cpython git:(master) ✗ ./python > > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > > [GCC 7.5.0] on linux > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import importlib.util > > >>> importlib.util.abc.Loader > > > > > > xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W --keep-going -j4" > > doctest suspicious html > > make: Entering directory '/root/cpython/Doc' > > make[1]: Entering directory '/root/cpython/Doc' > > mkdir -p build > > echo PATH=./venv/bin:$PATH blurb help > > PATH=./venv/bin:/root/.poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > > blurb help > > echo PATH=./venv/bin:$PATH sphinx-build --version > > PATH=./venv/bin:/root/.poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > > sphinx-build --version > > Building NEWS from Misc/NEWS.d with blurb > > PATH=./venv/bin:$PATH sphinx-build -b doctest -d build/doctrees -q -W > > --keep-going -j4 -W . build/doctest > > Traceback (most recent call last): > > File "/root/cpython/Doc/./venv/bin/sphinx-build", line 5, in > > from sphinx.cmd.build import main > > File > > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/cmd/build.py", > > line 23, in > > from sphinx.application import Sphinx > > File > > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/application.py", > > line 28, in > > from sphinx.config import Config > > File > > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/config.py", > > line 22, in > > from sphinx.ut
[Python-Dev] Re: Doc tests failing for many PRs on GitHub
Karthik, your analysis is correct. You just have to import importlib first to get the error: >>> import importlib >>> import importlib.util >>> importlib.util.abc.Loader Traceback (most recent call last): File "", line 1, in AttributeError: module 'importlib.util' has no attribute 'abc' On Mon, Aug 31, 2020 at 12:09 PM Karthikeyan wrote: > > The last successful build is > https://travis-ci.com/github/python/cpython/jobs/379161961 > Recent failure build is > https://travis-ci.com/github/python/cpython/jobs/379329436 > > I can see setuptools being upgraded from 49.6.0 to 50.0.0 as a difference. I > can see the below command to print sphinx-build version error out in the > makefile but unfortunately the error is being redirected to /dev/null in the > makefile and I needed to run it explicitly to see the output. There is a > recent change that moves Loader to _abc and imports Loader from _abc inside > abc is done with below commit. The same code is present in 49.6.0 too at > https://github.com/pypa/setuptools/blob/04e3df22df840c6bb244e9b27bc56750c44b7c85/_distutils_hack/__init__.py#L79 > . So I am not sure why this causes error. > > Pinning the dependency of setuptools to 49.6.0 in Doc/makefile runs the tests > fine without error. Can you please try pinning it and see if it helps? > > I am not sure if the below commit is the cause that is picked up in > setuptools but someone can correct me if I am wrong about the importlib > machinery error. > > commit 9e09849d20987c131b28bcdd252e53440d4cd1b3 > Author: Victor Stinner > Date: Wed Jun 17 23:15:59 2020 +0200 > > bpo-41006: importlib.util no longer imports typing (GH-20938) > > Create importlib._abc submodule to avoid importing typing when > importlib.util is imported. Move Loader ABC into importlib._abc. > > ./python > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > [GCC 7.5.0] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import importlib.util > >>> importlib.util.abc.Loader > Traceback (most recent call last): > File "", line 1, in > AttributeError: module 'importlib.util' has no attribute 'abc' > >>> > ➜ cpython git:(master) ✗ git checkout > 9e09849d20987c131b28bcdd252e53440d4cd1b3~1 Lib/importlib/util.py > ➜ cpython git:(master) ✗ ./python > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > [GCC 7.5.0] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import importlib.util > >>> importlib.util.abc.Loader > > > xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W --keep-going -j4" > doctest suspicious html > make: Entering directory '/root/cpython/Doc' > make[1]: Entering directory '/root/cpython/Doc' > mkdir -p build > echo PATH=./venv/bin:$PATH blurb help > PATH=./venv/bin:/root/.poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > blurb help > echo PATH=./venv/bin:$PATH sphinx-build --version > PATH=./venv/bin:/root/.poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > sphinx-build --version > Building NEWS from Misc/NEWS.d with blurb > PATH=./venv/bin:$PATH sphinx-build -b doctest -d build/doctrees -q -W > --keep-going -j4 -W . build/doctest > Traceback (most recent call last): > File "/root/cpython/Doc/./venv/bin/sphinx-build", line 5, in > from sphinx.cmd.build import main > File > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/cmd/build.py", > line 23, in > from sphinx.application import Sphinx > File > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/application.py", > line 28, in > from sphinx.config import Config > File > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/config.py", line > 22, in > from sphinx.util import logging > File > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/util/__init__.py", > line 40, in > from sphinx.util import smartypants # noqa > File > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/util/smartypants.py", > line 33, in > from sphinx.util.docutils import __version_info__ as docutils_version > File > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/util/docutils.py", > line 17, in > from distutils.version import LooseVersion > File "", line 1007, in _find_and_load > File "", line 982, in _find_and_load_unlocked > File "", line 925, in _find_spec > File > "/root/cpython/Doc/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py", > line 74, in find_spec > return method() > File > "/root/cpython/Doc/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py", > line 79, in spec_for_distutils > class DistutilsLoader(importlib.util.abc.Loader): > AttributeError: module 'importlib.util' has no attribute 'abc' > Makefile:49: recipe for target 'build' failed > make[1]: *** [build] Error 1 > make[1]:
[Python-Dev] Re: Detect memory leaks in unit tests
Aah, thanks for pointing this out. As a quick side note: are these cool
regrtest features documented anywhere? I remember seeing some mention of
the -R 3:3 argument in the dev guide but nothing else.
Probably useful to have this documented somewhere in the dev guide.
On Wed, May 13, 2020, 5:14 AM Pablo Galindo Salgado
wrote:
> > But again this is for PyObjects only.
>
> Not really, we check also memory blocks:
>
>
> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py#L72
>
> as long as you don't directly call malloc and use one of the Python
> specific APIs like PyMem_Malloc
> then the reflect code should catch that.
>
> On Wed, 13 May 2020 at 12:29, Giampaolo Rodola'
> wrote:
>
>> On Wed, May 13, 2020 at 9:17 AM Ammar Askar wrote:
>>
>>> > Py_DECREF calls in the C code
>>>
>>> I think this part specifically is already covered through refleak
>>> checks:
>>> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py
>>>
>>> Since it can involve the repetition of tests many times, these aren't
>>> run on the CI though, they do get run on the refleak buildbots so
>>> issues get caught eventually:
>>> https://buildbot.python.org/all/#/builders?tags=%2Brefleak
>>>
>>> But again this is for PyObjects only. Were you able to find any memory
>>> leaks with your proof-of-concept? I don't think there's a lot of
>>> chances of someone missing a PyMem_Free call and there's not a lot of
>>> other manual memory management but I could be wrong. Anything found
>>> there could help motivate adding this a bit more.
>>
>>
>> Yeah, I agree it depends on how many PyMem_* occurrences are there, and
>> it probably makes more sense to cover those ones only. Under Modules/* I
>> found:
>>
>> - 24 occurrences for PyMem_RawMalloc
>> - 2 for PyMem_RawCalloc
>> - 106 for PyMem_Malloc
>> - 12 for PyMem_Calloc
>> - 39 for PyMem_New
>> - 5 for " = malloc("
>>
>> I spent an hour covering around 20 of those and didn't find any leak.
>> It's a boring work. I will try to work on it over the next few weeks.
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/C7ZZRDPGIUS7Q6Q4AS4YFPD2OOF56JBO/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/GFY67CXZIJVGWITWXPJ3Y2OMKMAIPJ7Z/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Detect memory leaks in unit tests
> Py_DECREF calls in the C code I think this part specifically is already covered through refleak checks: https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py Since it can involve the repetition of tests many times, these aren't run on the CI though, they do get run on the refleak buildbots so issues get caught eventually: https://buildbot.python.org/all/#/builders?tags=%2Brefleak But again this is for PyObjects only. Were you able to find any memory leaks with your proof-of-concept? I don't think there's a lot of chances of someone missing a PyMem_Free call and there's not a lot of other manual memory management but I could be wrong. Anything found there could help motivate adding this a bit more. On Tue, May 12, 2020 at 4:59 PM Giampaolo Rodola' wrote: > > Hello there, > I would like to discuss a proposal regarding one aspect which AFAIK is > currently missing from cPython's test suite: the ability to detect memory > leaks of functions implemented in the C extension modules. > In psutil I use a test class/framework which calls a function many times, and > fails if the process memory increased after doing so. I do this in order to > quickly detect missing free() or Py_DECREF calls in the C code, but I suppose > there may be other use cases. Here's the class: > https://github.com/giampaolo/psutil/blob/913d4b1d6dcce88dea6ef9382b93883a04a66cd7/psutil/tests/__init__.py#L901 > > Detecting a memory leak is no easy task, and that's because the process > memory fluctuates. Sometimes it may increase (or even decrease!) even if > there's no leak, I suppose because of how the OS handles memory, the Python's > garbage collector, the fact that RSS is an approximation, and who knows what > else. In order to compensate fluctuations I did the following: in case of > failure (mem > 0 after calling fun() N times) I retry the test for up to 5 > times, increasing N (repetitions) each time, so I consider the test a failure > only if the memory keeps increasing across all runs. So for instance, here's > a legitimate failure: > > > psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_disk_partitions > ... > Run #1: extra-mem=696.0K, per-call=3.5K, calls=200 > Run #2: extra-mem=1.4M, per-call=3.5K, calls=400 > Run #3: extra-mem=2.1M, per-call=3.5K, calls=600 > Run #4: extra-mem=2.7M, per-call=3.5K, calls=800 > Run #5: extra-mem=3.4M, per-call=3.5K, calls=1000 > FAIL > > If, on the other hand, the memory increased on one run (say 200 calls) but > decreased on the next run (say 400 calls), then it clearly means it's a false > positive, because memory consumption may be > 0 on the second run, but if > it's lower than the previous run with less repetitions, then it cannot > possibly represent a leak (just a fluctuation): > > > psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_net_connections > ... > Run #1: extra-mem=568.0K, per-call=2.8K, calls=200 > Run #2: extra-mem=24.0K, per-call=61.4B, calls=400 > OK > > This is the best I could come up with as a simple leak detection mechanism to > integrate with CI services, and keep more advanced tools like Valgrind out of > the picture (I just wanted to know if there's a leak, not to debug the leak > itself). In addition, since psutil is able to get the number of fds (UNIX) > and handles (Windows) opened by a process, I also run a separate set of tests > to make sure I didn't forget to call close(2) or CloseHandle() in C. > > Would something like this make sense to have in cPython? Here's a quick PoC I > put together just to show how this thing would look like in practice: > https://github.com/giampaolo/cpython/pull/2/files > A proper work in terms of API coverage would result being quite huge (test > all C modules), and ideally should also include cases where functions raise > an exception when being fed with an improper input. The biggest stopper here > is, of course, psutil, since it's a third party dep, but before getting to > that I wanted to see how this idea is perceived in general. > > Cheers, > > -- > Giampaolo - http://grodola.blogspot.com > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/NFHW4TP3ALY3CVRBVKRI4SRG7BOLZLJH/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DFITPQJHVHFXVODGNBCJOJUZM7YG2WCI/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] Difference between Include/internal and Include/cpython ?
This is the discussion where it was named: https://discuss.python.org/t/poll-what-is-your-favorite-name-for-the-new-include-subdirectory/477?u=ammaraskar and the bug explaining the motivation: https://bugs.python.org/issue35134 >(and why the additional "pycore_XXX.h" naming convention for some ofthose >files?) "* Include/internal/pycore_*.h is the "internal" API" On Sun, Feb 3, 2019 at 10:20 AM Antoine Pitrou wrote: > > > Hello, > > Can someone explain why we have two separate directories > Include/internal and Include/cpython? What is the rule for declaring an > API inside one or another? > > At first sight, it seems to me we're having gratuitous complication > here. For example, I notice that PyFloat_Fini() is declared in > Include/cpython/pylifecycle.h but PyLong_Fini() is declared in > Include/internal/pycore_pylifecycle.h? > > (and why the additional "pycore_XXX.h" naming convention for some of > those files?) > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ammar%40ammaraskar.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] We now have C code coverage!
Oh whoops, sorry about that. I haven't really used mailing lists before and so I assumed hitting reply in gmail would send it to python-dev, not just your personal email. Just so the config file locations are publicly documented, here's what I responded with: > For IDLE's test suite, I use a customized .coveragerc. I strongly prefer to > not abandon that and litter the code with # pragmas. Yup, I agree completely. Having pragmas everywhere is really annoying, its really only useful for small one-off stuff. > In order to make sense of the coverage report and have it be truthful, one > needs to know what options are being used. > Is the --branch flag set? > Is .coveragerc or some other configuration file in use? > If so, what is the content? > Do we have any control over the use and content of exclusion settings? To answer all these questions at once, yeah, we have complete control over all the coverage parameters. Currently we aren't building with `--branch` and don't have a coveragrc file. We could put one in the root of CPython (which would get picked up automatically) but personally I think CI related crud should go into its own folder. Luckily we do have a ".github" folder, so I'd suggest putting the coveragerc file in there and then adding the parameter `--rcfile=.github/coveragerc` All the parameters to coverage.py can be configured here https://github.com/python/cpython/blob/master/.travis.yml#L82 If you wanna send over your IDLE coveragerc file, I can experiment and try to get it working for Travis, or you can explore it yourself if you want. On Fri, Jun 29, 2018 at 9:14 AM, Terry Reedy wrote: > On 6/29/2018 9:25 AM, Brett Cannon wrote: > >> On Thu, Jun 28, 2018, 21:28 Terry Reedy, > <mailto:tjre...@udel.edu>> wrote: > > [question about our coverage bot] > >> Everything is either covered by the Travis or codecov configuration files >> which are both checked into the cpython repo. (I'm on vacation or else I >> would provide links to the files themselves.) > > > Ammar Askar privately asked for my coveragerc file so he could experiment > with the configuration. > > -- > Terry Jan Reedy > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ammar%40ammaraskar.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] We now have C code coverage!
> Is it possible, given that we are not paying for those reports, to > customize the 'exclude_lines' definitions? Do you want to exclude python code or C code? For C code you can mark sections that exclude coverage in lcov with comments like "LCOV_EXCL_START" http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php For Python code, coverage.py also has some comments you can put down to exclude lines: http://coverage.readthedocs.io/en/coverage-4.2/excluding.html If you don't need line level granularity, you can always add files to ignore in our codecov.yml file: https://docs.codecov.io/docs/ignoring-paths ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
