Re: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear?
Victor Stinner writes: > 2013/10/29 Jim Jewett : > > reset() function: > > > > Clear traces of memory blocks allocated by Python. > > > > Does this do anything besides clear? If not, why not just re-use the > > 'clear' name from dicts? > > (I like the reset() name. Charles-François suggested this name > inspired by OProfile API.) Just "reset" implies to me that you're ready to start over. Not just traced memory blocks but accumulated statistics and any configuration (such as Filters) would also be reset. Also tracing would be disabled until started explicitly. If you want it to apply just to the traces, reset_traces() would be more appropriate. > > disable() function: > > > > Stop tracing Python memory allocations and clear traces of > > memory blocks allocated by Python. > > > > I would disable to stop tracing, but I would not expect it to clear > > out the traces it had already captured. If it has to do that, please > > put in some sample code showing how to save the current traces before > > disabling. > > For consistency, you cannot keep traces when tracing is disabled. The > free() must be enabled to remove allocated memory blocks, or next > malloc() may get the same address which would raise an assertion error > (you cannot have two memory blocks at the same address). Then I would not call this "disable". disable() should not "destroy" data. > Just call get_traces() to get traces before clearing them. I can > explain it in the doc. Shouldn't disable() do this automatically, perhaps with an optional discard_traces flag (which would be False by default)? But I definitely agree with Jim: You *must* provide an example here showing how to save the traces (even though it's trivial to do so), because that will make clear that disable() is a destructive operation. (It is not destructive in any other debugging tool that I've used.) Even with documentation, be prepared for user complaints. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
Nick Coghlan writes: > OK, time for me to stop trying to remember the details of the problem > I'm trying to solve and go look them up in the source code :) OMG! Even Nick does that. I feel better now! ;) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 454 (tracemalloc) disable ==> clear?
(Tue Oct 29 12:37:52 CET 2013) Victor Stinner wrote: > For consistency, you cannot keep traces when tracing is disabled. > The free() must be enabled to remove allocated memory blocks, or > next malloc() may get the same address which would raise an assertion > error (you cannot have two memory blocks at the same address). That seems like an a quirk of the implementation, particularly since the actual address is not returned to the user. Nor do I see any way of knowing when that allocation is freed. Well, unless I missed it... I don't see how to get anything beyond the return value of get_traces, which is a (time-ordered?) list of allocation size with then-current call stack. It doesn't mention any attribute for indicating that some entries are de-allocations, let alone the actual address of each allocation. > For the reason explained above, it's not possible to disable the whole > module temporarly. > Internally, tracemalloc uses a thread-local variable (called the > "reentrant" flag) to disable temporarly tracing allocations in the > current thread. It only disables tracing new allocations, > deallocations are still proceed. Even assuming the restriction is needed, this just seems to mean that disabling (or filtering) should not affect de-allocation events, for fear of corrupting tracemalloc's internal structures. In that case, I would expect disabling (and filtering) to stop capturing new allocation events for me, but I would still expect tracemalloc to do proper internal maintenance. It would at least explain why you need both disable *and* reset; reset would empty those internal structures, so that tracemalloc could shortcut that maintenance. I would NOT assume that I needed to call reset when changing the filters, nor would I assume that changing them threw out existing traces. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
On 30 October 2013 09:02, Eric Snow wrote: > On Tue, Oct 29, 2013 at 3:32 AM, Nick Coghlan wrote: >> On 29 Oct 2013 14:45, "Eric Snow" wrote: >>> I'd rather give loaders another optional method: >>> supports_reload(name). Complicating the spec methods signatures (to >>> support passing the old spec through) just for the sake of reload >>> seems less than optimal. I don't see any benefit to exposing the old >>> spec to loader.exec_module(). >> >> I do: it lets the loader tell if anything changed from the previous load >> operation, allowing it to make an informed decision on whether or not to >> permit the reload operation or throw an exception. >> >> However, I accept such a check would be better implemented as a separate >> yes/no method, so it makes sense to postpone this aspect to 3.5 at the >> earliest. > > With the PEP, loaders are no longer in charge of the module-related > boilerplate. This means that by the time exec_module() gets called, > they won't be able to determine for themselves if it's a reload by > checking sys.modules. That's the point of adding > loader.supports_reload(). (I haven't added that rationale to the PEP > yet). Ah, good point. > If it makes a difference to loaders to also look at the previous spec > during supports_reload(), then I think we should add that parameter > now. Otherwise if we add the parameter later it makes it messier. > The signature would now be: > > Loader.supports_reload(name, old_spec) OK, time for me to stop trying to remember the details of the problem I'm trying to solve and go look them up in the source code :) One of my goals here is to be able to migrate extension loading from the old API to the new plugin API. That means being able to break up the existing load_module implementation: http://hg.python.org/cpython/file/1787277915e9/Python/importdl.c#l23 For loading, that's a fairly straightforward create_module/exec_module split, but reloading gets a bit more interesting. Specifically, I'd like to be able to implement the relevant parts of _PyImport_FindExtensionObject as a precheck for reloading: http://hg.python.org/cpython/file/1787277915e9/Python/import.c#l533 That means just having access to the module name isn't enough: the extensions dictionary is keyed by a (name, filename) 2-tuple rather than just by the module name. Using the find_spec API, that filename would be stored in the loader state on the spec object rather than being looked up anew during the load operation. However, rereading this method also indicates that I really want to know *in exec_module* whether this is a reload or not, since extension loading needs to handle reloads differently from initial execution. So I'm back to my original preference: I'd like the previous spec to be passed to exec_module in the reloading case. If reloading is not supported at all, it's up to the loader to throw an appropriate exception when the the previous spec is not None. If loading and reloading doesn't make any difference, then they can just ignore it. But when both are supported, but handled differently (as for extension modules), then that can be detected, and the information from the old spec (including the original loader and loader state) is available if needed. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
On Tue, Oct 29, 2013 at 3:32 AM, Nick Coghlan wrote: > On 29 Oct 2013 14:45, "Eric Snow" wrote: >> I'd rather give loaders another optional method: >> supports_reload(name). Complicating the spec methods signatures (to >> support passing the old spec through) just for the sake of reload >> seems less than optimal. I don't see any benefit to exposing the old >> spec to loader.exec_module(). > > I do: it lets the loader tell if anything changed from the previous load > operation, allowing it to make an informed decision on whether or not to > permit the reload operation or throw an exception. > > However, I accept such a check would be better implemented as a separate > yes/no method, so it makes sense to postpone this aspect to 3.5 at the > earliest. With the PEP, loaders are no longer in charge of the module-related boilerplate. This means that by the time exec_module() gets called, they won't be able to determine for themselves if it's a reload by checking sys.modules. That's the point of adding loader.supports_reload(). (I haven't added that rationale to the PEP yet). If it makes a difference to loaders to also look at the previous spec during supports_reload(), then I think we should add that parameter now. Otherwise if we add the parameter later it makes it messier. The signature would now be: Loader.supports_reload(name, old_spec) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!
2013/10/26 Kristján Valur Jónsson : > In that case, how about adding a client/server feature? > > If you standardize the format, a minimal tracing client could write a log, > or send it to a socket, in a way that can be turned into a snapshot by a > corresponsing utility reading from a file or listenting to a socket. > > Just a though. Could be a future addition… tracemalloc maintains a dictionary of all allocated memory blocks, which is slow and eats a lot of memory. You don't need tracemalloc to log calls to malloc/realloc/free. You can write your own hook using the PEP 445 (malloc API). A code just writing to stderr should not be longer than 100 linues (tracemalloc is closer to 2500 lines). As I wrote, I began to log all calls into a file, but it was too slow. The memory leak occurred after between 2 hours and 1 day. To know the current state (ex: total memory currently allocated), you have to parse the whole log file, which can be really huge. Writing the log file, transfering the log file and analyzing the log file are too slow were too slow in my use cases. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RELEASED: Python 2.6.9 final
Thanks Barry! That was a great announcement note, right from the heart. Python 2.6 may retire, but Python is far from retired, and I am looking forward to many more great interactions with you here on python-dev. Cheers! --Guido On Tue, Oct 29, 2013 at 10:48 AM, Barry Warsaw wrote: > Hello Pythoneers and Pythonistas, > > Five years ago this month, we released Python 2.6. It was an important > version in Python's evolution, as it was developed in tandem with Python > 3.0 > and had the express intent of starting the transition toward Python 3. > > Amazingly, here we are five years later, and I am happy (and a little sad) > to > announce the Python 2.6.9 final release. > > Python 2.6.9 is a security-only source-only release, and with this, I > officially retire the Python 2.6 branch. All maintenance of Python 2.6, > including for security issues, has now ended. > > So too has my latest stint as Python Release Manager. Over the 19 years I > have been involved with Python, I've been honored, challenged, stressed, > and > immeasurably rewarded by managing several Python releases. I wish I could > thank each of you individually for all the support you've generously given > me > in this role. You deserve most of the credit for all these great releases; > I'm just a monkey pushing buttons. :) > > Here then are some release notes about 2.6.9. > > For ongoing maintenance, please see Python 2.7. > > Future issues with Python 2.6 may still be tracked in the Python bug > tracker, > and you may wish to visit specific issues to see if unofficial patches are > available for 2.6. > > http://bugs.python.org > > The source can be downloaded from: > > http://www.python.org/download/releases/2.6.9/ > > You can also see what's changed since Python 2.6.8: > > http://www.python.org/download/releases/2.6.9/NEWS.txt > > Users on OS X 10.9 (Mavericks) please note that issue 18458, which can > crash > the interactive interpreter, is *not* fixed in 2.6.9. If this issue > affects > you, please review the tracker for possible options: > > http://bugs.python.org/issue18458 > > Enjoy, > -Barry > (on behalf of the Python development community) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] RELEASED: Python 2.6.9 final
Hello Pythoneers and Pythonistas, Five years ago this month, we released Python 2.6. It was an important version in Python's evolution, as it was developed in tandem with Python 3.0 and had the express intent of starting the transition toward Python 3. Amazingly, here we are five years later, and I am happy (and a little sad) to announce the Python 2.6.9 final release. Python 2.6.9 is a security-only source-only release, and with this, I officially retire the Python 2.6 branch. All maintenance of Python 2.6, including for security issues, has now ended. So too has my latest stint as Python Release Manager. Over the 19 years I have been involved with Python, I've been honored, challenged, stressed, and immeasurably rewarded by managing several Python releases. I wish I could thank each of you individually for all the support you've generously given me in this role. You deserve most of the credit for all these great releases; I'm just a monkey pushing buttons. :) Here then are some release notes about 2.6.9. For ongoing maintenance, please see Python 2.7. Future issues with Python 2.6 may still be tracked in the Python bug tracker, and you may wish to visit specific issues to see if unofficial patches are available for 2.6. http://bugs.python.org The source can be downloaded from: http://www.python.org/download/releases/2.6.9/ You can also see what's changed since Python 2.6.8: http://www.python.org/download/releases/2.6.9/NEWS.txt Users on OS X 10.9 (Mavericks) please note that issue 18458, which can crash the interactive interpreter, is *not* fixed in 2.6.9. If this issue affects you, please review the tracker for possible options: http://bugs.python.org/issue18458 Enjoy, -Barry (on behalf of the Python development community) signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
> Gesendet: Dienstag, 29. Oktober 2013 um 10:54 Uhr
> Von: "Victor Stinner"
> An: "Georg Brandl"
> Cc: "Python Dev"
> Betreff: Re: [Python-Dev] cpython: Issue #18408: Fix
> PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
>
> 2013/10/29 Georg Brandl :
> >> diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
> >> --- a/Objects/unicodeobject.c
> >> +++ b/Objects/unicodeobject.c
> >> @@ -3766,6 +3766,7 @@
> >> return NULL;
> >> _PyUnicode_UTF8(unicode) =
> >> PyObject_MALLOC(PyBytes_GET_SIZE(bytes) + 1);
> >> if (_PyUnicode_UTF8(unicode) == NULL) {
> >> +PyErr_NoMemory();
> >> Py_DECREF(bytes);
> >> return NULL;
> >> }
> >
> > Shouldn't this (and related commits from #18408) have been committed to the
> > 3.3
> > branch?
>
> All changes of #18408 "should" be backported, but I don't plan to
> backport them. It is not trivial to backport them. Nobody complained
> before (memory allocation failure are usually bad handled anyway, it
> will crash later if it does not crash here). And I'm not 100%
> confident that these changes would not break anything.
OK, that's a good enough reason, if it's "only" no-memory-related errors.
> Examples of possible regression:
>
> - PyEval_GetLocals() now raises an exception in case of an error. This
> change "should" not break anything, because callers were already
> raising an exception when PyEval_GetLocals() returns NULL. The builtin
> locals() function was not raising an exception in case of error, but
> it's probably because it is very unlikely that the function is called
> without any frame (PyEval_GetFrame(); returns NULL).
>
> - many functions now fail with an assertion error when they are called
> with an exception set (assert(!PyErr_Occurred());) because they may
> replace the exception without noticing the caller
>
> - I tried to check reference counters, but I may have introduce a
> regression leak in the error handling code
>
> If there is really a regression, I prefer to limit it to the new
> version, not to a stable version.
>
> Note: I'm not saying that I'm 0% confident in my changes :-)
And I certainly think they have a better than 0% chance of being correct ;)
Georg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle
> Gesendet: Dienstag, 29. Oktober 2013 um 10:38 Uhr > Von: "Victor Stinner" > An: "Georg Brandl" > Cc: "Python Dev" > Betreff: Re: [Python-Dev] cpython: Issue #18408: Add a new > PyFrame_FastToLocalsWithError() function to handle > > 2013/10/29 Georg Brandl : > > Am 29.10.2013 01:19, schrieb victor.stinner: > >> http://hg.python.org/cpython/rev/4ef4578db38a > >> changeset: 86715:4ef4578db38a > >> user:Victor Stinner > >> date:Tue Oct 29 01:19:37 2013 +0100 > >> summary: > >> Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to > >> handle > >> exceptions when merging fast locals into f_locals of a frame. > >> PyEval_GetLocals() now raises an exception and return NULL on failure. > > > > You'll have to either make this private or document it. > > > > Georg > > I didn't find the documentation of PyFrame_FastToLocals(). Please open > an issue if you consider that these functions must be documented. I see you opened one already; thanks for that. In any case, our policy is that all public API function (those without _Py prefixes) *should* be documented. > IMO these functions should be private because very low-level, but it's > too late: PyFrame_FastToLocals() is part of the stable ABI, even if > it's not documented. Thankfully not; all of frameobject.h is not part of the stable ABI. cheers, Georg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tracemalloc.get_object_traceback() and "unclosed" ResourceWarning
2013/10/29 Nick Coghlan : > I was thinking you could turn on warnings->errors and then catch the > exception, and use that to get hold of the original object that > triggered the resource warning, and then fed *that* into > tracemalloc.get_object_traceback(). But, alas, even if we tweaked > ResourceWarning to make sure that was possible, exceptions from > __del__ methods are squashed by the interpreter, so you're only left > with the text output. > > Didn't an earlier version of the PEP have the ability to find a > traceback based on an object's ID rather than the object itself? If > that capability existed (including for snapshots), then an object id > printed in an error message could be matched up with the tracemalloc > output. (ResourceWarning currently only prints the id if the object > repr includes the id, but we could change that to ensure the object id > is always shown). An address is only valid while the object is alive. When an object is destroyed, its trace is immediatly removed. So it's safer to get the traceback when the warning is emitted, not before, not after. > Anyway, I don't think this feature needs to be included in the PEP, > but it's something we may decide we want to add later. I don't want to add the ResourceWarning hack to the PEP :-) I'm just asking for help to find the best solution. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tracemalloc.get_object_traceback() and "unclosed" ResourceWarning
On 29 October 2013 22:07, Victor Stinner wrote: > Hi, > > Since Python 3.3, when a file or socket object is destroyed before > being closed, a ResourceWarning is emitted. The warning is emitted > where the object is destroyed: it's common to see "gc.collect()" > location in the Python test suite for example. > > Tarek Ziadé asked on Twitter if there is a tool to track the origin of > the object: > "I wonder is there's a tool to trace back objects declarations that > raise ResourceWarning in Py" > https://twitter.com/tarek_ziade/status/394006754779877377 > > Nick Coghlan replied that tracemalloc may help: > "@tarek_ziade tracemalloc (http://www.python.org/dev/peps/pep-0454/ … > - not accepted yet) will likely make it possible in 3.4, dunno about > earlier versions." > https://twitter.com/ncoghlan_dev/status/394010756242350080 > > I checked the code and it's not so simple. FileIO destructor emits the > warning (fileio_dealloc_warn() in Modules/_io/fileio.c), but it's not > possible to "hook" this function. I tried something with weakref, but > I failed to set my hook. Ah, I wasn't clear about what I meant (yay, 140 characters!), but I was also wrong. I was thinking you could turn on warnings->errors and then catch the exception, and use that to get hold of the original object that triggered the resource warning, and then fed *that* into tracemalloc.get_object_traceback(). But, alas, even if we tweaked ResourceWarning to make sure that was possible, exceptions from __del__ methods are squashed by the interpreter, so you're only left with the text output. Didn't an earlier version of the PEP have the ability to find a traceback based on an object's ID rather than the object itself? If that capability existed (including for snapshots), then an object id printed in an error message could be matched up with the tracemalloc output. (ResourceWarning currently only prints the id if the object repr includes the id, but we could change that to ensure the object id is always shown). Anyway, I don't think this feature needs to be included in the PEP, but it's something we may decide we want to add later. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
On 29 October 2013 19:54, Victor Stinner wrote: > > If there is really a regression, I prefer to limit it to the new > version, not to a stable version. > > Note: I'm not saying that I'm 0% confident in my changes :-) Sounds like a solid rationale for limiting the changes to 3.4 to me :) /me should take another run at the known crasher tests one of these days. Maybe during the beta cycle... Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] tracemalloc.get_object_traceback() and "unclosed" ResourceWarning
Hi, Since Python 3.3, when a file or socket object is destroyed before being closed, a ResourceWarning is emitted. The warning is emitted where the object is destroyed: it's common to see "gc.collect()" location in the Python test suite for example. Tarek Ziadé asked on Twitter if there is a tool to track the origin of the object: "I wonder is there's a tool to trace back objects declarations that raise ResourceWarning in Py" https://twitter.com/tarek_ziade/status/394006754779877377 Nick Coghlan replied that tracemalloc may help: "@tarek_ziade tracemalloc (http://www.python.org/dev/peps/pep-0454/ … - not accepted yet) will likely make it possible in 3.4, dunno about earlier versions." https://twitter.com/ncoghlan_dev/status/394010756242350080 I checked the code and it's not so simple. FileIO destructor emits the warning (fileio_dealloc_warn() in Modules/_io/fileio.c), but it's not possible to "hook" this function. I tried something with weakref, but I failed to set my hook. Does anyone see how to reuse tracemalloc (which is optional and disabled by default) in FileIO destructor? The file/object destructor may retrieve somehow the most recent frame of the object and call PyErr_WarnExplicitObject() instead of PyErr_WarnFormat() to pass the filename and lineno of the object. But how would Python decide to use tracemalloc or not? Add a new genric "get object traceback" callback which would be set by tracemalloc when the module is enabled? A frame is maybe not enough, it would be better to get the whole traceback! Do you know other features where it would be nice to know the origin of a Python object? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear?
2013/10/29 Jim Jewett : > reset() function: > > Clear traces of memory blocks allocated by Python. > > Does this do anything besides clear? If not, why not just re-use the > 'clear' name from dicts? (I like the reset() name. Charles-François suggested this name inspired by OProfile API.) > disable() function: > > Stop tracing Python memory allocations and clear traces of > memory blocks allocated by Python. > > I would disable to stop tracing, but I would not expect it to clear > out the traces it had already captured. If it has to do that, please > put in some sample code showing how to save the current traces before > disabling. For consistency, you cannot keep traces when tracing is disabled. The free() must be enabled to remove allocated memory blocks, or next malloc() may get the same address which would raise an assertion error (you cannot have two memory blocks at the same address). Just call get_traces() to get traces before clearing them. I can explain it in the doc. 2013/10/29 Kristján Valur Jónsson : > I was thinking something similar. It would be useful to be able to "pause" > and "resume" > if one is doing any analysis work in the live environment. This would reduce > the > need to have "Filter" objects. For the reason explained above, it's not possible to disable the whole module temporarly. Internally, tracemalloc uses a thread-local variable (called the "reentrant" flag) to disable temporarly tracing allocations in the current thread. It only disables tracing new allocations, deallocations are still proceed. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
2013/10/29 Georg Brandl :
>> diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
>> --- a/Objects/unicodeobject.c
>> +++ b/Objects/unicodeobject.c
>> @@ -3766,6 +3766,7 @@
>> return NULL;
>> _PyUnicode_UTF8(unicode) = PyObject_MALLOC(PyBytes_GET_SIZE(bytes)
>> + 1);
>> if (_PyUnicode_UTF8(unicode) == NULL) {
>> +PyErr_NoMemory();
>> Py_DECREF(bytes);
>> return NULL;
>> }
>
> Shouldn't this (and related commits from #18408) have been committed to the
> 3.3
> branch?
All changes of #18408 "should" be backported, but I don't plan to
backport them. It is not trivial to backport them. Nobody complained
before (memory allocation failure are usually bad handled anyway, it
will crash later if it does not crash here). And I'm not 100%
confident that these changes would not break anything.
Examples of possible regression:
- PyEval_GetLocals() now raises an exception in case of an error. This
change "should" not break anything, because callers were already
raising an exception when PyEval_GetLocals() returns NULL. The builtin
locals() function was not raising an exception in case of error, but
it's probably because it is very unlikely that the function is called
without any frame (PyEval_GetFrame(); returns NULL).
- many functions now fail with an assertion error when they are called
with an exception set (assert(!PyErr_Occurred());) because they may
replace the exception without noticing the caller
- I tried to check reference counters, but I may have introduce a
regression leak in the error handling code
If there is really a regression, I prefer to limit it to the new
version, not to a stable version.
Note: I'm not saying that I'm 0% confident in my changes :-)
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear?
A > > > disable() function: > > Stop tracing Python memory allocations and clear traces of > memory blocks allocated by Python. > > I would disable to stop tracing, but I would not expect it to clear out the > traces it had already captured. If it has to do that, please put in some > sample > code showing how to save the current traces before disabling. I was thinking something similar. It would be useful to be able to "pause" and "resume" if one is doing any analysis work in the live environment. This would reduce the need to have "Filter" objects. K ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle
Le Tue, 29 Oct 2013 10:38:23 +0100, Victor Stinner a écrit : > 2013/10/29 Georg Brandl : > > Am 29.10.2013 01:19, schrieb victor.stinner: > >> http://hg.python.org/cpython/rev/4ef4578db38a > >> changeset: 86715:4ef4578db38a > >> user:Victor Stinner > >> date:Tue Oct 29 01:19:37 2013 +0100 > >> summary: > >> Issue #18408: Add a new PyFrame_FastToLocalsWithError() function > >> to handle exceptions when merging fast locals into f_locals of a > >> frame. PyEval_GetLocals() now raises an exception and return NULL > >> on failure. > > > > You'll have to either make this private or document it. > > > > Georg > > I didn't find the documentation of PyFrame_FastToLocals(). Please open > an issue if you consider that these functions must be documented. Just because something else isn't documented doesn't mean new APIs shouldn't be documented. Please let's not enter such a vicious circle. > IMO these functions should be private because very low-level, but it's > too late: PyFrame_FastToLocals() is part of the stable ABI, even if > it's not documented. It's not too late for PyFrame_FastToLocalsWithError() to be private, though. I agree with Georg: you should either make it private or document it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle
2013/10/29 Georg Brandl : > Am 29.10.2013 01:19, schrieb victor.stinner: >> http://hg.python.org/cpython/rev/4ef4578db38a >> changeset: 86715:4ef4578db38a >> user:Victor Stinner >> date:Tue Oct 29 01:19:37 2013 +0100 >> summary: >> Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle >> exceptions when merging fast locals into f_locals of a frame. >> PyEval_GetLocals() now raises an exception and return NULL on failure. > > You'll have to either make this private or document it. > > Georg I didn't find the documentation of PyFrame_FastToLocals(). Please open an issue if you consider that these functions must be documented. IMO these functions should be private because very low-level, but it's too late: PyFrame_FastToLocals() is part of the stable ABI, even if it's not documented. PyFrame_FastToLocalsWithError() is the same than PyFrame_FastToLocals(), except that it has a return value and keep the exception. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
On 29 Oct 2013 14:45, "Eric Snow" wrote:
>
> On Sat, Oct 26, 2013 at 11:03 PM, Nick Coghlan wrote:
> > I don't think we can postpone it to later, as we need to be clear on:
> >
> > 1. Does reload use __name__ or __spec__.name when both are available?
> > 2. Does __name__ get restored to its original value if reloading via
> > __spec__.name?
> > 3. Do other import related attributes get restored to their original
values?
> > 4. Does create_module get called if the loader has an exec_module
method?
> > 5. Does the loader have access to the previous spec when reloading a
module?
> >
> > My answers to these questions (note that my answer to 2 differs from
> > what I had in my initial sketch):
> >
> > 1. __spec__.name
> > 2. Yes, __name__ is updated to match __spec__.name, *expect* if it is
> > currently "__main__"
> > 3. Yes, other import related attributes are restored to their baseline
values
> > 4. No, create_module is never called when reloading
>
> I agree on all of these. I'm adding a "How reloading will work"
> section to the PEP.
>
> > 5. Currently no, but I'm thinking that may be worth changing (more on
> > that below)
> >
> > The reload() implementation in my message is actually based on the
> > current importlib.reload implementation. The only PEP 451 related
> > changes were:
> >
> > - using __spec__.name (if available) instead of __name__
> > - checking all parent modules exist rather than just the immediate
parent module
> > - calling import.find_spec() rather than using the __loader__ attribute
directly
> > - being explicit that __name__ is left at the value it had prior to the
reload
> > - handling the namespace package, exec_module and no exec_module cases
> >
> > I also added an explicit check that the module was re-used properly,
> > but I agree *that* change is out of scope for the PEP and should be
> > considered as a separate question.
> >
> > Now, regarding the signature of exec_module(): I'm back to believing
> > that loaders should receive a clear indication that a reload is taking
> > place. Legacy loaders have to figure that out for themselves (by
> > seeing that the module already exists in sys.modules), but we can do
> > better for the new API by making the exec_module signature look like:
> >
> > def exec_module(self, module, previous_spec=None):
> > # module is as per the current PEP 451 text
> > # previous_spec would be set *only* in the reload() case
> > # loaders that don't care still need to accept it, but can
> > just ignore it
> > ...
>
> I'd rather give loaders another optional method:
> supports_reload(name). Complicating the spec methods signatures (to
> support passing the old spec through) just for the sake of reload
> seems less than optimal. I don't see any benefit to exposing the old
> spec to loader.exec_module().
I do: it lets the loader tell if anything changed from the previous load
operation, allowing it to make an informed decision on whether or not to
permit the reload operation or throw an exception.
However, I accept such a check would be better implemented as a separate
yes/no method, so it makes sense to postpone this aspect to 3.5 at the
earliest.
Cheers,
Nick.
>
> >
> > So, with those revisions, the reload() semantics would be defined as:
> >
> > def reload(module):
> > # Get the name to reload from the spec if it is available,
> > # otherwise use __name__ directly
> > current_spec = getattr(module, "__spec__")
> > current_name = module.__name__
> > name_to_reload = current_name if current_spec is None else
> > current_spec.name
> >
> > # Check this module is properly imported before trying to
reload it
> > loaded_module = sys.modules.get(name_to_reload)
> > if loaded_module is not module:
> > msg = "module {} not in sys.modules"
> > raise ImportError(msg.format(name_to_reload),
name=name_to_reload)
> > parent_name = name_to_reload.rpartition('.')[0]
> > while parent_name:
> > if parent_name not in sys.modules:
> > msg = "parent {!r} not in sys.modules"
> > raise ImportError(msg.format(parent_name),
name=parent_name)
> > parent_name = parent_name.rpartition('.')[0]
> >
> > # Check for a modified spec (e.g. if the import hooks have
changed)
> > updated_spec = importlib.find_spec(name_to_reload)
> >
> > # The import-related module attributes get updated here
> > _init_module_attrs(loaded_module, updated_spec)
> > if current_name == "__main__":
> > loaded_module.__name__ = "__main__"
> >
> > # Now re-execute the module
> > loader = updated_spec.loader
> > if loader is None:
> > # Namespace package, already reinitialised above
> > return loaded_module
> > elif hasattr(loader, 'exec_module'):
> > loader.exec_module(module, current_spec
