Re: [Python-Dev] pathlib (PEP 428) status
Le Thu, 24 Oct 2013 00:38:02 +0200, Christian Heimes a écrit : > Am 23.10.2013 23:37, schrieb Charles-François Natali: > > Hi, > > > > What's the current status of pathlib? Is it targeted for 3.4? > > > > It would be a really nice addition, and AFAICT it has already been > > maturing a while on pypi, and discussed several times here. > > If I remember correctly, the only remaining issue was stat()'s > > result caching. > > Hi, > > I'd like to see pathlib in 3.4 as well. Last week at PyCon.DE in > Cologne several people have asked me about pathlib. We even had a > BarCamp session about path libraries for Python. A couple of German > Python users have promised to contribute doc improvements soonish. > > AFAIK stat caching and a os.listdir() generator with stat `recycling` > (dirent->d_type) are open issues. I suggest Python 3.4 should ignore > these features for now but prepare the API and documentation for > future enhancements. Thanks for the push. I have to cleanup / remove stat caching from the PEP (not sure yet which one, Nick expressed interest during the last discussion) and then polish it for pronouncement. Still, I need to find someone to pronounce on it. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed
On Tuesday, Apple released OS X 10.9 (a.k.a. Mavericks). There has already been a lot of interest in it, in part because Apple has made it available for free and has made it easy for users with 10.8, 10.7, and (most) 10.6 systems to upgrade directly to 10.9. Unfortunately, there are issues with our current maintenance releases (3.3.2 and 2.7.5) on OS X 10.9 that call for new maintenance releases as soon as possible. One is critical in that it causes the interpreter to crash when running in interactive mode (http://bugs.python.org/issue18458). The problem was due to a long-standing compatibility issue in libedit's readline compatibility layer that upstream has finally fixed and Apple has now shipped in 10.9. Because the python.org installers dynamically link to libedit, the original workaround in readline.c for the original design flaw in history indexing now causes a segfault on 10.9 when the user types in the second command interactively. Not good. Ronald devised a fix that allows readline.so at runtime to detect and work with either version of libedit so that we continue to have binary compatibility across multiple OS X releases. That fix is already out in the 3.4.0 alphas and backported to the 3.3 and 2.7 branches, awaiting release there. Just in the last 12 hours, there have been at least four duplicates of the issue reported by users. I've updated the original issue to explicitly mention 10.9, now that it is no longer under NDA, and to provide a downloadable script for inexperienced users to workaround the problem by "removing" readline.so. Presumably, as word gets out, there will be fewer duplicate issues opened but the impact will remain. The only other major 10.9-related issue that I'm aware of at the moment is one with the OS X native Tk (of course!) that can affect IDLE and other Tkinter applications. See http://bugs.python.org/issue19373 for details. The workaround for it is to run Tk in 32-bit mode. I've "engaged" the Tk project in the problem. Since it can affect all users of Tk, and not just Python ones, I'm hopeful that there will be a fix soon. Note that, for 3.4.0, we are now packaging a private copy of Tcl/Tk with the 10.6+ OS X installer to avoid the chronic problems caused by Apple not updating the buggy system Tcl/Tk and forcing users to download and install a third-party Tcl/Tk just to be able to use IDLE. With the concurrence of the release managers, I am in the process of backporting that packaging support into 3.3 and 2.7. That means any Tk fix will likely require an update to those private copies. Because of the criticality of the readline issue, we really need to get 2.7.6 and 3.3.3 out as soon as practical. Besides the Tk packaging, there are a few other 10.9 build-related issues that should go into these releases. I expect to complete testing across all releases and have everything checked-in sometime this weekend. I don't think we should wait for the Issue19373 Tk fix if it isn't available when everything else is. If necessary, we can later re-release the OS X installers with just the updated Tcl/Tk shared libraries. I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this point. But I'd like Benjamin and Georg to propose an aggressive schedule so we can get these fixes out there. -- Ned Deily, [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed
Am 24.10.2013 11:11, schrieb Ned Deily: > I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this > point. But I'd like Benjamin and Georg to propose an aggressive schedule so > we can get these fixes out there. > We've basically agreed to do rc releases this weekend. I don't have time the weekend of Nov 2, so 3.3.3 final would be scheduled for Nov 9. There are a few "unlimited read" issues still open; I will have a look at porting their patches tomorrow. Georg ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed
Am 24.10.2013 11:26, schrieb Georg Brandl: > Am 24.10.2013 11:11, schrieb Ned Deily: > >> I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this >> point. But I'd like Benjamin and Georg to propose an aggressive schedule so >> we can get these fixes out there. >> > > We've basically agreed to do rc releases this weekend. I don't have time the > weekend of Nov 2, so 3.3.3 final would be scheduled for Nov 9. There are a > few "unlimited read" issues still open; I will have a look at porting their > patches tomorrow. There seems to be a problem with the security fix "Re-seed OpenSSL's PRNG after fork": http://bugs.python.org/issue18747 http://bugs.python.org/issue19227 Perhaps it's best to disable the pthread_atfork() handler for the upcoming releases and replace it with security note in the ssl module, os.fork() and release notes. If you are using fork() and the ssl module in the same application then you must re-seed the PRNG with ssl.RAND_add() every now and then. Christian ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed
2013/10/24 Christian Heimes : > There seems to be a problem with the security fix "Re-seed OpenSSL's > PRNG after fork": > > http://bugs.python.org/issue18747 > http://bugs.python.org/issue19227 Yes, Charles Francois warned us that adding a pthread_atfork() hook to add entropy may add a deadlock. Such change should *not* be added to stable releases, even if it is supposed to improve the security. > Perhaps it's best to disable the pthread_atfork() handler for the > upcoming releases and replace it with security note in the ssl module, > os.fork() and release notes. IMO the best place to fix the bug is in OpenSSL directly: RAND_bytes() function of OpenSSL can detect a fork using getpid() and add more entropy (in the child or maybe in the parent process). OpenSSL has access to entropy sources and knows all mutexes, whereas Python can only guess the list of mutexes (there are probably many more private mutexes). OpenSSL may use pthread_atfork() internally. http://www.openwall.com/lists/oss-security/2013/04/12/3 "I believe it is wrong to fix this in PostgreSQL. Rather, this is a bug in the OpenSSL fork protection code. It should either install a fork hook, or reseed the PRNG from /dev/urandom if a PID change is detected." I wrote a patch for OpenSSL long time ago (when working on my Hasard project, PRNG library), but I don't remember if I proposed it upstream: https://bitbucket.org/haypo/hasard/src/tip/patches/openssl_rand_fork.patch > > If you are using fork() and the ssl module in the same application then > you must re-seed the PRNG with ssl.RAND_add() every now and then. > You should explain that the issue only occurs if you fork more than 2^16 times and two child processes get the same pid. I wouldn't say that it is unlikely, but just that there is no reason to scare users if they are not vulnerable. The issue can occur if a process spawn a lot of processes. Reminder: you are supposed to only call exec() after fork(), nothing else :-) (Only signal-safe functions are officially supported between exec() and fork()). Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
On 24 October 2013 16:05, Eric Snow wrote: > I've had some offline discussion with Brett and Nick about PEP 451 > which has led to some meaningful clarifications in the PEP. In the > interest of pulling further discussions back onto this > (archived/public) list, here's an update of what we'd discussed and > where things are at. :) There's also the fact Guido asked Brett and I to be co-delegates for this PEP, and we accepted (we both already agree with the overall concept, so it's just a matter of hammering out the final API details). > * path entry finders indicate that they found part of a possible > namespace package by returning a spec with no loader set (but with > submodule_search_locations set). Brett wanted some clarification on > this. > * The name/path signature and attributes of file-based finders in > importlib will no longer be changing. Brett had some suggestions on > the proposed change and it became clear that the the change was > actually pointless. > * I've asserted that there shouldn't be much difficulty in adjusting > pkgutil and other modules to work with ModuleSpec. More specifically: importlib finders will still expose the previous import plugin API for backwards compatibility, so the worst case scenario is that we miss something and there's an API somewhere that doesn't accept import plugins that only provide the new API and not the old one. However, the PEP should explicitly state that any such omissions will be treated as bugs in the 3.4.x series (although we'll aim to handle them all in the initial implementation). Thanks for recording the details of the earlier off-list discussion :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed
Am 24.10.2013 13:36, schrieb Victor Stinner: > IMO the best place to fix the bug is in OpenSSL directly: RAND_bytes() > function of OpenSSL can detect a fork using getpid() and add more > entropy (in the child or maybe in the parent process). OpenSSL has > access to entropy sources and knows all mutexes, whereas Python can > only guess the list of mutexes (there are probably many more private > mutexes). OpenSSL may use pthread_atfork() internally. That's not going to happen anytime soon (if ever). It doesn't solve old installation, too. You are forgetting that ssl.RAND_bytes() and RAND_pseudo_bytes() are our least concern. The problem also affects the TLS/SSL handshake and any internal OpenSSL code that needs a CPRNG. > Reminder: you are supposed to only call exec() after fork(), nothing > else :-) (Only signal-safe functions are officially supported between > exec() and fork()). How are forking web servers suppose to work if you can't use read() or write()? :) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pathlib (PEP 428) status
On 24 October 2013 18:17, Antoine Pitrou wrote: > Le Thu, 24 Oct 2013 00:38:02 +0200, > Christian Heimes a écrit : >> Am 23.10.2013 23:37, schrieb Charles-François Natali: >> > Hi, >> > >> > What's the current status of pathlib? Is it targeted for 3.4? >> > >> > It would be a really nice addition, and AFAICT it has already been >> > maturing a while on pypi, and discussed several times here. >> > If I remember correctly, the only remaining issue was stat()'s >> > result caching. >> >> Hi, >> >> I'd like to see pathlib in 3.4 as well. Last week at PyCon.DE in >> Cologne several people have asked me about pathlib. We even had a >> BarCamp session about path libraries for Python. A couple of German >> Python users have promised to contribute doc improvements soonish. >> >> AFAIK stat caching and a os.listdir() generator with stat `recycling` >> (dirent->d_type) are open issues. I suggest Python 3.4 should ignore >> these features for now but prepare the API and documentation for >> future enhancements. > > Thanks for the push. I have to cleanup / remove stat caching from the > PEP (not sure yet which one, Nick expressed interest during the last > discussion) and then polish it for pronouncement. A clean stat caching mechanism would potentially help resolve an architectural problem with walkdir (https://pypi.python.org/pypi/walkdir), where offering stat-based filtering as a pipeline of independent iterators would significantly increase the number of stat calls compared to doing such filtering directly outside the iterator. However, since walkdir doesn't currently offer such stat-based filters, and it's unlikely I'll be adding it any time soon, deferring a stat caching design sounds like a reasonable idea to me. > Still, I need to find someone to pronounce on it. I'd offer, but I already have a scary number of things I need to get done by beta 1 :P Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!
> -Original Message- > From: Victor Stinner [mailto:[email protected]] > Sent: 24. október 2013 01:03 > To: Kristján Valur Jónsson > Cc: Python Dev > Subject: Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! > > > The use case of get_traces() + get_object_trace() is to retrieve the traceback > of all alive Python objects for tools like Melia, Pympler or Heapy. The only > motivation is performance. Well, for me, the use of get_traces() is to get the raw data so that I can perform my own analysis on it. With this data, I foresee people wanting to try to analyse this data in novel ways, as I suggested to you privately. > > I wrote a benchmark using 10^6 objects and... get_traces() x 1 + > get_object_address() x N is 40% *slower* than calling > get_object_traceback() x N. So get_object_traceback() is faster for this use > case, especially if you don't want to the traceback of all objects, but only a > few of them. I understand your desire for things to be fast, but let me just re-iterate my view that for this kind of jobs, performance is completely secondary. Memory debugging and analysis is an off-line, laboratory task. In my opinion, performance should not be driving the design of a module like this. And in particular, it should not be the only reason to write code in C that could just as well be written in .py. This is a lorry. A lorry is for moving refrigerators, on those rare occasions when you need to have refrigerators moved. It doesn't need go-faster-stripes. Well, I think I've made my point on this amply clear now, in this email and the previous, so I won't dwell on it further. > > Charles-Francois already asked me to remove everything related to address, > so let's remove two more functions: Great. > > Test 1. With the Python test suite, 467,738 traces limited to 1 frame: ... > I'm surprised: it's faster than the benchmark I ran some weeks ago. > Maybe I optimized something? The most critical operation, taking a snapshot > takes half a second, so it's enough efficient. Well, to me anything that happens in under a second is fast :) > > Let's remove even more code: > > - remove get_stats() > - remove Snapshot.stats > Removal of code is always nice :) > > > 3) set_traceback_limit(). Truncating tracebacks is bad. Particularly if > > it is > truncated at the top end of the callstack, because then information looses > cohesion, namely, the common connection point, the root. If traceback > limits are required, I suggest being able to specifiy that we truncate the > leaf- > end of the tracebacks. > > If the traceback is truncated and 90% of all memory is allocated at the same > Python line: I prefer to have the get the most recent frame, than the n-th > function from main() which may indirectly call 100 different more functions... > In this case, how do you guess which function allocated the memory? You get > the same issue than > Melia/Pympler/Heapy: debug data doesn't help to identify the memory leak. Debugging memory leaks is not the only use case for your module. Analysing memory usage in a non-leaking application is also very important. In my work, I have been asked to reduce the memory overhead of a python application once it has started up. To do this, you need a top-down view of the application. You need to break it down from the "main" call down towards the leaves. Now, I would personally not truncate the stack, because I can afford the memory, but even if I would, for example, to hide a bunch of detail, I would want to throw away the _lower_ detals of the stack. It is unimportant to me to know if memory was allocated in ...;itertools.py;logging.py;stringutil.py but more important to know that it was allocated in main.py;databaseengine.py;enginesettings.py;... The "main" function here is the one that ties all the different allocations into one tree. If you take a tree, say a nice rowan, and truncate it by leaving only X nodes towards the leaves, you end up with a big heap of small branches. If on the other hand, you trim it so that you leave X nodes beginning at the root, you still have something resembling a tree, albeit a much coarser one. Anyway, this is not so important. I would run this with full traceback myself and truncate the tracebacks during the display stage anyway. > > > > 4) add_filter(). This is unnecessary. Information can be filtered on the > python side. Defining Filter as a C type is not necessary. Similarly, module > level filter functions can be dropped. > > Filters for capture are here for efficiency: attaching a trace to each memory > block is expensive. I tried pybench: when using tracemalloc, Python is 2x > slower. The memory usage is also doubled. Using filters, the overhead is > lower. I don't have numbers for the CPU, but for the > memory: ignored traces are not stored, so the memory usage is immediatly > reduced. Without filters for capture, I'm not sure that it is even poss
Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!
On 24 October 2013 22:34, Kristján Valur Jónsson wrote: > I understand your desire for things to be fast, but let me just re-iterate my > view > that for this kind of jobs, performance is completely secondary. Memory > debugging and analysis is an off-line, laboratory task. In my opinion, > performance should not be driving the design of a module like this. And in > particular, it should not be the only reason to write code in C that could > just as well be written in .py. > This is a lorry. A lorry is for moving refrigerators, on those rare > occasions when > you need to have refrigerators moved. It doesn't need go-faster-stripes. > > Well, I think I've made my point on this amply clear now, in this email and > the > previous, so I won't dwell on it further. Not everything is a PC that you can just add more memory to (or switch to a lab server with the same CPU architecture but more RAM). If Victor were only interested in analysing x86[_64] software, I'd agree with you, but embedded scenarios don't always offer that freedom to do resource consumption analysis on a more powerful system. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!
2013/10/24 Kristján Valur Jónsson :
>> Test 1. With the Python test suite, 467,738 traces limited to 1 frame:
> ...
>> I'm surprised: it's faster than the benchmark I ran some weeks ago.
>> Maybe I optimized something? The most critical operation, taking a snapshot
>> takes half a second, so it's enough efficient.
>
> Well, to me anything that happens in under a second is fast :)
In a previous version, taking a snapshot and computing the "top 10"
took up to 30 seconds. So it was justified to only store "stats"
(which is much smaller).
> (...) To do this, you need a top-down view of the application. You
> need to break it down from the "main" call down towards the leaves.
We don't have to agree, I may add an option to decide which frames are
stored. I have to check how truncated traceback would fit with the
current API, especially with Snapshot.group_by('line').
Which API do you propose to decide which frames are kept (most recent
frames or oldest frames)? It may be a new parameter of
set_traceback_limit() for example.
I have an idea how to implement it in C. But I'm not convinced yet
that there is a need for it :-)
> Anyway, this is not so important. I would run this with full traceback
> myself and truncate
> the tracebacks during the display stage anyway.
FYI tracemalloc has currently has arbitrary limit for 100 frames for a
technical reason: traces are stored in the stack in some C functions.
It should be possible to drop the limitation (without killing
performances). But storing 100 frames is already very slow.
Performances depends directly on the number of frames, because the
whole traceback is compared at each memory allocation to check if the
traceback is new or known in a hash table.
>> @staticmethod
>> def load(filename, traces=True):
>> with open(filename, "rb") as fp:
>> return pickle.load(fp)
>>
>
> What does the "traces" argument do in the load() function then?
A ghost from the past :-) I will remove it.
> Often, yes. But there are big black boxes that remain. The most numerous
> of those are those big mysterious allocations that can happen as a
> result of
> "import mymodule"
Yes, tracemalloc is not perfect. You may for example find a huge
allocation of 768 KB in a random "import module", it's the dictionary
of interned Unicode strings... It's hard to understand why "import
module" allocated 768 KB (and depending on the file modification time,
the allocation may occur somewhere else...).
> Not that we are likely to change PEP 445 at this stage, but this was the use
> case for my suggestion.
In tracemalloc, a traceback is a tuple of (str, int): you can probably
hack the the malloc API and tracemalloc to add C filename and line
number. (But I don't want to do it, because the Python traceback is
enough in my opinion.)
Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed
Kind of on a tangent (and I suspect I know what the answer to my question will be), but here at work they have asked people who use Macs not to upgrade to Mavericks until Apple fixes one or more networking problems which make it basically unusable, especially when connecting to our VPN. I realize y'all will be busy trying to fix these various problems as they relate to Python, but might it not be better to simply tell people to hold off on the Python installers until Apple gets their act together? Otherwise, I fear you'll just be trying to hit a moving target. Skip ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed
On Oct 24, 2013, at 02:11 AM, Ned Deily wrote: >I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this >point. But I'd like Benjamin and Georg to propose an aggressive schedule so >we can get these fixes out there. Does this problem affect 2.6? 2.6.9 final is scheduled for Monday, so if there's something we need to get in before then, please let me know asap. Cheers, -Barry ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!
> -Original Message- > From: Nick Coghlan [mailto:[email protected]] > Sent: 24. október 2013 12:44 > To: Kristján Valur Jónsson > Cc: Python Dev > Subject: Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! > Not everything is a PC that you can just add more memory to (or switch to a > lab server with the same CPU architecture but more RAM). > > If Victor were only interested in analysing x86[_64] software, I'd agree with > you, but embedded scenarios don't always offer that freedom to do > resource consumption analysis on a more powerful system. > Indeed not. In fact, I was faced with the same problem when developing for the PS3. My solution was to not do it. The memory profiler running on the PS3 performs no analysis whatsoever. For every operation (malloc/realloc/free) it simply records the address and the traceback and sends it along its merry way to a server which is listening on a TCP or UDP port If anyone is interested in adding that functionality to tracemalloc, I can contribute my code as an example. A corresponding server is a pure-python affair. An added benefit of a client-server approach is that there the memory profiling tool is non-intrusive (apart from slowing down the execution either due to cpu or network blockage) and so has to take no special steps to exclude itself from the profiling. K ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!
On 24/10/2013 15:40, Kristján Valur Jónsson wrote: -Original Message- From: Nick Coghlan [mailto:[email protected]] Sent: 24. október 2013 12:44 To: Kristján Valur Jónsson Cc: Python Dev Subject: Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! Not everything is a PC that you can just add more memory to (or switch to a lab server with the same CPU architecture but more RAM). If Victor were only interested in analysing x86[_64] software, I'd agree with you, but embedded scenarios don't always offer that freedom to do resource consumption analysis on a more powerful system. Indeed not. In fact, I was faced with the same problem when developing for the PS3. My solution was to not do it. The memory profiler running on the PS3 performs no analysis whatsoever. For every operation (malloc/realloc/free) it simply records the address and the traceback and sends it along its merry way to a server which is listening on a TCP or UDP port If anyone is interested in adding that functionality to tracemalloc, I can contribute my code as an example. A corresponding server is a pure-python affair. An added benefit of a client-server approach is that there the memory profiling tool is non-intrusive (apart from slowing down the execution either due to cpu or network blockage) and so has to take no special steps to exclude itself from the profiling. When I was looking for memory leaks in the regex module I simply wrote all of the allocations, reallocations and deallocations to a log file and then parsed it afterwards using a Python script. Simple, but effective. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!
> When I was looking for memory leaks in the regex module I simply wrote all of the allocations, reallocations and deallocations to a log file and then parsed it afterwards using a Python script. Simple, but effective. He he, it's funny because you described exactly my first implementation of tracemalloc! I wrote output using fprintf() into a text file. The parser was written in Python but too slow to be used it in practice. When running on the slow set top box, it took minutes (5 maybe 10) to analyze the file. Transfering the file to a PC took also minutes: the file was very large (maybe 1 GB, I don't remember) and SimpleHTTPServer too slow for the transfer. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!
2013/10/24 Kristján Valur Jónsson : > > Now, I would personally not truncate the stack, because I can afford the > memory, > but even if I would, for example, to hide a bunch of detail, I would want to > throw away > the _lower_ detals of the stack. It is unimportant to me to know if memory > was > allocated in > ...;itertools.py;logging.py;stringutil.py > but more important to know that it was allocated in > main.py;databaseengine.py;enginesettings.py;... Well, maybe to you, but if you look at valgrind for example, it keeps the top of the stack: and it makes a lot of sense to me, since otherwise you won't be able to find where the leak occurred. Anyway, since the stack depth is a tunable parameter, this shouldn't be an issue in practice: just save the whole stack. 2013/10/24 MRAB : > > When I was looking for memory leaks in the regex module I simply wrote all > of the allocations, reallocations and deallocations to a log file and then > parsed it afterwards using a Python script. Simple, but effective. We've all done that ;-) > 1) really, all that is required in terms of data is the > traceback.get_traces() function. Further, it _need_ not return addresses > since they are not required for analysis. It is sufficient for it to return > a list of (traceback, size, count) tuples. Sure. Since the beginning, I'm also leaning towards a minimal API, and let third-party tools do the analysis. It makes a lot of sense, since some people will want just basic snapshot information, some others will want to compute various statistics, some others will want to display the result in a GUI... But OTOT, it would be too bad not to ship the stdlib with a basic tool to process data, to as to make it usable out-of-the box. And in this regard, we should probably mimick what's done for CPU profiling: there are both low-level profiling data gathering infrastructure (profile and cProfile), but there's also a pstats.Stats class allowing basic operations/display on this raw data. That's IMO a reasonable balance. cf ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed
2013/10/24 Georg Brandl : > Am 24.10.2013 11:11, schrieb Ned Deily: > >> I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this >> point. But I'd like Benjamin and Georg to propose an aggressive schedule so >> we can get these fixes out there. >> > > We've basically agreed to do rc releases this weekend. I don't have time the > weekend of Nov 2, so 3.3.3 final would be scheduled for Nov 9. There are a > few "unlimited read" issues still open; I will have a look at porting their > patches tomorrow. What Georg said, but I'll probably be around Nov 2. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] 2.6.9 readline [Was: OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed]
In article <20131024094436.230220bf@anarchist>, Barry Warsaw wrote: > On Oct 24, 2013, at 02:11 AM, Ned Deily wrote: > >I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this > >point. But I'd like Benjamin and Georg to propose an aggressive schedule so > >we can get these fixes out there. > Does this problem affect 2.6? 2.6.9 final is scheduled for Monday, so if > there's something we need to get in before then, please let me know asap. Yes, this problem also affects 2.6. There are some mitigating factors. The support for libedit on OS X is only enabled when building for an OS X 10.5 or later ABI because in earlier releases, the readline emulation of libedit was judged too buggy. In 2.6 as is also the case in 2.7, when building from source, ./configure defaults to using a 10.4 ABI unless certain universal build options are selected or the user explicitly sets MACOSX_DEPLOYMENT_TARGET=10.5 or higher when running configure. With the default 10.4 setting, readline.so fails to build so there is no crash - no readline features, either. Also, if one supplies a version of GNU readline (which Apple does not ship) as many people do, there is also no crash. The 2.7 change of Issue18458 (http://hg.python.org/cpython/rev/1e03fd72e116) depends on some previous 2.7-only changes in Modules/readline.c so it does not apply cleanly to 2.6. However, there aren't *that* many other earlier changes to 2.7 readline that are not in 2.6 and, for the most part, they have to do with fixing memory leaks, including some introduced by using newer versions of GNU readline, and I didn't see any new features. Copying the current 2.7 tip of readline.c over to the current tip of 2.6 builds cleanly on 10.9. test_readline passes and the simple history scrolling seems to work without crashing. I'd recommend either doing nothing or backporting everything. -- Ned Deily, [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.6.9 readline [Was: OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed]
On Oct 24, 2013, at 01:12 PM, Ned Deily wrote: >Yes, this problem also affects 2.6. There are some mitigating factors. The >support for libedit on OS X is only enabled when building for an OS X 10.5 or >later ABI because in earlier releases, the readline emulation of libedit was >judged too buggy. In 2.6 as is also the case in 2.7, when building from >source, ./configure defaults to using a 10.4 ABI unless certain universal >build options are selected or the user explicitly sets >MACOSX_DEPLOYMENT_TARGET=10.5 or higher when running configure. With the >default 10.4 setting, readline.so fails to build so there is no crash - no >readline features, either. Also, if one supplies a version of GNU readline >(which Apple does not ship) as many people do, there is also no crash. > >The 2.7 change of Issue18458 (http://hg.python.org/cpython/rev/1e03fd72e116) >depends on some previous 2.7-only changes in Modules/readline.c so it does not >apply cleanly to 2.6. However, there aren't *that* many other earlier changes >to 2.7 readline that are not in 2.6 and, for the most part, they have to do >with fixing memory leaks, including some introduced by using newer versions of >GNU readline, and I didn't see any new features. Copying the current 2.7 tip >of readline.c over to the current tip of 2.6 builds cleanly on 10.9. >test_readline passes and the simple history scrolling seems to work without >crashing. I'd recommend either doing nothing or backporting everything. Thanks for the background Ned. Given that 2.6.9 will be the last release of the 2.6 series, we won't have much of a chance of going back to fix things if we/I mess it up. Our options then are to do another release candidate, which frankly will probably get about as much testing as the first one (i.e. close to zero, afaict) and push the final release out a few weeks. Or, as you say we can just do nothing. Maybe that means 2.6 will be broken on OS X 10.9 unless someone crafts and applies patches themselves, but I think that's acceptable given how old 2.6 is (EOL) and how new 10.9 is. If 10.9 had come out two weeks from now, "do nothing" would have been the obvious answer. :) So unless I hear strong objections before next Monday, I am not going to fix this in 2.6.9. Cheers, -Barry ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Close #19379: Lazily import linecache in the warnings module, to make startup
The diff doesn't show an addition to BENCH_GROUPS. Maybe apps? On Thu, Oct 24, 2013 at 4:24 PM, antoine.pitrou wrote: > http://hg.python.org/cpython/rev/8939c0196990 > changeset: 86603:8939c0196990 > user:Antoine Pitrou > date:Thu Oct 24 22:23:42 2013 +0200 > summary: > Close #19379: Lazily import linecache in the warnings module, to make > startup with warnings faster until a warning gets printed. > > files: > Lib/warnings.py | 6 ++ > Misc/NEWS | 3 +++ > 2 files changed, 5 insertions(+), 4 deletions(-) > > > diff --git a/Lib/warnings.py b/Lib/warnings.py > --- a/Lib/warnings.py > +++ b/Lib/warnings.py > @@ -1,9 +1,5 @@ > """Python part of the warnings subsystem.""" > > -# Note: function level imports should *not* be used > -# in this module as it may cause import lock deadlock. > -# See bug 683658. > -import linecache > import sys > > __all__ = ["warn", "showwarning", "formatwarning", "filterwarnings", > @@ -21,6 +17,7 @@ > > def formatwarning(message, category, filename, lineno, line=None): > """Function to format a warning the standard way.""" > +import linecache > s = "%s:%s: %s: %s\n" % (filename, lineno, category.__name__, > message) > line = linecache.getline(filename, lineno) if line is None else line > if line: > @@ -233,6 +230,7 @@ > > # Prime the linecache for formatting, in case the > # "file" is actually in a zipfile or something. > +import linecache > linecache.getlines(filename, module_globals) > > if action == "error": > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -21,6 +21,9 @@ > Library > --- > > +- Issue #19379: Lazily import linecache in the warnings module, to make > + startup with warnings faster until a warning gets printed. > + > - Issue #19327: Fixed the working of regular expressions with too big > charset. > > - Issue #17400: New 'is_global' attribute for ipaddress to tell if an > address > > -- > Repository URL: http://hg.python.org/cpython > > ___ > Python-checkins mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-checkins > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.6.9 readline [Was: OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed]
On 25 Oct 2013 06:42, "Barry Warsaw" wrote: > > On Oct 24, 2013, at 01:12 PM, Ned Deily wrote: > > >Yes, this problem also affects 2.6. There are some mitigating factors. The > >support for libedit on OS X is only enabled when building for an OS X 10.5 or > >later ABI because in earlier releases, the readline emulation of libedit was > >judged too buggy. In 2.6 as is also the case in 2.7, when building from > >source, ./configure defaults to using a 10.4 ABI unless certain universal > >build options are selected or the user explicitly sets > >MACOSX_DEPLOYMENT_TARGET=10.5 or higher when running configure. With the > >default 10.4 setting, readline.so fails to build so there is no crash - no > >readline features, either. Also, if one supplies a version of GNU readline > >(which Apple does not ship) as many people do, there is also no crash. > > > >The 2.7 change of Issue18458 ( http://hg.python.org/cpython/rev/1e03fd72e116) > >depends on some previous 2.7-only changes in Modules/readline.c so it does not > >apply cleanly to 2.6. However, there aren't *that* many other earlier changes > >to 2.7 readline that are not in 2.6 and, for the most part, they have to do > >with fixing memory leaks, including some introduced by using newer versions of > >GNU readline, and I didn't see any new features. Copying the current 2.7 tip > >of readline.c over to the current tip of 2.6 builds cleanly on 10.9. > >test_readline passes and the simple history scrolling seems to work without > >crashing. I'd recommend either doing nothing or backporting everything. > > Thanks for the background Ned. > > Given that 2.6.9 will be the last release of the 2.6 series, we won't have > much of a chance of going back to fix things if we/I mess it up. > > Our options then are to do another release candidate, which frankly will > probably get about as much testing as the first one (i.e. close to zero, > afaict) and push the final release out a few weeks. Or, as you say we can > just do nothing. Maybe that means 2.6 will be broken on OS X 10.9 unless > someone crafts and applies patches themselves, but I think that's acceptable > given how old 2.6 is (EOL) and how new 10.9 is. > > If 10.9 had come out two weeks from now, "do nothing" would have been the > obvious answer. :) So unless I hear strong objections before next Monday, I am > not going to fix this in 2.6.9. Since the default build settings work, that sounds reasonable. Perhaps include a note somewhere that targeting a more recent ABI may involve copying the 2.7 readline.c? Cheers, Nick. > > Cheers, > -Barry > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.6.9 readline [Was: OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed]
On Oct 25, 2013, at 07:54 AM, Nick Coghlan wrote: >Since the default build settings work, that sounds reasonable. Perhaps >include a note somewhere that targeting a more recent ABI may involve >copying the 2.7 readline.c? I'll probably add issue 18458 to the announcement and release page. That way we can still use the tracker to manage patches for 2.6, even though they won't be officially applied. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 3 to be default in Fedora 22
Progess on the Python3 as 'standard Python' front: Yesterday, FESCO (Fedora Steering Committee?) approved a proposal to make Python 3 (version unspecified, I presume 3.3 or 3.4) the default Python in Fedora 22 (I did not see a date anywhere). http://lwn.net/Articles/571528/ https://fedoraproject.org/wiki/Changes/Python_3_as_Default -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 to be default in Fedora 22
On 25 Oct 2013 09:02, "Terry Reedy" wrote: > > Progess on the Python3 as 'standard Python' front: Yesterday, FESCO (Fedora Steering Committee?) approved a proposal to make Python 3 (version unspecified, I presume 3.3 or 3.4) the default Python in Fedora 22 (I did not see a date anywhere). The current release cycle is F20, so this is aiming for late 2014 (and hence will be 3.4). > http://lwn.net/Articles/571528/ > https://fedoraproject.org/wiki/Changes/Python_3_as_Default Note that unlike Arch, the Fedora devs currently plan to leave "/usr/bin/python" referring to Python 2 (see the "User Experience" part of the proposal). Cheers, Nick. > > -- > Terry Jan Reedy > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
On Thu, Oct 24, 2013 at 12:05 AM, Eric Snow wrote: > * Nick suggested writing a draft patch for the language reference > changes (the import page). Such a patch will be a pretty good > indicator of the impact of PEP 451 on the import system and should > highlight any design flaws in the API. This is on my to-do list > (hopefully by tomorrow). Okay, I've posted the patch to http://bugs.python.org/issue18864. I'll get to the other open issues in the next couple days. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
On Thu, Oct 24, 2013 at 6:01 AM, Nick Coghlan wrote: > On 24 October 2013 16:05, Eric Snow wrote: >> * I've asserted that there shouldn't be much difficulty in adjusting >> pkgutil and other modules to work with ModuleSpec. > > More specifically: importlib finders will still expose the previous > import plugin API for backwards compatibility, so the worst case > scenario is that we miss something and there's an API somewhere that > doesn't accept import plugins that only provide the new API and not > the old one. > > However, the PEP should explicitly state that any such omissions will > be treated as bugs in the 3.4.x series (although we'll aim to handle > them all in the initial implementation). Good point. I'll add that. > > Thanks for recording the details of the earlier off-list discussion :) :) -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
