Re: [Python-Dev] s/hotshot/lsprof
Tim Peters wrote: > Center for Alcohol & Drug Treatment Besides Jamie Zawinski's definition, Google also told me it stands for Computer Aided Drafting Technology where "to draft" turns out to have two different meanings :-) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
Fredrik Lundh wrote: > is this some intrinsic property of profilers? if the existing tool has > problems, why not improve the tool itself? do we really need CADT- > based development in the standard library? It is, IMO, intrinsic to parts of the library that aren't used much. If bugs are in the heavily-used parts of the library, like regular expressions, it doesn't matter much if the original author goes away for some period of time - other contributors will fix the bugs that they care about, and not by rewriting the entire thing. If the library is less used, this kind of model is more likely, as resistance to replacing the existing library will be lower. > (on the other hand, I'm not sure we need a profiler as part of the > standard library either, but that's me...) It's a battery, to some. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch Req. # 1351020 & 1351036: PythonD modifications
[EMAIL PROTECTED] wrote: > The local python community here in Sydney indicated that python.org is > only upset when groups port the source to 'obscure' systems and *don't* > submit patches... It is possible that I was misinformed. I never heard such concerns. I personally wouldn't notice if somebody ported Python, and did not feed back the patches. Sometimes, people ask "there is this and that port, why isn't it integrated", to which the answer is in most cases "because authors didn't contribute". This is not being upset - it is merely a fact. This port (djgcc) is the first one in a long time (IIRC) where anybody proposed rejecting it. > I am not sure about the future myself. DJGPP 2.04 has been parked at beta > for two years now. It might be fair to say that the *general* DJGPP > developer base has shrunk a little bit. But the PythonD userbase has > actually grown since the first release three years ago. For the time > being, people get very angry when the servers go down here :-) It's not that much availability of the platform I worry about, but the commitment of the Python porter. We need somebody to forward bug reports to, and somebody to intervene if incompatible changes are made. This person would also indicate that the platform is no longer available, and hence the port can be removed. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ast status, memory leaks, etc
Neal Norwitz wrote: > I still think the total references at the end of a test run are high, > 342291. I don't have anything to base this number on. Some strategic > interning should help this number go down a bit. I suppose I > shouldn't worry much since these references don't seem to become > actual memory leaks. You could try to classify the objects remaining, counting them by type. Perhaps selectively clearing out sys.modules to what it is after startup might also give insights. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
Tim Peters <[EMAIL PROTECTED]> wrote: > We should note that hotshot didn't intend to reduce total time > overhead. What it's aiming at here is to be less disruptive (than > profile.py) to the code being profiled _while_ that code is running. A statistical profiler (e.g. http://wingolog.org/archives/2005/10/28/profiling) would be a nice addition, IMHO. I guess we should get the current profilers in shape first though. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
On Sunday 20 November 2005 21:02, Tim Peters wrote: > Since I didn't know what that meant, Google helpfully told me: > > Center for Alcohol & Drug Treatment On Sunday 20 November 2005 22:04, Steve Holden wrote: > I suspect you may already know that Fredrik referred to > > Cascade of Attention-Deficit Teenagers Yes, our former office in McLean, Virginia was known by many names. :-) > Where's the BDFL to say "yes" or "no" when you need one? Actually, he was just in the next room for HotShot. Guess he was distracted by the photons from the Window, which I was protected from (ironically, by his office). -Fred -- Fred L. Drake, Jr. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ast status, memory leaks, etc
Neal Norwitz wrote: [...] > To give you an example, I ran the entire regression suite through > Valgrind after configuring --without-pymalloc. I only found 3 > additional problems in new code. There was also one problem in older > code (Python/modsupport.c). > > The big benefit of running with pymalloc is that it only takes about > 1.25 to 1.50 hours to run on my box. When running without pymalloc, I > estimate it takes about 5 times longer. Plus it requires a lot of > extra work since I need to run the tests in batches. I only have 1 GB > of RAM and it takes a lot more than that when running without > pymalloc. > Is there maybe a machine in the SourceForge compile farm that could be used for this work? regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC www.holdenweb.com PyCon TX 2006 www.python.org/pycon/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
On Sun, Nov 20, 2005 at 11:33:42PM +0100, Fredrik Lundh wrote: > do we really need CADT-based development in the standard library? I didn't recognize the acronym, but Google told me CADT = "Cascade of Attention-Deficit Teenagers"; see http://www.jwz.org/doc/cadt.html for a rant. --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
Tim Peters wrote: > [Martin v. Löwis] > >>>I'm really concerned that the same fate will happen to any new >>>profiling library: anybody but the original author will hate it, >>>write his own, and then suggest to replace the existing one. > > > [Fredrik Lundh] > >>is this some intrinsic property of profilers? if the existing tool has >>problems, why not improve the tool itself? > > > How many regexp engines has Python gone through now? Profilers are > even more irritating to write and maintain than those -- and you > presumably know why you started over from scratch instead of improving > pcre, or whatever-the-heck-it-was that came before that ;-) > > >>do we really need CADT-based development in the standard library? > > > Since I didn't know what that meant, Google helpfully told me: > > Center for Alcohol & Drug Treatment > I suspect you may already know that Fredrik referred to Cascade of Attention-Deficit Teenagers Where's the BDFL to say "yes" or "no" when you need one? regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC www.holdenweb.com PyCon TX 2006 www.python.org/pycon/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
[Martin v. Löwis] >> I'm really concerned that the same fate will happen to any new >> profiling library: anybody but the original author will hate it, >> write his own, and then suggest to replace the existing one. [Fredrik Lundh] > is this some intrinsic property of profilers? if the existing tool has > problems, why not improve the tool itself? How many regexp engines has Python gone through now? Profilers are even more irritating to write and maintain than those -- and you presumably know why you started over from scratch instead of improving pcre, or whatever-the-heck-it-was that came before that ;-) > do we really need CADT-based development in the standard library? Since I didn't know what that meant, Google helpfully told me: Center for Alcohol & Drug Treatment Fits, anyway . ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
[Armin Rigo] ... > ... > 'hotshot', new from 2.2, is quite faster (reportedly, only 30% added > overhead). The log file is then loaded and turned into an instance of > the same 'pstats.Stats'. This loading takes ages. The reason is that > the log file only records events, and loading is done by instantiating a > 'profile.Profile' and sending it all the events. In other words, it > takes exactly as long as the time it spared in the first place! We should note that hotshot didn't intend to reduce total time overhead. What it's aiming at here is to be less disruptive (than profile.py) to the code being profiled _while_ that code is running. On modern boxes, any kind of profiling gimmick has the unfortunate side effect of _changing_ the runtime behavior of the code being profiled, at least by polluting I and D caches with droppings from the profiling code itself (or, in the case of profile.py, possibly overwhelming I and top-level D caches -- and distorting non-profiling runtime so badly that, e.g., networked apps may end up taking entirely different code paths). hotshot tries to stick with tiny little C functions that pack away a tiny amount of data each time, and avoid memory alloc/dealloc, to try to minimize this disruption. It looked like it was making real progress on this at one time ;-) > Moreover, for some reasons, the results given by hotshot seem sometimes > quite wrong. (I don't understand why, but I've seen it myself, and it's > been reported by various people, e.g. [2].) 'hotshot' doesn't know > about C calls, but it can log line events, although this information is > lost(!) in the final conversion to a 'pstats.Stats'. Ya, hotshot isn't finished. It had corporate support for its initial development, but lost that, and became an orphan then. That's the eventual fate of most profilers, alas. They're fiddly, difficult, and always wrong in some respect. Because of this, the existence of an eager maintainer without a real life is more important than the code ;-). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
Fredrik> (on the other hand, I'm not sure we need a profiler as part of Fredrik> the standard library either, but that's me...) Painful though hotshot can be at times, I occasionally find it extremely useful to zoom in on trouble spots. I haven't used profile in awhile and haven't tried lsprof yet. I would think having something readily available (whether in the standard library or not) would be handy when needed, hopefully with nothing more than "python setup.py install" required to make it available. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ast status, memory leaks, etc
I would really like it if someone could run Purify (or another memory tool) on Windows. Purify on any another (unix) platform would be nice, but I doubt it will show much more. By using different tools, problems not found by one tool may be found by the other. Plus there is windows specific code that isn't exercised at all right now. Any takers? I still think the total references at the end of a test run are high, 342291. I don't have anything to base this number on. Some strategic interning should help this number go down a bit. I suppose I shouldn't worry much since these references don't seem to become actual memory leaks. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ast status, memory leaks, etc
On 11/20/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > Can somebody please give a quick explanation how valgrind can give > *any* reasonable leak analysis when obmalloc is used? In the current > implementation, obmalloc never ever calls free(3), so all pool memory > should appear to have leaked. > > So if valgrind does *not* report all memory as leaked: how does it > find out? Thanks for reminding me I wanted to do the next step and test without pymalloc. Valgrind can't find certain kinds of leaks when pymalloc is holding on to memory, true. However, remember that lots of allocations are forwarded to the system malloc(). For example, any request > 256 bytes goes directly to system malloc. Also, PyMem_*() call the system functions. The core is pretty clean already, since I've been running Valgrind pretty regularly over the years. Before Valgrind I used purify going back to 2000 or 2001. Barry had used purify before me at some point AFAIK. So nearly all of the leaks have already been fixed. It's pretty much only new code that starts showing leaks. To give you an example, I ran the entire regression suite through Valgrind after configuring --without-pymalloc. I only found 3 additional problems in new code. There was also one problem in older code (Python/modsupport.c). The big benefit of running with pymalloc is that it only takes about 1.25 to 1.50 hours to run on my box. When running without pymalloc, I estimate it takes about 5 times longer. Plus it requires a lot of extra work since I need to run the tests in batches. I only have 1 GB of RAM and it takes a lot more than that when running without pymalloc. > This is the resizing of the list of arenas, which is a deliberate > leak. It just happened to be exhausted in this particular call > stack. Thanks I was going to look into the resizing and forgot about it. Running without pymalloc confirmed that there weren't more serious problems. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
Martin v. Löwis wrote: > The same could be said about hotshot, which was originally contributed > by Fred Drake, and hacked by Tim Peters, yourself, and others. Yet, now > people want to remove it again. > > I'm really concerned that the same fate will happen to any new > profiling library: anybody but the original author will hate it, > write his own, and then suggest to replace the existing one. is this some intrinsic property of profilers? if the existing tool has problems, why not improve the tool itself? do we really need CADT- based development in the standard library? (on the other hand, I'm not sure we need a profiler as part of the standard library either, but that's me...) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
Michael Hudson wrote: >>As stated, it certainly is a bad idea. > > > This is a bit extreme... Yes, my apologies :-( >>To make it a good idea, there should also be some commitment to >>maintain this library for a number of years. So who would be >>maintaining it, and what are their plans for doing so? > > > Well, the post was made by Armin who has been involved in CPython > development for quite a few years now, and mentioned that work on > lsprof was done by me who has been around for even longer -- neither > of us are going to quit anytime soon. The same could be said about hotshot, which was originally contributed by Fred Drake, and hacked by Tim Peters, yourself, and others. Yet, now people want to remove it again. I'm really concerned that the same fate will happen to any new profiling library: anybody but the original author will hate it, write his own, and then suggest to replace the existing one. It is the "let's build it from scratch" attitude which makes me nervous. Perhaps the library could be distributed separately for some time, e.g. as a package in the cheeseshop. When it proves to be mature, I probably would object less. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
"Martin v. Löwis" <[EMAIL PROTECTED]> writes: > Armin Rigo wrote: >> If anyone feels like this is a bad idea, please speak up. > > As stated, it certainly is a bad idea. This is a bit extreme... > To make it a good idea, there should also be some commitment to > maintain this library for a number of years. So who would be > maintaining it, and what are their plans for doing so? Well, the post was made by Armin who has been involved in CPython development for quite a few years now, and mentioned that work on lsprof was done by me who has been around for even longer -- neither of us are going to quit anytime soon. Cheers, mwh -- I think if we have the choice, I'd rather we didn't explicitly put flaws in the reST syntax for the sole purpose of not insulting the almighty.-- /will on the doc-sig ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
Fredrik Lundh <[EMAIL PROTECTED]> wrote: > Thomas Lee wrote: > >> Even if it meant we had just one function call - one, safe function call >> that deallocated all the memory allocated within a function - that we >> had to put before each and every return, that's better than what we >> have. > > alloca? Perhaps we should use the memory management technique that the rest of Python uses: reference counting. I don't see why the AST structures couldn't be PyObjects. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch Req. # 1351020 & 1351036: PythonD modifications
[EMAIL PROTECTED] wrote: > I've taken a glance at the patch. There are probably a few things to quarrel > over--for instance, it looks like a site.py change will cause python to print > a blank line when it's started, and the removal of a '#define HAVE_FORK 1' in > posixmodule.c---but this still doesn't mean the re-addition of DOS as a > supported > platform should be rejected out of hand. Well, my experience is that people contributing "minority" ports run away after getting their patches accepted more often than not (that so happened with the BeOS port and the VMS port, to take recent examples). So I would prefer to see some strong commitment from the porter. Even so, I don't think I'm willing to commit such a patch myself. If somebody else thinks this is worthwhile, I won't object. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch Req. # 1351020 & 1351036: PythonD modifications
On Sat, Nov 19, 2005 at 11:06:16PM +0100, "Martin v. Löwis" wrote: > [EMAIL PROTECTED] wrote: > > I would appreciate feedback concerning these patches before the next > > "PythonD" (for DOS/DJGPP) is released. > > PEP 11 says that DOS is not supported anymore since Python 2.0. So > I am -1 on reintroducing support for it. If we have someeone who is volunteering the time to make it work, not just today but in the future as well, we shouldn't rule out re-adding support. I've taken a glance at the patch. There are probably a few things to quarrel over--for instance, it looks like a site.py change will cause python to print a blank line when it's started, and the removal of a '#define HAVE_FORK 1' in posixmodule.c---but this still doesn't mean the re-addition of DOS as a supported platform should be rejected out of hand. Jeff pgpjEOaUPEZZv.pgp Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ast status, memory leaks, etc
Neal Norwitz wrote: > There are still 2 memory leaks while running the regression tests. > They show up when running test_fork1 and test_pty. There may be more, > valgrind crashed on me the last run which was also before I fixed some > of the reference leaks. It would be great if people could localize > the leaks. Can somebody please give a quick explanation how valgrind can give *any* reasonable leak analysis when obmalloc is used? In the current implementation, obmalloc never ever calls free(3), so all pool memory should appear to have leaked. So if valgrind does *not* report all memory as leaked: how does it find out? > 512 bytes in 1 blocks are definitely lost in loss record 319 of 548 >at 0x11B1AF13: malloc (vg_replace_malloc.c:149) >by 0x433CC4: new_arena (obmalloc.c:500) See http://mail.python.org/pipermail/python-dev/2004-June/045253.html This is the resizing of the list of arenas, which is a deliberate leak. It just happened to be exhausted in this particular call stack. > 1280 bytes in 2 blocks are definitely lost in loss record 383 of 548 >at 0x11B1AF13: malloc (vg_replace_malloc.c:149) >by 0x433CC4: new_arena (obmalloc.c:500) Likewise. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
