[Python-Dev] Allowing slicing of iterators
I just wrote a new C API function (PyItem_GetItem) that supports slicing for arbitrary iterators. A patch for current CVS is at http://www.python.org/sf/1108272 For simple indices it does the iteration manually, and for extended slices it returns an itertools.islice object. As a trivial example, here's how to skip the head of a zero-numbered list: for i, item in enumerate(ABCDEF)[1:]: print i, item Is this idea a non-starter, or should I spend my holiday on Wednesday finishing it off and writing the documentation and tests for it? Regards, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.skystorm.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improving the Python Memory Allocator
[Evan Jones] : -- 2. Every N memory operations (or some other measurement of time), reset this value and calculate a moving average of the number of pages. This estimates the current memory requirements of the application. The challenge is how to determine a good measurement of time. Ideally, if the application was idle for a while, you would perform some housekeeping like this. Does Python's cyclic garbage collector currently do this? If so, I could hook this management stuff on to its calls to gc.collect() IMVHO, any measurement of time chosen would hurt performance of non-memory greedy applications. OTOH, makes sense for the developers of memory greedy applications (they should be aware of it wink) to call gc.collect() periodically. Therefore, *hooking* gc.collect() sounds about right to me, let the janitoring pace be defined by those who really care about it. Looking forward to see this evolve, Senra -- Rodrigo Senra MSc Computer Engineer [EMAIL PROTECTED] GPr Sistemas Ltda http://www.gpr.com.br ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Allowing slicing of iterators
I just wrote a new C API function (PyItem_GetItem) that supports slicing for arbitrary iterators. A patch for current CVS is at http://www.python.org/sf/1108272 For simple indices it does the iteration manually, and for extended slices it returns an itertools.islice object. As a trivial example, here's how to skip the head of a zero-numbered list: for i, item in enumerate(ABCDEF)[1:]: print i, item Is this idea a non-starter, or should I spend my holiday on Wednesday finishing it off and writing the documentation and tests for it? Since we already have the islice iterator, what's the point? It seems to me that introducing this notation would mostly lead to confuse users, since in most other places a slice produces an independent *copy* of the data. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improving the Python Memory Allocator
Here my comments, from more general to more subtle:
- please don't post patches here; post them to SF
You may ask for comments here after you posted them to SF.
- please follow Python coding style. In particular, don't write
if ( available_arenas == NULL ) {
but write
if (available_arenas == NULL) {
Second, the previous allocator went out of its way to permit a module to
call PyObject_Free while another thread is executing PyObject_Malloc.
Apparently, this was a backwards compatibility hack for old Python
modules which erroneously call these functions without holding the GIL.
These modules will have to be fixed if this implementation is accepted
into the core.
I'm not certain it is acceptable to make this assumption. Why is it
not possible to use the same approach that was previously used (i.e.
leak the arenas array)?
- When allocating a page, it is taken from the arena that is the most
full. This gives arenas that are almost completely unused a chance to be
freed.
It would be helpful if that was documented in the data structures
somewhere. The fact that the nextarena list is sorted by nfreepools
is only mentioned in the place where this property is preserved;
it should be mentioned in the introductory comments as well.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Speed up function calls
Neal Norwitz wrote: Where are the Py_DECREFs done for the function arguments? The original code path still handles the Py_DECREFs. This is the while loop at the end of call_function(). Can you please elaborate? For METH_O and METH_ARGS, the arguments have already been popped off the stack, and the What does this do loop only pops off the function itself. So (without testing) methinks your code currently leaks references. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improving the Python Memory Allocator
On Jan 24, 2005, at 18:16, Martin v. Löwis wrote:
- please don't post patches here; post them to SF
You may ask for comments here after you posted them to SF.
Sure. This should be done even for patches which should absolutely not
be committed?
- please follow Python coding style. In particular, don't write
if ( available_arenas == NULL ) {
but write
if (available_arenas == NULL) {
Yikes! This is a bad habit of mine that is in the minority of coding
style . Thank you for catching it.
Second, the previous allocator went out of its way to permit a module
to call PyObject_Free while another thread is executing
PyObject_Malloc. Apparently, this was a backwards compatibility hack
for old Python modules which erroneously call these functions without
holding the GIL. These modules will have to be fixed if this
implementation is accepted into the core.
I'm not certain it is acceptable to make this assumption. Why is it
not possible to use the same approach that was previously used (i.e.
leak the arenas array)?
This is definitely a very important point of discussion. The main
problem is that leaking the arenas arena is not sufficient to make
the memory allocator thread safe. Back in October, Tim Peters suggested
that it might be possible to make the breakage easily detectable:
http://mail.python.org/pipermail/python-dev/2004-October/049502.html
If we changed PyMem_{Free, FREE, Del, DEL} to map to the system
free(), all would be golden (except for broken old code mixing
PyObject_ with PyMem_ calls). If any such broken code still exists,
that remapping would lead to dramatic failures, easy to reproduce; and
old code broken in the other, infinitely more subtle way (calling
PyMem_{Free, FREE, Del, DEL} when not holding the GIL) would continue
to work fine.
I'll be honest, I only have a theoretical understanding of why this
support is necessary, or why it is currently correct. For example, is
it possible to call PyMem_Free from two threads simultaneously? Since
the problem is that threads could call PyMem_Free without holding the
GIL, it seems to be that it is possible. Shouldn't it also be
supported? In the current memory allocator, I believe that situation
can lead to inconsistent state. For example, see obmalloc.c:746, where
it has been determined that a block needs to be put on the list of free
blocks:
*(block **)p = lastfree = pool-freeblock;
pool-freeblock = (block *)p;
Imagine two threads are simultaneously freeing blocks that belong to
the same pool. They both read the same value for pool-freeblock, and
assign that same value to p. The changes to pool-freeblock will have
some arbitrary ordering. The result? You have just leaked a block of
memory.
Basically, if a concurrent memory allocator is the requirement, then I
think some other approach is necessary.
- When allocating a page, it is taken from the arena that is the most
full. This gives arenas that are almost completely unused a chance to
be freed.
It would be helpful if that was documented in the data structures
somewhere. The fact that the nextarena list is sorted by nfreepools
is only mentioned in the place where this property is preserved;
it should be mentioned in the introductory comments as well.
This is one of those rough edges I mentioned before. If there is some
concensus that these changes should be accepted, then I will need to
severely edit the comments at the beginning of obmalloc.c.
Thanks for your feedback,
Evan Jones
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Speed up function calls
Neal Norwitz wrote: EXT_POP() modifies stack_pointer on the stack. In call_function(), stack_pointer is PyObject ***. But in new_fast_function(), stack_pointer is only PyObject **. So the modifications by EXT_POP to stack_pointer (moving it down) are lost in new_fast_function(). Thanks - that is the detail I was missing. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyCon: The Spam Continues ;-)
Dear python-dev: The current (as of even date) summary of my recent contributions to Python -dev appears to be spam about PyCon. Not being one to break habits, even not those of a lifetime sometimes, I spam you yet again to show you what a beautiful summary ActiveState have provided (I don't know whether this URL is cacheable or not): http://aspn.activestate.com/ASPN/Mail/Browse/ByAuthor/python-dev?author=cHljb25AcHl0aG9uLm9yZw-- If I remember Trent Lott (?) described at an IPC the SQL Server database that drives this system, and it was a great example of open source technology driving a proprietary (but I expect (?) relatively portable) repository. Since I have your attention (and if I haven't then it really doesn't matter what I write hereafter, goodbye ...) I will also point out that the current top hit on Google for Microsoft to Provide PyCon Opening Keynote is [Python-Dev] Microsoft to Provide PyCon Opening Keynote by Steve Holden (you can repeat the search to see whether this assertion is true as you read this mail, and read the opening keynote announcement [I hope...]). Space at PyCon is again enlarged, but it certainly isn't infinite. I'd love to see it filled in my third and last year as chair. The program committee have worked incredibly hard to make sure we all have to choose between far more technical content than a single individual can possibly take in on their own. They [disclaimer: I was program chair, but this should be kudos for the committee membership - without whom this conference would have failed in many dimensions] they have succeeded so well we all, I hope, have to agonize between two sumptuous but equidistant technical bales of hay. Only by providing such rich choice can we ensure that an even broader community forms around Python, with free interchange between the technical communities of the proprietary and open source worlds, and equitable participation in the benefit. Sorry I haven't made many CVS contributions lately. We really should be showcasing more Python technologies via www.python.org. targeted-marketing-to-talented-professionals-ly y'rs - steve -- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/ Holden Web LLC +1 703 861 4237 +1 800 494 3119 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: [PyCON-Organizers] PyCon: The Spam Continues ;-)
Steve Holden wrote: [some things followed by] If I remember Trent Lott (?) described at an IPC the SQL Server database that drives this system, and it was a great example of open source technology driving a proprietary (but I expect (?) relatively portable) repository. Please forgive me for this not-so-talented Transatlantic confusion, since I mistook one famous name for another. I did of course mean Trent Mick at ActiveState. Apologies for the confusion. regards Steve -- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/ Holden Web LLC +1 703 861 4237 +1 800 494 3119 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: PyCon: The Spam Continues ;-)
http://aspn.activestate.com/ASPN/Mail/Browse/ByAuthor/python-dev?author=cHljb25AcHl0aG9uLm9yZw--Huh? I get a mostly blank page. Perhaps there are no authors by thatname.tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange segfault in Python threads and linux kernel 2.6
On Thursday 20 January 2005 12:43, Donovan Baarda wrote: On Wed, 2005-01-19 at 13:37 +, Michael Hudson wrote: The main oddness about python threads (before 2.3) is that they run with all signals masked. You could play with a C wrapper (call setprocmask, then exec fop) to see if this is what is causing the problem. But please try 2.4. Python 2.4 does indeed fix the problem. Unfortunately we are using Zope 2.7.4, and I'm a bit wary of attempting to migrate it all from 2.3 to 2.4. Is there any wa this Fix can be back-ported to 2.3? It's extremely unlikely - I couldn't make myself comfortable with it when attempting to figure out it's backportedness. While the current behaviour on 2.3.4 is broken in some cases, I fear very much that the new behaviour will break other (working) code - and this is something I try very hard to avoid in a bugfix release, particularly in one that's probably the final one of a series. Fundamentally, the answer is don't do signals+threads, you will get burned. For your application, you might want to instead try something where you write requests to a file in a spool directory, and have a python script that loops looking for requests, and generates responses. This is likely to be much simpler to debug and work with. Anthony -- Anthony Baxter [EMAIL PROTECTED] It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
