Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Meador Inge]

On Tue, Jan 26, 2010 at 9:35 PM, David Lyon wrote:
> One problem is that in many places, users are trained specifically
> under windows to *never* run anything in a zip file. As it might
> contain a virus and "bring down the whole company network".

I have even hit cases where Windows flat out blocks me from opening a zip
file.  I believe XP SP2 added some security functionality [1] that,
depending on how the zip file is procured, blocks one from opening the
file.  It just throws the error: "Windows has blocked access to this file".

Regards,

-- Meador

[1] http://support.microsoft.com/default.aspx?scid=kb;en-us;883260
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[David Lyon]

Glen wrote:

> So let's further say that the .zip file was named .py, instead, but was
> a .zip internally.

The only one thing I have to say about that is that it makes
embedding of .py files recursive.

So, it begs the question "How many times can you embed a .py
within a .py?"

And then; "How do you check if a .py is a .py?"

Whereas, if you just associate to a .egg or a .eag or some
other extension suffix, it just keeps things very orthodox
and you can have a visual queue (for users with gui-
interfaces).

It also makes distribution on pypi and the like much more
straightforward.


David


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Terry Reedy]

On 1/26/2010 6:17 PM, Collin Winter wrote:


70MB of the increase was indeed debug information. Since the Linux
distros that I checked ship stripped Python binaries, I've stripped
the Unladen Swallow binaries as well, and while the size increase is
still significant, it's not as large as it once was.

Stripped CPython 2.6.4: 1.3 MB
Stripped CPython 3.1.1: 1.4 MB
Stripped Unladen r1041: 12 MB

A 9x increase is better than a 20x increase, but it's not great,


For downloading and installing for my own use, this would not bother me 
too much, especially since I expect you will be able to eliminate more 
that you do not need. People who package, say 500K or less of python 
code and resource files, using py2exe or whatever, might feel 
differently, especially if the time benefit is trivial for the app. If 
they can get, if they want, a no-U.S. Windows binary, which you have 
apparently now made possible, they should be happy. But I will not 
volunteer Martin to make two binaries unless he is able to complete the 
automation of the process, or unless someone volunteers to help.


Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Cameron Simpson]

On 26Jan2010 01:57, Terry Reedy  wrote:
| On 1/25/2010 9:32 PM, Nick Coghlan wrote:
| >However, as Cameron pointed out, the O() value for an operation is an
| >important characteristic of containers, and having people get used to an
| >O(1) list.pop(0) in CPython could create problems not only for other
| >current Python implementations but also for future versions of CPython
| >itself.
| 
| The idea that CPython should not be improved because it would spoil
| programmers strikes me as a thin, even desparate objection.

Hey, I even used the word "thin" to describe my concern!

My point was that I look on python builtins like list and dict as highly
optimised, highly efficient facilities. That means that I expect a "list"
to be very very much like a linear array as one would find in C-like
languages, with realloc() managment behind the scenes to handle the
resizing requirements on append/extend.

It also means that the O() cost of operations should be pretty obvious.
pop(0) "obviously" involves shuffling everything down one slot.

This means that the python programmer should expect that the common
"array" type operations such as __getitem__ should be as blazingly fast
as possible, and to expect that barring unusual programmer requirements,
if I'm thinking about "arrays" I should almost always reach for a python
list.

The proposed change to make pop(0) O(1) involves adding a base offset
to the internal list implementation. Doesn't that incur a (small)
overhead to _every_ list operation? Doesn't that weaken "list" as the
"as efficient as possible" implementation of choice for "array"-like
things?

| One
| could say that same thing about the recent optimization of string +=
| string so that repeated concats are O(n) instead of O(n*n). What a
| trap if people move code to other implementations (or older Python)
| without that new feature.
| 
| Of course, the whole purpose of adding a jit to CPython would be to
| spoil us.
| 
| It is a fact already that optimizing CPython code is specific to a
| particular interpreter, system, and workload.

Yes, optimisation is nice. Higher seed is nice. But there are
optimisations that simple fix inefficiencies or provide a special case
for a common operation, and there are those with side effects elsewhere
in the implementation (i.e. the base offset this pop(0) opt would incur,
which adds a (small) cost to everything).

The other aspect, mentioned elsewhere in this thread, is that
programmers should know the O() cost of these operations.

Cheers,
-- 
Cameron Simpson  DoD#743
http://www.cskk.ezoshosting.com/cs/

I am a kind of burr; I shall stick.
- William Shakespeare, _Measure for Measure_
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[David Lyon]

Glen wrote:

> So let's say that the .zip file was dropped onto the Desktop or start
> menu.  It would have an icon, then.

It would have an icon. But nothing to identify it as a python
application.

One problem is that in many places, users are trained specifically
under windows to *never* run anything in a zip file. As it might
contain a virus and "bring down the whole company network".

That is not a culture that a lot of people want to "undo"

>> But having a __main__.py file in a zip file is hardly a clear
>> and obvious way (to outside people) that it is a python
>> application.
>
> So let's further say that the .zip file was named .py, instead, but was
> a .zip internally.
>
> That would seem to go a long ways toward making the facility user
> friendly, at least on Windows, which is where your complaint about icons
> was based, and the only change to Python would be to recognize that if a
> .py contains a .zip signature, then process it the same way as this
> mysterious, not mentioned in the What's New .zip execution facility.

That is perfectly true to say that.

It is *very* clever.

So it begs the question then, why not use that as a form of application
packaging?

The only one downside (from an opinion perspective) is having an
external distribution format, like .egg, does make it easier to
see that the application is not yet installed.

>> Double click them and they get filed in the right place.
>>
>
> What platform files things in the right place when you double click them?

Windows and Linux do. When you double click an application installer. They
will go away and start the appropriate application installer. I am
refering to files like .msi's or .rpm's respectively.

But overall, I think this discussion is going in the right direction.

I hope Tarek can pick up on it and fnd a way to work it into distutils if
he has time as it does need to have a home to live.

David




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [Fwd: RE: [PSF-Board] Microsoft contributor agreement received?]

[Steve Holden]

Please note that both Dino Viehland and Dave Fugate are now signed up to
make commits in the name of Microsoft Corporation. I believe this just
formalizes existing working relationships, but what would *I* know ...

Thanks, everybody. As you were.

regards
 Steve

 Original Message 
Subject: RE: [PSF-Board] Microsoft contributor agreement received?
Date: Tue, 26 Jan 2010 23:25:45 +
From: Scott Wiltamuth 
To: Dino Viehland , Steve Holden 
CC: p...@python.org , VanL , Dave
Fugate 
References:
<1a472770e042064698cb5adc83a12acd04d90...@tk5ex14mbxc118.redmond.corp.microsoft.com>
<4b26e409.4080...@holdenweb.com>
<1a472770e042064698cb5adc83a12acd34c9e...@tk5ex14mbxc116.redmond.corp.microsoft.com>
<4b5f44ad.2040...@holdenweb.com>
<1a472770e042064698cb5adc83a12acd34c9e...@tk5ex14mbxc116.redmond.corp.microsoft.com>
<4b5f64ba.9000...@holdenweb.com>
<1a472770e042064698cb5adc83a12acd34c9f...@tk5ex14mbxc116.redmond.corp.microsoft.com>


Sure, no problem.  We would like to have Dino as the primary and Dave
Fugate as his backup.  It's always good to have a backup :-).  Thanks!

--Scott




-Original Message-
From: Dino Viehland
Sent: Tuesday, January 26, 2010 2:14 PM
To: Steve Holden; Scott Wiltamuth
Cc: p...@python.org; VanL; Dave Fugate
Subject: RE: [PSF-Board] Microsoft contributor agreement received?

Scott,

Can you respond and say it's alright to add Dave as our backup
submitter?

> -Original Message-
> From: Steve Holden [mailto:st...@holdenweb.com]
> Sent: Tuesday, January 26, 2010 1:55 PM
> To: Dino Viehland
> Cc: p...@python.org; VanL
> Subject: Re: [PSF-Board] Microsoft contributor agreement received?
> 
> Yes. We definitely want to see a line of authority from Microsoft
> stretching all the way to a legally-approved corporate signature.
> 
> I believe an email from Scott would be satisfactory.
> 
> regards
>  Steve
> 
> Dino Viehland wrote:
> > Correct, we submitted the agreement - note I am not the signer of the
> > agreement so if necessary I can have Scott Wiltamuth who was the signer
> > send an e-mail requesting to add Dave.
> >
> >> -Original Message-
> >> From: Steve Holden [mailto:st...@holdenweb.com]
> >> Sent: Tuesday, January 26, 2010 11:38 AM
> >> To: Dino Viehland
> >> Cc: p...@python.org; VanL
> >> Subject: Re: [PSF-Board] Microsoft contributor agreement received?
> >>
> >> I believe that shouldn't be a problem: you presented an agreement signed
> >> on behalf of Microsoft Corporation, right? And you are (or possibly
> >> should be) Microsoft's PSF representative, so you get to say who can and
> >> cannot submit on behalf of Microsoft. I think.
> >>
> >> IANAL, so copying this to someone who is (sorry, Van).
> >>
> >> regards
> >>  Steve
> >>
> >> Dino Viehland wrote:
> >>> Can we also add one backup contributor - Dave Fugate?
> >>>
>  -Original Message-
>  From: Dino Viehland
>  Sent: Monday, December 14, 2009 5:29 PM
>  To: 'Steve Holden'
>  Cc: p...@python.org
>  Subject: RE: [PSF-Board] Microsoft contributor agreement received?
> 
>  It actually doesn't cover all employees - we'd like to scope it to just
>  me for the time being and we'll funnel all of the check-ins via me.
> 
>  Sending it to Stephanie sounds good.
> 
>  Thanks Steve!
> 
> > -Original Message-
> > From: Steve Holden [mailto:st...@holdenweb.com]
> > Sent: Monday, December 14, 2009 5:19 PM
> > To: Dino Viehland
> > Cc: p...@python.org
> > Subject: Re: [PSF-Board] Microsoft contributor agreement received?
> >
> > Dino Viehland wrote:
> >> Guido suggested this would be better than checking on python-dev.  I’m
> >> told we faxed in our contributor agreement about a week and a half ago.
> >> I just wanted to make sure that it got received.  Thanks!
> >>
> > Dino:
> >
> > Thanks, it was received on December 4. Since it's in the name of
> > Microsoft Corporation can we assume that it covers contributions by all
> > employees?
> >
> > I'll execute it tonight. Should I send the signed copy to Stephanie
> > Vishneau at Redmond, or to you?
> >
> > regards
> >  Steve
> > --
> > Steve Holden   +1 571 484 6266   +1 800 494 3119
> > Holden Web LLC http://www.holdenweb.com/
> > UPCOMING EVENTS:http://holdenweb.eventbrite.com/
> >>
> >> --
> >> Steve Holden   +1 571 484 6266   +1 800 494 3119
> >> PyCon is coming! Atlanta, Feb 2010  http://us.pycon.org/
> >> Holden Web LLC http://www.holdenweb.com/
> >> UPCOMING EVENTS:http://holdenweb.eventbrite.com/
> >
> 
> 
> --
> Steve Holden   +1 571 484 6266   +1 800 494 3119
> PyCon is coming! Atlanta, Feb 2010  http://us.pycon.org/
> Holden Web LLC http://www.holdenweb.com/
> UPCOMING EVENTS:http://holdenweb.eventbrite.com/


-- 
Steve Holden   +1 571 484 6266   +1 800 4

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hey Martin,

On Thu, Jan 21, 2010 at 2:25 PM, "Martin v. Löwis"  wrote:
> Reid Kleckner wrote:
>> On Thu, Jan 21, 2010 at 4:34 PM, "Martin v. Löwis"  
>> wrote:
 How large is the LLVM shared library? One surprising data point is that the
 binary is much larger than some of the memory footprint measurements given 
 in
 the PEP.
>>> Could it be that you need to strip the binary, or otherwise remove
>>> unneeded debug information?
>>
>> Python is always built with debug information (-g), at least it was in
>> 2.6.1 which unladen is based off of, and we've made sure to build LLVM
>> the same way.  We had to muck with the LLVM build system to get it to
>> include debugging information.  On my system, stripping the python
>> binary takes it from 82 MB to 9.7 MB.  So yes, it contains extra debug
>> info, which explains the footprint measurements.  The question is
>> whether we want LLVM built with debug info or not.
>
> Ok, so if 70MB are debug information, I think a lot of the concerns are
> removed:
> - debug information doesn't consume any main memory, as it doesn't get
>  mapped when the process is started.
> - debug information also doesn't take up space in the system
>  distributions, as they distribute stripped binaries.
>
> As 10MB is still 10 times as large as a current Python binary,people
> will probably search for ways to reduce that further, or at least split
> it up into pieces.

70MB of the increase was indeed debug information. Since the Linux
distros that I checked ship stripped Python binaries, I've stripped
the Unladen Swallow binaries as well, and while the size increase is
still significant, it's not as large as it once was.

Stripped CPython 2.6.4: 1.3 MB
Stripped CPython 3.1.1: 1.4 MB
Stripped Unladen r1041: 12 MB

A 9x increase is better than a 20x increase, but it's not great,
either. There is still room to trim the set of LLVM libraries used by
Unladen Swallow, and we're continuing to investigate reducing on-disk
binary size (http://code.google.com/p/unladen-swallow/issues/detail?id=118
tracks this).

I've updated the PEP to reflect this configuration, since it's what
most users will pick up via their system package managers. The exact
change to the PEP wording is
http://codereview.appspot.com/186247/diff2/6001:6003/5002.

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[M.-A. Lemburg]

Benjamin Peterson wrote:
> 2010/1/25 Steve Howell :
>> I am interested in creating a patch to make deleting elements from the front
>> of Python list work in O(1) time by advancing the ob_item pointer.
> 
> How about just using a deque?

... or a stack:

http://www.egenix.com/products/python/mxBase/mxStack/

... or a queue:

http://www.egenix.com/products/python/mxBase/mxQueue/

Specialized implementations usually give the best performance -
of course, it all depends on what you're trying to achieve.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 26 2010)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hi Cesare,

On Tue, Jan 26, 2010 at 12:29 AM, Cesare Di Mauro
 wrote:
> Hi Collin,
>
> One more question: is it easy to support more opcodes, or a different opcode
> structure, in Unladen Swallow project?

I assume you're asking about integrating WPython. Yes, adding new
opcodes to Unladen Swallow is still pretty easy. The PEP includes a
section on this,
http://www.python.org/dev/peps/pep-3146/#experimenting-with-changes-to-python-or-cpython-bytecode,
though it doesn't cover something more complex like converting from
bytecode to wordcode, as a purely hypothetical example ;) Let me know
if that section is unclear or needs more data.

Converting from bytecode to wordcode should be relatively
straightforward, assuming that the arrangement of opcode arguments is
the main change. I believe the only real place you would need to
update is the JIT compiler's bytecode iterator (see
http://code.google.com/p/unladen-swallow/source/browse/trunk/Util/PyBytecodeIterator.cc).
Depending on the nature of the changes, the runtime feedback system
might need to be updated, too, but it wouldn't be too difficult, and
the changes should be localized.

Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Martin v. Löwis]

> This
> may not be a problem in the LLVM code base, but it is the typical
> problem that C++ devs run into with initialization of objects with
> static storage duration.

This *really* doesn't have to do anything with U-S, but I'd like to
point out that standard C++ has a very clear semantics in this matter:
Any global object can be used in the translation unit where it is
defined after the point where it is defined. So if you arrange for all
accessors to a global object to occur after its definition, you don't
need to worry about initialization order at all.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Raymond Hettinger]

>> 
>> Ah, but that applies for *large* lists. Adding 8 bytes to
>> each list
>> object applies to *all* lists. I betcha that many programs
>> use many
>> tiny lists.
>> 
> 
> Even most tiny-ish lists are wasting memory, though, according to this 
> sequence:
> 
> 4, 8, 16, 25, ...
> 


That is only is they are being grown with list.append().
Otherwise, list sizes are exact.  For example, [0,1,2]
uses space for just three entries and [] for none.

I get the impression that 1) you've already made up your
mind and are ignoring input from the other developers, 
2) that you're ignoring the python-dev discussions of long ago 
where this very idea was rejected and deques came in to
being instead, 3) over-estimating the prevalence of use
cases for pop(0), and 4) under-estimating the impact on
small lists, 5) under-estimating the impact on psyco or
other implementations of Python, 6) under-estimating the
impact on third-party extensions and debugging tools.

Deques already provide a solution to the FIFO problem
and they do so without huge wastes of memory or calls
to memmove().  They handle both insertion and deletion
from the front and back.  In comparison, the proposed
changes to lists seem like a complete hack.  Just because
it can be done, doesn't mean that it should.

ISTM that you're attacking an already solved problem
and that you're playing fast and loose with a core data
structure.


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Daniel Stutzbach]

On Tue, Jan 26, 2010 at 3:32 PM, Steve Howell  wrote:

> Even most tiny-ish lists are wasting memory, though, according to this
> sequence:
>
> 4, 8, 16, 25, ...
>

Several operations will allocate a list of exactly the right size, wasting
no memory.  In particular, a fixed-size list will always be exactly the
right size.

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Paul Moore]

2010/1/26 Ian Bicking :
> On Tue, Jan 26, 2010 at 2:44 PM, Glyph Lefkowitz 
> wrote:
>>
>> On Jan 26, 2010, at 3:20 PM, Ian Bicking wrote:
>>
>> Sadly you can't then do:
>>   chmod +x mz.py
>>   ./mz.py
>>
>> Unless I missed some subtlety earlier in the conversation, yes you can :).
>
> You are entirely correct; I accidentally was using Python 2.5 in my test.

Rats. There goes my "windows is better" example :-)
Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

> From: Guido van Rossum 
> Steve Howell 
> wrote:
> > It seems to me that the goal of keeping lists
> > super-compact from a memory standpoint is contradicted by
> > the code in list_resize that optimistically preallocates
> > extra memory on appends.
> 
> Ah, but that applies for *large* lists. Adding 8 bytes to
> each list
> object applies to *all* lists. I betcha that many programs
> use many
> tiny lists.
>

Even most tiny-ish lists are wasting memory, though, according to this sequence:

4, 8, 16, 25, ...


 
> If I had a program that was growing and shrinking a list at
> both ends,
> I'd consider a deque. 

I'd use a deque on current Python, but I'd use list with the proposed patch. :)

> If I had a program that was growing
> and
> shrinking a list at the front, I'd consider maintaining the
> data
> backwards. Extensive use of pop(0) (and insert(0, x) I
> suppose) is
> only defensible if you also need to access the data by
> index and
> negative indexes are not an option. Think about that.
> 

Maintaining the data backwards creates complications if you are popping off 
small chunks of the list and then subsequently operating on them.  A parser 
would be a good example.  If you reverse the original list, then grab the first 
four lines, for example, and then want to concatenate the first four lines in 
the original order, you will need to reverse them again.

Having to reverse a list to change its performance just seems backward to me.  

> > [...] My
> > gut says that the trend of the future is more CPU scarcity
> > than memory scarcity, but that's just a WAG.
> 
> I'm no expert in this area, but considering the cost of
> cache misses,
> I'm not so sure about that.

Ok, good point.
 
> > Just more food for thought--doing the memory
> management within listobject.c is really a workaround to the
> limitations of the underlying memory manager.  It seems
> conceptually possible to me to design a memory manager that
> allows you to shrink and extend memory chunks on both sides,
> without any extra memory overhead for bookkeeping.  I'm
> sure the devils in the details, of course.

I asked Doug Lea about this, and he just replied to me that the devil is indeed 
in the details--it would be hard to make memory managers do what I'm proposing 
in an efficient way.


 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Terry Reedy]

On 1/26/2010 7:51 AM, Steve Howell wrote:

From: Nick Coghlan

Steve, I suggest creating a new issue at bugs.python.org to
track your
proposal rather than sending diffs to the list.


I was about to suggest the same thing.

Even if your final patch is not currently accepted, it will remain on 
the tracker indefinitely, where interested parties can potentially use 
it in private customized versions that they compile themselves.


http://bugs.python.org/issue7784

Technical discussion of your patch should mostly migrate to the tracker, 
as with other issues. The issue will appear on both the new tracker 
issues list and the weekly (Friday) tracker summary, so interested 
persons can comment or just insert themselves on the nosy list.


Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Nick Coghlan]

Adam Olsen wrote:
> This is a much better optimization than the string appending
> optimization, as it is both portable and robust.
> 
> I find it shocking to change a semantic I've come to see as a core
> part of the language, but I can't come up with a rational reason to
> oppose it.  The approach is sane and the performance impact is
> (presumably) negligible.

Couldn't have said it better myself...

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Guido van Rossum]

On Tue, Jan 26, 2010 at 12:46 PM, Steve Howell  wrote:
> It seems to me that the goal of keeping lists super-compact from a memory 
> standpoint is contradicted by the code in list_resize that optimistically 
> preallocates extra memory on appends.

Ah, but that applies for *large* lists. Adding 8 bytes to each list
object applies to *all* lists. I betcha that many programs use many
tiny lists.

If I had a program that was growing and shrinking a list at both ends,
I'd consider a deque. If I had a program that was growing and
shrinking a list at the front, I'd consider maintaining the data
backwards. Extensive use of pop(0) (and insert(0, x) I suppose) is
only defensible if you also need to access the data by index and
negative indexes are not an option. Think about that.

> I'm not arguing against the tradeoff there, as I think that optimization has 
> a way more compelling CPU gain for most applications than the optimizations 
> I'm proposing.  What I'm really saying is that there is some precedent to 
> waste a little memory to make CPython lists run faster.  My gut says that the 
> trend of the future is more CPU scarcity than memory scarcity, but that's 
> just a WAG.

I'm no expert in this area, but considering the cost of cache misses,
I'm not so sure about that.

> Just more food for thought--doing the memory management within listobject.c 
> is really a workaround to the limitations of the underlying memory manager.  
> It seems conceptually possible to me to design a memory manager that allows 
> you to shrink and extend memory chunks on both sides, without any extra 
> memory overhead for bookkeeping.  I'm sure the devils in the details, of 
> course.

Maybe, but that's a project of a different magnitude altogether. For
better or for worse, the assumption is widely made that blocks are
represented by a pointer to the start plus a length. And considering
zero to have special properties has pretty good mathematical backup of
you ask me. :-)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Nick Coghlan]

Glyph Lefkowitz wrote:
> This use-case was specifically mentioned on
> , too.

Thanks glyph, I was going to dig up that link, but now I don't have to :)

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Ian Bicking]

On Tue, Jan 26, 2010 at 2:44 PM, Glyph Lefkowitz wrote:

>
> On Jan 26, 2010, at 3:20 PM, Ian Bicking wrote:
>
> Sadly you can't then do:
>
>   chmod +x mz.py
>   ./mz.py
>
>
> Unless I missed some subtlety earlier in the conversation, yes you can :).
>

You are entirely correct; I accidentally was using Python 2.5 in my test.

-- 
Ian Bicking  |  http://blog.ianbicking.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

From: Guido van Rossum 
> 
> So here's how you can fix it: go to "Edit Issue" and change
> the
> "Base:" field to the following:
> 
> http://svn.python.org/view/*checkout*/python/trunk/
>

I just deleted the issue altogether for now, since the preferred approach is to 
use a pointer, and that's gonna change the patch in lots of little places, so I 
think it would be wasteful to review it in its current form.
 
 
> I'd like you to revisit this design decision. It would make
> life
> harder if we ever switched to a conservative garbage
> collector. I feel
> much more comfortable having a hard pointer to the actual
> memory block
> in my hands rather than having to compute where that block
> starts
> using pointer arithmetic.

Ok, will do.
 
> Note that I'm not endorsing your patch -- I expect that the
> number of
> (real) use cases that aren't already served better or
> equally well by
> deques is very small, and I fear that the real cost of
> adding
> sizeof(ssize_t) bytes to every list object is real (unless
> there are 8
> bytes unused given the block size rounding happening in
> obmalloc). I
> don't believe a microbenchmark proves much of anything.
> 

Fair enough.

It seems to me that the goal of keeping lists super-compact from a memory 
standpoint is contradicted by the code in list_resize that optimistically 
preallocates extra memory on appends.  I'm not arguing against the tradeoff 
there, as I think that optimization has a way more compelling CPU gain for most 
applications than the optimizations I'm proposing.  What I'm really saying is 
that there is some precedent to waste a little memory to make CPython lists run 
faster.  My gut says that the trend of the future is more CPU scarcity than 
memory scarcity, but that's just a WAG.

Just more food for thought--doing the memory management within listobject.c is 
really a workaround to the limitations of the underlying memory manager.  It 
seems conceptually possible to me to design a memory manager that allows you to 
shrink and extend memory chunks on both sides, without any extra memory 
overhead for bookkeeping.  I'm sure the devils in the details, of course.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Glyph Lefkowitz]

On Jan 26, 2010, at 3:20 PM, Ian Bicking wrote:
> Sadly you can't then do:
> 
>   chmod +x mz.py
>   ./mz.py

Unless I missed some subtlety earlier in the conversation, yes you can :).

> because it doesn't have "#!/usr/bin/env python" like typical executable 
> Python scripts have.  You can put the shebang line at the beginning of the 
> zip file, and zip will complain about it but will still unpack the file, but 
> it won't be runnable as Python won't recognize it as a zip anymore.

python 2.6's zipfile module can cope with a shebang line in a zip file just 
fine, and since this is the first version of Python which supports this 
feature, that means the following works just fine (tested on OS X and Linux):

$ echo '#!/usr/bin/python
> ' > header.txt
$ echo 'import sys; sys.stdout.write("Hello, world!\n")' > __main__.py
$ zip go.zip __main__.py 
  adding: __main__.py (deflated 2%)
$ cat header.txt go.zip > go.py
$ chmod a+x go.py 
$ ./go.py 
Hello, world!

This use-case was specifically mentioned on 
, too.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Paul Moore]

2010/1/26 Ian Bicking :
> On Tue, Jan 26, 2010 at 1:40 PM, Paul Moore  wrote:
>> You're right, it works:
>>
>> >type __main__.py
>> print "Hello from a zip file"
>>
>> >zip mz.py __main__.py
>>  adding: __main__.py (172 bytes security) (stored 0%)
>>
>> >mz.py
>> Hello from a zip file
>
> Sadly you can't then do:
>   chmod +x mz.py
>   ./mz.py
> because it doesn't have "#!/usr/bin/env python" like typical executable
> Python scripts have.

I'm on Windows and I don't need to. (There's got to be some place
where Windows users get to win over Linux users :-))

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Ian Bicking]

On Tue, Jan 26, 2010 at 1:40 PM, Paul Moore  wrote:

> 2010/1/26 Nick Coghlan :
> > Glenn Linderman wrote:
> >> That would seem to go a long ways toward making the facility user
> >> friendly, at least on Windows, which is where your complaint about icons
> >> was based, and the only change to Python would be to recognize that if a
> >> .py contains a .zip signature,
> >
> > That should work today - the zipfile/directory support shouldn't care
> > about the filename at all (although the test suite doesn't currently
> > cover any extensions other than .zip, so I could be wrong about that...).
>
> You're right, it works:
>
> >type __main__.py
> print "Hello from a zip file"
>
> >zip mz.py __main__.py
>  adding: __main__.py (172 bytes security) (stored 0%)
>
> >mz.py
> Hello from a zip file
>

Sadly you can't then do:

  chmod +x mz.py
  ./mz.py

because it doesn't have "#!/usr/bin/env python" like typical executable
Python scripts have.  You can put the shebang line at the beginning of the
zip file, and zip will complain about it but will still unpack the file, but
it won't be runnable as Python won't recognize it as a zip anymore.  Now if
you could, say, put in "#!/usr/bin/env pythonz" (and then implement a
pythonz command that could do useful stuff) then that might work.  Though
generally #! is so broken that it's really hard to come up with a reasonable
option for these cases.

-- 
Ian Bicking  |  http://blog.ianbicking.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Guido van Rossum]

On Tue, Jan 26, 2010 at 11:36 AM, Steve Howell  wrote:
> --- On Tue, 1/26/10, Guido van Rossum  wrote:
>
>> From: Guido van Rossum 
>> Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
>> To: "Steve Howell" 
>> Cc: "Nick Coghlan" , python-dev@python.org
>> Date: Tuesday, January 26, 2010, 11:17 AM
>> On Tue, Jan 26, 2010 at 10:01 AM,
>> Steve Howell 
>> wrote:
>> > The patch is now on Rietveld.
>> >
>> > http://codereview.appspot.com/194083/show
>> >
>> > I wsa getting HTTP errors for certain operations, like
>> trying to publish comments, but I am able to see the patch
>> there.
>>
>> Hey Steve,
>>
>> You seem to be using Rietveld in a slightly odd fashion
>> which prevents
>> the side-by-side diff and commenting feature fro working.
>>
>> Try downloading Rietveld's "upload.py" script
>> (http://codereview.appspot.com/static/upload.py) and
>> uploading your
>> patch again from your SVN client, using
>>
>>   upload.py -i 194083
>>
>> (This will require the email and password you used to
>> upload the issue
>> in the first place.)
>>
>
> Ok, I just ran upload.py with my patch.  It shows up on Rietveld, but I am 
> still getting 500 errors when I attempt to publish my own comments, so I am 
> not sure what I'm doing wrong.

The first Patch Set is broken, and attempts to add comments to it will
fail. As the issue's owner you can delete the first patch set.

> The version that I just uploaded was my working copy, which was off of the 
> 3.x trunk.
>
> The version that I attempted to upload earlier, via the URL feature, was 
> slightly different--Florent had taken my changes, applied them to 2.x, 
> cleaned up tabs/spaces, and posted his diff to the issue tracker:
>
> http://bugs.python.org/issue7784

So here's how you can fix it: go to "Edit Issue" and change the
"Base:" field to the following:

http://svn.python.org/view/*checkout*/python/trunk/

Note the addition of "trunk/" at the end.

But it's best not to use the same Rietveld Issue for versions
pertaining to different branches.

> The content of both patches is the same otherwise.
>
> I suspect the patch has lots of minor cleanup that needs to be done, but I 
> should point out one major design decision that probably needs to be 
> addressed first.  I chose to store the number of orphans vs. storing the 
> pointer to the originally allocated memory.  There was no rhyme or reason for 
> my decision, other than it was just how I initially got my head around the 
> problem.  The tradeoffs should be pretty obvious--there are places where it's 
> convenient to work with the count, but it comes at the cost of pointer 
> arithmetic in other places.

I'd like you to revisit this design decision. It would make life
harder if we ever switched to a conservative garbage collector. I feel
much more comfortable having a hard pointer to the actual memory block
in my hands rather than having to compute where that block starts
using pointer arithmetic.

Note that I'm not endorsing your patch -- I expect that the number of
(real) use cases that aren't already served better or equally well by
deques is very small, and I fear that the real cost of adding
sizeof(ssize_t) bytes to every list object is real (unless there are 8
bytes unused given the block size rounding happening in obmalloc). I
don't believe a microbenchmark proves much of anything.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Paul Moore]

2010/1/26 Nick Coghlan :
> Glenn Linderman wrote:
>> That would seem to go a long ways toward making the facility user
>> friendly, at least on Windows, which is where your complaint about icons
>> was based, and the only change to Python would be to recognize that if a
>> .py contains a .zip signature,
>
> That should work today - the zipfile/directory support shouldn't care
> about the filename at all (although the test suite doesn't currently
> cover any extensions other than .zip, so I could be wrong about that...).

You're right, it works:

>type __main__.py
print "Hello from a zip file"

>zip mz.py __main__.py
  adding: __main__.py (172 bytes security) (stored 0%)

>mz.py
Hello from a zip file

Easy.
Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Tue, 1/26/10, Guido van Rossum  wrote:

> From: Guido van Rossum 
> Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
> To: "Steve Howell" 
> Cc: "Nick Coghlan" , python-dev@python.org
> Date: Tuesday, January 26, 2010, 11:17 AM
> On Tue, Jan 26, 2010 at 10:01 AM,
> Steve Howell 
> wrote:
> > The patch is now on Rietveld.
> >
> > http://codereview.appspot.com/194083/show
> >
> > I wsa getting HTTP errors for certain operations, like
> trying to publish comments, but I am able to see the patch
> there.
> 
> Hey Steve,
> 
> You seem to be using Rietveld in a slightly odd fashion
> which prevents
> the side-by-side diff and commenting feature fro working.
> 
> Try downloading Rietveld's "upload.py" script
> (http://codereview.appspot.com/static/upload.py) and
> uploading your
> patch again from your SVN client, using
> 
>   upload.py -i 194083
> 
> (This will require the email and password you used to
> upload the issue
> in the first place.)
>

Ok, I just ran upload.py with my patch.  It shows up on Rietveld, but I am 
still getting 500 errors when I attempt to publish my own comments, so I am not 
sure what I'm doing wrong.

The version that I just uploaded was my working copy, which was off of the 3.x 
trunk.

The version that I attempted to upload earlier, via the URL feature, was 
slightly different--Florent had taken my changes, applied them to 2.x, cleaned 
up tabs/spaces, and posted his diff to the issue tracker:

http://bugs.python.org/issue7784

The content of both patches is the same otherwise.

I suspect the patch has lots of minor cleanup that needs to be done, but I 
should point out one major design decision that probably needs to be addressed 
first.  I chose to store the number of orphans vs. storing the pointer to the 
originally allocated memory.  There was no rhyme or reason for my decision, 
other than it was just how I initially got my head around the problem.  The 
tradeoffs should be pretty obvious--there are places where it's convenient to 
work with the count, but it comes at the cost of pointer arithmetic in other 
places.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Guido van Rossum]

On Tue, Jan 26, 2010 at 10:01 AM, Steve Howell  wrote:
> The patch is now on Rietveld.
>
> http://codereview.appspot.com/194083/show
>
> I wsa getting HTTP errors for certain operations, like trying to publish 
> comments, but I am able to see the patch there.

Hey Steve,

You seem to be using Rietveld in a slightly odd fashion which prevents
the side-by-side diff and commenting feature fro working.

Try downloading Rietveld's "upload.py" script
(http://codereview.appspot.com/static/upload.py) and uploading your
patch again from your SVN client, using

  upload.py -i 194083

(This will require the email and password you used to upload the issue
in the first place.)

> Here is the issue tracker link:
>
> http://bugs.python.org/issue7784
>
> Here is a rough draft of a proposal to include the patch, which includes 
> discussion of possible objections:
>
> http://wiki.python.org/moin/ProposalToSpeedUpListOperations
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Adam Olsen]

On Mon, Jan 25, 2010 at 23:57, Terry Reedy  wrote:
> On 1/25/2010 9:32 PM, Nick Coghlan wrote:
>
>> However, as Cameron pointed out, the O() value for an operation is an
>> important characteristic of containers, and having people get used to an
>> O(1) list.pop(0) in CPython could create problems not only for other
>> current Python implementations but also for future versions of CPython
>> itself.
>
> The idea that CPython should not be improved because it would spoil
> programmers strikes me as a thin, even desparate objection. One could say
> that same thing about the recent optimization of string += string so that
> repeated concats are O(n) instead of O(n*n). What a trap if people move code
> to other implementations (or older Python) without that new feature.

This is a much better optimization than the string appending
optimization, as it is both portable and robust.

I find it shocking to change a semantic I've come to see as a core
part of the language, but I can't come up with a rational reason to
oppose it.  The approach is sane and the performance impact is
(presumably) negligible.


-- 
Adam Olsen, aka Rhamphoryncus
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

The patch is now on Rietveld.

http://codereview.appspot.com/194083/show

I wsa getting HTTP errors for certain operations, like trying to publish 
comments, but I am able to see the patch there.

Here is the issue tracker link:

http://bugs.python.org/issue7784

Here is a rough draft of a proposal to include the patch, which includes 
discussion of possible objections:

http://wiki.python.org/moin/ProposalToSpeedUpListOperations


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help wanted on a code generator project

[Alexandre Vassalotti]

On Tue, Jan 26, 2010 at 7:04 AM, Yingjie Lan  wrote:
>> note that this is quite off-topic for this list, which is
>> about the
>> development of the CPython interpreter and runtime
>> environment.
>
> Sorry if this is bothering you. I thought here are a lot of people who knows 
> how to write extensions, and has a lot of experiences. These are exactly the 
> best people that can perfect expy. On the other hand, expy, once perfected, 
> would be a nice tool to expedite adding runtime modules to Python. I am not 
> aware of other nice places to ask for help of such a sort. If you know, 
> please let me know, thanks in advance.
>

It is the third time now that people let you know that announcements
about your project are not welcome on this mailing list.

http://mail.python.org/pipermail/python-dev/2009-July/090699.html
http://mail.python.org/pipermail/python-dev/2009-August/091023.html

So please stop playing the ignorance card and behave appropriately.

-- Alexandre
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Glenn Linderman]

On approximately 1/26/2010 1:27 AM, came the following characters from 
the keyboard of Stefan Behnel:

Michael Foord, 26.01.2010 01:14:
   

How great is the complication? Making list.pop(0) efficient sounds like
a worthy goal, particularly given that the reason you don't use it is
because you *know* it is inefficient
 

I agree. Given a programmer the insight that lists are implemented as
arrays in CPython is essentially saying "don't use list.pop(0), it's
SLOW!". So they won't use it, and even avoid it for small lists where it
doesn't really matter. It basically stinks.

Making list.pop(0) fast removes another edge where programmers must
prematurely concentrate on implementation specifics of the interpreter they
use, rather than the functionality they want to implement. I think that's
an improvement to the simplicity that Python provides. It's basically
saying: "care about performance when you have to, but otherwise believe in
the core developers to make your code fast enough in most common cases".

Stefan
   


Being relatively new to Python, and not yet having discovered deque, 
I've coded a list.pop(0) just the other day.  Guess I better go find 
that, and see if my usage should be optimized somehow.


--
Glenn -- http://nevcal.com/
===
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Tue, 1/26/10, Antoine Pitrou  wrote:

> From: Antoine Pitrou 
> Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
> To: python-dev@python.org
> Date: Tuesday, January 26, 2010, 12:50 AM
> Terry Reedy 
> udel.edu> writes:
> > 
> > The idea that CPython should not be improved because
> it would spoil 
> > programmers strikes me as a thin, even desparate
> objection. One could 
> > say that same thing about the recent optimization of
> string += string so 
> > that repeated concats are O(n) instead of O(n*n).
> 
> Note that it's not even generally true. It only works if
> your platform's
> realloc() is sufficiently smart. That's why we once have
> had some
> Windows-specific performance problems in the py3k IO
> module.
> 


One thing that has struck me in working on the O(1) patch is that almost all of 
my code would be unneeded if C's memory manager were just smart enough to allow 
you to release memory off the top of a chunk equally as efficiently as 
releasing it off the bottom of a chunk.  It's not like memory is asymmetrical; 
it's just the 1980's C interface for memory management that makes us think we 
can only extend and shrink in one direction.  The memory management paradigm 
that CPython builds on top of is at least a quarter century outdated, as far as 
I am concerned.  It might even go back to the 1970s.

It's not Python's fault that OSes or compilers have not modernized their 
approach to memory management, but I do think there will come a time, maybe 25 
years from now, when CPython says enough is enough, it's time to treat memory 
like it's extendible and shrinkable in two directions.




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Tue, 1/26/10, Daniel Stutzbach  wrote:


> Just to be clear, when you say "all tests" do you
> mean "all of the list tests" or "the full
> Python test suite"?
> 

The full suite, minus some tests that skipped on my platform.  The last patch I 
posted was broken on test_sys.py, but I have since fixed that.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Daniel Stutzbach]

On Tue, Jan 26, 2010 at 1:17 AM, Steve Howell  wrote:

> Ok, I fixed the obvious segfaults, and I am now able to pass all the tests
> on my debug build.  A new diff is attached.
>

Just to be clear, when you say "all tests" do you mean "all of the list
tests" or "the full Python test suite"?

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

Hi Skip

For "relatively stable code" I talk about recent years.

My experience with CPython is limited, of course.

Cesare

2010/1/26 

>
>Cesare> ... but ceval.c has a relatively stable code ...
>
> I believe you are mistaken on several counts:
>
>* The names of the functions in there have changed over time.
>
>* The suite of byte code operations have changed dramatically over the
>  past ten years or so.
>
>* The relationship between the code in ceval.c and the Python threading
>  model has changed.
>
> Any or all of these aspects of the virtual machine, as well I'm sure as
> many
> other things I've missed would have to be tracked by any extension module
> which hoped to supplant or augment its function in some way.
>
> Skip
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help wanted on a code generator project

[Floris Bruynooghe]

On Tue, Jan 26, 2010 at 04:40:29AM -0800, Yingjie Lan wrote:
> I googled c.l.py but found 
> few pages (one link said it is a dead list, but anyway), 
> would you care give the information
> where to join it?

comp.lang.python newsgroup, see: http://python.org/community/lists/

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

> From: Nick Coghlan 
> 
> Steve, I suggest creating a new issue at bugs.python.org to
> track your
> proposal rather than sending diffs to the list. 

Agreed.  After sending out the second, still slightly broken diff, I realized 
that I probably needed to read up on the process.  You can find the issue 
tracked here:

http://bugs.python.org/issue7784

Florent Xicluna suggested applying my patch to 2.x, instead of 3.x, and was 
kind enough to fix up my tab/spacing issues.

> Putting the
> patch code
> up on Rietveld is also a good step in helping potential
> reviewers (I
> believe there is some info on using Rietveld in the dev
> FAQ).

Ok, I will look into Rietveld tomorrow.

> The patch may still end up being rejected, but I know I at
> least am lot
> less opposed to the idea than I was when this thread
> started.
> 

Thanks for the encouragement.  Regardless of the outcome, this has been a good 
learning experience for me.  I haven't touched gdb in over a decade; it was 
actually kind of fun to debug some C code.  As I mentioned earlier, Python has 
totally spoiled me for other languages, including C!


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help wanted on a code generator project

[Yingjie Lan]

> Note that pretty much everyone who reads this list will
> likely also read
> c.l.py, but c.l.py has a much broader audience, including a
> lot of people
> who write extension modules in one way or another. 

Thanks for the note. I googled c.l.py but found 
few pages (one link said it is a dead list, but anyway), 
would you care give the information
where to join it?

Yingjie


  
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help wanted on a code generator project

[Nick Coghlan]

Yingjie Lan wrote:
>> note that this is quite off-topic for this list, which is about the
>>  development of the CPython interpreter and runtime environment.
> 
> Sorry if this is bothering you. I thought here are a lot of people
> who knows how to write extensions, and has a lot of experiences.
> These are exactly the best people that can perfect expy. On the other
> hand, expy, once perfected, would be a nice tool to expedite adding
> runtime modules to Python.

The fact that a mailing list has an audience you are interested in
reaching is no excuse for posting deliberately off-topic messages. It is
instead a very good way to get yourself permanently ignored by the very
people you would most like to talk to.

> I am not aware of other nice places to ask
> for help of such a sort. If you know, please let me know, thanks in
> advance.

The list of mailing lists on mail.python.org is quite extensive and
includes both capi-sig and the general python-list.

The latter is also the appropriate place to post "Where should I ask
about Python topic ?" queries.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help wanted on a code generator project

[Stefan Behnel]

Yingjie Lan, 26.01.2010 13:04:
>> note that this is quite off-topic for this list, which is about the 
>> development of the CPython interpreter and runtime environment.
> 
> Sorry if this is bothering you.

No problem.


> I thought here are a lot of people who
> knows how to write extensions, and has a lot of experiences. These are
> exactly the best people that can perfect expy.

Note that pretty much everyone who reads this list will likely also read
c.l.py, but c.l.py has a much broader audience, including a lot of people
who write extension modules in one way or another. So you're better off
asking there than here. There's also a list dedicated to the C-API of
CPython, where this would be much better placed (it seems that you have
found it already).

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help wanted on a code generator project

[Michael Foord]

On 26/01/2010 12:04, Yingjie Lan wrote:

note that this is quite off-topic for this list, which is
about the
development of the CPython interpreter and runtime
environment.
 

Sorry if this is bothering you. I thought here are a lot of people who knows 
how to write extensions, and has a lot of experiences. These are exactly the 
best people that can perfect expy.
That doesn't make the topic suitable for this list - this list is 
exclusively for the development of Python.


All the best,

Michael


  On the other hand, expy, once perfected, would be a nice tool to expedite 
adding runtime modules to Python. I am not aware of other nice places to ask 
for help of such a sort. If you know, please let me know, thanks in advance.

Regards,

Yingjie



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Nick Coghlan]

Steve Howell wrote:
> Ok, I fixed the obvious segfaults, and I am now able to pass all the
> tests on my debug build.  A new diff is attached.

Steve, I suggest creating a new issue at bugs.python.org to track your
proposal rather than sending diffs to the list. Putting the patch code
up on Rietveld is also a good step in helping potential reviewers (I
believe there is some info on using Rietveld in the dev FAQ).

The patch may still end up being rejected, but I know I at least am lot
less opposed to the idea than I was when this thread started.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rich Comparison recipe wrong?

[Michael Foord]

On 26/01/2010 12:22, Nick Coghlan wrote:

Lennart Regebro wrote:
   

It never gets to that root comparison, as several of that types
comparisons just refer back to the type who returns NotImplemented. To
solve it you need to never refer to the other type in your
comparisons. And as far as I see that makes it impossible to implement
full comparisons with only one operator implemented.
 

Ah, OK, I'd missed the complication where the translations could reverse
the order of the arguments.

I'd suggest two things: putting comments on the recipes pointing out
they're vulnerable to infinite recursion if mixed with other types and
raising an issue on the tracker pointing back to this thread.
   


Note that the mail archives for this list are corrupted and not being 
updated [1], so any link will need to be to gmane (or equivalent) rather 
than the usual archive on mail.python.org.


All the best,

Michael

[1] http://mail.python.org/pipermail/python-dev/2010-January/thread.html



Cheers,
Nick.

   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rich Comparison recipe wrong?

[Nick Coghlan]

Lennart Regebro wrote:
> It never gets to that root comparison, as several of that types
> comparisons just refer back to the type who returns NotImplemented. To
> solve it you need to never refer to the other type in your
> comparisons. And as far as I see that makes it impossible to implement
> full comparisons with only one operator implemented.

Ah, OK, I'd missed the complication where the translations could reverse
the order of the arguments.

I'd suggest two things: putting comments on the recipes pointing out
they're vulnerable to infinite recursion if mixed with other types and
raising an issue on the tracker pointing back to this thread.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Nick Coghlan]

Glenn Linderman wrote:
> That would seem to go a long ways toward making the facility user
> friendly, at least on Windows, which is where your complaint about icons
> was based, and the only change to Python would be to recognize that if a
> .py contains a .zip signature,

That should work today - the zipfile/directory support shouldn't care
about the filename at all (although the test suite doesn't currently
cover any extensions other than .zip, so I could be wrong about that...).

> then process it the same way as this
> mysterious, not mentioned in the What's New .zip execution facility.

Agreed this is all a bit underdocumented though - it is only mentioned
in the online command line docs [1], with no mention anywhere in the
tutorial or the interpreter's own help message, nor is it mentioned in
the "Interpreter Changes" section for the What's New for 2.6 or 3.1.

Cheers,
Nick.

[1] http://docs.python.org/using/cmdline.html#command-line

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help wanted on a code generator project

[Yingjie Lan]

> note that this is quite off-topic for this list, which is
> about the
> development of the CPython interpreter and runtime
> environment.

Sorry if this is bothering you. I thought here are a lot of people who knows 
how to write extensions, and has a lot of experiences. These are exactly the 
best people that can perfect expy. On the other hand, expy, once perfected, 
would be a nice tool to expedite adding runtime modules to Python. I am not 
aware of other nice places to ask for help of such a sort. If you know, please 
let me know, thanks in advance.

Regards,

Yingjie


  
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help wanted on a code generator project

[Stefan Behnel]

Yingjie Lan, 26.01.2010 12:41:
> I am working on a project named expy, which intends to be an express way
> to extend Python (currently only supports extension in C, but could be
> easily expaned to support more languages). With expy you can write your
> code as a real python module, then expy would generate the actual code.
> I'm looking for people who are interested in this project to perfect it.
> For more details on this project, please see its homepage at
> sourceforge:
> 
> http://expy.sf.net/

Hi,

note that this is quite off-topic for this list, which is about the
development of the CPython interpreter and runtime environment.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Help wanted on a code generator project

[Yingjie Lan]

Hi, 

I am working on a project named expy, which intends to be an express way to 
extend Python (currently only supports extension in C, but could be easily 
expaned to support more languages). With expy you can write your code as a real 
python module, then expy would generate the actual code. I'm looking for people 
who are interested in this project to perfect it. For more details on this 
project, please see its homepage at sourceforge: 

http://expy.sf.net/

Best regards,

Yingjie


  
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[skip]

Cesare> ... but ceval.c has a relatively stable code ...

I believe you are mistaken on several counts:

* The names of the functions in there have changed over time.

* The suite of byte code operations have changed dramatically over the
  past ten years or so.  

* The relationship between the code in ceval.c and the Python threading
  model has changed.

Any or all of these aspects of the virtual machine, as well I'm sure as many
other things I've missed would have to be tracked by any extension module
which hoped to supplant or augment its function in some way.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Floris Bruynooghe]

Hello Collin

On Mon, Jan 25, 2010 at 05:27:38PM -0800, Collin Winter wrote:
> On Mon, Jan 25, 2010 at 1:25 PM, Floris Bruynooghe
>  wrote:
> > On Mon, Jan 25, 2010 at 10:14:35AM -0800, Collin Winter wrote:
> >> I'm working on a patch to completely remove all traces of C++ with
> >> configured with --without-llvm. It's a straightforward change, and
> >> should present no difficulties.
> >
> > Great to hear that, thanks for caring.
> 
> This has now been resolved. As of
> http://code.google.com/p/unladen-swallow/source/detail?r=1036,
> ./configure --without-llvm has no dependency on libstdc++:

Great, thanks for the work.

Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyCon Keynote

[Michael Foord]

On 26/01/2010 05:27, David Lyon wrote:

At 03:15 PM 1/26/2010 +1100, David Lyon wrote:
 

With all due respect, that process is a bit like a black magic
approach. Maybe the capability is there, but it isn't very well
documented and it isn't obvious.
   

I don't see what's so hard about:

1. Zip up your application in myapp.zip with a __main__.py
2. Run "python myapp.zip"

For development, you don't even need the zipfile.  Just do:

1. Put your app in myapp/ with a __main__.py
2. Run "python myapp"

 

Firstly, it doesn't create create desktop shortcuts - sorry users
need those. Where do the programs go?

   
That would typically be done by your installer. Compiling a .NET 
executable or even a C / C++ executable doesn't magically create 
shortcuts either.



Secondly, I never knew about it. Is this a distutils option?
   

Well, what does that tell us. ;-)


I'm sure you know that providing a zip file to a normal user
as an applicaton and you get back the immediate question.. "So,
what do I do with this?". Then as a sysadmin, you have to go
install it manually in their "Program Files" and setup all the
shortcuts so that they can actually use it.
   


Again - done by your installer, not provided by the language.


It just isn't up to modern standards.


What standards?

Even with my two years
experience in python I had no knowledge of this trick.

   


A whole two years? ;-)

All the best,

Michael Foord


Where is the distutils support to allow this to be built?

Hey look, if this capability is in there, then great.

But having a __main__.py file in a zip file is hardly a clear
and obvious way (to outside people) that it is a python
application.

Why can't we just be like the rest of the universe and have
one icon type for packages and one icon type for applications.

Double click them and they get filed in the right place.

This is what I am talking about, python packaging has become
just so obscure over the years.

Having two ways to do something (one way to package an egg
and one way to package an application) isn't true to python
spirit of having just one way, and one obvious way to do
something.

I'm very sure you know what I mean here.. and this is coming
from a big fan of yours.

David








___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Possible changes to handling of default encoding for text files (was Re: Proposed downstream change to site.py in Fedora (sys.defaultencoding))

[M.-A. Lemburg]

Nick Coghlan wrote:
> Tres Seaver wrote:
>>> Perhaps we could also add a warning to the open() API which warns
>>> in case a file is opened in text mode without specifying an
>>> encoding ?!
>>
>> That sounds like a good plan to me, given that backward-compatibility
>> requires keeping the guessing enabled by default.
> 
> Perhaps a switch along the lines of -t and -tt (warnings/errors for
> mixing tabs and spaces)?
> 
> So by default Python continues to guess based on the locale encoding
> (perhaps with a PYTHONTEXTENCODING to override the locale specifically
> for Python).

Right (that's why I suggested a few messages ago :-).

> Then -T might warn whenever a text file encoding is guessed rather than
> specified and -TT might raise an error.

I was thinking more along the lines of a deprecation warning,
but a command line option is good as well.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 26 2010)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rich Comparison recipe wrong?

[Lennart Regebro]

On Tue, Jan 26, 2010 at 09:15, Юлий Тихонов  wrote:
> It looks like __cmp__, not __lt__ method.

Oh, yeah, sorry. I copy/pasted code from my code where I use __cmp__, :)

> However, with this replacement test also falls into recursion.

Told ya. :)
-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Tue, 1/26/10, Stefan Behnel  wrote:
> From: Stefan Behnel 
> Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
> To: python-dev@python.org
> Date: Tuesday, January 26, 2010, 1:27 AM
> Michael Foord, 26.01.2010 01:14:
> > How great is the complication? Making list.pop(0)
> efficient sounds like
> > a worthy goal, particularly given that the reason you
> don't use it is
> > because you *know* it is inefficient
> 
> I agree. Giving a programmer the insight that lists are
> implemented as
> arrays in CPython is essentially saying "don't use
> list.pop(0), it's
> SLOW!". So they won't use it, and even avoid it for small
> lists where it
> doesn't really matter. It basically stinks.
> 
> Making list.pop(0) fast removes another edge where
> programmers must
> prematurely concentrate on implementation specifics of the
> interpreter they
> use, rather than the functionality they want to implement.
> I think that's
> an improvement to the simplicity that Python provides. It's
> basically
> saying: "care about performance when you have to, but
> otherwise believe in
> the core developers to make your code fast enough in most
> common cases".
> 

Thanks! I agree with you 100%, of course.

In fairness, I think my patch will only negligibly affect the performance of 
almost every Python program in existence.  I don't see any likelihood of 
negative effects for real world programs, but I also concede that the positive 
effects will usually be drowned out by other performance bottlenecks.

So I'd like to push for the patch on the grounds that it simply relieves Python 
programmers from worrying about a needless implementation detail.  But I am 
also prepared to have it rejected on the simple principle of keeping the 
CPython implementation less complex.  I've strived to make the changes as 
uninvasive as possible, but there is no getting around 1) increasing the size 
of PyListObject by an int, and 2) touching every major function in listobject.c 
related to memory management (w/list_resize getting the most surgery).  So 
there are certainly tradeoffs, unless I am overlooking some more clever way to 
implement this.

The core piece of the change that I made was to make deleting elements off the 
top of the list more efficient, by advancing a single pointer forward instead 
of moving an entire chunk of memory backward.  An incidental change was then to 
make inserting elements at the top of the list try to reclaim empty slots.  I 
think the former use case is valid and reasonably common (for some definition 
of "common").  The latter use case is probably a lot more rare, but I 
implemented the insert optimization to avoid the spiky double-memmove that the 
delete optimization would have otherwise introduced to prepends that succeeded 
pops.

My current strategy for giving memory back to the OS is overly simple and needs 
refinement, but it basically says that the number of orphaned pointers cannot 
exceed the number of active elements in the list.  This might sound wasteful, 
but the reality is that N list elements consume a lot more memory than N 
orphaned pointers, and the greater attractiveness of list.pop(0) would of 
course lead to earlier garbage collection of the popped elements in real world 
programs.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Stefan Behnel]

Michael Foord, 26.01.2010 01:14:
> How great is the complication? Making list.pop(0) efficient sounds like
> a worthy goal, particularly given that the reason you don't use it is
> because you *know* it is inefficient

I agree. Given a programmer the insight that lists are implemented as
arrays in CPython is essentially saying "don't use list.pop(0), it's
SLOW!". So they won't use it, and even avoid it for small lists where it
doesn't really matter. It basically stinks.

Making list.pop(0) fast removes another edge where programmers must
prematurely concentrate on implementation specifics of the interpreter they
use, rather than the functionality they want to implement. I think that's
an improvement to the simplicity that Python provides. It's basically
saying: "care about performance when you have to, but otherwise believe in
the core developers to make your code fast enough in most common cases".

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Antoine Pitrou]

Terry Reedy  udel.edu> writes:
> 
> The idea that CPython should not be improved because it would spoil 
> programmers strikes me as a thin, even desparate objection. One could 
> say that same thing about the recent optimization of string += string so 
> that repeated concats are O(n) instead of O(n*n).

Note that it's not even generally true. It only works if your platform's
realloc() is sufficiently smart. That's why we once have had some
Windows-specific performance problems in the py3k IO module.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rich Comparison recipe wrong?

[Jack Diederich]

On Mon, Jan 25, 2010 at 6:59 AM, Lennart Regebro  wrote:
[snip]
> If class A returns NotImplemented when compared to class B, and class
> B implements the recipe above, then we get infinite recursion, because
>
> 1. A() < B() will call A.__lt__(B) which will return NotImplemented.
> 2. which will mean that Python calls B.__ge__(A)
> 3. Which B implements by doing A < B
> 4. Start over at 1.

A small correction;  For the purposes of NotImplemented the opposite
of __lt__ is __gt__ because if A < B then B > A.  The pairs are ('==',
'!='), ('<', '>'), ('<=', '>=').

-Jack
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

Hi Collin,

One more question: is it easy to support more opcodes, or a different opcode
structure, in Unladen Swallow project?

Thanks,
Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rich Comparison recipe wrong?

[Юлий Тихонов]

# compare_test.py
@total_ordering
class StrictComparator(object):
<...>

def __lt__(self, other):
if not isinstance(other, StrictComparator):
return NotImplemented
else:
return (self.v > other.v) - (self.v < other.v)

It looks like __cmp__, not __lt__ method. __lt__ method should look like:

def __lt__(self, other):
if not isinstance(other, StrictComparator):
return NotImplemented
else:
return self.v < other.v

However, with this replacement test also falls into recursion.

2010/1/26 Lennart Regebro 

> On Tue, Jan 26, 2010 at 02:56, Nick Coghlan  wrote:
> > Lennart Regebro wrote:
> >> On Mon, Jan 25, 2010 at 15:30, Nick Coghlan  wrote:
> >>> Ah, you mean the case where both classes implement the recipe, but know
> >>> nothing about each other and hence both return NotImplemented from
> their
> >>> root comparison?
> >>
> >> Well, only one needs to return NotImplemented, actually.
> >
> > I'd like to see a test case that proved that. With two different types
> > and only one of them returning NotImplemented, the recursion should
> > terminate inside the one with the root comparison that can handle both
> > types.
>
> It never gets to that root comparison, as several of that types
> comparisons just refer back to the type who returns NotImplemented. To
> solve it you need to never refer to the other type in your
> comparisons. And as far as I see that makes it impossible to implement
> full comparisons with only one operator implemented.
>
> In short, the recipe assumes you never compare with other types, but
> it doesn't check for it.
>
> Test attached (assuming this list accepts attachements).
>
> --
> Lennart Regebro: Python, Zope, Plone, Grok
> http://regebro.wordpress.com/
> +33 661 58 14 64
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/july.tikh%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyCon Keynote

[Glenn Linderman]

On approximately 1/25/2010 9:27 PM, came the following characters from 
the keyboard of David Lyon:

Firstly, it doesn't create create desktop shortcuts - sorry users
need those. Where do the programs go?
   


So let's say that the .zip file was dropped onto the Desktop or start 
menu.  It would have an icon, then.



But having a __main__.py file in a zip file is hardly a clear
and obvious way (to outside people) that it is a python
application.
   


So let's further say that the .zip file was named .py, instead, but was 
a .zip internally.


That would seem to go a long ways toward making the facility user 
friendly, at least on Windows, which is where your complaint about icons 
was based, and the only change to Python would be to recognize that if a 
.py contains a .zip signature, then process it the same way as this 
mysterious, not mentioned in the What's New .zip execution facility.




Why can't we just be like the rest of the universe and have
one icon type for packages and one icon type for applications.

Double click them and they get filed in the right place.
   


What platform files things in the right place when you double click them?


--
Glenn -- http://nevcal.com/
===
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

Hi Collin,

2010/1/25 Collin Winter 

> Hi Cesare,
>
> On Sat, Jan 23, 2010 at 1:09 PM, Cesare Di Mauro
>  wrote:
> > Hi Collin
> >
> > IMO it'll be better to make Unladen Swallow project a module, to be
> > installed and used if needed, so demanding to users the choice of having
> it
> > or not. The same way psyco does, indeed.
> > Nowadays it requires too much memory, longer loading time, and fat
> binaries
> > for not-so-great performances. I know that some issues have being worked
> on,
> > but I don't think that they'll show something comparable to the current
> > CPython status.
>
> You're proposing that, even once the issues of memory usage and
> startup time are addressed, Unladen Swallow should still be an
>  extension module? I don't see why.


Absolutely not, of course.


> You're assuming that these issues
> cannot be fixed, which I disagree with.
>

No, it's my belief, from what I see until now, that it'll be very difficult
to have a situation that is comparable to the current one, in the terms that
I've talked about (memory usage, load time, and binaries size).

I hope to make a mistake. :)


>
> I think maintaining something like a JIT compiler out-of-line, as
> Psyco is, causes long-term maintainability problems. Such extension
> modules are forever playing catchup with the CPython code, depending
> on implementation details that the CPython developers are right to
>  regard as open to change.


I agree (especially for psyco), but ceval.c has a relatively stable code
(not mine, however :D).

It also limits what kind of optimizations
> you can implement or forces those optimizations to be implemented with
> workarounds that might be suboptimal or fragile. I'd recommend reading
> the Psyco codebase, if you haven't yet.
>

Optimizations are surely a point in favor of integrating U.S. project in the
main core.

Psyco, as I said before, is quite a mess. It's hard to add new back-ends for
other architectures. It's a bit less difficult to keep it in sync with
opcode changes (except for big changes), and a port to Python 3.x may be
suitable (but I don't know if the effort makes sense).

As others have requested, we are working hard to minimize the impact
> of the JIT so that it can be turned off entirely at runtime. We have
> an active issue tracking our progress at
> http://code.google.com/p/unladen-swallow/issues/detail?id=123.
>

I see, thanks.


> > Introducing C++ is a big step, also. Aside the problems it can bring on
> some
> > platforms, it means that C++ can now be used by CPython developers.
>
> Which platforms, specifically? What is it about C++ on those platforms
> that is problematic? Can you please provide details?
>

Others have talked about it.


> > It
> > doesn't make sense to force people use C for everything but the JIT part.
> In
> > the end, CPython could become a mix of C and C++ code, so a bit more
> > difficult to understand and manage.
>
> Whether CPython should allow wider usage of C++ or whether developer
> should be "force[d]" to use C is not our decision, and is not part of
> this PEP. With the exception of Python/eval.c, we deliberately have
> not converted any CPython code to C++ so that if you're not working on
> the JIT, python-dev's workflow remains the same. Even within eval.cc,
> the only C++ parts are related to the JIT, and so disappear completely
> with configured with --without-llvm (or if you're not working on the
> JIT).
>
> In any case, developers can easily tell which language to use based on
> file extension. The compiler errors that would result from compiling
> C++ with a C compiler would be a good indication as well.
>

OK, if CPython will be compilable without using C++ at all, I retire what I
said.


> > What I see is that LLVM is a too big project for the goal of having
> "just" a
> > JIT-ed Python VM. It can be surely easier to use and integrate into
> CPython,
> > but requires too much resources
>
> Which resources do you feel that LLVM would tax, machine resources or
> developer resources? Are you referring to the portions of LLVM used by
> Unladen Swallow, or the entire wider LLVM project, including the
> pieces Unladen Swallow doesn't use at runtime?
>

No, I'm referring to the portions of LLVM used by U.S..

Regards resources, I was talking about memory, loading time, and binaries
size (both with static and dynamic compilation).


>
> > (on the contrary, Psyco demands little
> > resources, give very good performances, but seems to be like a mess to
> > manage and extend).
>
> This is not my experience. For the workloads I have experience with,
> Psyco doubles memory usage while only providing a 15-30% speed
> improvement. Psyco's benefits are not uniform.
>

I made only computation intensive (integers, floats) tasks with Psyco, and
it worked fine.

I haven't made tests with U.S. benchmark suite.


>
> Unladen Swallow has been designed to be much more maintainable and
> easier to extend and modify than Psyco: the compiler and its attendant
>