Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Ulrich Eckhardt]

On Tuesday 26 January 2010, Steve Howell wrote:
> Here are the benefits of an O(1) implementation.
[...]
> Did I leave anything out?

Good summary, Steve, thanks!

Anyway, you left two out:

 * Inserting at the front gets the same complexity as inserting at the back.

 * Inserting and erasing anywhere in the middle can memmove() the smaller 
tail, so it changes from O(N) to O(N/2).


Finally, if you drop the invariant first<=last, you can simply wrap around the 
pointers. This allows creating a producer/consumer queue which never needs to 
move its content.


Cheers!

Uli

**
Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
**
   Visit our website at 
**
Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten 
bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen 
Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein 
sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, 
weitergeleitet, veröffentlicht oder anderweitig benutzt werden.
E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte 
Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht 
verantwortlich.
**

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of 2 years of Python fuzzing

[Stefan Behnel]

Neal Norwitz, 28.01.2010 08:31:
> One of my points to Victor and everyone else like him is that even
> though it may seem no one is listening to you or cares, you might be
> surprised to find out how many people really are paying attention and
> do care.  If you have something you want to do, make it happen.  One
> person really can do amazing things.  You may never get recognized for
> much of what you do, that doesn't make it any less important.

Sounds like someone watched a lot of Hollywood movies lately. ;)

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Gauging community appraisal for public efforts (was: Summary of 2 years of Python fuzzing)

[Ben Finney]

Neal Norwitz  writes:

> >> On Wed, Jan 27, 2010 at 2:54 AM, Ben Finney  
> >> wrote:
> >> > The book “Beautiful Testing”, published by O'Reilly, might help
> >> > http://oreilly.com/catalog/9780596159825>.
>
> Ben, thanks. Your comment gave me a great laugh and I really
> appreciated it. :-)
>
> The chapter is about the general Python development process, including
> testing, static analysis, dynamic analysis, including Fusil among
> other things.

That alone sounds worth the acquisition of the book. If anyone wants to
buy me a gift, that book would be a good choice.

> One of my points to Victor and everyone else like him is that even
> though it may seem no one is listening to you or cares, you might be
> surprised to find out how many people really are paying attention and
> do care. If you have something you want to do, make it happen. One
> person really can do amazing things. You may never get recognized for
> much of what you do, that doesn't make it any less important.

Indeed.

There is a saying something to the effect that one should read one's
feedback, whether praise or insult; but *keep* the praise, and forget
the insults. Though it superficially sounds trite, I would argue there's
a sound sociological basis for it:

People, in the main, tend to put much more effort into complaining about
things they don't like than they do praising things they do like.
Further, the complaints will be repeated many times, while the praise
will tend not to.

This has the familiar effect of hugely amplifying the volume of
displeased responses, far out of proportion to those giving praise. The
motivation to praise is weak, and occurs rarely; the motivation to
complain is strong, and occurs repeatedly until the complainant gets
satisfation. Thus, many expressed complaints are repetitions, and many
comments of praise never reach us.

Therefore, for any public action one performs, if we posit an
equal number of people pleased and displeased with it, the volume of
complaints will be vastly more than the volume of praise. Even if we
assume a great number of people are pleased and only a few are
displeased, we can *still* expect the expressed responses to have a
higher proportion of displeasure.

So, if one wants to have a good internal gauge of popular response to
one's action, one must weigh complaints vastly less than the weight of
praise.

Not only is it a good estimate of the truth, it will also have the
beneficial effect of making one feel much better about the feedback.
It's also a good reason to give public praise in response to a specific
praiseworthy action: it's worth far more :-)

-- 
 \   “One of the most important things you learn from the internet |
  `\   is that there is no ‘them’ out there. It's just an awful lot of |
_o__)‘us’.” —Douglas Adams |
Ben Finney

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Tim Wintle]

On Wed, 2010-01-27 at 10:25 -0800, Collin Winter wrote:
> On Wed, Jan 27, 2010 at 7:26 AM, William Dode  wrote:
> > I imagine that startup time and memory was also critical for V8.
> 
> Startup time and memory usage are arguably *more* critical for a
> Javascript implementation, since if you only spend a few milliseconds
> executing Javascript code, but your engine takes 10-20ms to startup,
> then you've lost. Also, a minimized memory profile is important if you
> plan to embed your JS engine on a mobile platform, for example, or you
> need to run in a heavily-multiprocessed browser on low-memory consumer
> desktops and netbooks.

(as a casual reader)

I'm not sure if this has been specifically mentioned before, but there
are certainly multiple uses for python - and from the arguments I've
seen on-list, U-S appears to mainly appeal for one of them.

The three main uses I see everyday are:

 * simple configuration-style applications (I'm thinking of gnome
configuration options etc.)

 * shell-script replacements

 * Long running processes (web apps, large desktop applications etc)

(in addition, I know several people working on python applications for
mobile phones)

I think the performance/memory tradeoffs being discussed are fine for
the long-running / server apps (20mb on a 8Gb machine is negligable) -
but I think the majority of applications probably fall into the other
options - extra startup time and memory usage each time you have a
server monitoring script / cronjob run becomes noticeable if you have
enough of them.

> Among other reasons we chose LLVM, we didn't want to write code
> generators for each platform we were targeting. LLVM has done this for
> us. V8, on the other hand, has to implement a new code generator for
> each new platform they want to target. This is non-trivial work: it
> takes a long time, has a lot of finicky details, and it greatly
> increases the maintenance burden on the team.

I'm not involved in either, but as someone who started watching the
commit list on V8, I'd thoroughly agree that it would be completely
impractical for python-dev to work as the V8 team seem to. 

What appear minor patches very frequently have to be modified by
reviewers over technicalities, and I can't imagine that level of
intricate work happening efficiently without a large (funded) core team
focusing on code generation.

Personally, picking up the (current) python code, I was able to
comfortably start modifying it within a few hours - with the V8 codebase
I still haven't got a clue what's going on where, to be honest.

Tim Wintle

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of 2 years of Python fuzzing

[Victor Stinner]

Le jeudi 28 janvier 2010 08:31:50, Neal Norwitz a écrit :
> Ben, thanks.  Your comment gave me a great laugh and I really
> appreciated it. :-)
> 
> The chapter is about the general Python development process, including
> testing, static analysis, dynamic analysis, including Fusil among
> other things.

including Fusil? Ok, I ordered the book :-) (anyway, the topic is interesting)

-- 
Victor Stinner
http://www.haypocalc.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[skip]

Tim> I think the performance/memory tradeoffs being discussed are fine
Tim> for the long-running / server apps (20mb on a 8Gb machine is
Tim> negligable) 

At work our apps' memory footprints are dominated by the Boost-wrapped C++
libraries we use.  100MB VM usage at run-time is pretty much the starting
point.  It just goes up from there.  We'd probably not notice an extra 20MB
if it was shared.

Skip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Daniel Fetchinson]

A question from someone writing C extension modules for python but not
involved in python-dev:

It has been said that compiling python with --without-llvm would not
include unladen swallow and would bypass llvm together with all C++.
Basically, as I understand it, --without-llvm gives the 'usual'
cpython we have today. Is this correct?

If this is correct, I still have one worry: since I wouldn't want to
touch the python install most linux distributions ship or most
windows/mac users install (or what MS/Apple ships) I will simply have
no choice than working with the python variant that is installed.

Is it anticipated that most linux distros and MS/Apple will ship the
python variant that comes with llvm/US? I suppose the goal of merging
llvm/US into python 3.x is this.

If this is the case then I, as a C extension author, will have no
choice than working with a python installation that includes llvm/US.
Which, as far as I undestand it, means dealing with C++ issues. Is
this correct? Or the same pure C extension module compiled with C-only
compilers would work with llvm-US-python and cpython?

Cheers,
Danil

-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Coghlan]

Daniel Fetchinson wrote:
> If this is the case then I, as a C extension author, will have no
> choice than working with a python installation that includes llvm/US.
> Which, as far as I undestand it, means dealing with C++ issues. Is
> this correct? Or the same pure C extension module compiled with C-only
> compilers would work with llvm-US-python and cpython?

As a C extension author you will be fine (the source and linker
interface will all still be C-only).

C++ extension authors may need to care about making the version of the
C++ runtime that they link against match that used internally by
CPython/LLVM (although I'm a little unclear on that point myself).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Stefan Behnel]

Daniel Fetchinson, 28.01.2010 13:19:
> A question from someone writing C extension modules for python

I doubt that this will have any impact on C extension developers.


> If this is correct, I still have one worry: since I wouldn't want to
> touch the python install most linux distributions ship or most
> windows/mac users install (or what MS/Apple ships) I will simply have
> no choice than working with the python variant that is installed.
> 
> Is it anticipated that most linux distros and MS/Apple will ship the
> python variant that comes with llvm/US? I suppose the goal of merging
> llvm/US into python 3.x is this.

Depends on the distro. My guess is that they will likely provide both as
separate packages (unless one turns out to be clearly 'better'), and
potentially even support their parallel installation. That's not
unprecedented, just think of different JVM implementations (or even just
different Python versions).


> If this is the case then I, as a C extension author, will have no
> choice than working with a python installation that includes llvm/US.
> Which, as far as I undestand it, means dealing with C++ issues.

I don't think so. Replacing the eval loop has no impact on the C-API
commonly used by binary extensions. It may have an impact on programs that
embed the Python interpreter, but not the other way round.

Remember that you usually don't have to compile the Python interpreter
yourself. Once it's a binary, it doesn't really matter anymore in what
language(s) it was originally written.


> Or the same pure C extension module compiled with C-only
> compilers would work with llvm-US-python and cpython?

That's to be expected.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Daniel Fetchinson]

>> If this is the case then I, as a C extension author, will have no
>> choice than working with a python installation that includes llvm/US.
>> Which, as far as I undestand it, means dealing with C++ issues. Is
>> this correct? Or the same pure C extension module compiled with C-only
>> compilers would work with llvm-US-python and cpython?
>
> As a C extension author you will be fine (the source and linker
> interface will all still be C-only).

Thanks, that is good to hear!

Cheers,
Daniel

> C++ extension authors may need to care about making the version of the
> C++ runtime that they link against match that used internally by
> CPython/LLVM (although I'm a little unclear on that point myself).


-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Daniel Fetchinson]

>> A question from someone writing C extension modules for python
>
> I doubt that this will have any impact on C extension developers.
>
>
>> If this is correct, I still have one worry: since I wouldn't want to
>> touch the python install most linux distributions ship or most
>> windows/mac users install (or what MS/Apple ships) I will simply have
>> no choice than working with the python variant that is installed.
>>
>> Is it anticipated that most linux distros and MS/Apple will ship the
>> python variant that comes with llvm/US? I suppose the goal of merging
>> llvm/US into python 3.x is this.
>
> Depends on the distro. My guess is that they will likely provide both as
> separate packages

Yes, it's clear that various packages will be available but what I was
asking about is the default python version that gets installed if a
user installs a vanilla version of a particular linux distro. It's a
big difference for developers of C extension modules to say "just
install this module and go" or "first download the python version
so-and-so and then install my module and go".

But as I understand from you and Nick, this will not be a problem for
C extension module authors.

> (unless one turns out to be clearly 'better'), and
> potentially even support their parallel installation. That's not
> unprecedented, just think of different JVM implementations (or even just
> different Python versions).
>
>
>> If this is the case then I, as a C extension author, will have no
>> choice than working with a python installation that includes llvm/US.
>> Which, as far as I undestand it, means dealing with C++ issues.
>
> I don't think so. Replacing the eval loop has no impact on the C-API
> commonly used by binary extensions. It may have an impact on programs that
> embed the Python interpreter, but not the other way round.
>
> Remember that you usually don't have to compile the Python interpreter
> yourself. Once it's a binary, it doesn't really matter anymore in what
> language(s) it was originally written.
>
>> Or the same pure C extension module compiled with C-only
>> compilers would work with llvm-US-python and cpython?
>
> That's to be expected.

Okay, that's great, basically Nick confirmed the same thing.

Cheers,
Daniel

-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Improved Traceback Module

[Benjamin Schweizer]

Hello,

I've updated the traceback.py module; my improved version dumps all
local variabes from the stack trace, which helps in debugging rare
problems. You can find details in my latest blog post here:

  http://benjamin-schweizer.de/improved-python-traceback-module.html

and the source code there:

  http://hg.sickos.org/python-traceback/

I'm intrested in comments and improvements - and possibly inclusion in
the source repository.


Greetings
Benjamin

-- 
http://benjamin-schweizer.de/contact
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Facundo Batista]

On Thu, Jan 28, 2010 at 10:33 AM, Benjamin Schweizer
 wrote:

> I've updated the traceback.py module; my improved version dumps all
> local variabes from the stack trace, which helps in debugging rare
> problems. You can find details in my latest blog post here:

This is like the django traceback if debug mode is on?

I would love to get tracebacks with all variables in all levels of the stack.

However, this may be too much information for standard tracebacks, so
what do you think to enable it on demand? Like setting a flag or
importing a module at the beginning of the file?

-- 
.Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Maciej Fijalkowski]

On Thu, Jan 28, 2010 at 2:33 PM, Benjamin Schweizer
 wrote:
> Hello,
>
> I've updated the traceback.py module; my improved version dumps all
> local variabes from the stack trace, which helps in debugging rare
> problems. You can find details in my latest blog post here:
>
>  http://benjamin-schweizer.de/improved-python-traceback-module.html
>
> and the source code there:
>
>  http://hg.sickos.org/python-traceback/
>
> I'm intrested in comments and improvements - and possibly inclusion in
> the source repository.
>
>
> Greetings
> Benjamin
>

Hello.

I could not find any tests in the source repository, are they somewhere else?

Cheers,
fijal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Antoine Pitrou]

Le Thu, 28 Jan 2010 14:33:11 +0100, Benjamin Schweizer a écrit :
> 
> I've updated the traceback.py module; my improved version dumps all
> local variabes from the stack trace, which helps in debugging rare
> problems. You can find details in my latest blog post here:
> 

As Facundo said it shouldn't replace the default output but it could be 
enabled through e.g. an additional argument.
Would you be interested in reworking your code for this, and also add 
tests to Lib/test_traceback.py? You can then post your patch on http://
bugs.python.org

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Barry Warsaw]

On Jan 28, 2010, at 8:33 AM, Benjamin Schweizer wrote:

> I've updated the traceback.py module; my improved version dumps all
> local variabes from the stack trace, which helps in debugging rare
> problems. You can find details in my latest blog post here:
> 
>  http://benjamin-schweizer.de/improved-python-traceback-module.html

Cool.  Does it work with Python 3's chained exceptions?

http://www.python.org/dev/peps/pep-3134/

-Barry

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Fred Drake]

The traceback support from the zc.twist package might be interesting
as well; not sure how well that's isolated from the rest of the
package though:

http://pypi.python.org/pypi/zc.twist/


  -Fred

-- 
Fred L. Drake, Jr.
"Chaos is the score upon which reality is written." --Henry Miller
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Kristján Valur Jónsson]

As it happens, a few months ago we did the same here.
I have a traceback2.py module with the same api as traceback.  Displaying local 
variables is optional through keyword arguments.
I was also able to refactor the original significantly, making it much clearer.
I've pasted it here:  http://dpaste.com/151597/
Its currently in production use and has had some of the sharp edges worn off.

K

> -Original Message-
> From: python-dev-bounces+kristjan=ccpgames@python.org
> [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf
> Of Antoine Pitrou
> Sent: 28. janúar 2010 14:15
> To: python-dev@python.org
> Subject: Re: [Python-Dev] Improved Traceback Module
> 
> Le Thu, 28 Jan 2010 14:33:11 +0100, Benjamin Schweizer a écrit :
> >
> > I've updated the traceback.py module; my improved version dumps all
> > local variabes from the stack trace, which helps in debugging rare
> > problems. You can find details in my latest blog post here:
> >
> 
> As Facundo said it shouldn't replace the default output but it could be
> enabled through e.g. an additional argument.
> Would you be interested in reworking your code for this, and also add
> tests to Lib/test_traceback.py? You can then post your patch on http://
> bugs.python.org
> 
> Regards
> 
> Antoine.
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-
> dev/kristjan%40ccpgames.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Guido van Rossum]

On Thu, Jan 28, 2010 at 5:33 AM, Benjamin Schweizer
 wrote:
> I've updated the traceback.py module; my improved version dumps all
> local variabes from the stack trace, which helps in debugging rare
> problems. You can find details in my latest blog post here:
>
>  http://benjamin-schweizer.de/improved-python-traceback-module.html
>
> and the source code there:
>
>  http://hg.sickos.org/python-traceback/
>
> I'm intrested in comments and improvements - and possibly inclusion in
> the source repository.

This is a nice idea, but to be 100% robust is very hard. I assume
you've already added something to clip large values. But still...
sometimes you find variables with a broken __repr__, and you must
catch various exceptions, and even be prepared for __repr__ not
returning a string. Sometimes there are too many variables to print,
or their names are excessively long. Sometimes __repr__ takes a very
long time. How many of these have you encountered yet? And dealt with?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Benjamin Schweizer]

Hello Facuno,

I would love to get tracebacks with all variables in all levels of the
> stack.
>
> However, this may be too much information for standard tracebacks, so
> what do you think to enable it on demand? Like setting a flag or
> importing a module at the beginning of the file?
>

I've added an optional argument "with_vars" that makes variable output
optional and defaults to the traditional view. I think this is a good idea
and should prevent security issues where end users see tracebacks with
otherwise sensitive information.

Further, I've updated the test cases (I've tested on Python2.6 on Linux with
UTF-8 locales - may be some people can test the code?)


Greetings

-- 
http://benjamin-schweizer.de/contact
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Benjamin Schweizer]

Hi Kristján,

I have a traceback2.py module with the same api as traceback.  Displaying
> local variables is optional through keyword arguments.
> I was also able to refactor the original significantly, making it much
> clearer.
>

traceback2.py was my first attempt; but I finally came out with a patch to
the original module - in the hope we could get this into the main tree as an
update, not an replacement.

I've also noticed a lot of duplicate code, but I assume that there are lots
of special cases buried deep down in the source. So, I've decided to keep
the original structure.


Greetings,

-- 
http://benjamin-schweizer.de/contact
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Benjamin Schweizer]

Hi Guido,

On Thu, Jan 28, 2010 at 5:02 PM, Guido van Rossum  wrote:

> On Thu, Jan 28, 2010 at 5:33 AM, Benjamin Schweizer
>  wrote:
> > I've updated the traceback.py module; my improved version dumps all
> > local variabes from the stack trace, which helps in debugging rare
> > problems. You can find details in my latest blog post here:
> >
> >  http://benjamin-schweizer.de/improved-python-traceback-module.html
> >
> > and the source code there:
> >
> >  http://hg.sickos.org/python-traceback/
> >
> > I'm intrested in comments and improvements - and possibly inclusion in
> > the source repository.
>
> This is a nice idea, but to be 100% robust is very hard. I assume
> you've already added something to clip large values. But still...
> sometimes you find variables with a broken __repr__, and you must
> catch various exceptions, and even be prepared for __repr__ not
> returning a string. Sometimes there are too many variables to print,
> or their names are excessively long. Sometimes __repr__ takes a very
> long time. How many of these have you encountered yet? And dealt with?
>

That's a good point; the code needs some testing, especially on different
platforms. Though, the changes are very unobtrusive and I hope I won't break
compatibility.
I've recently updated it, making the additional format_vars() and
print_vars() optional, keeping the default output untouched.


Greetings

-- 
http://benjamin-schweizer.de/contact
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[P.J. Eby]

At 02:33 PM 1/28/2010 +0100, Benjamin Schweizer wrote:

Hello,

I've updated the traceback.py module; my improved version dumps all
local variabes from the stack trace, which helps in debugging rare
problems. You can find details in my latest blog post here:

  http://benjamin-schweizer.de/improved-python-traceback-module.html


May I suggest a possible improvement to fix the "sensitive 
information" question you raised, as well as offering an improvement 
in compactness/readability?


If you look for a local variable in each frame containing a format 
string, let's say __trace__, you could apply that format string to a 
locals+globals dictionary for the frame, in place of dumping all the 
locals by default.  This would allow one to customize the tracing 
information along the lines of:


__trace__ = "Discombobulating %(thing)s"
for thing in other_thing:
 discombobulate(thing)

And you would then get a very nice, nearly user-readable traceback 
showing the context of what's going on.  (It should probably use 
new-style string formatting instead of old-style, of course, but you 
get the idea.)


This is an idea that's been on my back-burner for ages, as some of my 
libraries and frameworks make calls into user code in such a way that 
the context of what's happening is not so obvious.  A traceback that 
can state the *intention* of what's going on in a given stack frame 
could add a lot of valuable context for interpreting what's going on 
-- especially if this became a standard available feature in the language.


(In principle, a Python VM could later omit assignments to __trace__ 
when -O or -OO is used, or even optimize away constant __trace__ 
assignments altogether, by saving them with associated line ranges, 
similar to how co_lnotab allowed optimizing away the SET_LINENO opcode.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Steve Howell wrote:
> --- On Wed, 1/27/10, Alex Gaynor  wrote:
> 
>> "Python lists implement a pretty useless data structure"
>>
>> It's very difficult for ideas to gain traction when they
>> contain such
>> useless, and obviously wrong, rhetoric.  There's an
>> enormous body of
>> code out there that begs to disagree with this assertion.
>>
> 
> The statement was meant 99% tongue in cheek. Like probably 99.99% of
> Python programmers, I use lists all the time; that's why I want them to
> be more versatile.
> 
> There's also a 1% element of truth to the statement. To the extent
> that people are arguing for alternative data structures to list,
> particularly deque, I wonder if there actually is some merit in
> discouraging the use of lists in favor of better alternatives.

Your tongue must surely still be firmly wedged to the side:  I would say
it was more like "five nines" of the uses of Python lists never even
contemplate pop(0), and not because the programmer is worried about
performance:  most lists aren't being used as FIFOs.  The overwhelming
majority of uses is going to be for cases where pure iteration is all
that is required;  a smaller but significant class needs random access,
and a very tiny set would use pop at either end.

If I wanted a scalable / performant FIFO, then I would use a deque,
because that is what deques are for.  Note that in this case I wouldn't
care about O(1) indexing, because a FIFO is not useful when random
access is needed.  If I was just fooling around, then the cost of pop(0)
on a list wouldn't matter.


Tres.
- --
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkthvLcACgkQ+gerLs4ltQ6omwCfQBKyn1n2Sk+oknrQ+RQbZNp+
I2cAnRyutpRq7Y4LgXCQQyWFy3DX4ZxP
=0ekf
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[A.M. Kuchling]

cgitb can also produce text tracebacks:

>>> import cgitb
>>> cgitb.enable(format='text')
>>> import urllib
>>> f=urllib.urlopen('bogus://foo')

Python 2.7a1+: /home/amk/source/p/python/python
Thu Jan 28 11:35:04 2010

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /home/amk/source/p/python/ in ()


 /home/amk/source/p/python/Lib/urllib.py in urlopen(url='bogus://foo', 
data=None, proxies=None)
   85 opener = _urlopener
   86 if data is None:
   87 return opener.open(url)
   88 else:
   89 return opener.open(url, data)
opener = 
opener.open = >
url = 'bogus://foo'



--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[skip]

pje> If you look for a local variable in each frame containing a format
pje> string, let's say __trace__, you could apply that format string to
pje> a locals+globals dictionary for the frame, in place of dumping all
pje> the locals by default

I commented on the blog post before noticing all the replies here.  I'll
embellish that suggestion by suggesting that instance attributes can be as
valuable when debugging instance methods.  Perhaps __trace_self__ (or
similar) could be fed from self.__dict__ if it exists?

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Ian Bicking]

On Thu, Jan 28, 2010 at 11:01 AM,  wrote:

>
>pje> If you look for a local variable in each frame containing a format
>pje> string, let's say __trace__, you could apply that format string to
>pje> a locals+globals dictionary for the frame, in place of dumping all
>pje> the locals by default
>
> I commented on the blog post before noticing all the replies here.  I'll
> embellish that suggestion by suggesting that instance attributes can be as
> valuable when debugging instance methods.  Perhaps __trace_self__ (or
> similar) could be fed from self.__dict__ if it exists?
>

It seems reasonable to special case the variable named "self".  You might
also want other hooks.  For instance in weberror we take a convention from
Zope of looking or __traceback_supplement__, which is a factory for an
object that informs the traceback (a factory so you don't have to actually
instantiate it until there's an error).  I then extended its protocol a bit,
and use it for putting request information into the traceback.  I can
imagine two lighter ways to do this.  One is something like:

__traceback_inspect__ = ['self', 'request']

which indicates those two local variables should be inspected.  Another
might be some magic method on the request object.  Of course if
repr(request) is sufficient then you are golden.  But it almost certainly
isn't sufficient.  There's usually key objects that deserve special
attention in the case of an error, but which you don't want to flood the
output just because you happen to print their repr.  (With WebOb actually
str(request) would be quite good, while repr(request) would be too brief.)

To echo Guido, in my own traceback extensions I have at least a couple
levels of try:except: around anything fancy.  repr() definitely fails.
 Unicode errors happen at a lot of different levels (repr() returning
unicode, for example).  And everything you do may break simply by an error
in the code, and you still shouldn't lose at least the old traceback, so
putting one big try:except:traceback.print_exc() around your code is also
appropriate.  Well... not quite appropriate because that would show the
exception in the traceback machinery.  Instead you should save exc_info and
show both tracebacks.

Given the amount of data involved you also don't want the traceback to
become too hard to read for simple bugs.  What is really useful for an
unattended process that occasionally fails with unexpected input, may be
excessive for development; either it has to be easy to switch on and off, or
there needs to be some compromise.  In HTML it's easy to make a compromise
(put in a little Javascript to hide the extended detail until asked for, for
instance).  Of course, in some contexts (an email, a web page) the stuff at
the top is most visible, and a great place for an abbreviated view, while in
other contexts (mostly at a console) the bottom is easiest.

Oh, and you even should consider: will you get a unicode error on output?
 I'd actually suggest returning a unicode subclass that won't ever emit
UnicodeEncodeError when it is converted to a str or bytes.  Correctness at
that stage is not as important as not losing the exception.

So... a few suggestions.

-- 
Ian Bicking  |  http://blog.ianbicking.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[P.J. Eby]

At 11:01 AM 1/28/2010 -0600, s...@pobox.com wrote:


pje> If you look for a local variable in each frame containing a format
pje> string, let's say __trace__, you could apply that format string to
pje> a locals+globals dictionary for the frame, in place of dumping all
pje> the locals by default

I commented on the blog post before noticing all the replies here.  I'll
embellish that suggestion by suggesting that instance attributes can be as
valuable when debugging instance methods.  Perhaps __trace_self__ (or
similar) could be fed from self.__dict__ if it exists?


New style formatting allows attributes and item indexes to be used:

http://docs.python.org/library/string.html#formatstrings

e.g. "{self.foo}" would render the foo attribute.  So, I don't think 
trace_self is strictly necessary.  (If one wanted to go hogwild, one 
could always just allow __trace__ to be a dictionary mapping locals 
to format strings, though it loses much of the attractive simplicity 
of setting a trace string.)



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Paul Moore]

On 28 January 2010 12:58, Daniel Fetchinson  wrote:
>>> If this is the case then I, as a C extension author, will have no
>>> choice than working with a python installation that includes llvm/US.
>>> Which, as far as I undestand it, means dealing with C++ issues. Is
>>> this correct? Or the same pure C extension module compiled with C-only
>>> compilers would work with llvm-US-python and cpython?
>>
>> As a C extension author you will be fine (the source and linker
>> interface will all still be C-only).
>
> Thanks, that is good to hear!

So, just to extend the question a little (or reiterate, it may be that
this is already covered and I didn't fully understand):

On Windows, would a C extension author be able to distribute a single
binary (bdist_wininst/bdist_msi) which would be compatible with
with-LLVM and without-LLVM builds of Python?

Actually, if we assume that only a single Windows binary, presumably
with-LLVM, will be distributed on python.org, I'm probably being
over-cautious here, as distributing binaries compatible with the
python.org release should be sufficient. Nevertheless, I'd be
interested in the answer.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Reid Kleckner]

On Thu, Jan 28, 2010 at 1:14 PM, Paul Moore  wrote:
> So, just to extend the question a little (or reiterate, it may be that
> this is already covered and I didn't fully understand):
>
> On Windows, would a C extension author be able to distribute a single
> binary (bdist_wininst/bdist_msi) which would be compatible with
> with-LLVM and without-LLVM builds of Python?
>
> Actually, if we assume that only a single Windows binary, presumably
> with-LLVM, will be distributed on python.org, I'm probably being
> over-cautious here, as distributing binaries compatible with the
> python.org release should be sufficient. Nevertheless, I'd be
> interested in the answer.

We have broken ABI compatibility with Python 2.6, but unladen
--without-llvm should be ABI compatible with itself.  In the future we
would probably want to set up a buildbot to make sure we don't mess
this up.  One thing we have done is to add #ifdef'd attributes to
things like the code object, but so long as you don't touch those
attributes, you should be fine.

Reid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Benjamin Schweizer]

Hello pje,

On Thu, Jan 28, 2010 at 12:43:13PM -0500, P.J. Eby wrote:
> At 11:01 AM 1/28/2010 -0600, s...@pobox.com wrote:
>
>> pje> If you look for a local variable in each frame containing a format
>> pje> string, let's say __trace__, you could apply that format string to
>> pje> a locals+globals dictionary for the frame, in place of dumping all
>> pje> the locals by default
>>

I like the idea of configuring the list of variables with using a
convention like __trace__, though this requires me to specify what
variables cause an exception and that would be hard?

May be we could add a loglevel or something; or we keep it simple as it
is now but respect __trace__ if it is there.

Guido and Ian emphasize stability and I guess they are right. Unicode
was one of the problems I had in mind when I implemented this: strings
are properly backslash-escaped, using sys.locale. Further, if __repr__
fails, the code uses __type__. Tough, there might be problems when
variables have unicode names (what hopefully never happens) or similar.


Greetings,

-- 
http://benjamin-schweizer.de/contact
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Kristján Valur Jónsson]

We have
import traceback2 as traceback all over the place.
I found that there is no need for the redundancy.  There really are no special 
cases.

I‘ve added traceback2.py at google code, in case anyone is interested.
K

From: benjamin.schwei...@gmail.com [mailto:benjamin.schwei...@gmail.com] On 
Behalf Of Benjamin Schweizer
Sent: 28. janúar 2010 16:03
To: Kristján Valur Jónsson
Cc: Antoine Pitrou; python-dev@python.org
Subject: Re: [Python-Dev] Improved Traceback Module

Hi Kristján,
I have a traceback2.py module with the same api as traceback.  Displaying local 
variables is optional through keyword arguments.
I was also able to refactor the original significantly, making it much clearer.

traceback2.py was my first attempt; but I finally came out with a patch to the 
original module - in the hope we could get this into the main tree as an 
update, not an replacement.

I've also noticed a lot of duplicate code, but I assume that there are lots of 
special cases buried deep down in the source. So, I've decided to keep the 
original structure.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[P.J. Eby]

At 07:47 PM 1/28/2010 +0100, Benjamin Schweizer wrote:

Hello pje,

On Thu, Jan 28, 2010 at 12:43:13PM -0500, P.J. Eby wrote:
> At 11:01 AM 1/28/2010 -0600, s...@pobox.com wrote:
>
>> pje> If you look for a local variable in each frame 
containing a format
>> pje> string, let's say __trace__, you could apply that 
format string to
>> pje> a locals+globals dictionary for the frame, in place of 
dumping all

>> pje> the locals by default
>>

I like the idea of configuring the list of variables with using a
convention like __trace__, though this requires me to specify what
variables cause an exception and that would be hard?


The idea is that you simply use __trace__ to format the local 
variables for that frame.  In other words, the programmer specifies 
what they want to see at that point in the traceback, by setting a 
variable in their code.  If there's a __trace__ variable in the frame 
locals at that point in the traceback, simply format the traceback 
locals+globals using that string's .safe_format() method.




May be we could add a loglevel or something; or we keep it simple as it
is now but respect __trace__ if it is there.


Right - I'm saying, if there's a __trace__, you would just use that, 
and otherwise either include local variables or not, depending on 
whether they were requested in the traceback formatting call.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improved Traceback Module

[Kristján Valur Jónsson]

> -Original Message-
> From: python-dev-bounces+kristjan=ccpgames@python.org
> [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf
> Of Guido van Rossum
> 
> This is a nice idea, but to be 100% robust is very hard. I assume
> you've already added something to clip large values. But still...
> sometimes you find variables with a broken __repr__, and you must
> catch various exceptions, and even be prepared for __repr__ not
> returning a string. Sometimes there are too many variables to print,
> or their names are excessively long. Sometimes __repr__ takes a very
> long time. How many of these have you encountered yet? And dealt with?

We have been using this approach in EVE online for seven years at least and 
encountered most possible problems :)
Only recently did I refactor it into a traceback2.py module.
Here is the value printing code:

def _format_locals(f_locals, format):
lines = []
if f_locals is None:
return lines
for key, value in f_locals.iteritems():
if format & FORMAT_LOGSRV:
extra = '%s = ' % (key,)
else:
extra = '%20s = ' % (key,)
try:
width = 253-len(extra) #limit according to log server
val = pprint.pformat(value, depth=1, width=width)
#truncate each var's dump to 1024 bytes
if len(val) > 1024:
val = val[:1024] + "..."
vlines = val.splitlines()
#show at most 4 lines
if len(vlines) > 4:
vlines[4:] = ["..."]
#indent lines
for i in xrange(1, len(vlines)):
vlines[i] = " "*23+vlines[i]
extra += "\n".join(vlines)+"\n"
except Exception, e:
try:
extra += ""%(e,)
except Exception:
extra += ""
lines.append(extra)
return lines

You'll notice the pprint thingie to format output.  There is also truncation of 
long data as well as double exception handling on the output.  There is no 
unicode encoding at this level, we pass on the any unicode objects directly (in 
our framework, that only happens at the final output stream if necessary)

K
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Josiah Carlson]

Having read the entirety of the thread (which is a rare case these
days, I need more spare time), and being that I'm feeling particularly
snarky today, I'm going to agree 100% with everything that Raymond has
said in this message and his few subsequent messages.  Snarky comments
to follow.

I would also point out that the way these things are typically done is
that programmers/engineers have use-cases that are not satisfied by
existing structures, they explain the issues they have with existing
structures, and they propose modifications.  So far, Steve has not
offered any use-cases for why his proposed change is necessary; merely
a (paraphrased) "It would be great if list.pop(0) was O(1) instead of
O(n)."  It's a solution looking for a problem that doesn't exist.
That said, even if he were to magically come up with twenty examples
where neither a deque nor a list were the right solution, I'd still be
a -1, because in the decade+ that I've been using Python and
needed an ordered sequence; lists were the right solution 99% of the
time, deques got .99%, and the remaining .01% would not have been
satisfied with what Steve is proposing (this is obvious hyperbole and
made-up numbers, but the number of sequence types I've created (easily
10-20) is still a few orders of magnitude lower than how often I use
plain lists and deques).

Don't get me wrong, I'm all about nifty "optimizations" (I still get a
chuckle out of proper string lengths with surrogate pairs, string
views, "Some", ...).  But in this case there is no problem; merely the
use of a data structure for something it was not designed.

It's great that Steve wants to help.  And I <3 innovation in data
structures.  But this patch isn't innovation, and it isn't helping
99.99% of use-cases.


Going back to not having free time,
 - Josiah

On Tue, Jan 26, 2010 at 1:52 PM, Raymond Hettinger
 wrote:
>
> Ah, but that applies for *large* lists. Adding 8 bytes to
>
> each list
>
> object applies to *all* lists. I betcha that many programs
>
> use many
>
> tiny lists.
>
>
> Even most tiny-ish lists are wasting memory, though, according to this
> sequence:
>
> 4, 8, 16, 25, ...
>
>
> That is only is they are being grown with list.append().
> Otherwise, list sizes are exact.  For example, [0,1,2]
> uses space for just three entries and [] for none.
> I get the impression that 1) you've already made up your
> mind and are ignoring input from the other developers,
> 2) that you're ignoring the python-dev discussions of long ago
> where this very idea was rejected and deques came in to
> being instead, 3) over-estimating the prevalence of use
> cases for pop(0), and 4) under-estimating the impact on
> small lists, 5) under-estimating the impact on psyco or
> other implementations of Python, 6) under-estimating the
> impact on third-party extensions and debugging tools.
> Deques already provide a solution to the FIFO problem
> and they do so without huge wastes of memory or calls
> to memmove().  They handle both insertion and deletion
> from the front and back.  In comparison, the proposed
> changes to lists seem like a complete hack.  Just because
> it can be done, doesn't mean that it should.
> ISTM that you're attacking an already solved problem
> and that you're playing fast and loose with a core data
> structure.
>
> Raymond
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] scripts, applets, and applications (Was Re: PyCon Keynote)

[David Lyon]

> Eric Smith wrote:
>> This discussion probably belongs on the distutils list.
>
> Yes, the discussion should be.

Except that distutils doesn't have very much support for doing
applets or applications. So it's logical to see why most posts which
ask the same question, often go unanswered there.

Probably better to try on comp.lang.python which is more general. There's
perhaps a lot more real world experience available there.

I think this little sub-thread has been about some excellent
progress that was started to put an applet or script in a
'zip' box.

I think you've hit upon some work that has been done which
is just past the development status. It is waiting for better
documentation so that its' use can be more widespread.

>>> Are there any best practices I should keep in mind if I intend
 to distribute my work?

Make sure it has a nice splash screen.. a test suite.. and try to
run various code analysis tools like cheesecake over it.. haha

Well 'best practices' definitely means comp.lang.python is the
best list.. unless you can help us all to figure out how to get
this wonderful 'zip box' applet support into a form that is
more ready for developers to start using.

I think it needs more tests, more examples, more documentation
to show what a wonderfully useful thing it could be.

David



David


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] O(1) random access to deque? (Re: patch to make list.pop(0) work in O(1) time)

[Josiah Carlson]

If one doesn't care about slicing, the obvious implementation using a
dictionary and two counters works great for a deque with random
indexing.  Well... except for the doubling in memory usage.

 - Josiah

On Wed, Jan 27, 2010 at 4:15 PM, Raymond Hettinger
 wrote:
>
> On Jan 27, 2010, at 3:55 PM, Maciej Fijalkowski wrote:
>
> FWIW, deque indexing for small deques is already O(1)
>
> and somewhat fast.  You only get O(n) degradation
>
> (with a small contant factor) on large deques.
>
>
> Hi.
>
> For small dequeues (smaller than a constant), you have to have O(1)
> operations, by definition :-)
>
>
> LOL, of course that's true.
> What I meant though is that for deques that fit in a single block, the
> lookup is direct (no searching).
> For multi-block deques, the code traverses links (one link for every 62
> elements).
> So the indexing time would be i//62 + 1.  However , if i is more than
> half-way through the deque, the search starts from the other end.  So the
> indexing time is:  min(i, len(d)-i)//2 + 1.
> It's still O(n) for large deques, but the constant factor isn't large and it
> is fast for small deques, and always O(1) for accesses near either end (i.e.
>  d[0] and d[-1] are direct lookups with no searching) -- those cases are
> typical.
> What use case requires a large deque, growing on end, shrinking on the
> other, and needs O(1) random access to elements in the middle?  Stored
> indicies lose their meaning whenever the deque mutates.
>
> Raymond
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Thu, 1/28/10, Josiah Carlson  wrote:
> [...] in the decade+ that I've been using
> Python and
> needed an ordered sequence; lists were the right solution
> 99% of the
> time [...]

What do you think of LISP, and "car" in particular (apart from the stupidly 
cryptic name)?




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Terry Reedy]

On 1/28/2010 6:30 PM, Josiah Carlson wrote:


I would also point out that the way these things are typically done is
that programmers/engineers have use-cases that are not satisfied by
existing structures, they explain the issues they have with existing
structures, and they propose modifications.  So far, Steve has not
offered any use-cases for why his proposed change is necessary; merely


Use of a list as a queue rather than as a stack, as in breadth-first 
search, where one only needs to pop off the front but never push to the 
front. That is not to say that this is common or that a deque or other 
options may no be pretty satisfactory. But it would certainly be easier, 
when presenting such algorithms, to just be able to use a list, which 
has already been taught, than to introduce another structure. Currently 
a deque is not a drop-in replacement for a list in that one cannot use 
all list methods with a deque.


As I understand it, his proposal is simpler than the one rejected a 
couple of years ago is that it does not include intentional 
over-allocation at the front of the list, as would be needed for 
guaranteed O(1) behavior for deque-like insertion at the front. I may 
consider a Python version of his idea for one of my needs, where speed 
is not an issue.


I agree that the discussion has gone on too long here and that some of 
Steve's rhetoric has been unnecessarily abrasive and off-putting. He has 
been told this and acknowledged it once on Python-list, but habits die 
hard. For both reasons, I suggested a few days ago that further 
discussion should focus on the patch and be moved to the issue on the 
tracker. So I will not say more here.


Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Scott Dial]

On 1/28/2010 11:57 PM, Steve Howell wrote:
> --- On Thu, 1/28/10, Josiah Carlson  wrote:
>> [...] in the decade+ that I've been using
>> Python and
>> needed an ordered sequence; lists were the right solution
>> 99% of the
>> time [...]
> 
> What do you think of LISP, and "car" in particular (apart from the stupidly 
> cryptic name)?

A LISP list/pair has nothing to do with a Python list. The list of LISP
is a *singly-linked* list. You cannot O(1) index a LISP list. A Python
list is more equivalent to a LISP vector, which "car" does not work
with; in fact, there is not even a "pop" operator -- all size changes of
a vector O(n) unless the implementation is playing games (like the one
you are proposing for the start and the one Python already uses for the
end of a list).

(And with this, clearly uninformed reply by you, I am now ignoring your
trolling.)

-- 
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

Hi Collin,

Thanks for the useful links.

I think that superinstructions require a bit more work, because they aren't
just opcode arguments rearrangement. For example, in wpython 1.1 (that I'll
release next month) I've introduced a CALL_SUB opcode to handle all kind of
function types, so the 2 words pack together:
- the opcode (CALL_SUB);
- the function type and flags (normal, VAR, KW, procedure);
- the number of arguments;
- the number of keywords arguments.

Superinstructions aren't intended to be a simple drop-in replacement of
existing bytecodes. They can carry new ideas and implement them in a
versatile and efficient way.

Anyway, I don't think to continue with wpython: 1.1 will be the last version
I'll release (albeit I initially have planned 1.2 and 1.3 for this year, and
2.0 for 2011) for several reasons.

2.7 is the last planned 2.x release, and once it got alpha state, there's no
chance to introduce wordcodes model in it.

3.2 or later will be good candidates, but I don't want to make a new project
and fork again. Forking is a waste of time and resources (I spent over 1
year of my spare time just to prove an idea).

I think that wpython as a proof-of-concept have done its work, showing its
potentials.

If python dev community is interested, I can work on a 3.x branch, porting
all optimizations I made (and many others that I've planned to implement)
one step at the time, in order to carefully check and validate any change
with expert people monitoring it.

Cesare

2010/1/26 Collin Winter 

> Hi Cesare,
>
> On Tue, Jan 26, 2010 at 12:29 AM, Cesare Di Mauro
>  wrote:
> > Hi Collin,
> >
> > One more question: is it easy to support more opcodes, or a different
> opcode
> > structure, in Unladen Swallow project?
>
> I assume you're asking about integrating WPython. Yes, adding new
> opcodes to Unladen Swallow is still pretty easy. The PEP includes a
> section on this,
>
> http://www.python.org/dev/peps/pep-3146/#experimenting-with-changes-to-python-or-cpython-bytecode
> ,
> though it doesn't cover something more complex like converting from
> bytecode to wordcode, as a purely hypothetical example ;) Let me know
> if that section is unclear or needs more data.
>
> Converting from bytecode to wordcode should be relatively
> straightforward, assuming that the arrangement of opcode arguments is
> the main change. I believe the only real place you would need to
> update is the JIT compiler's bytecode iterator (see
>
> http://code.google.com/p/unladen-swallow/source/browse/trunk/Util/PyBytecodeIterator.cc
> ).
> Depending on the nature of the changes, the runtime feedback system
> might need to be updated, too, but it wouldn't be too difficult, and
> the changes should be localized.
>
> Collin Winter
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com