[Python-Dev] LXR site for Python CVS

[Hye-Shik Chang]

Hi,

I just set up a LXR instance for Python CVS for my personal use:
  http://pxr.openlook.org/pxr/

If you find it useful, feel free to use the site. :) The source files will
be updated twice a day.

Hye-Shik
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] int(string) (was: DRAFT: python-dev Summary for 2005-09-01 through 2005-09-16)

[Tim Peters]

...
> -
> Speeding up list append calls
> -
>
> A `comp.lang.python message from Tim Peters`_ prompted Neal Norwitz
> to investigate how the code that Tim posted could be sped up.  He
> hacked the code to replace var.append() with the LIST_APPEND opcode,
> 

Someone want a finite project that would _really_ help their Uncle
Timmy in his slow-motion crusade to get Python on the list of "solved
it!" languages for each problem on that magnificent site?

http://spoj.sphere.pl

It turns out that many of the problems there have input encoded as
vast quantities of integers (stdin is a mass of decimal integers on
one or more lines).  Most infamous for Python is this tutorial (you
don't get points for solving it) problem, which is _trying_ to test
whether your language of choice can read from stdin "fast enough":

http://spoj.sphere.pl/problems/INTEST/

"""
The input begins with two positive integers n k (n, k<=10**7). The
next n lines of input contain one positive integer t_i, not greater
than 10**9, each.

Output
Write a single integer to output, denoting how many integers t_i are
divisable by k.

Example

Input:
7 3
1
51
966369
7
9
96
11

Output:
4
"""

There's an 8-second time limit, and I believe stdin is about 8MB
(you're never allowed to see the actual input they use).  They have a
slower machine than you use ;-), so it's harder than it sounds.  To
date, 975 people have submitted a program that passed, but only a few
managed to do it using Python.  I did, and it required every trick in
the book, including using psyco.

Turns out it's _not_ input speed that's the problem here, and not even
mainly the speed of integer mod:  the bulk of the time is spent in
int(string) (and, yes, that's also far more important to the problem
Neal was looking at than list.append time). If you can even track all
the levels of C function calls that ends up invoking , you find
yourself in PyOS_strtoul(), which is a nifty all-purpose routine that
accepts inputs in bases 2 thru 36, can auto-detect base, and does
platform-independent overflow checking at the cost of a division per
digit.  All those are features, but it makes for slw conversion.

I assume it's the overflow-checking that's the major time sink, and
it's not correct anyway:  it does the check slightly differently for
base 10 than for any other base, explained only in the checkin comment
for rev 2.13, 8 years ago:

For base 10, cast unsigned long to long before testing overflow.
This prevents 4294967296 from being an acceptable way to spell zero!

So what are the odds that base 10 was the _only_ base that had a "bad
input" case for the overflow-check method used?  If you thought
"slim", you were right ;-)  Here are other bad cases, under all Python
versions to date (on a 32-bit box; if sizeof(long) == 8, there are
different bad cases):

int('10200202220122211', 3) = 0
int('32244002423141', 5) = 0
int('1550104015504', 6) = 0
int('211301422354', 7) = 0
int('12068657454', 9) = 0
int('1904440554', 11) = 0
int('9ba461594', 12) = 0
int('535a79889', 13) = 0
int('2ca5b7464', 14) = 0
int('1a20dcd81', 15) = 0
int('a7ffda91', 17) = 0
int('704he7g4', 18) = 0
int('4f5aff66', 19) = 0
int('3723ai4g', 20) = 0
int('281d55i4', 21) = 0
int('1fj8b184', 22) = 0
int('1606k7ic', 23) = 0
int('mb994ag', 24) = 0
int('hek2mgl', 25) = 0
int('dnchbnm', 26) = 0
int('b28jpdm', 27) = 0
int('8pfgih4', 28) = 0
int('76beigg', 29) = 0
int('5qmcpqg', 30) = 0
int('4q0jto4', 31) = 0
int('3aokq94', 33) = 0
int('2qhxjli', 34) = 0
int('2br45qb', 35) = 0
int('1z141z4', 36) = 0

IOW, the only bases that _aren't_ "bad" are powers of 2, and 10
because it's special-cased (BTW, I'm not sure that base 10 doesn't
have a different bad case now, but don't care enough to prove it one
way or the other).

Now fixing that is easy:  the problem comes from being too clever,
doing both a multiply and an addition before checking for overflow. 
Check each operation on its own and it would be bulletproof, without
special-casing.  But that might be even slower (it would remove the
branch special-casing 10, but add a cheap integer addition overflow
check with its own branch).

The challenge (should you decide to accept it ) is to replace
the overflow-checking with something both correct _and_ much faster
than doing n integer divisions for an n-character input.  For example,
36**6 < 2**32-1, so whenever the input has no more than 6 digits
overflow is impossible regardless of base and regardless of platform. 
That's simple and exploitable.  For extra credit, make int(string) go
faster than preparing your taxes ;-)


BTW, Python as-is can be used to solve many (I'd bet most) of these
problems in the time limit imposed, although it may take some effort,
and it may not be possible without using psyco.  A Python triumph I'm
particularly fond of:

 http://spoj.sphere.pl/problems/FAMILY/

The legend at the bottom:

Warning: large Input/Output data, 

Re: [Python-Dev] DRAFT: python-dev Summary for 2005-09-01 through 2005-09-16

[Guido van Rossum]

On 10/21/05, Tony Meyer <[EMAIL PROTECTED]> wrote:
> This is over a month late, sorry, but here it is (Steve did his
> threads ages ago; I've fallen really behind).

Better late than never! These summaries are awesome.

Just one nit:
> --
> Responsiveness of IDLE development
> --
>
> Noam Raphael posted a request for help getting a large patch to IDLE
> committed to CVS.  He was concerned that there hasn't been any IDLE
> development recently, and that patches are not being considered.  He
> indicated that his group was considering offering a fork of IDLE with
> the improvements, but that they would much prefer integrating the
> improvements into the core distribution.
>
> It was pointed out that a fork might be the best solution, for
> various reasons (e.g. the improvements may not be of general
> interest, the release time would be much quicker), and that this was
> how the current version of IDLE was developed.  The dicussion died
> out, so it seems likely that a fork will be the resulting solution.

Later, it turned out that Kurt Kaiser had missed this message on
python-dev (which he only reads occasionally); he redirected the
thread to idle-dev where it seems that his issues with the
contribution are being resolved and a fork is averted. Whew!

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] DRAFT: python-dev Summary for 2005-09-01 through 2005-09-16

[Tony Meyer]

This is over a month late, sorry, but here it is (Steve did his  
threads ages ago; I've fallen really behind).  Summaries for the  
second half of September and the first half of October will soon  
follow.  As always, if anyone is able to give this a quick look that  
would be great.  Feedback to me or Steve ([EMAIL PROTECTED]).   
Thanks!

=
Announcements
=

-
QOTF: Quotes of the Fortnight
-

In the thread on the print statement, Charles Cazabon provided some  
nice imagery for Guido's Python 3.0 strategy.  Our first QOTF is his  
comment about the print statement:

 It's an anomaly.  It stands out in the language as a sore thumb  
waiting for Guido's hammer.

We also learned something important about the evolution of Python  
thanks to Paul Moore.  In the thread on the Python 3.0 executable  
name, Greg Ewing worried that if the Python 3.0 executable is named  
"py":

 Python 4.0 is going to just be called "p", and by the time we  
get to Python 5.0, the name will have vanished altogether!

Fortunately, as Paul Moore explains in our second QOTF, these naming  
conventions are exactly as we should expect them:

 That's OK, by the time Python 5.0 comes out, it will have taken  
over the world and be the default language for everything. So  
omitting the name is exactly right :-)

[SJB]

Contributing threads:

- `Replacement for print in Python 3.0 `__
- `Python 3 executable name `__

--
The "Swiss Army Knife (...Not)" API design pattern
--

This fortnight saw a number of different discussions on what Guido's  
guiding principles are in making design decisions about Python. Guido  
introduced the "Swiss Army Knife (...Not)" API design pattern, which  
has been lauded by some as `the long-lost 20th principle from the Zen  
of Python`_. A direct quote from Guido:

 [I]nstead of a single "swiss-army-knife" function with various  
options that choose different behavior variants, it's better to have  
different dedicated functions for each of the major functionality types.

This principle is the basis for pairs like str.split() and str.rsplit 
() or str.find() and str.rfind().  The goal is to keep cognitive  
overhead down by associating with each use case a single function  
with a minimal number of parameters.

[SJB]

.. _the long-lost 20th principle from the Zen of Python: http:// 
mail.python.org/pipermail/python-dev/2005-September/056228.html

Contributing threads:

- `Replacement for print in Python 3.0 `__
- `Replacement for print in Python 3.0 `__


A Python-to-C++ compiler


Mark Dufour announced `Shed Skin`_, an experimental Python-to-C++  
compiler, which can convert many Python programs into optimized C++  
code, using static type inference techniques.  It works best for  
Python programs written in a relatively static C++-style; much work  
remains to be done, and Mark would like anyone interested in getting  
involved to contact him.  Shed Skin was one of the recent `Google`_  
`Summer of Code`_ projects.
.. _Shed Skin: http://shedskin.sourceforge.net
.. _Google: http://www.google.com
.. _Summer of Code: http://code.google.com/summerofcode.html

[TAM]

Contributing thread:

- `First release of Shed Skin, a Python-to-C++ compiler. `__

--
python-checkins followups now stay on the python-checkins list
--

In a follow-up to a `thread in early July`_, the python-checkins  
mailing list Reply-To header munging has been turned off.   
Previously, follow-ups to python-checkins would be addressed to  
python-dev; now, follow-ups will stay on the python-checkins list by  
default.

.. _thread in early July: http://www.python.org/dev/summary/ 
2005-07-01_2005-07-15.html#behavior-of-sourceforge-when-replying-to- 
python-checkins

[TAM]

Contributing thread:

- `python-checkins reply-to `__


=
Summaries
=


Converting print to a function in Python 3.0


In Python 3.0, Guido wants to change print from a statement to a  
function.  Some of his motivation for this change:

* Converting code that uses the print statement to instead use the  
logging package, a UI package, etc. is complicated because of the  
syntax.  Parentheses, commas and >>s all behave differently in the  
print statement t

Re: [Python-Dev] PEP 267 -- is the semantics change OK?

[Jeremy Hylton]

On 10/21/05, Jim Jewett <[EMAIL PROTECTED]> wrote:
> (In http://mail.python.org/pipermail/python-dev/2005-October/057501.html)
> Neil Schemenauer suggested PEP 267 as an example of something that
> might be easier with the AST compiler.
>
> As written, PEP 267 does propose a slight semantics change -- but it
> might be an improvement, if it is acceptable.

No, it does not.  PEP 267 suggests a way to preserve the existing
semantics.  You could probably come up with a much simpler approach if
you agreed to change semantics.

Jeremy

> Today, after
>
> from othermod import val1
> import othermod
> val2 = othermod.val2
> othermod.val3  # Just making sure it was referenced early
>
> othermod.val1 = "new1"
> othermod.val2 = "new2"
> othermod.val3 = "new3"
>
> print val1, val2, othermod.val3
>
> The print statement will see the updated val3, but will still have
> the original values for val1 and val2.
>
> Under PEP267, all three variables would be compiled to a slot
> access in othermod, and would see the updated objects.
>
> In many cases, this would be a *good* thing.  It might allow
> reload to be rewritten to do what people expect.  On the other
> hand, it would be a change.  Would it be an acceptable change?
>
> -jJ
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 267 -- is the semantics change OK?

[Jim Jewett]

(In http://mail.python.org/pipermail/python-dev/2005-October/057501.html)
Neil Schemenauer suggested PEP 267 as an example of something that
might be easier with the AST compiler.

As written, PEP 267 does propose a slight semantics change -- but it
might be an improvement, if it is acceptable.

Today, after

from othermod import val1
import othermod
val2 = othermod.val2
othermod.val3  # Just making sure it was referenced early

othermod.val1 = "new1"
othermod.val2 = "new2"
othermod.val3 = "new3"

print val1, val2, othermod.val3

The print statement will see the updated val3, but will still have
the original values for val1 and val2.

Under PEP267, all three variables would be compiled to a slot
access in othermod, and would see the updated objects.

In many cases, this would be a *good* thing.  It might allow
reload to be rewritten to do what people expect.  On the other
hand, it would be a change.  Would it be an acceptable change?

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] New codecs checked in

[M.-A. Lemburg]

I've checked in a whole bunch of newly generated codecs
which now make use of the faster charmap decoding variant added
by Walter a short while ago.

Please let me know if you find any problems.

Some codecs (esp. the Mac OS X ones) have minor changes.
These originate from updated mapping files on ftp.unicode.org.

I also added an alias iso8859_1 -> latin_1, so that applications
using the iso8859_1 encoding name can benefit from the faster
native implementation of the latin_1 codec.

Regards,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 22 2005)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Questionable AST wibbles

[M.-A. Lemburg]

Neal Norwitz wrote:
> Jeremy,
> 
> There are a bunch of mods from the AST branch that got integrated into
> head.  Hopefully, by doing this on python-dev more people will get
> involved.  I'll describe  high level things first, but there will be a
> ton of details later on.  If people don't want to see this crap on
> python-dev, I can take this offline.
> 
> Highlevel overview of code size (rough decrease of 300 C source lines):
>  * Python/compile.c -2733 (was 6822 now 4089)
>  * Python/Python-ast.c +2281 (new file)
>  * Python/asdl.c +92 (new file)
>  * plus other minor mods

FYI, I'm getting these warnings:

Python/Python-ast.c: In function `marshal_write_expr_context':
Python/Python-ast.c:1995: warning: unused variable `i'
Python/Python-ast.c: In function `marshal_write_boolop':
Python/Python-ast.c:2070: warning: unused variable `i'
Python/Python-ast.c: In function `marshal_write_operator':
Python/Python-ast.c:2085: warning: unused variable `i'
Python/Python-ast.c: In function `marshal_write_unaryop':
Python/Python-ast.c:2130: warning: unused variable `i'
Python/Python-ast.c: In function `marshal_write_cmpop':
Python/Python-ast.c:2151: warning: unused variable `i'
Python/Python-ast.c: In function `marshal_write_keyword':
Python/Python-ast.c:2261: warning: unused variable `i'
Python/Python-ast.c: In function `marshal_write_alias':
Python/Python-ast.c:2270: warning: unused variable `i'

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 21 2005)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Questionable AST wibbles

[Nick Coghlan]

Jeremy Hylton wrote:
> I would like to keep them separate.  The compiler produces code
> objects, but consumers of code objects don't need to know anything
> about the compiler.

Please do keep these separate - the only reason I've ever had to muck with 
code objects is to check if a function is a generator or not, and including 
the entire compiler header just for that seemed like overkill.

It's not a huge issue for me, but the separate header files do give better 
'separation of concerns'.

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://boredomandlaziness.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AST branch is in?

[Guido van Rossum]

On 10/21/05, Neil Schemenauer <[EMAIL PROTECTED]> wrote:
> Also, the concrete syntax tree (CST) generated by Python's parser is
> not a convenient data structure to deal with. Anyone who's used the
> 'parser' module probably experienced the pain:
>
> >>> parser.ast2list(parser.suite('a = 1'))
> [257, [266, [267, [268, [269, [320, [298, [299, [300, [301,
> [303, [304, [305, [306, [307, [308, [309, [310, [311, [1,
> 'a']]], [22, '='], [320, [298, [299, [300, [301, [303,
> [304, [305, [306, [307, [308, [309, [310, [311, [2,
> '1'], [4, '']]], [0, '']]

That's the fault of the 'parser' extension module though, and this
affects tools using the parser module, not the bytecode compiler
itself. The CST exposed to C programmers is slightly higher level.
(But the new AST is higher level still, of course.)

BTW, Elemental is letting me open-source a reimplementation of pgen in
Python. This also includes a nifty way to generate ASTs. This should
become available within a few weeks.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AST branch is in?

[Neil Schemenauer]

Anthony Baxter <[EMAIL PROTECTED]> wrote:
> Could someone involved give a short email laying out what concrete (no 
> pun intended) advantages this new compiler gives us?

One advantage is that it decreases the coupling between the parser
and the backend of the compiler. For example, it should be possible
to replace the parser without modifying a lot of the compiler.
Also, the concrete syntax tree (CST) generated by Python's parser is
not a convenient data structure to deal with. Anyone who's used the
'parser' module probably experienced the pain:

>>> parser.ast2list(parser.suite('a = 1'))
[257, [266, [267, [268, [269, [320, [298, [299, [300, [301,
[303, [304, [305, [306, [307, [308, [309, [310, [311, [1,
'a']]], [22, '='], [320, [298, [299, [300, [301, [303,
[304, [305, [306, [307, [308, [309, [310, [311, [2,
'1'], [4, '']]], [0, '']]

> Does it just allow us to do new and interesting manipulations of
> the code during compilation?

Well, that's a pretty big deal, IMHO. For example, adding
pychecker-like functionality should be straight forward now. I also
hope some of the namespace optimizations get explored (e.g. PEP
267).

> Cleaner, easier to maintain, or the like?

At this point, the old and new compiler are pretty similar in terms
of complexity. However, the new compiler is a much better base to
build upon.

  Neil

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AST branch is in?

[Jeremy Hylton]

On 10/20/05, Guido van Rossum <[EMAIL PROTECTED]> wrote:
> On 10/20/05, Anthony Baxter <[EMAIL PROTECTED]> wrote:
> > So it looks like the AST branch has landed. Wooo! Well done to all who
> > were involved - it seems like it's been a huge amount of work.
>
> Hear, hear. Great news! Thanks to Jeremy, Neil and all the others. I
> can't wait to check it out!

I want to thank all the people who made it possible by writing code
and debugging.  I hope this is a complete list:

Armin Rigo
Brett Cannon
Grant Edwards
John Ehresman
Kurt Kaiser
Neal Norwitz
Neil Schemenauer
Nick Coghlan
Tim Peters

And thanks to the PSF and PyCon organizers for hosting the formerly
annual ast-branch sprints!

Jeremy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AST branch is in?

[Jeremy Hylton]

On 10/20/05, Neal Norwitz <[EMAIL PROTECTED]> wrote:
> On 10/20/05, Anthony Baxter <[EMAIL PROTECTED]> wrote:
> >
> > Could someone involved give a short email laying out what concrete (no
> > pun intended) advantages this new compiler gives us? Does it just
> > allow us to do new and interesting manipulations of the code during
> > compilation? Cleaner, easier to maintain, or the like?
>

I just wanted to clarify that Neal meant the abstract syntax, not the
grammar.  It should allow people to write tools to analyze Python
source code without having to worry about the often irrelevant details
of the exact tokens or the way they are parsed.  We should be able to
get to a state where tools using the AST work with Python and Jython
(and maybe IronPython, who knows).  The tokenize and parser modules
still exist for tools for which those details aren't irrelevant.

We should also think about how to migrate the compiler module from its
current AST to the new AST, although the backwards compatibility
issues there are a bit tricky.

> The Grammar is (was at one point at least) shared between Jython and
> would allow more tools to be able to share infrastructure.  The idea
> is to eventually be able to have [JP]ython output the same AST to
> tools.  There is quite a bit of generated code based on the Grammar.
> So some stuff should be easier.  Other stuff is just moved.  You still
> need to convert from the AST to the byte code.
>
> Hopefully it will be easier to do various sorts of optimization and
> general manipulation of an AST rather than what existed before.

I think it should be a lot easier to write tools for the C Python
compiler that do extra analysis or optimization.  The existing
peephole optimizer could be improved by integrating it with the
bytecode assembler (for example, eliminating all NOP bytecodes).

Jeremy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Questionable AST wibbles

[Phillip J. Eby]

At 11:13 AM 10/21/2005 -0400, Jeremy Hylton wrote:
>I don't have a strong sense for how important these changes are.  I
>don't think the old behavior was documented, but I can imagine some
>code depending on these implementation details.

I'm pretty sure I've seen code in the field (e.g. recipes in the online 
Python cookbook) that checked for a function's name being 
''.  That's also a thing that's likely to show up in people's doctests.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Questionable AST wibbles

[Guido van Rossum]

On 10/21/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote:
> On 10/21/05, Neal Norwitz <[EMAIL PROTECTED]> wrote:
> > This probably is not a big deal, but I was surprised by this change:
> >
> > +++ test_repr.py20 Oct 2005 19:59:24 -  1.20
> > @@ -123,7 +123,7 @@
> >
> > def test_lambda(self):
> > self.failUnless(repr(lambda x: x).startswith(
> > -" > +"", please change it back. The angle
brackets make it stand out more, and I imagine people might be
checking for this to handle it specially.

> > This one may be only marginally worse (names w/parameter unpacking):
> >
> > test_grammar.py
> >
> > -verify(f4.func_code.co_varnames == ('two', '.2', 'compound',
> > -'argument',  'list'))
> > +vereq(f4.func_code.co_varnames,
> > +  ('two', '.1', 'compound', 'argument',  'list'))

This doesn't bother me.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Questionable AST wibbles

[Jeremy Hylton]

On 10/21/05, Neal Norwitz <[EMAIL PROTECTED]> wrote:
> There are a bunch of mods from the AST branch that got integrated into
> head.  Hopefully, by doing this on python-dev more people will get
> involved.  I'll describe  high level things first, but there will be a
> ton of details later on.  If people don't want to see this crap on
> python-dev, I can take this offline.

Thanks for all the notes and questions, Neal.  There were a lot of
changes made over a long time, and it's good to discuss some of them.

> Highlevel overview of code size (rough decrease of 300 C source lines):
>  * Python/compile.c -2733 (was 6822 now 4089)
>  * Python/Python-ast.c +2281 (new file)
>  * Python/asdl.c +92 (new file)
>  * plus other minor mods
>
> symtable.h has lots of changes to structs and APIs.  Not sure what
> needs to be doc'ed.

The old symtable wasn't well documented and the API it exposed to
Python programmers was lousy.  We need to figure out a good Python API
and document it.

> I was very glad to see that ./python compileall.py Lib took virtually
> the same time before and after AST.  Yeah!  Unfortunately, I can't say
> the same for memory usage for running compileall:
>
> Before AST: [10120 refs]
> After AST:  [916096 refs]

That's great news!  That is, I expected it to be a lot slower to
compile and didn't have any particulary good ideas about how to speed
it up.  I expected there to be a lot of memory bloat and think we can
fix that without undue effort :-).

> A bunch of APIs changed and there is some additional name pollution.
> Since these are pretty internal APIs, I'm not sure that part is a big
> deal.  I will try to find more name pollution and eliminate it by
> prefixing with Py.

Right.  The code isn't binary compatible with Python 2.4 right now,
but given the APIs that changed I wasn't too concerned about that. 
I'm not sure who should make the final decision there.

> One API change which I think was a mistake was _Py_Mangle() losing 2
> parameters (I think this was how it was a long time ago).  See
> typeobject.c, Python.h, compile.c.

I don't mind this one since it's an _Py function.  I don't think code
outside the core should use it.

> pythonrun.h has a bunch of changes.  I think a lot of the APIs
> changed, but there might be backwards compatible macros.  I'm not
> sure.  I need to review closely.

We should double-check.  I tried to get rid of the nest of different
functions that call each other by replacing the old ones with macros
that call the newest ones (the functions that take the most
arguments).  It's not really a related change, except that it seemed
like cleanup of compiler-related code.  Also, a bunch of functions
started taking const char* instead of char*.  I think that's a net
win, too.

> code.h was added, but it mostly contains stuff from compile.h.  Should
> we remove code.h and just put everything in compile.h?  This will
> remove lots little changes.
> code.h & compile.h are tightly coupled.  If we keep them separate, I
> would like to see some other changes.

I would like to keep them separate.  The compiler produces code
objects, but consumers of code objects don't need to know anything
about the compiler.  You did remind me that I intended to remove the
#include "compile.h" lines from a bunch of files that merely consume
code objects.  What other changes would you like to see?

> This probably is not a big deal, but I was surprised by this change:
>
> +++ test_repr.py20 Oct 2005 19:59:24 -  1.20
> @@ -123,7 +123,7 @@
>
> def test_lambda(self):
> self.failUnless(repr(lambda x: x).startswith(
> -" +"
> This one may be only marginally worse (names w/parameter unpacking):
>
> test_grammar.py
>
> -verify(f4.func_code.co_varnames == ('two', '.2', 'compound',
> -'argument',  'list'))
> +vereq(f4.func_code.co_varnames,
> +  ('two', '.1', 'compound', 'argument',  'list'))
>
> There are still more things I need to review.  These were the biggest
> issues I found.  I don't think most are that big of a deal, just
> wanted to point stuff out.

I don't have a strong sense for how important these changes are.  I
don't think the old behavior was documented, but I can imagine some
code depending on these implementation details.

Jeremy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Task-local variables

[Nick Coghlan]

Guido van Rossum wrote:
> If it weren't for Python's operator overloading, the decimal module
> would have used explicit contexts (like the Java version); but since
> it would be really strange to have such a fundamental numeric type
> without the ability to use the conventional operator notation, we
> resorted to per-thread context. Even that doesn't always do the right
> thing -- handling decimal contexts is surprisingly subtle (as Nick can
> testify based on his experiences attempting to write a decimal context
> manager for the with-statement!).

Indeed. Fortunately it isn't as complicated as I feared last night (it turned 
out to be a problem with me trying to hit a small nail with the new 
sledgehammer I was playing with, forgetting entirely about the trusty old 
normal hammer still in the toolkit).

> But I haven't seen the use case yet for mixing coroutines with changes
> to decimal context settings; somehow it doesn't strike me as a likely
> use case (not that you can't construct one, so don't bother -- I can
> imagine it too, I just think YAGNI).

For Python 2.5, I think the approach of generators explicitly reverting 
altered contexts around yield expressions is a reasonable way to go.

This concept is workable for generators, because they *know* when they're 
going to lose control (i.e., by invoking yield), whereas it's impossible for 
threads to know when the eval loop is going to drop them in favour of a 
different thread.

I think the parallel between __iter__ and __with__ continues to work here, too 
- alternate context managers to handle reversion of the context (e.g., 
Lock.released()) can be provided as separate methods, just as alternative 
iterators are provided (e.g., dict.iteritems(), dict.itervalues()).

Also, just as we eventually added "itertools" to support specific ways of 
working with iterators, I expect to eventually see "contexttools" to support 
specific ways of working with contexts (e.g. duck-typed contexts like 
"closing", or a 'nested' context that allowed multiple resources to be 
properly managed by a single with statement).

contexttools would also be the place for ideas like suspending and resuming a 
context - rather than requiring specific syntax, it could be implemented as a 
context manager:

   ctx = suspendable_context(EXPR)
   with ctx as VAR:
 # VAR would still be the result of (EXPR).__with__().__enter__()
 # It's just that suspendable_context would be taking care of
 # making that happen, rather than it happening the usual way
 with ctx.suspended():
   # Context is suspended here
 # Context is resumed here

I do *not* think we should add contexttools in Python 2.5, because there's far 
too much chance of YAGNI. We need experience with the 'with' statement before 
we can really identify the tools that are appropriate.

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://boredomandlaziness.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AST branch is in?

[Nick Coghlan]

Anthony Baxter wrote:
> So it looks like the AST branch has landed. Wooo! Well done to all who 
> were involved - it seems like it's been a huge amount of work.

Congratulations from this quarter, too.

I really liked the structure of the new compiler in the limited time I spent 
working with it on the AST branch, and am glad it has made its way onto the 
HEAD for Python 2.5.

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://boredomandlaziness.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Task-local variables

[Nick Coghlan]

Phillip J. Eby wrote:
> Actually, it's fairly simple to write a generator decorator using 
> context.swap() that saves and restores the current execution state 
> around next()/send()/throw() calls, if you prefer it to be the 
> generator's responsibility to maintain such context.

Yeah, I also realised there's a fairly obvious solution to my decimal.Context 
"problem" too:

 def iter_sin(iterable):
orig_ctx = decimal.getcontext()
with orig_ctx as ctx:
ctx.prec += 10
for r in iterable:
y = sin(r) # Very high precision during calculation
with orig_ctx:
yield +y # Interim results have normal precision
# We get "ctx" back here
# We get "orig_ctx" back here

That is, if you want to be able to restore the original context just *save* 
the damn thing. . .

Ah well, chalk up the __suspend__/__resume__ idea up as another case of me 
getting overly enthusiastic about a complex idea without looking for simpler 
solutions first. It's not like it would be the first time ;)

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://boredomandlaziness.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Task-local variables

[Michael Hudson]

Guido van Rossum <[EMAIL PROTECTED]> writes:

> On 10/20/05, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
>> At 08:57 AM 10/20/2005 -0700, Guido van Rossum wrote:
>> >Whoa, folks! Can I ask the gentlemen to curb their enthusiasm?
>> >
>> >PEP 343 is still (back) on the drawing table, PEP 342 has barely been
>> >implemented (did it survive the AST-branch merge?), and already you
>> >are talking about adding more stuff. Please put on the brakes!
>>
>> Sorry.  I thought that 343 was just getting a minor tune-up.
>
> Maybe, but the issues on the table are naming issues -- is __with__
> the right name, or should it be __context__? Should the decorator be
> applied implicitly? Should the decorator be called @context or
> @contextmanager?
>
>> In the months
>> since the discussion and approval (and implementation; Michael Hudson
>> actually had a PEP 343 patch out there),
>
> Which he described previously as "a hack"

Err, that was the code I used for my talk at EuroPython.  That really
*was* a hack.  The code on SF is much better.

> and apparently didn't feel comfortable checking in.

Well, I was kind of hoping for a review, or positive comment on the
tracker, or *something* (Phillip posted half a review here a couple of
weeks ago, but I've been stupidly stupidly busy since then).

> At least some of it will have to be redone, (a) for the AST code,

Indeed.  Not much, I hope, the compiler changes were fairly simple.

> and (b) for the revised PEP.

Which I still haven't digested :-/

Cheers,
mwh

-- 
  I'm about to search Google for contract assassins to go to Iomega
  and HP's programming groups and kill everyone there with some kind
  of electrically charged rusty barbed thing.
-- http://bofhcam.org/journal/journal.html, 2002-01-08
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com