Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Mon, Jan 25, 2010 at 10:44 AM, Tres Seaver  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Collin Winter wrote:
>
>> For reference, what are these "obscure platforms" where static
>> initializers cause problems?
>
> It's been a long while since I had to deal with it, but the "usual
> suspets" back in the day were HP-UX, AIX, and Solaris with non-GCC
> compilers, as well as Windows when different VC RT libraries got into
> the mix.

So then the question is, will this cause any problems we care about?
Do the problems still exist, or were they eliminated in the time
between "back in the day" and now? In what circumstances do static
initializers have problems? What problems do they have? Can the
obscure platforms work around the problems by configuring with
--without-llvm? If we eliminate static initializers in LLVM, are there
any other problems?

We really do need precise descriptions of the problems so we can avoid them.

Thanks,
Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Floris Bruynooghe]

On Mon, Jan 25, 2010 at 10:14:35AM -0800, Collin Winter wrote:
> I'm working on a patch to completely remove all traces of C++ with
> configured with --without-llvm. It's a straightforward change, and
> should present no difficulties.

Great to hear that, thanks for caring.

> For reference, what are these "obscure platforms" where static
> initializers cause problems?

I've had serious trouble on AIX 5.3 TL 04 with a GCC toolchain
(apparently the IBM xlc toolchain is better for that instance).  The
problem seems to be that gcc stores the initialisation code in a
section (_GLOBAL__DI IIRC) which the system loader does not execute.
Altough this was involving dlopen() from a C main() which U-S would
not need AFAIK, having a C++ main() might make the loader do the right
thing.  I must also note that on more recent versions (TL 07) this was
no problem at all.  But you don't always have the luxury of being able
to use recent OSes.


Regards
Floris

PS: For completeness sake this was trying to use the omniorbpy module
with Python 2.5.

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Floris Bruynooghe]

On Mon, Jan 25, 2010 at 11:48:56AM -0800, Jeffrey Yasskin wrote:
> On Mon, Jan 25, 2010 at 10:44 AM, Tres Seaver  wrote:
> > Collin Winter wrote:
> >
> >> For reference, what are these "obscure platforms" where static
> >> initializers cause problems?
> >
> > It's been a long while since I had to deal with it, but the "usual
> > suspets" back in the day were HP-UX, AIX, and Solaris with non-GCC
> > compilers, as well as Windows when different VC RT libraries got into
> > the mix.
> 
> So then the question is, will this cause any problems we care about?
> Do the problems still exist, or were they eliminated in the time
> between "back in the day" and now? In what circumstances do static
> initializers have problems? What problems do they have? Can the
> obscure platforms work around the problems by configuring with
> --without-llvm? If we eliminate static initializers in LLVM, are there
> any other problems?

When Collin's patch is finished everything will be lovely since if
there's no C++ then there's no problem.  Since I was under the
impression that the JIT/LLVM can't emit machine code for the platforms
where these C++ problems would likely occur nothing would be lost.  So
trying to change the LLVM to avoid static initialisers would not seem
like a good use of someones time.

> We really do need precise descriptions of the problems so we can avoid them.

Sometimes these "precise descriptions" are hard to come by.  :-)


Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Martin v. Löwis]

> We really do need precise descriptions of the problems so we can avoid them.

One family of problems is platform lack of initializer support in the
object file format; any system with traditional a.out (or b.out) is
vulnerable (also, COFF is, IIRC).

The solution e.g. g++ came up with is to have the collect2 linker
replacement combine all such initializers into a synthesized function
__main; this function then gets "magically" called by main(), provided
that main() itself gets compiled by a C++ compiler. Python used to have
a ccpython.cc entry point to support such systems.

This machinery is known to fail in the following ways:
a) main() is not compiled with g++: static objects get not constructed
b) code that gets linked into shared libraries (assuming the system
   supports them) does not get its initializers invoked.
c) compilation of main() with a C++ compiler, but then linking with ld
   results in an unresolved symbol __main.

Not sure whether U-S has any global C++ objects that need construction
(but I would be surprised if it didn't).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Martin v. Löwis]

Floris Bruynooghe wrote:
> Since I was under the
> impression that the JIT/LLVM can't emit machine code for the platforms
> where these C++ problems would likely occur nothing would be lost.

AFAICT, LLVM doesn't support Itanium or HPPA, and apparently not POWER,
either (although they do support PPC - not sure what that means for
POWER). So that rules out AIX and HP-UX as sources of problems (beyond
the problems that they cause already :-)

LLVM does support SPARC, so I'd be curious about reports that it worked
or didn't work on a certain Solaris release, with either SunPRO or the
gcc release that happened to be installed on the system (Solaris
installation sometimes feature fairly odd g++ versions).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Jeffrey Yasskin wrote:
> On Mon, Jan 25, 2010 at 10:44 AM, Tres Seaver  wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Collin Winter wrote:
>>
>>> For reference, what are these "obscure platforms" where static
>>> initializers cause problems?
>> It's been a long while since I had to deal with it, but the "usual
>> suspets" back in the day were HP-UX, AIX, and Solaris with non-GCC
>> compilers, as well as Windows when different VC RT libraries got into
>> the mix.
> 
> So then the question is, will this cause any problems we care about?
> Do the problems still exist, or were they eliminated in the time
> between "back in the day" and now? In what circumstances do static
> initializers have problems? What problems do they have? Can the
> obscure platforms work around the problems by configuring with
> --without-llvm? If we eliminate static initializers in LLVM, are there
> any other problems?
> 
> We really do need precise descriptions of the problems so we can avoid them.

Yup, sorry:  I was trying to kick in the little I could remember, but it
has been eight years since I wrote / compiled C++ in anger (a decade of
work in it before that).  MvLs reply sounds *exactly* like the memories
I have, though.


Tres.
- --
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkteGGcACgkQ+gerLs4ltQ5SHwCfcQOswX0StFS32U3fFE6RZ5rr
z0QAmgKUECEhdZPQhgsNACkRiWrWX0t0
=eXM+
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Mon, Jan 25, 2010 at 1:50 PM, "Martin v. Löwis"  wrote:
>> We really do need precise descriptions of the problems so we can avoid them.
>
> One family of problems is platform lack of initializer support in the
> object file format; any system with traditional a.out (or b.out) is
> vulnerable (also, COFF is, IIRC).
>
> The solution e.g. g++ came up with is to have the collect2 linker
> replacement combine all such initializers into a synthesized function
> __main; this function then gets "magically" called by main(), provided
> that main() itself gets compiled by a C++ compiler. Python used to have
> a ccpython.cc entry point to support such systems.
>
> This machinery is known to fail in the following ways:
> a) main() is not compiled with g++: static objects get not constructed
> b) code that gets linked into shared libraries (assuming the system
>   supports them) does not get its initializers invoked.
> c) compilation of main() with a C++ compiler, but then linking with ld
>   results in an unresolved symbol __main.

Thank you for the details. I'm pretty confident that (a) and (c) will
not be a problem for the Unladen Swallow merge because we switched
python.c (which holds main()) and linking to use the C++ compiler when
LLVM's enabled at all:
http://code.google.com/p/unladen-swallow/source/browse/trunk/Makefile.pre.in.
Python already had some support for this through the LINKCC configure
variable, but it wasn't being used to compile main(), just to link.

(b) could be a problem if we depend on LLVM as a shared library on one
of these platforms (and, of course, if LLVM's JIT supports these
systems at all). The obvious answers are: 1) --without-llvm on these
systems, 2) link statically on these systems, 3) eliminate the static
constructors. There may also be less obvious answers.

> Not sure whether U-S has any global C++ objects that need construction
> (but I would be surprised if it didn't).

I checked with them, and LLVM would welcome patches to remove these if
we need to. They already have a llvm::ManagedStatic class
(http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Support/ManagedStatic.h?view=markup)
with no constructor that we can replace most of them with.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Martin v. Löwis]

>> a) main() is not compiled with g++: static objects get not constructed
>> b) code that gets linked into shared libraries (assuming the system
>>   supports them) does not get its initializers invoked.
>> c) compilation of main() with a C++ compiler, but then linking with ld
>>   results in an unresolved symbol __main.
> 
> Thank you for the details. I'm pretty confident that (a) and (c) will
> not be a problem for the Unladen Swallow merge because we switched
> python.c (which holds main()) and linking to use the C++ compiler when
> LLVM's enabled at all:
> http://code.google.com/p/unladen-swallow/source/browse/trunk/Makefile.pre.in.
> Python already had some support for this through the LINKCC configure
> variable, but it wasn't being used to compile main(), just to link.

Ok - but then this *is* a problem for people embedding Python on such
systems. They have to adjust their build process as well.
(they probably will need to make adjustments for C++ even on ELF
systems, as you still need to run C++ as the linker when linking
libpythonxy.a, to get the C++ runtime linked)

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hey Floris,

On Mon, Jan 25, 2010 at 1:25 PM, Floris Bruynooghe
 wrote:
> On Mon, Jan 25, 2010 at 10:14:35AM -0800, Collin Winter wrote:
>> I'm working on a patch to completely remove all traces of C++ with
>> configured with --without-llvm. It's a straightforward change, and
>> should present no difficulties.
>
> Great to hear that, thanks for caring.

This has now been resolved. As of
http://code.google.com/p/unladen-swallow/source/detail?r=1036,
./configure --without-llvm has no dependency on libstdc++:

Before: $ otool -L ./python.exe
./python.exe:
/usr/lib/libSystem.B.dylib
/usr/lib/libstdc++.6.dylib
/usr/lib/libgcc_s.1.dylib


After: $ otool -L ./python.exe
./python.exe:
/usr/lib/libSystem.B.dylib
/usr/lib/libgcc_s.1.dylib

I've explicitly noted this in the PEP (see
http://codereview.appspot.com/186247/diff2/2001:4001/5001).

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Meador Inge]

> We really do need precise descriptions of the problems so we can avoid
them.

Initialization of objects with static storage duration typically get a bad
wrap for two main reasons: (1) each toolchain implements them differently
(but typically by storing initialization thunks in a table that is walked by
the C++RT before entry to 'main'), which may lead to subtle differences when
compiling for different platforms and (2) there is no guaranteed
initialization order across translation unit boundaries.

(1) is just a fact of multi-platform development.  (2) is a bit more
interesting.  Consider two translation units 'a.cpp' and 'b.cpp':

   // a.cpp
   T { T() {} };
   ...
   static T obj1;

   // b.cpp
   S { S() {} };
   ...
   static S obj2;

When 'obj1' and 'obj2' get linked into the final image there are no
guarantees on whose constructor (T::T or S::S) will be called first.
Sometimes folks write code where this initialization order matters.  It may
cause strange behavior at run-time that is hard to pin down.  This may not
be a problem in the LLVM code base, but it is the typical problem that C++
devs run into with initialization of objects with static storage duration.

Also related to reduced code size with C++ I was wondering whether or not
anyone has explored using the ability of some toolchains to remove unused
code and data?  In GCC this can be enabled by compiling with
'-ffunction-sections' and '-fdata-sections' and linking with
'--gc-sections'.  In MS VC++ you can compile with '/Gy' and link with
'/OPT'.  This feature can lead to size reductions sometimes with C++ due to
things like template instantation causing multiple copies of the same
function to be linked in.  I played around with compiling CPython with this
(gcc + Darwin) and saw about a 200K size drop.  I want to try compiling all
of U-S (e.g. including LLVM) with these options next.

Thanks,

-- Meador
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Coghlan]

Jeffrey Yasskin wrote:
> (b) could be a problem if we depend on LLVM as a shared library on one
> of these platforms (and, of course, if LLVM's JIT supports these
> systems at all). The obvious answers are: 1) --without-llvm on these
> systems, 2) link statically on these systems, 3) eliminate the static
> constructors. There may also be less obvious answers.

Could the necessary initialisation be delayed until the Py_Initialize()
call? (although I guess that is just a particular implementation
strategy for option 3).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Coghlan]

Meador Inge wrote:
> When 'obj1' and 'obj2' get linked into the final image there are no
> guarantees on whose constructor (T::T or S::S) will be called first. 
> Sometimes folks write code where this initialization order matters.  It
> may cause strange behavior at run-time that is hard to pin down.  This
> may not be a problem in the LLVM code base, but it is the typical
> problem that C++ devs run into with initialization of objects with
> static storage duration.

Avoiding this problem is actually one of the original reasons for the
popularity of the singleton design pattern in C++. With instantiation on
first use, it helps ensure the constructors are all executed in the
right order. (There are other problems with the double-checked locking
required under certain aggressive compiler optimisation strategies, but
the static initialisation ordering problem occurs even when optimisation
is completely disabled).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Reid Kleckner]

On Mon, Jan 25, 2010 at 9:05 PM, Meador Inge  wrote:
> Also related to reduced code size with C++ I was wondering whether or not
> anyone has explored using the ability of some toolchains to remove unused
> code and data?  In GCC this can be enabled by compiling with
> '-ffunction-sections' and '-fdata-sections' and linking with
> '--gc-sections'.  In MS VC++ you can compile with '/Gy' and link with
> '/OPT'.  This feature can lead to size reductions sometimes with C++ due to
> things like template instantation causing multiple copies of the same
> function to be linked in.  I played around with compiling CPython with this
> (gcc + Darwin) and saw about a 200K size drop.  I want to try compiling all
> of U-S (e.g. including LLVM) with these options next.

I'm sure someone has looked at this before, but I was also considering
this the other day.  One catch is that C extension modules need to be
able to link against any symbol declared with the PyAPI_* macros, so
you're not allowed to delete PyAPI_DATA globals or any code reachable
from a PyAPI_FUNC.

Someone would need to modify the PyAPI_* macros to include something
like __attribute__((used)) with GCC and then tell the linker to strip
unreachable code.  Apple calls it "dead stripping":
http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man1/ld.1.html

This seems to have a section on how to achieve the same effect with a
gnu toolchain:
http://utilitybase.com/article/show/2007/04/09/225/Size+does+matter:+Optimizing+with+size+in+mind+with+GCC

I would guess that we have a fair amount of unused LLVM code linked in
to unladen, so stripping it would reduce our size.  However, we can
only do that if we link LLVM statically.  If/When we dynamically link
against LLVM, we lose our ability to strip out unused symbols.  The
best we can do is only link with the libraries we use, which is what
we already do.

Reid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Martin v. Löwis]

Nick Coghlan wrote:
> Jeffrey Yasskin wrote:
>> (b) could be a problem if we depend on LLVM as a shared library on one
>> of these platforms (and, of course, if LLVM's JIT supports these
>> systems at all). The obvious answers are: 1) --without-llvm on these
>> systems, 2) link statically on these systems, 3) eliminate the static
>> constructors. There may also be less obvious answers.
> 
> Could the necessary initialisation be delayed until the Py_Initialize()
> call? (although I guess that is just a particular implementation
> strategy for option 3).

Exactly. The convenience of constructors is that you don't need to know
what all your global objects. If you arrange it so that you can trigger
initialization, you must have already eliminated them.

The question now is what state they carry: if some of them act as
dynamic containers, e.g. for machine code, it would surely be desirable
to have them constructed with Py_Initialize and destroyed with
Py_Finalize (so that the user sees the memory being released). If they
are merely global flags of some kind, they don't matter much.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

Hi Collin,

2010/1/25 Collin Winter 

> Hi Cesare,
>
> On Sat, Jan 23, 2010 at 1:09 PM, Cesare Di Mauro
>  wrote:
> > Hi Collin
> >
> > IMO it'll be better to make Unladen Swallow project a module, to be
> > installed and used if needed, so demanding to users the choice of having
> it
> > or not. The same way psyco does, indeed.
> > Nowadays it requires too much memory, longer loading time, and fat
> binaries
> > for not-so-great performances. I know that some issues have being worked
> on,
> > but I don't think that they'll show something comparable to the current
> > CPython status.
>
> You're proposing that, even once the issues of memory usage and
> startup time are addressed, Unladen Swallow should still be an
>  extension module? I don't see why.


Absolutely not, of course.


> You're assuming that these issues
> cannot be fixed, which I disagree with.
>

No, it's my belief, from what I see until now, that it'll be very difficult
to have a situation that is comparable to the current one, in the terms that
I've talked about (memory usage, load time, and binaries size).

I hope to make a mistake. :)


>
> I think maintaining something like a JIT compiler out-of-line, as
> Psyco is, causes long-term maintainability problems. Such extension
> modules are forever playing catchup with the CPython code, depending
> on implementation details that the CPython developers are right to
>  regard as open to change.


I agree (especially for psyco), but ceval.c has a relatively stable code
(not mine, however :D).

It also limits what kind of optimizations
> you can implement or forces those optimizations to be implemented with
> workarounds that might be suboptimal or fragile. I'd recommend reading
> the Psyco codebase, if you haven't yet.
>

Optimizations are surely a point in favor of integrating U.S. project in the
main core.

Psyco, as I said before, is quite a mess. It's hard to add new back-ends for
other architectures. It's a bit less difficult to keep it in sync with
opcode changes (except for big changes), and a port to Python 3.x may be
suitable (but I don't know if the effort makes sense).

As others have requested, we are working hard to minimize the impact
> of the JIT so that it can be turned off entirely at runtime. We have
> an active issue tracking our progress at
> http://code.google.com/p/unladen-swallow/issues/detail?id=123.
>

I see, thanks.


> > Introducing C++ is a big step, also. Aside the problems it can bring on
> some
> > platforms, it means that C++ can now be used by CPython developers.
>
> Which platforms, specifically? What is it about C++ on those platforms
> that is problematic? Can you please provide details?
>

Others have talked about it.


> > It
> > doesn't make sense to force people use C for everything but the JIT part.
> In
> > the end, CPython could become a mix of C and C++ code, so a bit more
> > difficult to understand and manage.
>
> Whether CPython should allow wider usage of C++ or whether developer
> should be "force[d]" to use C is not our decision, and is not part of
> this PEP. With the exception of Python/eval.c, we deliberately have
> not converted any CPython code to C++ so that if you're not working on
> the JIT, python-dev's workflow remains the same. Even within eval.cc,
> the only C++ parts are related to the JIT, and so disappear completely
> with configured with --without-llvm (or if you're not working on the
> JIT).
>
> In any case, developers can easily tell which language to use based on
> file extension. The compiler errors that would result from compiling
> C++ with a C compiler would be a good indication as well.
>

OK, if CPython will be compilable without using C++ at all, I retire what I
said.


> > What I see is that LLVM is a too big project for the goal of having
> "just" a
> > JIT-ed Python VM. It can be surely easier to use and integrate into
> CPython,
> > but requires too much resources
>
> Which resources do you feel that LLVM would tax, machine resources or
> developer resources? Are you referring to the portions of LLVM used by
> Unladen Swallow, or the entire wider LLVM project, including the
> pieces Unladen Swallow doesn't use at runtime?
>

No, I'm referring to the portions of LLVM used by U.S..

Regards resources, I was talking about memory, loading time, and binaries
size (both with static and dynamic compilation).


>
> > (on the contrary, Psyco demands little
> > resources, give very good performances, but seems to be like a mess to
> > manage and extend).
>
> This is not my experience. For the workloads I have experience with,
> Psyco doubles memory usage while only providing a 15-30% speed
> improvement. Psyco's benefits are not uniform.
>

I made only computation intensive (integers, floats) tasks with Psyco, and
it worked fine.

I haven't made tests with U.S. benchmark suite.


>
> Unladen Swallow has been designed to be much more maintainable and
> easier to extend and modify than Psyco: the compiler and its attendant
> 

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

Hi Collin,

One more question: is it easy to support more opcodes, or a different opcode
structure, in Unladen Swallow project?

Thanks,
Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Floris Bruynooghe]

Hello Collin

On Mon, Jan 25, 2010 at 05:27:38PM -0800, Collin Winter wrote:
> On Mon, Jan 25, 2010 at 1:25 PM, Floris Bruynooghe
>  wrote:
> > On Mon, Jan 25, 2010 at 10:14:35AM -0800, Collin Winter wrote:
> >> I'm working on a patch to completely remove all traces of C++ with
> >> configured with --without-llvm. It's a straightforward change, and
> >> should present no difficulties.
> >
> > Great to hear that, thanks for caring.
> 
> This has now been resolved. As of
> http://code.google.com/p/unladen-swallow/source/detail?r=1036,
> ./configure --without-llvm has no dependency on libstdc++:

Great, thanks for the work.

Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[skip]

Cesare> ... but ceval.c has a relatively stable code ...

I believe you are mistaken on several counts:

* The names of the functions in there have changed over time.

* The suite of byte code operations have changed dramatically over the
  past ten years or so.  

* The relationship between the code in ceval.c and the Python threading
  model has changed.

Any or all of these aspects of the virtual machine, as well I'm sure as many
other things I've missed would have to be tracked by any extension module
which hoped to supplant or augment its function in some way.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

Hi Skip

For "relatively stable code" I talk about recent years.

My experience with CPython is limited, of course.

Cesare

2010/1/26 

>
>Cesare> ... but ceval.c has a relatively stable code ...
>
> I believe you are mistaken on several counts:
>
>* The names of the functions in there have changed over time.
>
>* The suite of byte code operations have changed dramatically over the
>  past ten years or so.
>
>* The relationship between the code in ceval.c and the Python threading
>  model has changed.
>
> Any or all of these aspects of the virtual machine, as well I'm sure as
> many
> other things I've missed would have to be tracked by any extension module
> which hoped to supplant or augment its function in some way.
>
> Skip
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Martin v. Löwis]

> This
> may not be a problem in the LLVM code base, but it is the typical
> problem that C++ devs run into with initialization of objects with
> static storage duration.

This *really* doesn't have to do anything with U-S, but I'd like to
point out that standard C++ has a very clear semantics in this matter:
Any global object can be used in the translation unit where it is
defined after the point where it is defined. So if you arrange for all
accessors to a global object to occur after its definition, you don't
need to worry about initialization order at all.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hi Cesare,

On Tue, Jan 26, 2010 at 12:29 AM, Cesare Di Mauro
 wrote:
> Hi Collin,
>
> One more question: is it easy to support more opcodes, or a different opcode
> structure, in Unladen Swallow project?

I assume you're asking about integrating WPython. Yes, adding new
opcodes to Unladen Swallow is still pretty easy. The PEP includes a
section on this,
http://www.python.org/dev/peps/pep-3146/#experimenting-with-changes-to-python-or-cpython-bytecode,
though it doesn't cover something more complex like converting from
bytecode to wordcode, as a purely hypothetical example ;) Let me know
if that section is unclear or needs more data.

Converting from bytecode to wordcode should be relatively
straightforward, assuming that the arrangement of opcode arguments is
the main change. I believe the only real place you would need to
update is the JIT compiler's bytecode iterator (see
http://code.google.com/p/unladen-swallow/source/browse/trunk/Util/PyBytecodeIterator.cc).
Depending on the nature of the changes, the runtime feedback system
might need to be updated, too, but it wouldn't be too difficult, and
the changes should be localized.

Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hey Martin,

On Thu, Jan 21, 2010 at 2:25 PM, "Martin v. Löwis"  wrote:
> Reid Kleckner wrote:
>> On Thu, Jan 21, 2010 at 4:34 PM, "Martin v. Löwis"  
>> wrote:
 How large is the LLVM shared library? One surprising data point is that the
 binary is much larger than some of the memory footprint measurements given 
 in
 the PEP.
>>> Could it be that you need to strip the binary, or otherwise remove
>>> unneeded debug information?
>>
>> Python is always built with debug information (-g), at least it was in
>> 2.6.1 which unladen is based off of, and we've made sure to build LLVM
>> the same way.  We had to muck with the LLVM build system to get it to
>> include debugging information.  On my system, stripping the python
>> binary takes it from 82 MB to 9.7 MB.  So yes, it contains extra debug
>> info, which explains the footprint measurements.  The question is
>> whether we want LLVM built with debug info or not.
>
> Ok, so if 70MB are debug information, I think a lot of the concerns are
> removed:
> - debug information doesn't consume any main memory, as it doesn't get
>  mapped when the process is started.
> - debug information also doesn't take up space in the system
>  distributions, as they distribute stripped binaries.
>
> As 10MB is still 10 times as large as a current Python binary,people
> will probably search for ways to reduce that further, or at least split
> it up into pieces.

70MB of the increase was indeed debug information. Since the Linux
distros that I checked ship stripped Python binaries, I've stripped
the Unladen Swallow binaries as well, and while the size increase is
still significant, it's not as large as it once was.

Stripped CPython 2.6.4: 1.3 MB
Stripped CPython 3.1.1: 1.4 MB
Stripped Unladen r1041: 12 MB

A 9x increase is better than a 20x increase, but it's not great,
either. There is still room to trim the set of LLVM libraries used by
Unladen Swallow, and we're continuing to investigate reducing on-disk
binary size (http://code.google.com/p/unladen-swallow/issues/detail?id=118
tracks this).

I've updated the PEP to reflect this configuration, since it's what
most users will pick up via their system package managers. The exact
change to the PEP wording is
http://codereview.appspot.com/186247/diff2/6001:6003/5002.

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Terry Reedy]

On 1/26/2010 6:17 PM, Collin Winter wrote:


70MB of the increase was indeed debug information. Since the Linux
distros that I checked ship stripped Python binaries, I've stripped
the Unladen Swallow binaries as well, and while the size increase is
still significant, it's not as large as it once was.

Stripped CPython 2.6.4: 1.3 MB
Stripped CPython 3.1.1: 1.4 MB
Stripped Unladen r1041: 12 MB

A 9x increase is better than a 20x increase, but it's not great,


For downloading and installing for my own use, this would not bother me 
too much, especially since I expect you will be able to eliminate more 
that you do not need. People who package, say 500K or less of python 
code and resource files, using py2exe or whatever, might feel 
differently, especially if the time benefit is trivial for the app. If 
they can get, if they want, a no-U.S. Windows binary, which you have 
apparently now made possible, they should be happy. But I will not 
volunteer Martin to make two binaries unless he is able to complete the 
automation of the process, or unless someone volunteers to help.


Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[William Dode]

Hi (as a simple user),

I'd like to know why you didn't followed the same way as V8 Javascript, 
or the opposite, why for V8 they didn't choose llvm ?

I imagine that startup time and memory was also critical for V8.

thanks


-- 
William Dodé - http://flibuste.net
Informaticien Indépendant

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[David Malcolm]

On Thu, 2010-01-21 at 14:46 -0800, Jeffrey Yasskin wrote:
> On Thu, Jan 21, 2010 at 10:09 AM, Hanno Schlichting  wrote:
> > I'm a relative outsider to core development (I'm just a Plone release
> > manager), but'll allow myself a couple of questions. Feel free to
> > ignore them, if you think they are not relevant at this point :-) I'd
> > note that I'm generally enthusiastic and supportive of the proposal :)
> > As a data point, I can add that all tests of Zope 2.12 / Plone 4.0 and
> > their dependency set run fine under Unladen Swallow.
> 
> Hi, thanks for checking that!
> 
> > On Wed, Jan 20, 2010 at 11:27 PM, Collin Winter  
> > wrote:
> >> We have chosen to reuse a set of existing compiler libraries called LLVM
> >> [#llvm]_ for code generation and code optimization.
> >
> > Would it be prudent to ask for more information about the llvm
> > project? Especially in terms of its non-code related aspects. I can
> > try to hunt down this information myself, but as a complete outsider
> > to the llvm project this takes much longer, compared to someone who
> > has interacted with the project as closely as you have.
> >
> > Questions like:
> >

[snip]

> >> Managing LLVM Releases, C++ API Changes
> >> ---
> >>
> >> LLVM is released regularly every six months. This means that LLVM may be
> >> released two or three times during the course of development of a CPython 
> >> 3.x
> >> release. Each LLVM release brings newer and more powerful optimizations,
> >> improved platform support and more sophisticated code generation.
> >
> > How does the support and maintenance policy of llvm releases look
> > like? If a Python version is pegged to a specific llvm release, it
> > needs to be able to rely on critical bug fixes and security fixes to
> > be made for that release for a rather prolonged time. How does this
> > match the llvm policies given their frequent time based releases?
> 
> LLVM doesn't currently do "dot" releases. So, once 2.7 is released,
> it's very unlikely there would be a 2.6.1. They do make release
> branches, and they've said they're open to dot releases if someone
> else does them, so if we need a patch release for some issue we could
> make it ourselves. I recognize that's not ideal, but I also expect
> that we'll be able to work around LLVM bugs with changes in Python,
> rather than needing to change LLVM.

[snip]

(I don't think the following has specifically been asked yet, though
this thread has become large)

As a downstream distributor of Python, a major pain point for me is when
Python embeds a copy of a library's source code, rather than linking
against a system library (zlib, libffi and expat spring to mind): if
bugs (e.g. security issues) arise in a library, I have to go chasing
down all of the embedded copies of the library, rather than having
dynamic linking deal with it for me.

So I have some concerns about having a copy of LLVM embedded in Python's
source tree, which I believe other distributors of Python would echo;
our rough preference ordering is:

   dynamic linking > static linking > source code copy

I would like CPython to be faster, and if it means dynamically linking
against the system LLVM, that's probably OK (though I have some C++
concerns already discussed elsewhere in this thread).  If it means
statically linking, or worse, having a separate copy of the LLVM source
as an implementation detail of CPython, that would be painful.

I see that the u-s developers have been run into issues in LLVM itself,
and fixed them (bravo!), and seem to have done a good job of sending
those fixes back to LLVM for inclusion. [1]

Some questions for the U-S devs:
  - will it be possible to dynamically link against the system LLVM?
(the PEP currently seems to speak of statically linking against it)
  - does the PEP anticipate that the Python source tree will start
embedding a copy of the LLVM source tree?
  - if so, what can be done to mitigate the risk of drift from upstream?
(this is the motivation for some of the following questions)
  - to what extent do you anticipate further changes needed in LLVM for
U-S? (given the work you've already put in, I expect the answer is
"probably a lot, but we can't know what those will be yet")
  - do you anticipate all of these changes being accepted by the
upstream LLVM maintainers?
  - to what extent would these changes be likely to break API and/or ABI
compat with other users of LLVM (i.e. would a downstream distributor of
CPython be able to simply apply the necessary patches to the system LLVM
in order to track? if they did so, would it require a recompilation of
all of the other users of the system LLVM?)
  - if Python needed to make a dot-release of LLVM, would LLVM allow
Python to increment the SONAME version identifying the ABI within the
DSO (".so") files, and guarantee not to reuse that SONAME version? (so
that automated ABI dependency tracking in e.g. RPM can identify the ABI
incompatibilities without be

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hi William,

On Wed, Jan 27, 2010 at 7:26 AM, William Dode  wrote:
> Hi (as a simple user),
>
> I'd like to know why you didn't followed the same way as V8 Javascript,
> or the opposite, why for V8 they didn't choose llvm ?
>
> I imagine that startup time and memory was also critical for V8.

Startup time and memory usage are arguably *more* critical for a
Javascript implementation, since if you only spend a few milliseconds
executing Javascript code, but your engine takes 10-20ms to startup,
then you've lost. Also, a minimized memory profile is important if you
plan to embed your JS engine on a mobile platform, for example, or you
need to run in a heavily-multiprocessed browser on low-memory consumer
desktops and netbooks.

Among other reasons we chose LLVM, we didn't want to write code
generators for each platform we were targeting. LLVM has done this for
us. V8, on the other hand, has to implement a new code generator for
each new platform they want to target. This is non-trivial work: it
takes a long time, has a lot of finicky details, and it greatly
increases the maintenance burden on the team. We felt that requiring
python-dev to understand code generation on multiple platforms was a
distraction from what python-dev is trying to do -- develop Python. V8
still doesn't have x86-64 code generation working on Windows
(http://code.google.com/p/v8/issues/detail?id=330), so I wouldn't
underestimate the time required for that kind of project.

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[skip]

David> As a downstream distributor of Python, a major pain point for me
David> is when Python embeds a copy of a library's source code, rather
David> than linking against a system library (zlib, libffi and expat
David> spring to mind): if bugs (e.g. security issues) arise in a
David> library, I have to go chasing down all of the embedded copies of
David> the library, rather than having dynamic linking deal with it for
David> me.

The Unladen Swallow developers can correct me if I'm wrong, but I believe
the Subversion checkout holds a copy of LLVM strictly to speed development.
If the U-S folks find and fix a bug in LLVM they can shoot the fix upstream,
apply it locally, then keep moving forward without waiting for a new release
of LLVM.  Support exists now for building with an external LLVM, and I would
expect that as Unladen Swallow moves out of active development into a more
stable phase of its existence that it will probably stop embedding LLVM.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[William Dode]

On 27-01-2010, Collin Winter wrote:
> Hi William,
>
> On Wed, Jan 27, 2010 at 7:26 AM, William Dode  wrote:
>> Hi (as a simple user),
>>
>> I'd like to know why you didn't followed the same way as V8 Javascript,
>> or the opposite, why for V8 they didn't choose llvm ?
>>
>> I imagine that startup time and memory was also critical for V8.
>
> Startup time and memory usage are arguably *more* critical for a
> Javascript implementation, since if you only spend a few milliseconds
> executing Javascript code, but your engine takes 10-20ms to startup,
> then you've lost. Also, a minimized memory profile is important if you
> plan to embed your JS engine on a mobile platform, for example, or you
> need to run in a heavily-multiprocessed browser on low-memory consumer
> desktops and netbooks.
>
> Among other reasons we chose LLVM, we didn't want to write code
> generators for each platform we were targeting. LLVM has done this for
> us. V8, on the other hand, has to implement a new code generator for
> each new platform they want to target. This is non-trivial work: it
> takes a long time, has a lot of finicky details, and it greatly
> increases the maintenance burden on the team. We felt that requiring
> python-dev to understand code generation on multiple platforms was a
> distraction from what python-dev is trying to do -- develop Python. V8
> still doesn't have x86-64 code generation working on Windows
> (http://code.google.com/p/v8/issues/detail?id=330), so I wouldn't
> underestimate the time required for that kind of project.

Thanks for this answer.

The startup time and memory comsumption are a limitation of llvm that 
their developers plan to resolve or is it only specific to the current 
python integration ? I mean the work to correct this is more on U-S or 
on llvm ?

thanks (and sorry for my english !)

-- 
William Dodé - http://flibuste.net
Informaticien Indépendant

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hi David,

On Wed, Jan 27, 2010 at 8:37 AM, David Malcolm  wrote:
[snip]
> As a downstream distributor of Python, a major pain point for me is when
> Python embeds a copy of a library's source code, rather than linking
> against a system library (zlib, libffi and expat spring to mind): if
> bugs (e.g. security issues) arise in a library, I have to go chasing
> down all of the embedded copies of the library, rather than having
> dynamic linking deal with it for me.
>
> So I have some concerns about having a copy of LLVM embedded in Python's
> source tree, which I believe other distributors of Python would echo;
> our rough preference ordering is:
>
>   dynamic linking > static linking > source code copy
>
> I would like CPython to be faster, and if it means dynamically linking
> against the system LLVM, that's probably OK (though I have some C++
> concerns already discussed elsewhere in this thread).  If it means
> statically linking, or worse, having a separate copy of the LLVM source
> as an implementation detail of CPython, that would be painful.

We absolutely do not want CPython to include a copy of LLVM in its
source tree. Unladen Swallow has done this to make it easier to pick
up changes to LLVM's codebase as we make them, but this is not a
viable model for CPython's long-term development. As mentioned in
http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes,
one of our full-time engineers is tasked with fixing all critical
issues in LLVM before LLVM's 2.7 release so that CPython can simply
use that release.

We are currently statically linking against LLVM because that's what
LLVM best supports, but that's not set in stone. We can make LLVM
better support shared linking; it's one of the open issues as to
whether this is important to python-dev
(http://www.python.org/dev/peps/pep-3146/#open-issues), and it sounds
like the answer is "Yes". Our priorities will adjust accordingly.

To answer the individual bullet points:

> I see that the u-s developers have been run into issues in LLVM itself,
> and fixed them (bravo!), and seem to have done a good job of sending
> those fixes back to LLVM for inclusion. [1]
>
> Some questions for the U-S devs:
>  - will it be possible to dynamically link against the system LLVM?
> (the PEP currently seems to speak of statically linking against it)

We currently link statically, but we will fix LLVM to better support
shared linking.

>  - does the PEP anticipate that the Python source tree will start
> embedding a copy of the LLVM source tree?

No, that would be a terrible idea, and we do not endorse it.

>  - to what extent do you anticipate further changes needed in LLVM for
> U-S? (given the work you've already put in, I expect the answer is
> "probably a lot, but we can't know what those will be yet")

We have fixed all the critical issues that our testing has exposed and
have now moved on to "nice to have" items that will aid future
optimizations. LLVM 2.7 will probably be released in May, so we still
have time to fix LLVM as needed.

>  - do you anticipate all of these changes being accepted by the
> upstream LLVM maintainers?

Three of the Unladen Swallow committers are also LLVM committers. Our
patches have historically been accepted enthusiastically by the LLVM
maintainers, and we believe this warm relationship will continue.

>  - to what extent would these changes be likely to break API and/or ABI
> compat with other users of LLVM (i.e. would a downstream distributor of
> CPython be able to simply apply the necessary patches to the system LLVM
> in order to track? if they did so, would it require a recompilation of
> all of the other users of the system LLVM?)

As mentioned in
http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes,
every LLVM release introduces incompatibilities to the C++ API. Our
experience has been that these API changes are easily remedied. We
recommend that CPython depend on a single version of LLVM and not try
to track LLVM trunk.

>  - if Python needed to make a dot-release of LLVM, would LLVM allow
> Python to increment the SONAME version identifying the ABI within the
> DSO (".so") files, and guarantee not to reuse that SONAME version? (so
> that automated ABI dependency tracking in e.g. RPM can identify the ABI
> incompatibilities without being stomped on by a future upstream LLVM
> release)

Nick Lewycky is better-positioned to answer that.

>  - is your aim to minimize the difference between upstream LLVM and the
> U-S copy of LLVM?  ("yes", probably)
>  - will you publish/track the differences between upstream LLVM and the
> U-S copy of LLVM somewhere? I see that you currently do this here: [2],
> though it strikes me that the canonical representation of the LLVM
> you're using is in the embedded copy in Utils/llvm, rather than e.g. a
> recipe like "grab upstream release foo and apply this short list of
> patches" (a distributed SCM might help here, or a tool like jhbuild [3])

We g

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hi William,

On Wed, Jan 27, 2010 at 11:02 AM, William Dode  wrote:
> The startup time and memory comsumption are a limitation of llvm that
> their developers plan to resolve or is it only specific to the current
> python integration ? I mean the work to correct this is more on U-S or
> on llvm ?

Part of it is LLVM, part of it is Unladen Swallow. LLVM is very
flexible, and there's a price for that. We have also found and fixed
several cases of quadratic memory usage in LLVM optimization passes,
and there may be more of those lurking around. On the Unladen Swallow
side, there are doubtless things we can do to improve our usage of
LLVM; http://code.google.com/p/unladen-swallow/issues/detail?id=68 has
most of our work on this, and there are still more ideas to implement.

Part of the issue is that Unladen Swallow is using LLVM's JIT
infrastructure in ways that it really hasn't been used before, and so
there's a fair amount of low-hanging fruit left in LLVM that no-one
has needed to pick yet.

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Wed, Jan 27, 2010 at 11:16 AM, Collin Winter  wrote:
> We absolutely do not want CPython to include a copy of LLVM in its
> source tree. Unladen Swallow has done this to make it easier to pick
> up changes to LLVM's codebase as we make them, but this is not a
> viable model for CPython's long-term development. As mentioned in
> http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes,
> one of our full-time engineers is tasked with fixing all critical
> issues in LLVM before LLVM's 2.7 release so that CPython can simply
> use that release.

I'm now tracking my to-do list for LLVM 2.7 in
http://code.google.com/p/unladen-swallow/issues/detail?id=131.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[David Malcolm]

On Wed, 2010-01-27 at 11:34 -0800, Jeffrey Yasskin wrote:
> On Wed, Jan 27, 2010 at 11:16 AM, Collin Winter  
> wrote:
> > We absolutely do not want CPython to include a copy of LLVM in its
> > source tree. Unladen Swallow has done this to make it easier to pick
> > up changes to LLVM's codebase as we make them, but this is not a
> > viable model for CPython's long-term development. As mentioned in
> > http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes,
> > one of our full-time engineers is tasked with fixing all critical
> > issues in LLVM before LLVM's 2.7 release so that CPython can simply
> > use that release.
> 
> I'm now tracking my to-do list for LLVM 2.7 in
> http://code.google.com/p/unladen-swallow/issues/detail?id=131.

Many thanks for addressing these concerns!

Dave

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Lewycky]

On 27 January 2010 08:37, David Malcolm  wrote:

> On Thu, 2010-01-21 at 14:46 -0800, Jeffrey Yasskin wrote:
> > On Thu, Jan 21, 2010 at 10:09 AM, Hanno Schlichting 
> wrote:
> > > I'm a relative outsider to core development (I'm just a Plone release
> > > manager), but'll allow myself a couple of questions. Feel free to
> > > ignore them, if you think they are not relevant at this point :-) I'd
> > > note that I'm generally enthusiastic and supportive of the proposal :)
> > > As a data point, I can add that all tests of Zope 2.12 / Plone 4.0 and
> > > their dependency set run fine under Unladen Swallow.
> >
> > Hi, thanks for checking that!
> >
> > > On Wed, Jan 20, 2010 at 11:27 PM, Collin Winter <
> collinwin...@google.com> wrote:
> > >> We have chosen to reuse a set of existing compiler libraries called
> LLVM
> > >> [#llvm]_ for code generation and code optimization.
> > >
> > > Would it be prudent to ask for more information about the llvm
> > > project? Especially in terms of its non-code related aspects. I can
> > > try to hunt down this information myself, but as a complete outsider
> > > to the llvm project this takes much longer, compared to someone who
> > > has interacted with the project as closely as you have.
> > >
> > > Questions like:
> > >
>
> [snip]
>
> > >> Managing LLVM Releases, C++ API Changes
> > >> ---
> > >>
> > >> LLVM is released regularly every six months. This means that LLVM may
> be
> > >> released two or three times during the course of development of a
> CPython 3.x
> > >> release. Each LLVM release brings newer and more powerful
> optimizations,
> > >> improved platform support and more sophisticated code generation.
> > >
> > > How does the support and maintenance policy of llvm releases look
> > > like? If a Python version is pegged to a specific llvm release, it
> > > needs to be able to rely on critical bug fixes and security fixes to
> > > be made for that release for a rather prolonged time. How does this
> > > match the llvm policies given their frequent time based releases?
> >
> > LLVM doesn't currently do "dot" releases. So, once 2.7 is released,
> > it's very unlikely there would be a 2.6.1. They do make release
> > branches, and they've said they're open to dot releases if someone
> > else does them, so if we need a patch release for some issue we could
> > make it ourselves. I recognize that's not ideal, but I also expect
> > that we'll be able to work around LLVM bugs with changes in Python,
> > rather than needing to change LLVM.
>
> [snip]
>
> (I don't think the following has specifically been asked yet, though
> this thread has become large)
>
> As a downstream distributor of Python, a major pain point for me is when
> Python embeds a copy of a library's source code, rather than linking
> against a system library (zlib, libffi and expat spring to mind): if
> bugs (e.g. security issues) arise in a library, I have to go chasing
> down all of the embedded copies of the library, rather than having
> dynamic linking deal with it for me.
>
> So I have some concerns about having a copy of LLVM embedded in Python's
> source tree, which I believe other distributors of Python would echo;
> our rough preference ordering is:
>
>   dynamic linking > static linking > source code copy
>
> I would like CPython to be faster, and if it means dynamically linking
> against the system LLVM, that's probably OK (though I have some C++
> concerns already discussed elsewhere in this thread).  If it means
> statically linking, or worse, having a separate copy of the LLVM source
> as an implementation detail of CPython, that would be painful.
>
> I see that the u-s developers have been run into issues in LLVM itself,
> and fixed them (bravo!), and seem to have done a good job of sending
> those fixes back to LLVM for inclusion. [1]
>

Hi David, I'm unladen-swallow's resident "llvm consultant", so I'll answer
the questions I can from an llvm upstream point of view.


> Some questions for the U-S devs:
>  - will it be possible to dynamically link against the system LLVM?
> (the PEP currently seems to speak of statically linking against it)
>

Sadly, LLVM still only offers static libraries (there are two exceptions,
but they don't fit unladen-swallow). This is a something we want but
nobody's stepped up to make it happen yet.


>  - does the PEP anticipate that the Python source tree will start
> embedding a copy of the LLVM source tree?
>  - if so, what can be done to mitigate the risk of drift from upstream?
> (this is the motivation for some of the following questions)
>

Jeff and I are working to make sure that everything unladen-swallow needs is
in LLVM 2.7 when it releases.

 - to what extent do you anticipate further changes needed in LLVM for
> U-S? (given the work you've already put in, I expect the answer is
> "probably a lot, but we can't know what those will be yet")
>

Unladen-swallow works with upstream LLVM now. The changes we're mak

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Tim Wintle]

On Wed, 2010-01-27 at 10:25 -0800, Collin Winter wrote:
> On Wed, Jan 27, 2010 at 7:26 AM, William Dode  wrote:
> > I imagine that startup time and memory was also critical for V8.
> 
> Startup time and memory usage are arguably *more* critical for a
> Javascript implementation, since if you only spend a few milliseconds
> executing Javascript code, but your engine takes 10-20ms to startup,
> then you've lost. Also, a minimized memory profile is important if you
> plan to embed your JS engine on a mobile platform, for example, or you
> need to run in a heavily-multiprocessed browser on low-memory consumer
> desktops and netbooks.

(as a casual reader)

I'm not sure if this has been specifically mentioned before, but there
are certainly multiple uses for python - and from the arguments I've
seen on-list, U-S appears to mainly appeal for one of them.

The three main uses I see everyday are:

 * simple configuration-style applications (I'm thinking of gnome
configuration options etc.)

 * shell-script replacements

 * Long running processes (web apps, large desktop applications etc)

(in addition, I know several people working on python applications for
mobile phones)

I think the performance/memory tradeoffs being discussed are fine for
the long-running / server apps (20mb on a 8Gb machine is negligable) -
but I think the majority of applications probably fall into the other
options - extra startup time and memory usage each time you have a
server monitoring script / cronjob run becomes noticeable if you have
enough of them.

> Among other reasons we chose LLVM, we didn't want to write code
> generators for each platform we were targeting. LLVM has done this for
> us. V8, on the other hand, has to implement a new code generator for
> each new platform they want to target. This is non-trivial work: it
> takes a long time, has a lot of finicky details, and it greatly
> increases the maintenance burden on the team.

I'm not involved in either, but as someone who started watching the
commit list on V8, I'd thoroughly agree that it would be completely
impractical for python-dev to work as the V8 team seem to. 

What appear minor patches very frequently have to be modified by
reviewers over technicalities, and I can't imagine that level of
intricate work happening efficiently without a large (funded) core team
focusing on code generation.

Personally, picking up the (current) python code, I was able to
comfortably start modifying it within a few hours - with the V8 codebase
I still haven't got a clue what's going on where, to be honest.

Tim Wintle

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[skip]

Tim> I think the performance/memory tradeoffs being discussed are fine
Tim> for the long-running / server apps (20mb on a 8Gb machine is
Tim> negligable) 

At work our apps' memory footprints are dominated by the Boost-wrapped C++
libraries we use.  100MB VM usage at run-time is pretty much the starting
point.  It just goes up from there.  We'd probably not notice an extra 20MB
if it was shared.

Skip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Daniel Fetchinson]

A question from someone writing C extension modules for python but not
involved in python-dev:

It has been said that compiling python with --without-llvm would not
include unladen swallow and would bypass llvm together with all C++.
Basically, as I understand it, --without-llvm gives the 'usual'
cpython we have today. Is this correct?

If this is correct, I still have one worry: since I wouldn't want to
touch the python install most linux distributions ship or most
windows/mac users install (or what MS/Apple ships) I will simply have
no choice than working with the python variant that is installed.

Is it anticipated that most linux distros and MS/Apple will ship the
python variant that comes with llvm/US? I suppose the goal of merging
llvm/US into python 3.x is this.

If this is the case then I, as a C extension author, will have no
choice than working with a python installation that includes llvm/US.
Which, as far as I undestand it, means dealing with C++ issues. Is
this correct? Or the same pure C extension module compiled with C-only
compilers would work with llvm-US-python and cpython?

Cheers,
Danil

-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Coghlan]

Daniel Fetchinson wrote:
> If this is the case then I, as a C extension author, will have no
> choice than working with a python installation that includes llvm/US.
> Which, as far as I undestand it, means dealing with C++ issues. Is
> this correct? Or the same pure C extension module compiled with C-only
> compilers would work with llvm-US-python and cpython?

As a C extension author you will be fine (the source and linker
interface will all still be C-only).

C++ extension authors may need to care about making the version of the
C++ runtime that they link against match that used internally by
CPython/LLVM (although I'm a little unclear on that point myself).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Stefan Behnel]

Daniel Fetchinson, 28.01.2010 13:19:
> A question from someone writing C extension modules for python

I doubt that this will have any impact on C extension developers.


> If this is correct, I still have one worry: since I wouldn't want to
> touch the python install most linux distributions ship or most
> windows/mac users install (or what MS/Apple ships) I will simply have
> no choice than working with the python variant that is installed.
> 
> Is it anticipated that most linux distros and MS/Apple will ship the
> python variant that comes with llvm/US? I suppose the goal of merging
> llvm/US into python 3.x is this.

Depends on the distro. My guess is that they will likely provide both as
separate packages (unless one turns out to be clearly 'better'), and
potentially even support their parallel installation. That's not
unprecedented, just think of different JVM implementations (or even just
different Python versions).


> If this is the case then I, as a C extension author, will have no
> choice than working with a python installation that includes llvm/US.
> Which, as far as I undestand it, means dealing with C++ issues.

I don't think so. Replacing the eval loop has no impact on the C-API
commonly used by binary extensions. It may have an impact on programs that
embed the Python interpreter, but not the other way round.

Remember that you usually don't have to compile the Python interpreter
yourself. Once it's a binary, it doesn't really matter anymore in what
language(s) it was originally written.


> Or the same pure C extension module compiled with C-only
> compilers would work with llvm-US-python and cpython?

That's to be expected.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Daniel Fetchinson]

>> If this is the case then I, as a C extension author, will have no
>> choice than working with a python installation that includes llvm/US.
>> Which, as far as I undestand it, means dealing with C++ issues. Is
>> this correct? Or the same pure C extension module compiled with C-only
>> compilers would work with llvm-US-python and cpython?
>
> As a C extension author you will be fine (the source and linker
> interface will all still be C-only).

Thanks, that is good to hear!

Cheers,
Daniel

> C++ extension authors may need to care about making the version of the
> C++ runtime that they link against match that used internally by
> CPython/LLVM (although I'm a little unclear on that point myself).


-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Daniel Fetchinson]

>> A question from someone writing C extension modules for python
>
> I doubt that this will have any impact on C extension developers.
>
>
>> If this is correct, I still have one worry: since I wouldn't want to
>> touch the python install most linux distributions ship or most
>> windows/mac users install (or what MS/Apple ships) I will simply have
>> no choice than working with the python variant that is installed.
>>
>> Is it anticipated that most linux distros and MS/Apple will ship the
>> python variant that comes with llvm/US? I suppose the goal of merging
>> llvm/US into python 3.x is this.
>
> Depends on the distro. My guess is that they will likely provide both as
> separate packages

Yes, it's clear that various packages will be available but what I was
asking about is the default python version that gets installed if a
user installs a vanilla version of a particular linux distro. It's a
big difference for developers of C extension modules to say "just
install this module and go" or "first download the python version
so-and-so and then install my module and go".

But as I understand from you and Nick, this will not be a problem for
C extension module authors.

> (unless one turns out to be clearly 'better'), and
> potentially even support their parallel installation. That's not
> unprecedented, just think of different JVM implementations (or even just
> different Python versions).
>
>
>> If this is the case then I, as a C extension author, will have no
>> choice than working with a python installation that includes llvm/US.
>> Which, as far as I undestand it, means dealing with C++ issues.
>
> I don't think so. Replacing the eval loop has no impact on the C-API
> commonly used by binary extensions. It may have an impact on programs that
> embed the Python interpreter, but not the other way round.
>
> Remember that you usually don't have to compile the Python interpreter
> yourself. Once it's a binary, it doesn't really matter anymore in what
> language(s) it was originally written.
>
>> Or the same pure C extension module compiled with C-only
>> compilers would work with llvm-US-python and cpython?
>
> That's to be expected.

Okay, that's great, basically Nick confirmed the same thing.

Cheers,
Daniel

-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Paul Moore]

On 28 January 2010 12:58, Daniel Fetchinson  wrote:
>>> If this is the case then I, as a C extension author, will have no
>>> choice than working with a python installation that includes llvm/US.
>>> Which, as far as I undestand it, means dealing with C++ issues. Is
>>> this correct? Or the same pure C extension module compiled with C-only
>>> compilers would work with llvm-US-python and cpython?
>>
>> As a C extension author you will be fine (the source and linker
>> interface will all still be C-only).
>
> Thanks, that is good to hear!

So, just to extend the question a little (or reiterate, it may be that
this is already covered and I didn't fully understand):

On Windows, would a C extension author be able to distribute a single
binary (bdist_wininst/bdist_msi) which would be compatible with
with-LLVM and without-LLVM builds of Python?

Actually, if we assume that only a single Windows binary, presumably
with-LLVM, will be distributed on python.org, I'm probably being
over-cautious here, as distributing binaries compatible with the
python.org release should be sufficient. Nevertheless, I'd be
interested in the answer.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Reid Kleckner]

On Thu, Jan 28, 2010 at 1:14 PM, Paul Moore  wrote:
> So, just to extend the question a little (or reiterate, it may be that
> this is already covered and I didn't fully understand):
>
> On Windows, would a C extension author be able to distribute a single
> binary (bdist_wininst/bdist_msi) which would be compatible with
> with-LLVM and without-LLVM builds of Python?
>
> Actually, if we assume that only a single Windows binary, presumably
> with-LLVM, will be distributed on python.org, I'm probably being
> over-cautious here, as distributing binaries compatible with the
> python.org release should be sufficient. Nevertheless, I'd be
> interested in the answer.

We have broken ABI compatibility with Python 2.6, but unladen
--without-llvm should be ABI compatible with itself.  In the future we
would probably want to set up a buildbot to make sure we don't mess
this up.  One thing we have done is to add #ifdef'd attributes to
things like the code object, but so long as you don't touch those
attributes, you should be fine.

Reid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

Hi Collin,

Thanks for the useful links.

I think that superinstructions require a bit more work, because they aren't
just opcode arguments rearrangement. For example, in wpython 1.1 (that I'll
release next month) I've introduced a CALL_SUB opcode to handle all kind of
function types, so the 2 words pack together:
- the opcode (CALL_SUB);
- the function type and flags (normal, VAR, KW, procedure);
- the number of arguments;
- the number of keywords arguments.

Superinstructions aren't intended to be a simple drop-in replacement of
existing bytecodes. They can carry new ideas and implement them in a
versatile and efficient way.

Anyway, I don't think to continue with wpython: 1.1 will be the last version
I'll release (albeit I initially have planned 1.2 and 1.3 for this year, and
2.0 for 2011) for several reasons.

2.7 is the last planned 2.x release, and once it got alpha state, there's no
chance to introduce wordcodes model in it.

3.2 or later will be good candidates, but I don't want to make a new project
and fork again. Forking is a waste of time and resources (I spent over 1
year of my spare time just to prove an idea).

I think that wpython as a proof-of-concept have done its work, showing its
potentials.

If python dev community is interested, I can work on a 3.x branch, porting
all optimizations I made (and many others that I've planned to implement)
one step at the time, in order to carefully check and validate any change
with expert people monitoring it.

Cesare

2010/1/26 Collin Winter 

> Hi Cesare,
>
> On Tue, Jan 26, 2010 at 12:29 AM, Cesare Di Mauro
>  wrote:
> > Hi Collin,
> >
> > One more question: is it easy to support more opcodes, or a different
> opcode
> > structure, in Unladen Swallow project?
>
> I assume you're asking about integrating WPython. Yes, adding new
> opcodes to Unladen Swallow is still pretty easy. The PEP includes a
> section on this,
>
> http://www.python.org/dev/peps/pep-3146/#experimenting-with-changes-to-python-or-cpython-bytecode
> ,
> though it doesn't cover something more complex like converting from
> bytecode to wordcode, as a purely hypothetical example ;) Let me know
> if that section is unclear or needs more data.
>
> Converting from bytecode to wordcode should be relatively
> straightforward, assuming that the arrangement of opcode arguments is
> the main change. I believe the only real place you would need to
> update is the JIT compiler's bytecode iterator (see
>
> http://code.google.com/p/unladen-swallow/source/browse/trunk/Util/PyBytecodeIterator.cc
> ).
> Depending on the nature of the changes, the runtime feedback system
> might need to be updated, too, but it wouldn't be too difficult, and
> the changes should be localized.
>
> Collin Winter
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[skip]

Cesare> I think that wpython as a proof-of-concept have done its work,
Cesare> showing its potentials.

If you haven't alreayd is there any chance you can run the Unladen Swallow
performance test suite and post the results?  The code is separate from U-S
and should work with wpython:

http://unladen-swallow.googlecode.com/svn/tests

-- 
Skip Montanaro - s...@pobox.com - http://www.smontanaro.net/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

2010/1/29  

>
>Cesare> I think that wpython as a proof-of-concept have done its work,
>Cesare> showing its potentials.
>
> If you haven't alreayd is there any chance you can run the Unladen Swallow
> performance test suite and post the results?  The code is separate from U-S
> and should work with wpython:
>
>http://unladen-swallow.googlecode.com/svn/tests
>
> --
> Skip Montanaro - s...@pobox.com - http://www.smontanaro.net/
>

I work on a Windows machine, so I don't know if I can run the U-S test suite
on it (the first time I tried, it failed since U-S used a module available
on Unix machines only).

If it works now, I can provide results with wpython 1.0 final and the
current 1.1 I'm working on (which has additional optimizations; I've also
moved all peephole optimizer code on compile.c).

Anyway, Mart Sõmermaa provided some
resultsbased
on wpython 1.0 alpha (you can find the wpython 1.0 final
here ).

Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[skip]

Cesare> ... (you can find the wpython 1.0 final here
Cesare> ).

I tried downloading it.  Something about wpython10.7z and wpython10_fix.7z.
What's a 7z file?  What tool on my Mac will unpack that?  Can I build and
run wpython on my Mac or is it Windows only?

Thx,

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Michael Foord]

On 29/01/2010 13:24, s...@pobox.com wrote:

 Cesare>  ... (you can find the wpython 1.0 final here
 Cesare>  ).

I tried downloading it.  Something about wpython10.7z and wpython10_fix.7z.
What's a 7z file?  What tool on my Mac will unpack that?  Can I build and
run wpython on my Mac or is it Windows only?
   


7z (7zip) is a semi-popular compression format, and yes there are 
cross-platform tools to decompress them. A quick google should reveal them.


Michael


Thx,

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

2010/1/29  

>
>Cesare> ... (you can find the wpython 1.0 final here
>Cesare> ).
>
> I tried downloading it.  Something about wpython10.7z and wpython10_fix.7z.
> What's a 7z file?  What tool on my Mac will unpack that?  Can I build and
> run wpython on my Mac or is it Windows only?
>
> Thx,
>
> Skip
>

You can find 7-Zip tools here .

If you use Mercurial, you can grab a local copy this way:

hg clone https://wpython10.wpython2.googlecode.com/hg/ wpython2-wpython10

Wpython is intended to run on any platform where CPython 2.6.4 runs.

Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Antoine Pitrou]

Cesare Di Mauro  gmail.com> writes:
> 
> If python dev community is interested, I can work on a 3.x branch, porting
> all optimizations I made (and many others that I've planned to implement) one 
> step at the time, in order to carefully check and validate any change with
> expert people monitoring it.

We are certainly more interested in a 3.x branch than in a 2.x one ;-)
You can start by cloning http://code.python.org/hg/branches/py3k/

Or you could submit patches piecewise on http://bugs.python.org
I think the first step would be to switch to 16-bit bytecodes. It would be
uncontroversial (the increase in code size probably has no negative effect) and
would provide the foundation for all of your optimizations.

Are you going to PyCon?


Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[skip]

Cesare> You can find 7-Zip tools here
Cesare> .

Thanks.  Found a tool named 7za in MacPorts which I was able to install.

One strong suggestion for future releases: Please put a top-level directory
in your archives.  It is annoying to expect that only to have an archive
expand into the current directory without creating a directory of its own.
I've been burned often enough that I always check before expanding source
archives from new (to me) sources, so no harm, no foul in this case.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

2010/1/29 Antoine Pitrou 

> Cesare Di Mauro  gmail.com> writes:
> >
> > If python dev community is interested, I can work on a 3.x branch,
> porting
> > all optimizations I made (and many others that I've planned to implement)
> one
> > step at the time, in order to carefully check and validate any change
> with
> > expert people monitoring it.
>
> We are certainly more interested in a 3.x branch than in a 2.x one ;-)
> You can start by cloning http://code.python.org/hg/branches/py3k/
>
> Or you could submit patches piecewise on http://bugs.python.org


I prefer to make a branch with Mercurial, which I found a comfortable tool.
:)


> I think the first step would be to switch to 16-bit bytecodes. It would be
> uncontroversial (the increase in code size probably has no negative effect)
> and
> would provide the foundation for all of your optimizations.
>

I agree. At the beginning I need to disable the peepholer, so performances
are better to be compared when all peephole optimizations will be ported to
the wordcode model.

I'll make the branch after I release wpython 1.1, which I'll do ASAP.


> Are you going to PyCon?
>
>  Antoine.


No, I don't. But if there's a python-dev meeting, I can make a (long) jump.
May be it can be easier to talk about the superinstructions model, and I can
show and comment all optimizations that I made.

Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

2010/1/29  

>
> One strong suggestion for future releases: Please put a top-level directory
> in your archives.  It is annoying to expect that only to have an archive
> expand into the current directory without creating a directory of its own.
> I've been burned often enough that I always check before expanding source
> archives from new (to me) sources, so no harm, no foul in this case.
>
> Skip
>

You're right. Excuse me. I'll do it next time.

Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Coghlan]

Antoine Pitrou wrote:
> Or you could submit patches piecewise on http://bugs.python.org
> I think the first step would be to switch to 16-bit bytecodes. It would be
> uncontroversial (the increase in code size probably has no negative effect) 
> and
> would provide the foundation for all of your optimizations.

I wouldn't consider changing from bytecode to wordcode uncontroversial -
the potential to have an effect on cache hit ratios means it needs to be
benchmarked (the U-S performance tests should be helpful there).

It's the same basic problem where any changes to the ceval loop can have
surprising performance effects due to the way they affect the compiled
switch statements ability to fit into the cache and other low level
processor weirdness.

If there was an old style map of the CPython code base, the whole area
would have "'ware, here be dragons" written over the top of it ;)

Cheers,
Nick.

P.S. Note that I'm not saying I'm fundamentally opposed to a change to
wordcode - just that it needs to be benchmarked fairly thoroughly first.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Antoine Pitrou]

Nick Coghlan  gmail.com> writes:
> 
> Antoine Pitrou wrote:
> > Or you could submit patches piecewise on http://bugs.python.org
> > I think the first step would be to switch to 16-bit bytecodes. It 
would be
> > uncontroversial (the increase in code size probably has no negative 
effect) and
> > would provide the foundation for all of your optimizations.
> 
> I wouldn't consider changing from bytecode to wordcode uncontroversial -
> the potential to have an effect on cache hit ratios means it needs to be
> benchmarked (the U-S performance tests should be helpful there).

Well I said "/probably/ has no negative effect" :-)

Actually, "wordcode" could allow accesses in the eval loop to be done on 
aligned words, so as to fetch operands in one step on little-endian CPUs 
(instead of recombining bytes manually).

The change would be "uncontroversial", however, in that it wouldn't 
modify code complexity or increase the maintenance burden.

Of course, performance checks have to be part of the review.

> If there was an old style map of the CPython code base, the whole area
> would have "'ware, here be dragons" written over the top of it ;)

Just FYI: weakrefs and memoryviews are guarded by a mantichore, and only
Benjamin can tame it.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

2010/1/29 Nick Coghlan 

> I wouldn't consider changing from bytecode to wordcode uncontroversial -
> the potential to have an effect on cache hit ratios means it needs to be
> benchmarked (the U-S performance tests should be helpful there).
>

It's quite strange, but from the tests made it seems that wpython perform
better with old architectures (such as my Athlon64 socket 754), which have
less resources like caches.

It'll be interesting to check how it works on more limited ISAs. I'm
especially curious about ARMs.


> It's the same basic problem where any changes to the ceval loop can have
> surprising performance effects due to the way they affect the compiled
> switch statements ability to fit into the cache and other low level
> processor weirdness.
>
> Cheers,
> Nick.
>

Sure, but consider that with wpython wordcodes require less space on
average. Also, less instructions are executed inside the ceval loop, thanks
to some natural instruction grouping.

For example, I recently introduced in wpython 1.1 a new opcode to handle
more efficiently expression generators. It's mapped as a unary operator, so
it exposes interesting properties which I'll show you with an example.

def f(a):
return sum(x for x in a)

With CPython 2.6.4 it generates:

  0 LOAD_GLOBAL 0 (sum)
  3 LOAD_CONST 1 ( at 00512EC8, file "", line
1>)
  6 MAKE_FUNCTION 0
  9 LOAD_FAST 0 (a)
12 GET_ITER
13 CALL_FUNCTION 1
16 CALL_FUNCTION 1
19 RETURN_VALUE

With wpython 1.1:

0 LOAD_GLOBAL 0 (sum)
1 LOAD_CONST 1 ( at 01F13208, file "", line 1>)
2 MAKE_FUNCTION 0
3 FAST_BINOP get_generator a
5 QUICK_CALL_FUNCTION 1
6 RETURN_VALUE

The new opcode is GET_GENERATOR, which is equivalent (but more efficient,
using a faster internal function call) to:

GET_ITER
CALL_FUNCTION 1

The compiler initially generated the following opcodes:

LOAD_FAST 0 (a)
GET_GENERATOR

then the peepholer recognized the pattern UNARY(FAST), and produced the
single opcode:

FAST_BINOP get_generator a

In the end, the ceval loop executes a single instruction instead of three.
The wordcode requires 14 bytes to be stored instead of 20, so it will use 1
data cache line instead of 2 on CPUs with 16 bytes lines data cache.

The same grouping behavior happens with binary operators as well. Opcodes
aggregation is a natural and useful concept with the new wordcode structure.

Cheers,
Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

I made a mistake talking about the example. Exchange binary operator and
unary operator.

Cesare

2010/1/29 Cesare Di Mauro 

> 2010/1/29 Nick Coghlan 
>
> I wouldn't consider changing from bytecode to wordcode uncontroversial -
>> the potential to have an effect on cache hit ratios means it needs to be
>> benchmarked (the U-S performance tests should be helpful there).
>>
>
> It's quite strange, but from the tests made it seems that wpython perform
> better with old architectures (such as my Athlon64 socket 754), which have
> less resources like caches.
>
> It'll be interesting to check how it works on more limited ISAs. I'm
> especially curious about ARMs.
>
>
>> It's the same basic problem where any changes to the ceval loop can have
>> surprising performance effects due to the way they affect the compiled
>> switch statements ability to fit into the cache and other low level
>> processor weirdness.
>>
>> Cheers,
>> Nick.
>>
>
> Sure, but consider that with wpython wordcodes require less space on
> average. Also, less instructions are executed inside the ceval loop, thanks
> to some natural instruction grouping.
>
> For example, I recently introduced in wpython 1.1 a new opcode to handle
> more efficiently expression generators. It's mapped as a unary operator, so
> it exposes interesting properties which I'll show you with an example.
>
> def f(a):
> return sum(x for x in a)
>
> With CPython 2.6.4 it generates:
>
>   0 LOAD_GLOBAL 0 (sum)
>   3 LOAD_CONST 1 ( at 00512EC8, file "", line
> 1>)
>   6 MAKE_FUNCTION 0
>   9 LOAD_FAST 0 (a)
> 12 GET_ITER
> 13 CALL_FUNCTION 1
> 16 CALL_FUNCTION 1
> 19 RETURN_VALUE
>
> With wpython 1.1:
>
> 0 LOAD_GLOBAL 0 (sum)
> 1 LOAD_CONST 1 ( at 01F13208, file "", line
> 1>)
> 2 MAKE_FUNCTION 0
> 3 FAST_BINOP get_generator a
> 5 QUICK_CALL_FUNCTION 1
> 6 RETURN_VALUE
>
> The new opcode is GET_GENERATOR, which is equivalent (but more efficient,
> using a faster internal function call) to:
>
> GET_ITER
> CALL_FUNCTION 1
>
> The compiler initially generated the following opcodes:
>
> LOAD_FAST 0 (a)
> GET_GENERATOR
>
> then the peepholer recognized the pattern UNARY(FAST), and produced the
> single opcode:
>
> FAST_BINOP get_generator a
>
> In the end, the ceval loop executes a single instruction instead of three.
> The wordcode requires 14 bytes to be stored instead of 20, so it will use 1
> data cache line instead of 2 on CPUs with 16 bytes lines data cache.
>
> The same grouping behavior happens with binary operators as well. Opcodes
> aggregation is a natural and useful concept with the new wordcode structure.
>
> Cheers,
> Cesare
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

2010/1/29 Antoine Pitrou 

> Actually, "wordcode" could allow accesses in the eval loop to be done on
> aligned words, so as to fetch operands in one step on little-endian CPUs
> (instead of recombining bytes manually).
>
>  Regards
>
> Antoine.
>

I think that big-endians CPUs can get benefits too, since a single word load
operation is needed, followed by an instruction such as ROL #8 to "adjust"
the result (supposing that the compiler is smart enough to recognize the
pattern).

Using bytecodes, two loads are needed to retrieve the two bytes, and some
SHIFT & OR instructions to combine them getting the correct word. Loads are
generally more expensive / limited.

All that not counting the operations needed to advance the instruction
pointer.

Regards,

Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[M.-A. Lemburg]

Collin Winter wrote:
> I added startup benchmarks for Mercurial and Bazaar yesterday
> (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we
> can use them as more macro-ish benchmarks, rather than merely starting
> the CPython binary over and over again. If you have ideas for better
> Mercurial/Bazaar startup scenarios, I'd love to hear them. The new
> hg_startup and bzr_startup benchmarks should give us some more data
> points for measuring improvements in startup time.
> 
> One idea we had for improving startup time for apps like Mercurial was
> to allow the creation of hermetic Python "binaries", with all
> necessary modules preloaded. This would be something like Smalltalk
> images. We haven't yet really fleshed out this idea, though.

In Python you can do the same with the freeze.py utility. See

http://www.egenix.com/www2002/python/mxCGIPython.html

for an old project where we basically put the Python
interpreter and stdlib into a single executable.

We've recently revisited that project and created something
we call "pyrun". It fits Python 2.5 into a single executable
and a set of shared modules (which for various reasons cannot
be linked statically)... 12MB in total.

If you load lots of modules from the stdlib this does provide
a significant improvement over standard Python.

Back to the PEP's proposal:

Looking at the data you currently have, the negative results
currently don't really look good in the light of the small
performance improvements.

Wouldn't it be possible to have the compiler approach work
in three phases in order to reduce the memory footprint and
startup time hit, ie.

 1. run an instrumented Python interpreter to collect all
the needed compiler information; write this information into
a .pys file (Python stats)

 2. create compiled versions of the code for various often
used code paths and type combinations by reading the
.pys file and generating an .so file as regular
Python extension module

 3. run an uninstrumented Python interpreter and let it
use the .so files instead of the .py ones

In production, you'd then only use step 3 and avoid the
overhead of steps 1 and 2.

Moreover, the .so file approach
would only load the code for code paths and type combinations
actually used in a particular run of the Python code into
memory and allow multiple Python processes to share it.

As side effect, you'd probably also avoid the need to have
C++ code in the production Python runtime - that is unless
LLVM requires some kind of runtime support which is written
in C++.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 29 2010)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlan  wrote:
> Antoine Pitrou wrote:
>> Or you could submit patches piecewise on http://bugs.python.org
>> I think the first step would be to switch to 16-bit bytecodes. It would be
>> uncontroversial (the increase in code size probably has no negative effect) 
>> and
>> would provide the foundation for all of your optimizations.
>
> I wouldn't consider changing from bytecode to wordcode uncontroversial -
> the potential to have an effect on cache hit ratios means it needs to be
> benchmarked (the U-S performance tests should be helpful there).
>
> It's the same basic problem where any changes to the ceval loop can have
> surprising performance effects due to the way they affect the compiled
> switch statements ability to fit into the cache and other low level
> processor weirdness.

Agreed. We originally switched Unladen Swallow to wordcode in our
2009Q1 release, and saw a performance improvement from this across the
board. We switched back to bytecode for the JIT compiler to make
upstream merger easier. The Unladen Swallow benchmark suite should
provided a thorough assessment of the impact of the wordcode ->
bytecode switch. This would be complementary to a JIT compiler, rather
than a replacement for it.

I would note that the switch will introduce incompatibilities with
libraries like Twisted. IIRC, Twisted has a traceback prettifier that
removes its trampoline functions from the traceback, parsing CPython's
bytecode in the process. If running under CPython, it assumes that the
bytecode is as it expects. We broke this in Unladen's wordcode switch.
I think parsing bytecode is a bad idea, but any switch to wordcode
should be advertised widely.

Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Terry Reedy]

On 1/29/2010 4:19 PM, Collin Winter wrote:

On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlan  wrote:



Agreed. We originally switched Unladen Swallow to wordcode in our
2009Q1 release, and saw a performance improvement from this across the
board. We switched back to bytecode for the JIT compiler to make
upstream merger easier. The Unladen Swallow benchmark suite should
provided a thorough assessment of the impact of the wordcode ->
bytecode switch. This would be complementary to a JIT compiler, rather
than a replacement for it.

I would note that the switch will introduce incompatibilities with
libraries like Twisted. IIRC, Twisted has a traceback prettifier that
removes its trampoline functions from the traceback, parsing CPython's
bytecode in the process. If running under CPython, it assumes that the
bytecode is as it expects. We broke this in Unladen's wordcode switch.
I think parsing bytecode is a bad idea, but any switch to wordcode
should be advertised widely.


Several years, there was serious consideration of switching to a 
registerbased vm, which would have been even more of a change. Since I 
learned 1.4, Guido has consistently insisted that the CPython vm is not 
part of the language definition and, as far as I know, he has rejected 
any byte-code hackery in the stdlib. While he is not one to, say, 
randomly permute the codes just to frustrate such hacks, I believe he 
has always considered vm details private and subject to change and any 
usage thereof 'at one's own risk'.


tjr

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hey Terry,

On Fri, Jan 29, 2010 at 2:47 PM, Terry Reedy  wrote:
> On 1/29/2010 4:19 PM, Collin Winter wrote:
>>
>> On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlan  wrote:
>
>> Agreed. We originally switched Unladen Swallow to wordcode in our
>> 2009Q1 release, and saw a performance improvement from this across the
>> board. We switched back to bytecode for the JIT compiler to make
>> upstream merger easier. The Unladen Swallow benchmark suite should
>> provided a thorough assessment of the impact of the wordcode ->
>> bytecode switch. This would be complementary to a JIT compiler, rather
>> than a replacement for it.
>>
>> I would note that the switch will introduce incompatibilities with
>> libraries like Twisted. IIRC, Twisted has a traceback prettifier that
>> removes its trampoline functions from the traceback, parsing CPython's
>> bytecode in the process. If running under CPython, it assumes that the
>> bytecode is as it expects. We broke this in Unladen's wordcode switch.
>> I think parsing bytecode is a bad idea, but any switch to wordcode
>> should be advertised widely.
>
> Several years, there was serious consideration of switching to a
> registerbased vm, which would have been even more of a change. Since I
> learned 1.4, Guido has consistently insisted that the CPython vm is not part
> of the language definition and, as far as I know, he has rejected any
> byte-code hackery in the stdlib. While he is not one to, say, randomly
> permute the codes just to frustrate such hacks, I believe he has always
> considered vm details private and subject to change and any usage thereof
> 'at one's own risk'.

No, I agree entirely: bytecode is an implementation detail that could
be changed at any time. But like reference counting, it's an
implementation detail that people have -- for better or worse -- come
to rely on. My only point was that a switch to wordcode should be
announced prominently in the release notes and not assumed to be
without impact on user code. That people are directly munging CPython
bytecode means that CPython should provide a better, more abstract way
to do the same thing that's more resistant to these kinds of changes.

Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[exarkun]

On 10:47 pm, tjre...@udel.edu wrote:

On 1/29/2010 4:19 PM, Collin Winter wrote:
On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlan 
wrote:



Agreed. We originally switched Unladen Swallow to wordcode in our
2009Q1 release, and saw a performance improvement from this across the
board. We switched back to bytecode for the JIT compiler to make
upstream merger easier. The Unladen Swallow benchmark suite should
provided a thorough assessment of the impact of the wordcode ->
bytecode switch. This would be complementary to a JIT compiler, rather
than a replacement for it.

I would note that the switch will introduce incompatibilities with
libraries like Twisted. IIRC, Twisted has a traceback prettifier that
removes its trampoline functions from the traceback, parsing CPython's
bytecode in the process. If running under CPython, it assumes that the
bytecode is as it expects. We broke this in Unladen's wordcode switch.
I think parsing bytecode is a bad idea, but any switch to wordcode
should be advertised widely.


Several years, there was serious consideration of switching to a 
registerbased vm, which would have been even more of a change. Since I 
learned 1.4, Guido has consistently insisted that the CPython vm is not 
part of the language definition and, as far as I know, he has rejected 
any byte- code hackery in the stdlib. While he is not one to, say, 
randomly permute the codes just to frustrate such hacks, I believe he 
has always considered vm details private and subject to change and any 
usage thereof 'at one's own risk'.


Language to such effect might be a useful addition to this page (amongst 
others, perhaps):


 http://docs.python.org/library/dis.html

which very clearly and helpfully lays out quite a number of APIs which 
can be used to get pretty deep into the bytecode.  If all of this is 
subject to be discarded at the first sign that doing so might be 
beneficial for some reason, don't keep it a secret that people need to 
join python-dev to learn.


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[exarkun]

On 10:55 pm, collinwin...@google.com wrote:


That people are directly munging CPython
bytecode means that CPython should provide a better, more abstract way
to do the same thing that's more resistant to these kinds of changes.


Yes, definitely!  Requesting a supported way to do the kind of 
introspection that you mentioned earlier (I think it's a little simpler 
than you recollect) has been on my todo list for a while.  The hold-up 
is just figuring out exactly what kind of introspection API would make 
sense.


It might be helpful to hear more about how the wordcode implementation 
differs from the bytecode implementation.  It's challenging to abstract 
from a single data point. :)


For what it's worth, I think this is the code in Twisted that Collin was 
originally referring to:


   # it is only really originating from
   # throwExceptionIntoGenerator if the bottom of the traceback
   # is a yield.
   # Pyrex and Cython extensions create traceback frames
   # with no co_code, but they can't yield so we know it's okay to 
just

   # return here.
   if ((not lastFrame.f_code.co_code) or
   lastFrame.f_code.co_code[lastTb.tb_lasti] != 
cls._yieldOpcode):

   return

And it's meant to be used in code like this:

def foo():
   try:
   yield
   except:
   # raise a new exception

def bar():
   g = foo()
   g.next()
   try:
   g.throw(...)
   except:
   # Code above is invoked to determine if the exception raised
   # was the one that was thrown into the generator or a different
   # one.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Martin v. Löwis]

> On Windows, would a C extension author be able to distribute a single
> binary (bdist_wininst/bdist_msi) which would be compatible with
> with-LLVM and without-LLVM builds of Python?

When PEP 384 gets implemented, you not only get that, but you will also
be able to use the same extension module for 3.2, 3.3, 3.4, etc, with
or without U-S.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Terry Reedy]

On 1/29/2010 5:55 PM, Collin Winter wrote:

Hey Terry,

On Fri, Jan 29, 2010 at 2:47 PM, Terry Reedy  wrote:


Several years, there was serious consideration of switching to a
registerbased vm, which would have been even more of a change. Since I
learned 1.4, Guido has consistently insisted that the CPython vm is not part
of the language definition and, as far as I know, he has rejected any
byte-code hackery in the stdlib. While he is not one to, say, randomly
permute the codes just to frustrate such hacks, I believe he has always
considered vm details private and subject to change and any usage thereof
'at one's own risk'.


No, I agree entirely: bytecode is an implementation detail that could
be changed at any time. But like reference counting, it's an
implementation detail that people have -- for better or worse -- come
to rely on. My only point was that a switch to wordcode should be
announced prominently in the release notes and not assumed to be
without impact on user code.


Of course. If it does not already, What's New could routinely have a 
section Changes in the CPython Implementation that would include any 
such change that might impact people. It could also include a warning to 
not depend on certain details even if they are mentioned.


tjr


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Terry Reedy]

On 1/29/2010 6:45 PM, "Martin v. Löwis" wrote:

On Windows, would a C extension author be able to distribute a single
binary (bdist_wininst/bdist_msi) which would be compatible with
with-LLVM and without-LLVM builds of Python?


When PEP 384 gets implemented, you not only get that, but you will also
be able to use the same extension module for 3.2, 3.3, 3.4, etc, with
or without U-S.


Even if CPython changes VC compiler version? In any case, greater 
usability of binaries would be really nice.


tjr


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Scott Dial]

On 1/29/2010 8:43 AM, Cesare Di Mauro wrote:
> If you use Mercurial, you can grab a local copy this way:
> 
> hg clone https://wpython10.wpython2.googlecode.com/hg/ wpython2-wpython10
> 
> Wpython is intended to run on any platform where CPython 2.6.4 runs.
> 

Cesare, just FYI, your Hg repository has lost the execute bits on some
files (namely "./configure" and "./Parser/asdl_c.py"), so it does not
quite build out-of-the-box.

I took the liberty of cloning your repo into my laptop's VirtualBox
instance of Ubuntu. I ran the default performance tests from the U-S
repo, with VirtualBox at highest priority. As a sanity check, I ran it
against the U-S trunk. I think the numbers speak for themselves.

$ ./perf.py -r -b default \
  ../python-2.6.4/python \
  ../wpython2-wpython10/python

Report on Linux narf-ubuntu 2.6.31-16-generic #53-Ubuntu SMP Tue Dec 8
04:01:29 UTC 2009 i686
Total CPU cores: 1

### 2to3 ###
Min: 21.629352 -> 20.893306: 1.0352x faster
Avg: 22.245390 -> 21.061316: 1.0562x faster
Significant (t=4.416683)
Stddev: 0.58810 -> 0.11618: 5.0620x smaller
Timeline: http://tinyurl.com/yawzt5z

### django ###
Min: 1.105662 -> 1.115090: 1.0085x slower
Avg: 1.117930 -> 1.131781: 1.0124x slower
Significant (t=-11.024923)
Stddev: 0.00729 -> 0.01023: 1.4027x larger
Timeline: http://tinyurl.com/ydzn6e6

### nbody ###
Min: 0.535204 -> 0.559320: 1.0451x slower
Avg: 0.558861 -> 0.572902: 1.0251x slower
Significant (t=-7.484374)
Stddev: 0.01309 -> 0.01344: 1.0272x larger
Timeline: http://tinyurl.com/ygjnh5x

### slowpickle ###
Min: 0.788558 -> 0.757067: 1.0416x faster
Avg: 0.799407 -> 0.774368: 1.0323x faster
Significant (t=12.772246)
Stddev: 0.00686 -> 0.01836: 2.6759x larger
Timeline: http://tinyurl.com/y8g3zjg

### slowspitfire ###
Min: 1.200616 -> 1.218915: 1.0152x slower
Avg: 1.229028 -> 1.255978: 1.0219x slower
Significant (t=-8.165772)
Stddev: 0.02133 -> 0.02519: 1.1808x larger
Timeline: http://tinyurl.com/y9gg2x5

### slowunpickle ###
Min: 0.355483 -> 0.347013: 1.0244x faster
Avg: 0.369828 -> 0.359714: 1.0281x faster
Significant (t=6.817449)
Stddev: 0.01008 -> 0.01089: 1.0804x larger
Timeline: http://tinyurl.com/ybf3qg9

### spambayes ###
Min: 0.316724 -> 0.314673: 1.0065x faster
Avg: 0.327262 -> 0.332370: 1.0156x slower
Significant (t=-3.690136)
Stddev: 0.00598 -> 0.01248: 2.0876x larger
Timeline: http://tinyurl.com/ydck59l

$ ./perf.py -r -b default \
  ../python-2.6.4/python \
  ../unladen-swallow/python

### 2to3 ###
Min: 24.833552 -> 24.433527: 1.0164x faster
Avg: 25.241577 -> 24.878355: 1.0146x faster
Not significant
Stddev: 0.39099 -> 0.28158: 1.3886x smaller
Timeline: http://tinyurl.com/yc7nm79

### django ###
Min: 1.153900 -> 0.892072: 1.2935x faster
Avg: 1.198777 -> 0.926776: 1.2935x faster
Significant (t=61.465586)
Stddev: 0.03914 -> 0.02065: 1.8949x smaller
Timeline: http://tinyurl.com/ykm6lnk

### nbody ###
Min: 0.541474 -> 0.307949: 1.7583x faster
Avg: 0.564526 -> 0.327311: 1.7247x faster
Significant (t=57.615664)
Stddev: 0.01784 -> 0.03711: 2.0798x larger
Timeline: http://tinyurl.com/ylmezw5

### slowpickle ###
Min: 0.832452 -> 0.607266: 1.3708x faster
Avg: 0.860438 -> 0.651645: 1.3204x faster
Significant (t=20.779110)
Stddev: 0.01559 -> 0.09926: 6.3665x larger
Timeline: http://tinyurl.com/yaktykw

### slowspitfire ###
Min: 1.204681 -> 1.038169: 1.1604x faster
Avg: 1.236843 -> 1.085254: 1.1397x faster
Significant (t=20.203736)
Stddev: 0.02417 -> 0.07103: 2.9391x larger
Timeline: http://tinyurl.com/ykgmop5

### slowunpickle ###
Min: 0.374148 -> 0.279743: 1.3375x faster
Avg: 0.398137 -> 0.315630: 1.2614x faster
Significant (t=16.069155)
Stddev: 0.01333 -> 0.04958: 3.7203x larger
Timeline: http://tinyurl.com/y9b5rza

### spambayes ###
Min: 0.330391 -> 0.302988: 1.0904x faster
Avg: 0.349153 -> 0.394819: 1.1308x slower
Not significant
Stddev: 0.01158 -> 0.35049: 30.2739x larger
Timeline: http://tinyurl.com/ylq8sef

-- 
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Coghlan]

Terry Reedy wrote:
> On 1/29/2010 6:45 PM, "Martin v. Löwis" wrote:
>> When PEP 384 gets implemented, you not only get that, but you will also
>> be able to use the same extension module for 3.2, 3.3, 3.4, etc, with
>> or without U-S.
> 
> Even if CPython changes VC compiler version? In any case, greater
> usability of binaries would be really nice.

Yep, one of the main goals of PEP 384 is to identify and cordon off all
of the APIs that are dependent on the version of the C runtime and
exclude them from the stable ABI.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

2010/1/30  

> On 10:55 pm, collinwin...@google.com wrote:
>
>>
>> That people are directly munging CPython
>> bytecode means that CPython should provide a better, more abstract way
>> to do the same thing that's more resistant to these kinds of changes.
>>
>
> It might be helpful to hear more about how the wordcode implementation
> differs from the bytecode implementation.  It's challenging to abstract from
> a single data point. :)
>
>  Jean-Paul


Wordcodes structure is simple. You can find information
here.
Slide 6 provide a description, 7 details about the structure and some
examples. Slide 9 explains how I mapped most bytecodes grouping in 6
"families".

However, wordcodes internals can be complicated to "pretty print", because
they may carry many information. You can take a look at
opcode.hand
dis.py(function
"common_disassemble") to understand why this happens.

Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

2010/1/30 Scott Dial

>

> Cesare, just FYI, your Hg repository has lost the execute bits on some
> files (namely "./configure" and "./Parser/asdl_c.py"), so it does not
> quite build out-of-the-box.
>

That's probably because I worked on Windows. I have to address this issue.
Thanks.


> I took the liberty of cloning your repo into my laptop's VirtualBox
> instance of Ubuntu. I ran the default performance tests from the U-S
> repo, with VirtualBox at highest priority. As a sanity check, I ran it
> against the U-S trunk. I think the numbers speak for themselves.
>
>  --
>  Scott Dial


I see. I don't know why you got those numbers. Until now, what I saw are
better performances on average with wpython.

In a previous mail, Collin stated that when they implemented wordcode con
U-S they got benefits from the new opcode structure.

May be more tests on different hardware / platforms will help.

Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Paul Moore]

On 29 January 2010 23:45, "Martin v. Löwis"  wrote:
>> On Windows, would a C extension author be able to distribute a single
>> binary (bdist_wininst/bdist_msi) which would be compatible with
>> with-LLVM and without-LLVM builds of Python?
>
> When PEP 384 gets implemented, you not only get that, but you will also
> be able to use the same extension module for 3.2, 3.3, 3.4, etc, with
> or without U-S.

Ah! That's the point behind PEP 384! Sorry, I'd only skimmed that PEP
when it came up, and completely missed the implications.

In which case a HUGE +1 from me for PEP 384.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

I'm back with some tests that I made with the U-S test suite.

2010/1/30 Scott Dial

>

> Cesare, just FYI, your Hg repository has lost the execute bits on some
> files (namely "./configure" and "./Parser/asdl_c.py"), so it does not
> quite build out-of-the-box.
>

Unfortunately, I haven't found a solution to this problem. If somebody
working with Windows and Mercurial (I use TortoiseHg graphical client) can
give help on this issue, I'll release wpython 1.1 final.


> I took the liberty of cloning your repo into my laptop's VirtualBox
> instance of Ubuntu. I ran the default performance tests from the U-S
> repo, with VirtualBox at highest priority. As a sanity check, I ran it
> against the U-S trunk. I think the numbers speak for themselves.
>
>  --
> Scott Dial
> sc...@scottdial.com
> scod...@cs.indiana.edu
>

I downloaded U-S test suite, and made some benchmarks with my machine.
Django and Spambayes tests didn't run:

Running django...
INFO:root:Running D:\Projects\wpython\wpython10_test\PCbuild\python
performance/bm_django.py -n 100
Traceback (most recent call last):
File "perf.py", line 1938, in 
main(sys.argv[1:])
File "perf.py", line 1918, in main
options)))
File "perf.py", line 1193, in BM_Django
return SimpleBenchmark(MeasureDjango, *args, **kwargs)
File "perf.py", line 590, in SimpleBenchmark
*args, **kwargs)
File "perf.py", line 1189, in MeasureDjango
return MeasureGeneric(python, options, bm_path, bm_env)
File "perf.py", line 960, in MeasureGeneric
inherit_env=options.inherit_env)
File "perf.py", line 916, in CallAndCaptureOutput
raise RuntimeError("Benchmark died: " + err)
RuntimeError: Benchmark died: Traceback (most recent call last):
File "performance/bm_django.py", line 25, in 
from django.template import Context, Template
ImportError: No module named template

Running spambayes...
INFO:root:Running D:\Projects\wpython\wpython10_test\PCbuild\python
performance/bm_spambayes.py -n 50
Traceback (most recent call last):
File "perf.py", line 1938, in 
main(sys.argv[1:])
File "perf.py", line 1918, in main
options)))
File "perf.py", line 1666, in BM_spambayes
return SimpleBenchmark(MeasureSpamBayes, *args, **kwargs)
File "perf.py", line 590, in SimpleBenchmark
*args, **kwargs)
File "perf.py", line 1662, in MeasureSpamBayes
return MeasureGeneric(python, options, bm_path, bm_env)
File "perf.py", line 960, in MeasureGeneric
inherit_env=options.inherit_env)
File "perf.py", line 916, in CallAndCaptureOutput
raise RuntimeError("Benchmark died: " + err)
RuntimeError: Benchmark died: Traceback (most recent call last):
File "performance/bm_spambayes.py", line 18, in 
from spambayes import hammie, mboxutils
ImportError: No module named spambayes

Anyway, I run all others with wpython 1.0 final:

C:\Temp\unladen-swallow-tests>C:\temp\Python-2.6.4\PCbuild\python perf.py -r
-b default,-django,-spambayes C:\temp\Python-2.6.4\PCbuild\python
D:\Projects\wpython\wpython10_test\PCbuild\python

Report on Windows Conan post2008Server 6.1.7600 x86 AMD64 Family 15 Model 12
Stepping 0, AuthenticAMD
Total CPU cores: 1

### 2to3 ###
Min: 43.408000 -> 38.528000: 1.1267x faster
Avg: 44.448600 -> 39.391000: 1.1284x faster
Significant (t=10.582185)
Stddev: 0.84415 -> 0.65538: 1.2880x smaller
Timeline: http://tinyurl.com/ybdwese

### nbody ###
Min: 1.124000 -> 1.109000: 1.0135x faster
Avg: 1.167630 -> 1.148190: 1.0169x faster
Not significant
Stddev: 0.09607 -> 0.09544: 1.0065x smaller
Timeline: http://tinyurl.com/yex7dfv

### slowpickle ###
Min: 1.237000 -> 1.067000: 1.1593x faster
Avg: 1.283800 -> 1.109070: 1.1575x faster
Significant (t=11.393574)
Stddev: 0.11086 -> 0.10596: 1.0462x smaller
Timeline: http://tinyurl.com/y8t5ess

### slowspitfire ###
Min: 2.079000 -> 1.928000: 1.0783x faster
Avg: 2.148920 -> 1.987540: 1.0812x faster
Significant (t=7.731224)
Stddev: 0.15384 -> 0.14108: 1.0904x smaller
Timeline: http://tinyurl.com/yzexcqa

### slowunpickle ###
Min: 0.617000 -> 0.568000: 1.0863x faster
Avg: 0.645420 -> 0.590790: 1.0925x faster
Significant (t=7.087322)
Stddev: 0.05478 -> 0.05422: 1.0103x smaller
Timeline: http://tinyurl.com/ycsoouq


I also made some tests with wpython 1.1, leaving bytecode peepholer enabled:

C:\Temp\unladen-swallow-tests>C:\temp\Python-2.6.4\PCbuild\python perf.py -r
-b default,-django,-spambayes C:\temp\Python-2.6.4\PCbuild\python
D:\Projects\wpython\wpython_test\PCbuild\python

Report on Windows Conan post2008Server 6.1.7600 x86 AMD64 Family 15 Model 12
Stepping 0, AuthenticAMD
Total CPU cores: 1

### 2to3 ###
Min: 43.454000 -> 39.912000: 1.0887x faster
Avg: 44.301000 -> 40.766800: 1.0867x faster
Significant (t=8.188533)
Stddev: 0.65325 -> 0.71041: 1.0875x larger
Timeline: http://tinyurl.com/ya5z9mg

### nbody ###
Min: 1.125000 -> 1.07: 1.0514x faster
Avg: 1.169270 -> 1.105530: 1.0577x faster
Significant (t=4.774702)
Stddev: 0.09655 -> 0.09219: 1.0473x smaller
Timeline: http://tinyurl.com/y8udjmk

### slowpickle ###
Min: 1.235000 -> 1.094000: 1.1289x faster
Avg: 1.275860 -> 1.132740: 1.1263

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Brett Cannon]

On Fri, Jan 29, 2010 at 15:04,   wrote:
> On 10:47 pm, tjre...@udel.edu wrote:
>>
>> On 1/29/2010 4:19 PM, Collin Winter wrote:
>>>
>>> On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlan wrote:
>>
>>> Agreed. We originally switched Unladen Swallow to wordcode in our
>>> 2009Q1 release, and saw a performance improvement from this across the
>>> board. We switched back to bytecode for the JIT compiler to make
>>> upstream merger easier. The Unladen Swallow benchmark suite should
>>> provided a thorough assessment of the impact of the wordcode ->
>>> bytecode switch. This would be complementary to a JIT compiler, rather
>>> than a replacement for it.
>>>
>>> I would note that the switch will introduce incompatibilities with
>>> libraries like Twisted. IIRC, Twisted has a traceback prettifier that
>>> removes its trampoline functions from the traceback, parsing CPython's
>>> bytecode in the process. If running under CPython, it assumes that the
>>> bytecode is as it expects. We broke this in Unladen's wordcode switch.
>>> I think parsing bytecode is a bad idea, but any switch to wordcode
>>> should be advertised widely.
>>
>> Several years, there was serious consideration of switching to a
>> registerbased vm, which would have been even more of a change. Since I
>> learned 1.4, Guido has consistently insisted that the CPython vm is not part
>> of the language definition and, as far as I know, he has rejected any byte-
>> code hackery in the stdlib. While he is not one to, say, randomly permute
>> the codes just to frustrate such hacks, I believe he has always considered
>> vm details private and subject to change and any usage thereof 'at one's own
>> risk'.
>
> Language to such effect might be a useful addition to this page (amongst
> others, perhaps):
>
>  http://docs.python.org/library/dis.html
>
> which very clearly and helpfully lays out quite a number of APIs which can
> be used to get pretty deep into the bytecode.  If all of this is subject to
> be discarded at the first sign that doing so might be beneficial for some
> reason, don't keep it a secret that people need to join python-dev to learn.
>

Can you file a bug and assign it to me?

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[M.-A. Lemburg]

M.-A. Lemburg wrote:
> Collin Winter wrote:
>> I added startup benchmarks for Mercurial and Bazaar yesterday
>> (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we
>> can use them as more macro-ish benchmarks, rather than merely starting
>> the CPython binary over and over again. If you have ideas for better
>> Mercurial/Bazaar startup scenarios, I'd love to hear them. The new
>> hg_startup and bzr_startup benchmarks should give us some more data
>> points for measuring improvements in startup time.
>>
>> One idea we had for improving startup time for apps like Mercurial was
>> to allow the creation of hermetic Python "binaries", with all
>> necessary modules preloaded. This would be something like Smalltalk
>> images. We haven't yet really fleshed out this idea, though.
> 
> In Python you can do the same with the freeze.py utility. See
> 
> http://www.egenix.com/www2002/python/mxCGIPython.html
> 
> for an old project where we basically put the Python
> interpreter and stdlib into a single executable.
> 
> We've recently revisited that project and created something
> we call "pyrun". It fits Python 2.5 into a single executable
> and a set of shared modules (which for various reasons cannot
> be linked statically)... 12MB in total.
> 
> If you load lots of modules from the stdlib this does provide
> a significant improvement over standard Python.
> 
> Back to the PEP's proposal:
> 
> Looking at the data you currently have, the negative results
> currently don't really look good in the light of the small
> performance improvements.
> 
> Wouldn't it be possible to have the compiler approach work
> in three phases in order to reduce the memory footprint and
> startup time hit, ie.
> 
>  1. run an instrumented Python interpreter to collect all
> the needed compiler information; write this information into
> a .pys file (Python stats)
> 
>  2. create compiled versions of the code for various often
> used code paths and type combinations by reading the
> .pys file and generating an .so file as regular
> Python extension module
> 
>  3. run an uninstrumented Python interpreter and let it
> use the .so files instead of the .py ones
> 
> In production, you'd then only use step 3 and avoid the
> overhead of steps 1 and 2.
> 
> Moreover, the .so file approach
> would only load the code for code paths and type combinations
> actually used in a particular run of the Python code into
> memory and allow multiple Python processes to share it.
> 
> As side effect, you'd probably also avoid the need to have
> C++ code in the production Python runtime - that is unless
> LLVM requires some kind of runtime support which is written
> in C++.

BTW: Some years ago we discussed the idea of pluggable VMs for
Python. Wouldn't U-S be a good motivation to revisit this idea ?

We could then have a VM based on byte code using a stack
machines, one based on word code using a register machine
and perhaps one that uses the Stackless approach.

Each VM type could use the PEP 3147 approach to store
auxiliary files to store byte code, word code or machine
compiled code.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 01 2010)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hey MA,

On Mon, Feb 1, 2010 at 9:58 AM, M.-A. Lemburg  wrote:
> BTW: Some years ago we discussed the idea of pluggable VMs for
> Python. Wouldn't U-S be a good motivation to revisit this idea ?
>
> We could then have a VM based on byte code using a stack
> machines, one based on word code using a register machine
> and perhaps one that uses the Stackless approach.

What is the usecase for having pluggable VMs? Is the idea that, at
runtime, the user would select which virtual machine they want to run
their code under? How would the user make that determination
intelligently?

I think this idea underestimates a) how deeply the current CPython VM
is intertwined with the rest of the implementation, and b) the nature
of the changes required by these separate VMs. For example, Unladen
Swallow adds fields to the C-level structs for dicts, code objects and
frame objects; how would those changes be pluggable? Stackless
requires so many modifications that it is effectively a fork; how
would those changes be pluggable?

Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[M.-A. Lemburg]

Collin Winter wrote:
> Hey MA,
> 
> On Mon, Feb 1, 2010 at 9:58 AM, M.-A. Lemburg  wrote:
>> BTW: Some years ago we discussed the idea of pluggable VMs for
>> Python. Wouldn't U-S be a good motivation to revisit this idea ?
>>
>> We could then have a VM based on byte code using a stack
>> machines, one based on word code using a register machine
>> and perhaps one that uses the Stackless approach.
> 
> What is the usecase for having pluggable VMs? Is the idea that, at
> runtime, the user would select which virtual machine they want to run
> their code under? How would the user make that determination
> intelligently?

The idea back then (IIRC) was to have a compile time option to select
one of a few available VMs, in order to more easily experiment with
new or optimized implementations such as e.g. a register based VM.

It should even be possible to factor out the VM into a DLL/SO which
is then selected and loaded via a command line option.

> I think this idea underestimates a) how deeply the current CPython VM
> is intertwined with the rest of the implementation, and b) the nature
> of the changes required by these separate VMs. For example, Unladen
> Swallow adds fields to the C-level structs for dicts, code objects and
> frame objects; how would those changes be pluggable? Stackless
> requires so many modifications that it is effectively a fork; how
> would those changes be pluggable?

They wouldn't be pluggable. Such changes would have to be made
in a more general way in order to serve more than just one VM.

Getting the right would certainly require a major effort, but it
would also reduce the need to have several branches of C-based
Python implementations.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 01 2010)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

On Mon, Feb 1, 2010 at 11:17 AM, M.-A. Lemburg  wrote:
> Collin Winter wrote:
>> I think this idea underestimates a) how deeply the current CPython VM
>> is intertwined with the rest of the implementation, and b) the nature
>> of the changes required by these separate VMs. For example, Unladen
>> Swallow adds fields to the C-level structs for dicts, code objects and
>> frame objects; how would those changes be pluggable? Stackless
>> requires so many modifications that it is effectively a fork; how
>> would those changes be pluggable?
>
> They wouldn't be pluggable. Such changes would have to be made
> in a more general way in order to serve more than just one VM.

I believe these VMs would have little overlap. I cannot imagine that
Unladen Swallow's needs have much in common with Stackless's, or with
those of a hypothetical register machine to replace the current stack
machine.

Let's consider that last example in more detail: a register machine
would require completely different bytecode. This would require
replacing the bytecode compiler, the peephole optimizer, and the
bytecode eval loop. The frame object would need to be changed to hold
the registers and a new blockstack design; the code object would have
to potentially hold a new bytecode layout.

I suppose making all this pluggable would be possible, but I don't see
the point. This kind of experimentation is ideal for a branch: go off,
test your idea, report your findings, merge back. Let the branch be
long-lived, if need be. The Mercurial migration will make all this
easier.

> Getting the right would certainly require a major effort, but it
> would also reduce the need to have several branches of C-based
> Python implementations.

If such a restrictive plugin-based scheme had been available when we
began Unladen Swallow, I do not doubt that we would have ignored it
entirely. I do not like the idea of artificially tying the hands of
people trying to make CPython faster. I do not see any part of Unladen
Swallow that would have been made easier by such a scheme. If
anything, it would have made our project more difficult.

Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Terry Reedy]

On 2/1/2010 1:32 PM, Collin Winter wrote:

Hey MA,

On Mon, Feb 1, 2010 at 9:58 AM, M.-A. Lemburg  wrote:

BTW: Some years ago we discussed the idea of pluggable VMs for
Python. Wouldn't U-S be a good motivation to revisit this idea ?

We could then have a VM based on byte code using a stack
machines, one based on word code using a register machine
and perhaps one that uses the Stackless approach.


What is the usecase for having pluggable VMs?


Running an application full time on multiple machines


Is the idea that, at runtime, the user would select which virtual

>  machine they want to run their code under?

From you comments below, I would presume the selection should be at 
startup, from the command line, before building Python objects.



How would the user make that determination intelligently?


The same way people would determine whether to select JIT or not -- by 
testing space and time performance for their app, with test time 
proportioned to expected run time.


Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Cesare Di Mauro]

2010/2/1 Collin Winter 

> I believe these VMs would have little overlap. I cannot imagine that
> Unladen Swallow's needs have much in common with Stackless's, or with
> those of a hypothetical register machine to replace the current stack
> machine.
>
> Let's consider that last example in more detail: a register machine
> would require completely different bytecode. This would require
> replacing the bytecode compiler, the peephole optimizer, and the
> bytecode eval loop. The frame object would need to be changed to hold
> the registers and a new blockstack design; the code object would have
> to potentially hold a new bytecode layout.
>
> I suppose making all this pluggable would be possible, but I don't see
> the point. This kind of experimentation is ideal for a branch: go off,
> test your idea, report your findings, merge back. Let the branch be
> long-lived, if need be. The Mercurial migration will make all this
> easier.
>
> > Getting the right would certainly require a major effort, but it
> > would also reduce the need to have several branches of C-based
> > Python implementations.
>
> If such a restrictive plugin-based scheme had been available when we
> began Unladen Swallow, I do not doubt that we would have ignored it
> entirely. I do not like the idea of artificially tying the hands of
> people trying to make CPython faster. I do not see any part of Unladen
> Swallow that would have been made easier by such a scheme. If
> anything, it would have made our project more difficult.
>
>  Collin Winter


I completely agree. Working with wpython I have changed a lot of code
ranging from the ASDL grammar to the eval loop, including some library
module and tests (primarily the Python-based parser and the disassembly
tools; module finder required work, too).
I haven't changed the Python objects or the object model (except in the
alpha release; then I dropped this "invasive" change), but I've added some
helper functions in object.c, dict.c, etc.

A pluggable VM isn't feasible because we are talking about a brand new
CPython (library included), to be chosen each time.

If approved, this model will limit a lot the optimizations that can be
implemented to make CPython running faster.

Cesare Di Mauro
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[M.-A. Lemburg]

Collin Winter wrote:
> On Mon, Feb 1, 2010 at 11:17 AM, M.-A. Lemburg  wrote:
>> Collin Winter wrote:
>>> I think this idea underestimates a) how deeply the current CPython VM
>>> is intertwined with the rest of the implementation, and b) the nature
>>> of the changes required by these separate VMs. For example, Unladen
>>> Swallow adds fields to the C-level structs for dicts, code objects and
>>> frame objects; how would those changes be pluggable? Stackless
>>> requires so many modifications that it is effectively a fork; how
>>> would those changes be pluggable?
>>
>> They wouldn't be pluggable. Such changes would have to be made
>> in a more general way in order to serve more than just one VM.
> 
> I believe these VMs would have little overlap. I cannot imagine that
> Unladen Swallow's needs have much in common with Stackless's, or with
> those of a hypothetical register machine to replace the current stack
> machine.
> 
> Let's consider that last example in more detail: a register machine
> would require completely different bytecode. This would require
> replacing the bytecode compiler, the peephole optimizer, and the
> bytecode eval loop. The frame object would need to be changed to hold
> the registers and a new blockstack design; the code object would have
> to potentially hold a new bytecode layout.
> 
> I suppose making all this pluggable would be possible, but I don't see
> the point. This kind of experimentation is ideal for a branch: go off,
> test your idea, report your findings, merge back. Let the branch be
> long-lived, if need be. The Mercurial migration will make all this
> easier.
> 
>> Getting the right would certainly require a major effort, but it
>> would also reduce the need to have several branches of C-based
>> Python implementations.
> 
> If such a restrictive plugin-based scheme had been available when we
> began Unladen Swallow, I do not doubt that we would have ignored it
> entirely. I do not like the idea of artificially tying the hands of
> people trying to make CPython faster. I do not see any part of Unladen
> Swallow that would have been made easier by such a scheme. If
> anything, it would have made our project more difficult.

I don't think that it has to be restrictive - much to the contrary,
it would provide a consistent API to those CPython internals and
also clarify the separation between the various parts. Something
which currently does not exist in CPython.

Note that it may be easier for you (and others) to just take
CPython and patch it as necessary. However, this doesn't relieve
you from the needed maintenance - which, I presume, is one of the
reasons why you are suggesting to merge U-S back into CPython ;-)

The same problem exists for all other branches, such as e.g.
Stackless. Now, why should we merge in your particular branch
and make it harder for those other teams ?

Instead of having 2-3 teams maintain complete branches, it would
be more efficient to just have them take care of their particular
VM implementation. Furthermore, we wouldn't need to decide
which VM variant to merge into the core, since all of them
would be equally usable.

The idea is based on a big picture perspective and focuses on long
term benefits for more than just one team of VM implementers.

BTW: I also doubt that Mercurial will make any of this easier.
It makes creating branches easier for non-committers, but the
problem of having to maintain the branches remains.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 02 2010)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Coghlan]

M.-A. Lemburg wrote:
> BTW: I also doubt that Mercurial will make any of this easier.
> It makes creating branches easier for non-committers, but the
> problem of having to maintain the branches remains.

It greatly simplifies the process of syncing the branch with the main
line of development so yes, it should help with branch maintenance
(svnmerge is a pale shadow of what a true DVCS can handle). That aspect
is one of the DVCS selling points that also applies to core development.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hey Dirkjan,

[Circling back to this part of the thread]

On Thu, Jan 21, 2010 at 1:37 PM, Dirkjan Ochtman  wrote:
> On Thu, Jan 21, 2010 at 21:14, Collin Winter  wrote:
[snip]
>> My quick take on Cython and Shedskin is that they are
>> useful-but-limited workarounds for CPython's historically-poor
>> performance. Shedskin, for example, does not support the entire Python
>> language or standard library
>> (http://shedskin.googlecode.com/files/shedskin-tutorial-0.3.html).
>
> Perfect, now put something like this in the PEP, please. ;)

Done. The diff is at
http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed
Cython, Shedskin and a bunch of other alternatives to pure CPython.
Some of that information is based on conversations I've had with the
respective developers, and I'd appreciate corrections if I'm out of
date.

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Craig Citro]

> Done. The diff is at
> http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed
> Cython, Shedskin and a bunch of other alternatives to pure CPython.
> Some of that information is based on conversations I've had with the
> respective developers, and I'd appreciate corrections if I'm out of
> date.
>

Well, it's a minor nit, but it might be more fair to say something
like "Cython provides the biggest improvements once type annotations
are added to the code." After all, Cython is more than happy to take
arbitrary Python code as input -- it's just much more effective when
it knows something about types. The code to make Cython handle
closures has just been merged ... hopefully support for the full
Python language isn't so far off. (Let me know if you want me to
actually make a comment on Rietveld ...)

Now what's more interesting is whether or not U-S and Cython could
play off one another -- take a Python program, run it with some
"generic input data" under Unladen and record info about which
functions are hot, and what types they tend to take, then let
Cython/gcc -O3 have a go at these, and lather, rinse, repeat ... JIT
compilation and static compilation obviously serve different purposes,
but I'm curious if there aren't other interesting ways to take
advantage of both.

-cc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hey MA,

On Fri, Jan 29, 2010 at 11:14 AM, M.-A. Lemburg  wrote:
> Collin Winter wrote:
>> I added startup benchmarks for Mercurial and Bazaar yesterday
>> (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we
>> can use them as more macro-ish benchmarks, rather than merely starting
>> the CPython binary over and over again. If you have ideas for better
>> Mercurial/Bazaar startup scenarios, I'd love to hear them. The new
>> hg_startup and bzr_startup benchmarks should give us some more data
>> points for measuring improvements in startup time.
>>
>> One idea we had for improving startup time for apps like Mercurial was
>> to allow the creation of hermetic Python "binaries", with all
>> necessary modules preloaded. This would be something like Smalltalk
>> images. We haven't yet really fleshed out this idea, though.
>
> In Python you can do the same with the freeze.py utility. See
>
> http://www.egenix.com/www2002/python/mxCGIPython.html
>
> for an old project where we basically put the Python
> interpreter and stdlib into a single executable.
>
> We've recently revisited that project and created something
> we call "pyrun". It fits Python 2.5 into a single executable
> and a set of shared modules (which for various reasons cannot
> be linked statically)... 12MB in total.
>
> If you load lots of modules from the stdlib this does provide
> a significant improvement over standard Python.

Good to know there are options. One feature we had in mind for a
system of this sort would be the ability to take advantage of the
limited/known set of modules in the image to optimize the application
further, similar to link-time optimizations in gcc/LLVM
(http://www.airs.com/blog/archives/100).

> Back to the PEP's proposal:
>
> Looking at the data you currently have, the negative results
> currently don't really look good in the light of the small
> performance improvements.

The JIT compiler we are offering is more than just its current
performance benefit. An interpreter loop will simply never be as fast
as machine code. An interpreter loop, no matter how well-optimized,
will hit a performance ceiling and before that ceiling will run into
diminishing returns. Machine code is a more versatile optimization
target, and as such, allows many optimizations that would be
impossible or prohibitively difficult in an interpreter.

Unladen Swallow offers a platform to extract increasing performance
for years to come. The current generation of modern, JIT-based
JavaScript engines are instructive in this regard: V8 (which I'm most
familiar with) delivers consistently improving performance
release-over-release (see the graphs at the top of
http://googleblog.blogspot.com/2009/09/google-chrome-after-year-sporting-new.html).
 I'd like to see CPython be able to achieve the same thing, like the
new implementations of JavaScript and Ruby are able to do.

We are aware that Unladen Swallow is not finished; that's why we're
not asking to go into py3k directly. Unladen Swallow's memory usage
will continue to decrease, and its performance will only go up. The
current state is not its permanent state; I'd hate to see the perfect
become the enemy of the good.

> Wouldn't it be possible to have the compiler approach work
> in three phases in order to reduce the memory footprint and
> startup time hit, ie.
>
>  1. run an instrumented Python interpreter to collect all
>    the needed compiler information; write this information into
>    a .pys file (Python stats)
>
>  2. create compiled versions of the code for various often
>    used code paths and type combinations by reading the
>    .pys file and generating an .so file as regular
>    Python extension module
>
>  3. run an uninstrumented Python interpreter and let it
>    use the .so files instead of the .py ones
>
> In production, you'd then only use step 3 and avoid the
> overhead of steps 1 and 2.

That is certainly a possibility if we are unable to reduce memory
usage to a satisfactory level. I've added a "Contingency Plans"
section to the PEP, including this option:
http://codereview.appspot.com/186247/diff2/8004:7005/8006.

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Reid Kleckner]

On Tue, Feb 2, 2010 at 8:57 PM, Collin Winter  wrote:
>> Wouldn't it be possible to have the compiler approach work
>> in three phases in order to reduce the memory footprint and
>> startup time hit, ie.
>>
>>  1. run an instrumented Python interpreter to collect all
>>    the needed compiler information; write this information into
>>    a .pys file (Python stats)
>>
>>  2. create compiled versions of the code for various often
>>    used code paths and type combinations by reading the
>>    .pys file and generating an .so file as regular
>>    Python extension module
>>
>>  3. run an uninstrumented Python interpreter and let it
>>    use the .so files instead of the .py ones
>>
>> In production, you'd then only use step 3 and avoid the
>> overhead of steps 1 and 2.
>
> That is certainly a possibility if we are unable to reduce memory
> usage to a satisfactory level. I've added a "Contingency Plans"
> section to the PEP, including this option:
> http://codereview.appspot.com/186247/diff2/8004:7005/8006.

This would be another good research problem for someone to take and
run.  The trick is that you would need to add some kind of "linking"
step to loading the .so.  Right now, we just collect PyObject*'s, and
don't care whether they're statically allocated or user-defined
objects.  If you wanted to pursue offline feedback directed
compilation, you would need to write something that basically can map
from the pointers in the feedback data to something like a Python
dotted name import path, and then when you load the application, look
up those names and rewrite the new pointers into the generated machine
code.  It sounds a lot like writing a dynamic loader.  :)

It sounds like a huge amount of work, and we haven't approached it.
On the other hand, it sounds like it might be rewarding.

Reid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Dirkjan Ochtman]

On Tue, Feb 2, 2010 at 23:54, Collin Winter  wrote:
> Done. The diff is at
> http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed
> Cython, Shedskin and a bunch of other alternatives to pure CPython.
> Some of that information is based on conversations I've had with the
> respective developers, and I'd appreciate corrections if I'm out of
> date.

Thanks, that's a very good list (and I think it makes for a useful
addition to the PEP).

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[M.-A. Lemburg]

Reid Kleckner wrote:
> On Tue, Feb 2, 2010 at 8:57 PM, Collin Winter  wrote:
>>> Wouldn't it be possible to have the compiler approach work
>>> in three phases in order to reduce the memory footprint and
>>> startup time hit, ie.
>>>
>>>  1. run an instrumented Python interpreter to collect all
>>>the needed compiler information; write this information into
>>>a .pys file (Python stats)
>>>
>>>  2. create compiled versions of the code for various often
>>>used code paths and type combinations by reading the
>>>.pys file and generating an .so file as regular
>>>Python extension module
>>>
>>>  3. run an uninstrumented Python interpreter and let it
>>>use the .so files instead of the .py ones
>>>
>>> In production, you'd then only use step 3 and avoid the
>>> overhead of steps 1 and 2.
>>
>> That is certainly a possibility if we are unable to reduce memory
>> usage to a satisfactory level. I've added a "Contingency Plans"
>> section to the PEP, including this option:
>> http://codereview.appspot.com/186247/diff2/8004:7005/8006.
> 
> This would be another good research problem for someone to take and
> run.  The trick is that you would need to add some kind of "linking"
> step to loading the .so.  Right now, we just collect PyObject*'s, and
> don't care whether they're statically allocated or user-defined
> objects.  If you wanted to pursue offline feedback directed
> compilation, you would need to write something that basically can map
> from the pointers in the feedback data to something like a Python
> dotted name import path, and then when you load the application, look
> up those names and rewrite the new pointers into the generated machine
> code.  It sounds a lot like writing a dynamic loader.  :)

You lost me there :-)

I am not familiar with how U-S actually implements the compilation
step and was thinking of it working at the functions/methods level
and based on input/output parameter type information.

Most Python functions and methods have unique names (when
combined with the module and class name), so these could
be used for the referencing and feedback writing.

The only cases where this doesn't work too well is dynamic
programming of the sort done in namedtuples: where you
dynamically create a class and then instantiate it.

Type information for basic types and their subclasses can
be had dynamically (there's also a basic type bitmap for
faster lookup) or in a less robust way by name.

For the feedback file the names should be a good reference.

> It sounds like a huge amount of work, and we haven't approached it.
> On the other hand, it sounds like it might be rewarding.

Indeed. Perhaps this could be further investigated in a SoC
project ?!

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 03 2010)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Reid Kleckner]

On Wed, Feb 3, 2010 at 6:51 AM, M.-A. Lemburg  wrote:
> You lost me there :-)
>
> I am not familiar with how U-S actually implements the compilation
> step and was thinking of it working at the functions/methods level
> and based on input/output parameter type information.

Yes, but it's more like for every attribute lookup we ask "what was
the type of the object we did the lookup on?"  So, we simply take a
reference to obj->ob_type and stuff it in our feedback record, which
is limited to just three pointers.  Then when we generate code, we may
emit a guard that compares obj->ob_type in the compiled lookup to the
pointer we recorded.  We also need to place a weak reference to the
code object on the type so that when the type is mutated or deleted,
we invalidate the code, since its assumptions are invalid.

> Most Python functions and methods have unique names (when
> combined with the module and class name), so these could
> be used for the referencing and feedback writing.

Right, so when building the .so to load, you would probably want to
take all the feedback data and find these dotted names for the
pointers in the feedback data.  If you find any pointers that can't be
reliably identified, you could drop it from the feedback and flag that
site as polymorphic (ie don't optimize this site).  Then you generate
machine code from the feedback and stuff it in a .so, with special
relocation information.

When you load the .so into a fresh Python process with a different
address space layout, you try to recover the pointers to the PyObjects
mentioned in the relocation information and patch up the machine code
with the new pointers, which is very similar to the job of a linker.
If you can't find the name or things don't look right, you just drop
that piece of native code.

> The only cases where this doesn't work too well is dynamic
> programming of the sort done in namedtuples: where you
> dynamically create a class and then instantiate it.

It might actually work if the namedtuple is instantiated at module
scope before loading the .so.

> Type information for basic types and their subclasses can
> be had dynamically (there's also a basic type bitmap for
> faster lookup) or in a less robust way by name.

If I understand you correctly, you're thinking about looking the types
up by name or bitmap in the machine code.  I think it would be best to
just do the lookup once at load time and patch the native code.

>> It sounds like a huge amount of work, and we haven't approached it.
>> On the other hand, it sounds like it might be rewarding.
>
> Indeed. Perhaps this could be further investigated in a SoC
> project ?!

Or maybe a thesis.  I'm really walking out on a limb, and this idea is
quite hypothetical.  :)

Reid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hi Craig,

On Tue, Feb 2, 2010 at 4:42 PM, Craig Citro  wrote:
>> Done. The diff is at
>> http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed
>> Cython, Shedskin and a bunch of other alternatives to pure CPython.
>> Some of that information is based on conversations I've had with the
>> respective developers, and I'd appreciate corrections if I'm out of
>> date.
>>
>
> Well, it's a minor nit, but it might be more fair to say something
> like "Cython provides the biggest improvements once type annotations
> are added to the code." After all, Cython is more than happy to take
> arbitrary Python code as input -- it's just much more effective when
> it knows something about types. The code to make Cython handle
> closures has just been merged ... hopefully support for the full
> Python language isn't so far off. (Let me know if you want me to
> actually make a comment on Rietveld ...)

Indeed, you're quite right. I've corrected the description here:
http://codereview.appspot.com/186247/diff2/7005:9001/10001

> Now what's more interesting is whether or not U-S and Cython could
> play off one another -- take a Python program, run it with some
> "generic input data" under Unladen and record info about which
> functions are hot, and what types they tend to take, then let
> Cython/gcc -O3 have a go at these, and lather, rinse, repeat ... JIT
> compilation and static compilation obviously serve different purposes,
> but I'm curious if there aren't other interesting ways to take
> advantage of both.

Definitely! Someone approached me about possibly reusing the profile
data for a feedback-enhanced code coverage tool, which has interesting
potential, too. I've added a note about this under the "Future Work"
section: http://codereview.appspot.com/186247/diff2/9001:10002/9003

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

To follow up on some of the open issues:

On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter  wrote:
[snip]
> Open Issues
> ===
>
> - *Code review policy for the ``py3k-jit`` branch.* How does the CPython
>  community want us to procede with respect to checkins on the ``py3k-jit``
>  branch? Pre-commit reviews? Post-commit reviews?
>
>  Unladen Swallow has enforced pre-commit reviews in our trunk, but we realize
>  this may lead to long review/checkin cycles in a purely-volunteer
>  organization. We would like a non-Google-affiliated member of the CPython
>  development team to review our work for correctness and compatibility, but we
>  realize this may not be possible for every commit.

The feedback we've gotten so far is that at most, only larger, more
critical commits should be sent for review, while most commits can
just go into the branch. Is that broadly agreeable to python-dev?

> - *How to link LLVM.* Should we change LLVM to better support shared linking,
>  and then use shared linking to link the parts of it we need into CPython?

The consensus has been that we should link shared against LLVM.
Jeffrey Yasskin is now working on this in upstream LLVM. We are
tracking this at
http://code.google.com/p/unladen-swallow/issues/detail?id=130 and
http://llvm.org/PR3201.

> - *Prioritization of remaining issues.* We would like input from the CPython
>  development team on how to prioritize the remaining issues in the Unladen
>  Swallow codebase. Some issues like memory usage are obviously critical before
>  merger with ``py3k``, but others may fall into a "nice to have" category that
>  could be kept for resolution into a future CPython 3.x release.

The big-ticket items here are what we expected: reducing memory usage
and startup time. We also need to improve profiling options, both for
oProfile and cProfile.

> - *Create a C++ style guide.* Should PEP 7 be extended to include C++, or
>  should a separate C++ style PEP be created? Unladen Swallow maintains its own
>  style guide [#us-styleguide]_, which may serve as a starting point; the
>  Unladen Swallow style guide is based on both LLVM's [#llvm-styleguide]_ and
>  Google's [#google-styleguide]_ C++ style guides.

Any thoughts on a CPython C++ style guide? My personal preference
would be to extend PEP 7 to cover C++ by taking elements from
http://code.google.com/p/unladen-swallow/wiki/StyleGuide and the LLVM
and Google style guides (which is how we've been developing Unladen
Swallow). If that's broadly agreeable, Jeffrey and I will work on a
patch to PEP 7.

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Brett Cannon]

On Tue, Feb 9, 2010 at 14:47, Collin Winter  wrote:
> To follow up on some of the open issues:
>
> On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter  
> wrote:
> [snip]
>> Open Issues
>> ===
>>
>> - *Code review policy for the ``py3k-jit`` branch.* How does the CPython
>>  community want us to procede with respect to checkins on the ``py3k-jit``
>>  branch? Pre-commit reviews? Post-commit reviews?
>>
>>  Unladen Swallow has enforced pre-commit reviews in our trunk, but we realize
>>  this may lead to long review/checkin cycles in a purely-volunteer
>>  organization. We would like a non-Google-affiliated member of the CPython
>>  development team to review our work for correctness and compatibility, but 
>> we
>>  realize this may not be possible for every commit.
>
> The feedback we've gotten so far is that at most, only larger, more
> critical commits should be sent for review, while most commits can
> just go into the branch. Is that broadly agreeable to python-dev?
>
>> - *How to link LLVM.* Should we change LLVM to better support shared linking,
>>  and then use shared linking to link the parts of it we need into CPython?
>
> The consensus has been that we should link shared against LLVM.
> Jeffrey Yasskin is now working on this in upstream LLVM. We are
> tracking this at
> http://code.google.com/p/unladen-swallow/issues/detail?id=130 and
> http://llvm.org/PR3201.
>
>> - *Prioritization of remaining issues.* We would like input from the CPython
>>  development team on how to prioritize the remaining issues in the Unladen
>>  Swallow codebase. Some issues like memory usage are obviously critical 
>> before
>>  merger with ``py3k``, but others may fall into a "nice to have" category 
>> that
>>  could be kept for resolution into a future CPython 3.x release.
>
> The big-ticket items here are what we expected: reducing memory usage
> and startup time. We also need to improve profiling options, both for
> oProfile and cProfile.
>
>> - *Create a C++ style guide.* Should PEP 7 be extended to include C++, or
>>  should a separate C++ style PEP be created? Unladen Swallow maintains its 
>> own
>>  style guide [#us-styleguide]_, which may serve as a starting point; the
>>  Unladen Swallow style guide is based on both LLVM's [#llvm-styleguide]_ and
>>  Google's [#google-styleguide]_ C++ style guides.
>
> Any thoughts on a CPython C++ style guide? My personal preference
> would be to extend PEP 7 to cover C++ by taking elements from
> http://code.google.com/p/unladen-swallow/wiki/StyleGuide and the LLVM
> and Google style guides (which is how we've been developing Unladen
> Swallow). If that's broadly agreeable, Jeffrey and I will work on a
> patch to PEP 7.
>

I have found the Google C++ style guide good so I am fine with taking
ideas from that and adding them to PEP 7.

-Brett



> Thanks,
> Collin Winter
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Maciej Fijalkowski]

Snippet from:

http://codereview.appspot.com/186247/diff2/5014:8003/7002

*PyPy*: PyPy [#pypy]_ has good performance on numerical code, but is
slower than Unladen Swallow on non-numerical workloads. PyPy only
supports 32-bit x86 code generation. It has poor support for CPython
extension modules, making migration for large applications
prohibitively expensive.

That part at the very least has some sort of personal opinion
"prohibitively", while the other part is not completely true "slower
than US on non-numerical workloads". Fancy providing a proof for that?
I'm well aware that there are benchmarks on which PyPy is slower than
CPython or US, however, I would like a bit more weighted opinion in
the PEP.

Cheers,
fijal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hey Maciej,

On Thu, Feb 11, 2010 at 6:39 AM, Maciej Fijalkowski  wrote:
> Snippet from:
>
> http://codereview.appspot.com/186247/diff2/5014:8003/7002
>
> *PyPy*: PyPy [#pypy]_ has good performance on numerical code, but is
> slower than Unladen Swallow on non-numerical workloads. PyPy only
> supports 32-bit x86 code generation. It has poor support for CPython
> extension modules, making migration for large applications
> prohibitively expensive.
>
> That part at the very least has some sort of personal opinion
> "prohibitively",

Of course; difficulty is always in the eye of the person doing the
work. Simply put, PyPy is not a drop-in replacement for CPython: there
is no embedding API, much less the same one exported by CPython;
important libraries, such as MySQLdb and pycrypto, do not build
against PyPy; PyPy is 32-bit x86 only.

All of these problems can be overcome with enough time/effort/money,
but I think you'd agree that, if all I'm trying to do is speed up my
application, adding a new x86-64 backend or implementing support for
CPython extension modules is certainly north of "prohibitively
expensive". I stand by that wording. I'm willing to enumerate all of
PyPy's deficiencies in this regard in the PEP, rather than the current
vaguer wording, if you'd prefer.

> while the other part is not completely true "slower
> than US on non-numerical workloads". Fancy providing a proof for that?
> I'm well aware that there are benchmarks on which PyPy is slower than
> CPython or US, however, I would like a bit more weighted opinion in
> the PEP.

Based on the benchmarks you're running at
http://codespeak.net:8099/plotsummary.html, PyPy is slower than
CPython on many non-numerical workloads, which Unladen Swallow is
faster than CPython at. Looking at the benchmarks there at which PyPy
is faster than CPython, they are primarily numerical; this was the
basis for the wording in the PEP.

My own recent benchmarking of PyPy and Unladen Swallow (both trunk;
PyPy wouldn't run some benchmarks):

| Benchmark| PyPy  | Unladen | Change  |
+==+===+=+=+
| ai   | 0.61  | 0.51|  1.1921x faster |
| django   | 0.68  | 0.8 |  1.1898x slower |
| float| 0.03  | 0.07|  2.7108x slower |
| html5lib | 20.04 | 16.42   |  1.2201x faster |
| pickle   | 17.7  | 1.09| 16.2465x faster |
| rietveld | 1.09  | 0.59|  1.8597x faster |
| slowpickle   | 0.43  | 0.56|  1.2956x slower |
| slowspitfire | 2.5   | 0.63|  3.9853x faster |
| slowunpickle | 0.26  | 0.27|  1.0585x slower |
| unpickle | 28.45 | 0.78| 36.6427x faster |

I'm happy to change the wording to "slower than US on some workloads".

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Coghlan]

Collin Winter wrote:
> Hey Maciej,
> 
> On Thu, Feb 11, 2010 at 6:39 AM, Maciej Fijalkowski  wrote:
>> Snippet from:
>>
>> http://codereview.appspot.com/186247/diff2/5014:8003/7002
>>
>> *PyPy*: PyPy [#pypy]_ has good performance on numerical code, but is
>> slower than Unladen Swallow on non-numerical workloads. PyPy only
>> supports 32-bit x86 code generation. It has poor support for CPython
>> extension modules, making migration for large applications
>> prohibitively expensive.
>>
>> That part at the very least has some sort of personal opinion
>> "prohibitively",
> 
> Of course; difficulty is always in the eye of the person doing the
> work. Simply put, PyPy is not a drop-in replacement for CPython: there
> is no embedding API, much less the same one exported by CPython;
> important libraries, such as MySQLdb and pycrypto, do not build
> against PyPy; PyPy is 32-bit x86 only.

I think pointing out at least these two restrictions explicitly would be
helpful (since they put some objective bounds on the meaning of
"prohibitive" in this context).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Maciej Fijalkowski]

On Fri, Feb 12, 2010 at 7:04 PM, Collin Winter  wrote:
> Hey Maciej,
>
> On Thu, Feb 11, 2010 at 6:39 AM, Maciej Fijalkowski  wrote:
>> Snippet from:
>>
>> http://codereview.appspot.com/186247/diff2/5014:8003/7002
>>
>> *PyPy*: PyPy [#pypy]_ has good performance on numerical code, but is
>> slower than Unladen Swallow on non-numerical workloads. PyPy only
>> supports 32-bit x86 code generation. It has poor support for CPython
>> extension modules, making migration for large applications
>> prohibitively expensive.
>>
>> That part at the very least has some sort of personal opinion
>> "prohibitively",
>
> Of course; difficulty is always in the eye of the person doing the
> work. Simply put, PyPy is not a drop-in replacement for CPython: there
> is no embedding API, much less the same one exported by CPython;
> important libraries, such as MySQLdb and pycrypto, do not build
> against PyPy; PyPy is 32-bit x86 only.

I like this wording far more. It's at the very least far more precise.
Those examples are fair enough (except the fact that PyPy is not 32bit
x86 only, the JIT is).

> All of these problems can be overcome with enough time/effort/money,
> but I think you'd agree that, if all I'm trying to do is speed up my
> application, adding a new x86-64 backend or implementing support for
> CPython extension modules is certainly north of "prohibitively
> expensive". I stand by that wording. I'm willing to enumerate all of
> PyPy's deficiencies in this regard in the PEP, rather than the current
> vaguer wording, if you'd prefer.
>
>> while the other part is not completely true "slower
>> than US on non-numerical workloads". Fancy providing a proof for that?
>> I'm well aware that there are benchmarks on which PyPy is slower than
>> CPython or US, however, I would like a bit more weighted opinion in
>> the PEP.
>
> Based on the benchmarks you're running at
> http://codespeak.net:8099/plotsummary.html, PyPy is slower than
> CPython on many non-numerical workloads, which Unladen Swallow is
> faster than CPython at. Looking at the benchmarks there at which PyPy
> is faster than CPython, they are primarily numerical; this was the
> basis for the wording in the PEP.
>
> My own recent benchmarking of PyPy and Unladen Swallow (both trunk;
> PyPy wouldn't run some benchmarks):
>
> | Benchmark    | PyPy  | Unladen | Change          |
> +==+===+=+=+
> | ai           | 0.61  | 0.51    |  1.1921x faster |
> | django       | 0.68  | 0.8     |  1.1898x slower |
> | float        | 0.03  | 0.07    |  2.7108x slower |
> | html5lib     | 20.04 | 16.42   |  1.2201x faster |
> | pickle       | 17.7  | 1.09    | 16.2465x faster |
> | rietveld     | 1.09  | 0.59    |  1.8597x faster |
> | slowpickle   | 0.43  | 0.56    |  1.2956x slower |
> | slowspitfire | 2.5   | 0.63    |  3.9853x faster |
> | slowunpickle | 0.26  | 0.27    |  1.0585x slower |
> | unpickle     | 28.45 | 0.78    | 36.6427x faster |
>
> I'm happy to change the wording to "slower than US on some workloads".
>
> Thanks,
> Collin Winter
>

"slower than US on some workloads" is true, while not really telling
much to a potential reader. For any X and Y implementing the same
language "X is faster than Y on some workloads" is usually true.

To be precise you would need to include the above table in the PEP,
which is probably a bit too much, given that PEP is not about PyPy at
all. I'm fine with any wording that is at least correct.

Cheers,
fijal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

On Sat, Feb 13, 2010 at 12:12 AM, Maciej Fijalkowski  wrote:
> I like this wording far more. It's at the very least far more precise.
> Those examples are fair enough (except the fact that PyPy is not 32bit
> x86 only, the JIT is).
[snip]
> "slower than US on some workloads" is true, while not really telling
> much to a potential reader. For any X and Y implementing the same
> language "X is faster than Y on some workloads" is usually true.
>
> To be precise you would need to include the above table in the PEP,
> which is probably a bit too much, given that PEP is not about PyPy at
> all. I'm fine with any wording that is at least correct.

I've updated the language:
http://codereview.appspot.com/186247/diff2/9005:11001/11002. Thanks
for the clarifications.

Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com