Re: On-topic: alternate Python implementations

[Ulrich Eckhardt]

Am 04.08.2012 15:53, schrieb Stefan Behnel:

So, if a C++ compiler takes a .c file and compiles it with C language
semantics, it doesn't qualify as a C compiler? That implies a rather weird
definition of a C compiler, I'd say.


I'd say that even a brainfuck compiler compiling a .py file with C 
language semantics can shamelessly call itself a C compiler. :P


If a C++ compiler is given C code, it may or may not produce equivalent 
executables. In most non-trivial cases it will just barf on the valid C 
/ invalid C++ code and refuse to compile it. In few rare cases, it will 
compile the code and produce different behaviour at runtime (e.g. for 
"sizeof 'a'").



Uli
--
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[John Nagle]

On 8/4/2012 7:19 PM, Steven D'Aprano wrote:
> On Sat, 04 Aug 2012 18:38:33 -0700, Paul Rubin wrote:
> 
>> Steven D'Aprano  writes:
>>> Runtime optimizations that target the common case, but fall back to
>>> unoptimized code in the rare cases that the optimization doesn't apply,
>>> offer the opportunity of big speedups for most code at the cost of
>>> trivial slowdowns when you do something unusual.
>>
>> The problem is you can't always tell if the unusual case is being
>> exercised without an expensive dynamic check, which in some cases must
>> be repeated in every iteration of a critical inner loop, even though it
>> turns out that the program never actually uses the unusual case.

   There are other approaches. PyPy uses two interpreters and a JIT
compiler to handle the hard cases.  When code does something unexpected
to other code, the backup interpreter is used to get control out of
the trouble spot so that the JIT compiler can then recompile the
code.  (I think; I've read the paper but haven't looked at the
internals.)

   This is hard to implement and hard to get right.

John Nagle
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Jürgen A. Erhard, 05.08.2012 14:28:
> On Sun, Aug 05, 2012 at 07:46:59AM +0200, Stefan Behnel wrote:
>> Jürgen A. Erhard, 05.08.2012 01:25:
>>> None of the other implementations require Python for actually
>>> compiling or running Python source.
>>
>> Nuitka was on the list as well.
> 
> True, which I realized only after my missive.  But doesn't change
> much, only that the list is wrong.

Agreed.


> My definition, to also answer your following post, is "does not rely
> on any executable part of the CPython source (which includes .c files
> and executable code in header files if any, but of course can exclude
> the stdlib)".  Not sure that's precise enough, but... if it can't
> run/work on a system that has no shred of CPython installed, it's not
> an alternative *implementation*.

I can live with that definition. Cython is (by design) not an independent
reimplementation of Python.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

alex23, 06.08.2012 05:40:
> On Aug 4, 4:15 pm, Steven D'Aprano  +comp.lang.pyt...@pearwood.info> wrote:
>> But the Python ecosystem is a lot bigger than just those four. Here are
>> just a few other implementations that you might be interested in:
> 
> There's also HotPy:
> 
> http://code.google.com/p/hotpy/
> http://www.hotpy.org/

And just in case anyone was wondering where the others are:

http://wiki.python.org/moin/PythonImplementations

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[alex23]

On Aug 4, 4:15 pm, Steven D'Aprano  wrote:
> But the Python ecosystem is a lot bigger than just those four. Here are
> just a few other implementations that you might be interested in:

There's also HotPy:

http://code.google.com/p/hotpy/
http://www.hotpy.org/

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Ben Finney]

Mark Lawrence  writes:

> With arrogance like that German by any chance?

Please keep derogatory national stereotypes off this forum and out of
our community. They are counter to our goals of diversity
http://www.python.org/community/diversity/>; you don't have to
subscribe to that, but if not then you thereby exclude yourself.

-- 
 \   “You can stand tall without standing on someone. You can be a |
  `\ victor without having victims.” —Harriet Woods, 1927–2007 |
_o__)  |
Ben Finney
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Ethan Furman]

Mark Lawrence wrote:

With arrogance like that German by any chance?


Comments like that are not appropriate on this list.  Please don't make 
them.


~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Jürgen A . Erhard]

On Sun, Aug 05, 2012 at 07:46:59AM +0200, Stefan Behnel wrote:
> Jürgen A. Erhard, 05.08.2012 01:25:
> > On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote:
> >> Steven D'Aprano, 04.08.2012 08:15:
> >>> Most people are aware, if only vaguely, of the big Four Python 
> >>> implementations:
> >>
> >> And not to forget Cython, which is the only static Python compiler that is
> >> widely used. Compiles and optimises Python to C code that uses the CPython
> >> runtime and allows for easy manual optimisations to get C-like performance
> >> out of it.
> > 
> > Cython is certainly *not* a Python *implementation*, since it always
> > uses the CPython runtime (and compiling Cython C files requires
> > Python.h).
> 
> Yes, it avoids an unnecessary duplication of effort as well as a
> substantial loss of compatibility that all non-CPython based
> implementations suffer from.

But it's not an Python *implementation*, "just" an extension.

Mind you, this is not intended as a slight of Cython as such.  I
really like it, though I haven't had need for it yet, but I sure
prefer it to writing extensions in pure C. *b*

> > None of the other implementations require Python for actually
> > compiling or running Python source.
> 
> Nuitka was on the list as well.

True, which I realized only after my missive.  But doesn't change
much, only that the list is wrong.

> > Oh, yes, you can create a stand-alone... wait, a "stand-alone" app.
> > By embedding the Python runtime (dynamic linking with libpythonX.Y...
> > maybe static too?
> 
> Sure, that works.

My definition, to also answer your following post, is "does not rely
on any executable part of the CPython source (which includes .c files
and executable code in header files if any, but of course can exclude
the stdlib)".  Not sure that's precise enough, but... if it can't
run/work on a system that has no shred of CPython installed, it's not
an alternative *implementation*.  The big three don't need CPython
(except PyPy for building, and even it can use a precompile PyPy I think).

Grits, J
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Stefan Behnel, 05.08.2012 07:46:
> Jürgen A. Erhard, 05.08.2012 01:25:
>> None of the other implementations require Python for actually
>> compiling or running Python source.
> 
> Nuitka was on the list as well.

Oh, and Stackless was also on Steven's list, as well as WPython. That means
that 50% of the "other implementations" that Steven presented are not
"implementations" according to your apparent definition.

BTW, what is you definition?

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Jürgen A. Erhard, 05.08.2012 01:25:
> On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote:
>> Steven D'Aprano, 04.08.2012 08:15:
>>> Most people are aware, if only vaguely, of the big Four Python 
>>> implementations:
>>
>> And not to forget Cython, which is the only static Python compiler that is
>> widely used. Compiles and optimises Python to C code that uses the CPython
>> runtime and allows for easy manual optimisations to get C-like performance
>> out of it.
> 
> Cython is certainly *not* a Python *implementation*, since it always
> uses the CPython runtime (and compiling Cython C files requires
> Python.h).

Yes, it avoids an unnecessary duplication of effort as well as a
substantial loss of compatibility that all non-CPython based
implementations suffer from.

You'd be surprised to see how much of Python we implement, though,
including some of the builtins. You might want to revise your opinion once
you start digging into it. It's always easy to disagree at the surface.


> None of the other implementations require Python for actually
> compiling or running Python source.

Nuitka was on the list as well.


> Oh, yes, you can create a stand-alone... wait, a "stand-alone" app.
> By embedding the Python runtime (dynamic linking with libpythonX.Y...
> maybe static too?

Sure, that works.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Paul Rubin, 05.08.2012 03:38:
> Steven D'Aprano writes:
>> Runtime optimizations that target the common case, but fall back to 
>> unoptimized code in the rare cases that the optimization doesn't apply, 
>> offer the opportunity of big speedups for most code at the cost of 
>> trivial slowdowns when you do something unusual.
> 
> The problem is you can't always tell if the unusual case is being
> exercised without an expensive dynamic check, which in some cases must
> be repeated in every iteration of a critical inner loop, even though it
> turns out that the program never actually uses the unusual case.

Cython does a lot of optimistic optimisations. That's where a large part of
that huge C file comes from that Cython generates from even simple Python code.

For example, in CPython, C function calls are so ridiculously faster than
Python function calls that it's worth some effort if it saves you from
packing an argument tuple to call into a Python function. In fact, we've
been thinking about ways to export C signatures from Python function
objects, so that code implemented in C (or a C compatible language) can be
called directly from other code implemented in C. That's very common in the
CPython ecosystem.

There are a lot of simple things that quickly add up into a much better
performance on average.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Steven D'Aprano]

On Sat, 04 Aug 2012 18:38:33 -0700, Paul Rubin wrote:

> Steven D'Aprano  writes:
>> Runtime optimizations that target the common case, but fall back to
>> unoptimized code in the rare cases that the optimization doesn't apply,
>> offer the opportunity of big speedups for most code at the cost of
>> trivial slowdowns when you do something unusual.
> 
> The problem is you can't always tell if the unusual case is being
> exercised without an expensive dynamic check, which in some cases must
> be repeated in every iteration of a critical inner loop, even though it
> turns out that the program never actually uses the unusual case.

I never said optimizing Python was easy :)

Obviously if the check is expensive enough, the optimization isn't going 
to be worth doing. But often the check is not so expensive, or is just a 
matter of tedious and careful book-keeping.

I don't wish to dispute that optimizing Python is hard, but it's not a 
Hard Problem like factorizing huge integers, or solving the Palestine/
Israeli conflict. It's hard like cleaning your house after a gang of 
drunken frat boys have partied all weekend.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Paul Rubin]

Steven D'Aprano  writes:
> Runtime optimizations that target the common case, but fall back to 
> unoptimized code in the rare cases that the optimization doesn't apply, 
> offer the opportunity of big speedups for most code at the cost of 
> trivial slowdowns when you do something unusual.

The problem is you can't always tell if the unusual case is being
exercised without an expensive dynamic check, which in some cases must
be repeated in every iteration of a critical inner loop, even though it
turns out that the program never actually uses the unusual case.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Steven D'Aprano]

On Sat, 04 Aug 2012 08:59:18 -0700, Paul Rubin wrote:

> C isn't so great for high-assurance stuff either, compared to (say) Ada.
> People do use it in critical apps, but that's just because it is (or
> anyway used to be) so ubiquitous.

And then they are shocked, SHOCKED I say!, when their app has enough 
buffer overflow security vulnerabilities to sink a battleship.

[half a wink]


> Haskell doesn't sound all that great as a translation target for Python
> either, unfortunately, because its execution semantics are so different.

I have no opinion on that either way, except to say that if some 
developer wants to experiment with Python-in-Haskell, good on him or her. 
Trying something new is how progress is made.


[...]
> Finally, Python itself isn't all that well suited for compilation, given
> its high dynamicity.  It will be interesting to see if the language
> evolves due to PyPy.

Python is a dynamic language, but most Python code is relatively static. 
Runtime optimizations that target the common case, but fall back to 
unoptimized code in the rare cases that the optimization doesn't apply, 
offer the opportunity of big speedups for most code at the cost of 
trivial slowdowns when you do something unusual.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Jürgen A . Erhard]

On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote:
> Steven D'Aprano, 04.08.2012 08:15:
> > Most people are aware, if only vaguely, of the big Four Python 
> > implementations:
> > 
> 
> And not to forget Cython, which is the only static Python compiler that is
> widely used. Compiles and optimises Python to C code that uses the CPython
> runtime and allows for easy manual optimisations to get C-like performance
> out of it.

Cython is certainly *not* a Python *implementation*, since it always
uses the CPython runtime (and compiling Cython C files requires
Python.h).

None of the other implementations require Python for actually
compiling or running Python source.

Oh, yes, you can create a stand-alone... wait, a "stand-alone" app.
By embedding the Python runtime (dynamic linking with libpythonX.Y...
maybe static too?  Didn't test, because it's irrelevant for making the
point).

Grits, J
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[jwp]

On Friday, August 3, 2012 11:15:20 PM UTC-7, Steven D'Aprano wrote:
> WPython - another optimizing version of Python with wordcodes instead of 
> bytecodes.
> 
> http://code.google.com/p/wpython/

I remember reading about this a while ago. I thought this was eventually going 
to be committed to CPython... =\
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Paul Rubin, 04.08.2012 22:43:
> Stefan Behnel writes:
>>> Calling CPython hardly counts as compiling Python into C.
>> CPython is written in C, though. So anything that CPython does can be
>> done in C. It's not like the CPython project used a completely unusual
>> way of writing C code.
> 
> CPython is a relatively simple interpreter, and executing code 
> by invoking such an interpreter IMHO doesn't count as "compiling" it in
> any meaningful way.

Oh, CPython is substantially more than an interpreter. The eval loop is
only *one* way to use the runtime environment. Remember that it has many
builtin types and functions as well as a huge standard library. Much of the
runtime environment is already written in C or can be compiled down to C.
If you compile Python code into C code that avoids the eval loop and only
uses the CPython runtime environment (which is what Cython does), I think
that qualifies as compiling Python code to C. It's definitely the most
practical and user friendly way to do it.


>> You will always need some kind of runtime infrastructure when you
>> "compile Python into C", so you can just as well use CPython for that
>> instead of reimplementing it completely from scratch. 
> 
> Maybe there's parts of Cpython you can re-use, but having the CPython
> interpreter be the execution engine for "compiled" Python generators
> again fails the seriousness test of what it means to compile code.  If
> you mean something other than that, you might explain more clearly.

See above.


>> Both Cython and Nuitka do exactly that, 
> 
> I didn't know about Nuitka; it looks interesting but (at least after a
> few minutes looking) I don't have much sense of how it works.

It's mostly like Cython but without the type system, i.e. without all the
stuff that makes it useful in real life. Just a bare
Python-to-C++-in-CPython compiler, without much of a way to make it do what
you want.


>> Last I heard, PyPy had a couple of GCs to choose from,
> 
> PyPy doesn't compile to C

RPython (usually) does, though, and my guess is that the memory management
part of the runtime is written in RPython.


> but I guess compiling to C doesn't preclude
> precise GC, as long as the generated C code carefully tracks what C
> objects can contain GC-able pointers, and follows some constraints about
> when the GC can run.  Some other compilers do this so it's not as big a
> deal as it sounded like at first.  OK.

Yep, C really becomes a lot nicer when you generate it.


>> Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT
>> compiler specifically for Lua code. How is that similar to the Jython
>> runtime that runs *on top of* the JVM with its generic byte code based
>> JIT compiler?
> 
> I thought LuaJIT compiles the existing Lua VM code, but I haven't
> looked at it closely or used it.

Ok. It obviously reuses code, but the VM part of it is really different
from standard Lua.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Paul Rubin]

Stefan Behnel  writes:
>> Calling CPython hardly counts as compiling Python into C.
> CPython is written in C, though. So anything that CPython does can be
> done in C. It's not like the CPython project used a completely unusual
> way of writing C code.

CPython is a relatively simple interpreter, and executing code 
by invoking such an interpreter IMHO doesn't count as "compiling" it in
any meaningful way.  

> You will always need some kind of runtime infrastructure when you
> "compile Python into C", so you can just as well use CPython for that
> instead of reimplementing it completely from scratch. 

Maybe there's parts of Cpython you can re-use, but having the CPython
interpreter be the execution engine for "compiled" Python generators
again fails the seriousness test of what it means to compile code.  If
you mean something other than that, you might explain more clearly.

> Both Cython and Nuitka do exactly that, 

I didn't know about Nuitka; it looks interesting but (at least after a
few minutes looking) I don't have much sense of how it works.

> No, you are going to compile only the generator function into a function
> that uses gotos, maybe with an additional in-out struct parameter that
> holds its state.

Yeah, ok, I guess that can work, given python generators are limited
to returning through just one stack level.  You might want to avoid
copying locals by just putting everything into a struct, that has to
be retained across entries/exits.

> If you don't like that, you can experiment with anything from a dedicated
> GC to transactional memory.

OK, but then CPython is no longer managing the memory.

> Last I heard, PyPy had a couple of GCs to choose from,

PyPy doesn't compile to C, but I guess compiling to C doesn't preclude
precise GC, as long as the generated C code carefully tracks what C
objects can contain GC-able pointers, and follows some constraints about
when the GC can run.  Some other compilers do this so it's not as big a
deal as it sounded like at first.  OK.
>
>>> or provide none at all.
>> You're going to let the program just leak memory until it crashes??
> Well, it's not like CPython leaks memory until it crashes...

I was counting CPython's reference counting as a rudimentary form of GC,
though I guess that's terminology that not everyone agrees on.

> Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT
> compiler specifically for Lua code. How is that similar to the Jython
> runtime that runs *on top of* the JVM with its generic byte code based
> JIT compiler?

I thought LuaJIT compiles the existing Lua VM code, but I haven't
looked at it closely or used it.

>> It seems very hard to do reasonable optimizations in the presence of
>> standard Python techniques
>
> Sure. Even when targeting the CPython runtime with the generated C
> code (like Cython or Nuitka), you can still do a lot. And sure, static
> code analysis will never be able to infer everything that a JIT
> compiler can see.

I think even a JIT can't avoid a lot of pain and slowdown, without
complex whole-program analysis and requiring the application to follow
some special conventions, like never importing at "runtime".
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Tim Roberts]

Steven D'Aprano  wrote:
>
>Most people are aware, if only vaguely, of the big Four Python 
>implementations:
>
>CPython, or just Python, the reference implementation written in C.
>IronPython, written in .NET.

Technicality:  .NET is not a language, it is a run-time framework.
IronPython is written in C#.  It generates code that runs in the .NET
Framework.
-- 
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[MRAB]

On 04/08/2012 20:06, Stefan Behnel wrote:

Paul Rubin, 04.08.2012 20:18:

Stefan Behnel writes:

C is pretty poor as a compiler target: how would you translate Python
generators into C, for example?

Depends. If you have CPython available, that'd be a straight forward
extension type.


Calling CPython hardly counts as compiling Python into C.


CPython is written in C, though. So anything that CPython does can be done
in C. It's not like the CPython project used a completely unusual way of
writing C code.

Besides, I find your above statement questionable. You will always need
some kind of runtime infrastructure when you "compile Python into C", so
you can just as well use CPython for that instead of reimplementing it
completely from scratch. Both Cython and Nuitka do exactly that, and one of
the major advantages of that approach is that they can freely interact with
arbitrary code (Python or not) that was written for CPython, regardless of
its native dependencies. What good would it be to throw all of that away,
just for the sake of having "pure C code generation"?



For the yielding, you can use labels and goto. Given that you generate
the code, that's pretty straight forward as well.


You're going to compile the whole Python program into a single C
function so that you can do gotos inside of it?  What happens if the
program imports a generator?


No, you are going to compile only the generator function into a function
that uses gotos, maybe with an additional in-out struct parameter that
holds its state. Then, on entry, you read the label (or its ID) from the
previous state, reset local variables and jump to the label. On exit, you
store the state back end return. Cython does it that way. Totally straight
forward, as I said.



How would you handle garbage collection?

CPython does it automatically for us at least.


You mean you're going to have all the same INCREF/DECREF stuff on every
operation in compiled data?  Ugh.


If you don't like that, you can experiment with anything from a dedicated
GC to transactional memory.



Lacking that, you'd use one of the available garbage collection
implementations,


What implementations would those be?  There's the Boehm GC which is
useful for some purposes but not really suitable at large scale, from
what I can tell.  Is there something else?


No idea - I'll look it up when I need one. Last I heard, PyPy had a couple
of GCs to choose from, but I don't know how closely the are tied into its
infrastructure.



or provide none at all.


You're going to let the program just leak memory until it crashes??


Well, it's not like CPython leaks memory until it crashes, now does it? And
it's written in C. So there must be ways to handle this also in C.

Remember that CPython didn't even have a GC before something around 2.0,
IIRC. That worked quite ok in most cases and simply left the tricky cases
to the programmers. It really depends on what your requirements are. Small
embedded systems, time critical code and real-time systems are often much
better off without garbage collection. It's pure convenience, after all.


[snip]
CPython relied entirely on reference counting, so memory could leak you 
if inadvertently created a cycle of memory references. That problem was

fixed when a mark-and-sweep mechanism was added (it's called
occasionally to collect any unreachable cycles).

--
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Paul Rubin, 04.08.2012 20:18:
> Stefan Behnel writes:
>>> C is pretty poor as a compiler target: how would you translate Python
>>> generators into C, for example?
>> Depends. If you have CPython available, that'd be a straight forward
>> extension type.
> 
> Calling CPython hardly counts as compiling Python into C.

CPython is written in C, though. So anything that CPython does can be done
in C. It's not like the CPython project used a completely unusual way of
writing C code.

Besides, I find your above statement questionable. You will always need
some kind of runtime infrastructure when you "compile Python into C", so
you can just as well use CPython for that instead of reimplementing it
completely from scratch. Both Cython and Nuitka do exactly that, and one of
the major advantages of that approach is that they can freely interact with
arbitrary code (Python or not) that was written for CPython, regardless of
its native dependencies. What good would it be to throw all of that away,
just for the sake of having "pure C code generation"?


>> For the yielding, you can use labels and goto. Given that you generate
>> the code, that's pretty straight forward as well.
> 
> You're going to compile the whole Python program into a single C
> function so that you can do gotos inside of it?  What happens if the
> program imports a generator?

No, you are going to compile only the generator function into a function
that uses gotos, maybe with an additional in-out struct parameter that
holds its state. Then, on entry, you read the label (or its ID) from the
previous state, reset local variables and jump to the label. On exit, you
store the state back end return. Cython does it that way. Totally straight
forward, as I said.


>>> How would you handle garbage collection?
>> CPython does it automatically for us at least.
> 
> You mean you're going to have all the same INCREF/DECREF stuff on every
> operation in compiled data?  Ugh.

If you don't like that, you can experiment with anything from a dedicated
GC to transactional memory.


>> Lacking that, you'd use one of the available garbage collection
>> implementations,
> 
> What implementations would those be?  There's the Boehm GC which is
> useful for some purposes but not really suitable at large scale, from
> what I can tell.  Is there something else?

No idea - I'll look it up when I need one. Last I heard, PyPy had a couple
of GCs to choose from, but I don't know how closely the are tied into its
infrastructure.


>> or provide none at all.
> 
> You're going to let the program just leak memory until it crashes??

Well, it's not like CPython leaks memory until it crashes, now does it? And
it's written in C. So there must be ways to handle this also in C.

Remember that CPython didn't even have a GC before something around 2.0,
IIRC. That worked quite ok in most cases and simply left the tricky cases
to the programmers. It really depends on what your requirements are. Small
embedded systems, time critical code and real-time systems are often much
better off without garbage collection. It's pure convenience, after all.


>> you shouldn't expect too much of a performance gain from what the
>> platform gives you for the underlying implementation. It can optimise
>> the emulator, but it won't see enough of the Python code to make
>> anything efficient out of it. Jython is an example for that.
> 
> Compare that to the performance gain of LuaJIT and it starts to look
> like something is wrong with that approach, or maybe some issue inherent
> in Python itself.

Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT
compiler specifically for Lua code. How is that similar to the Jython
runtime that runs *on top of* the JVM with its generic byte code based JIT
compiler?

Basically, LuaJIT's JIT compiler works at the same level as the one in
PyPy, which is why both can theoretically provide the same level of
performance gains.


>> You can get pretty far with static code analysis, optimistic
>> optimisations and code specialisation.
> 
> It seems very hard to do reasonable optimizations in the presence of
> standard Python techniques like dynamically poking class instance
> attributes.  I guess some optimizations are still possible, like storing
> attributes named as literals in the program in fixed slots, saving some
> dictionary lookups even though the slot contents would have to still be
> mutable.

Sure. Even when targeting the CPython runtime with the generated C code
(like Cython or Nuitka), you can still do a lot. And sure, static code
analysis will never be able to infer everything that a JIT compiler can see.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Zero Piraeus]

:

On 4 August 2012 14:50, Mark Lawrence  wrote:
>
> No.  Next question?

*plonk*

 -[]z.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Mark Lawrence]

 From: Zero Piraeus 
To: Mark Lawrence  
Cc: python-list@python.org 
Sent: Saturday, 4 August 2012, 19:42
Subject: Re: On-topic: alternate Python implementations
 
:

On 4 August 2012 14:24, Mark Lawrence  wrote:
>
> With arrogance like that German by any chance?

I didn't give a monkeys about the beer conversation personally, but
can we leave the national stereotypes out of it?

-[]z.

No.  Next question?-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Zero Piraeus]

:

On 4 August 2012 14:24, Mark Lawrence  wrote:
>
> With arrogance like that German by any chance?

I didn't give a monkeys about the beer conversation personally, but
can we leave the national stereotypes out of it?

 -[]z.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Temia Eszteri]

On Sat, 04 Aug 2012 19:24:12 +0100, Mark Lawrence
 wrote:

>On 04/08/2012 11:59, Stefan Behnel wrote:
>> Mark Lawrence, 04.08.2012 12:05:
>>> I agree so it's off topic and can't be discussed here.  Isn't that right,
>>> Stefan?
>>
>> Hmm, in case you are referring to a recent friendly and diplomatic request
>> of mine regarding a couple of people who were burdening a public high
>> volume mailing list with a purely private back-and-forth chat about having
>> beer and getting drunk - then, no, I don't think the discussion in this
>> thread qualifies as yet another example for that so far.
>>
>> Stefan
>>
>>
>
>With arrogance like that German by any chance?

Hey now, cool it with the passive-aggression. We're here to discuss
code, right? If you want to fight it out, you can gladly do it by
e-mail.

~Temia
-- 
Invective! Verb your expletive nouns!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Mark Lawrence]

On 04/08/2012 11:59, Stefan Behnel wrote:

Mark Lawrence, 04.08.2012 12:05:

I agree so it's off topic and can't be discussed here.  Isn't that right,
Stefan?


Hmm, in case you are referring to a recent friendly and diplomatic request
of mine regarding a couple of people who were burdening a public high
volume mailing list with a purely private back-and-forth chat about having
beer and getting drunk - then, no, I don't think the discussion in this
thread qualifies as yet another example for that so far.

Stefan




With arrogance like that German by any chance?

--
Cheers.

Mark Lawrence.

--
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Paul Rubin]

Stefan Behnel  writes:
>> C is pretty poor as a compiler target: how would you translate Python
>> generators into C, for example?
> Depends. If you have CPython available, that'd be a straight forward
> extension type.

Calling CPython hardly counts as compiling Python into C.

> For the yielding, you can use labels and goto. Given that you generate
> the code, that's pretty straight forward as well.

You're going to compile the whole Python program into a single C
function so that you can do gotos inside of it?  What happens if the
program imports a generator?

>> How would you handle garbage collection?
> CPython does it automatically for us at least. 

You mean you're going to have all the same INCREF/DECREF stuff on every
operation in compiled data?  Ugh.

> Lacking that, you'd use one of the available garbage collection
> implementations,

What implementations would those be?  There's the Boehm GC which is
useful for some purposes but not really suitable at large scale, from
what I can tell.  Is there something else?

> or provide none at all.

You're going to let the program just leak memory until it crashes??

> you shouldn't expect too much of a performance gain from what the
> platform gives you for the underlying implementation. It can optimise
> the emulator, but it won't see enough of the Python code to make
> anything efficient out of it. Jython is an example for that.

Compare that to the performance gain of LuaJIT and it starts to look
like something is wrong with that approach, or maybe some issue inherent
in Python itself.

> You can get pretty far with static code analysis, optimistic
> optimisations and code specialisation.

It seems very hard to do reasonable optimizations in the presence of
standard Python techniques like dynamically poking class instance
attributes.  I guess some optimizations are still possible, like storing
attributes named as literals in the program in fixed slots, saving some
dictionary lookups even though the slot contents would have to still be
mutable.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Paul Rubin, 04.08.2012 17:59:
> Stefan Krah writes:
>> In the free software world, apparently many people like C. C is also
>> quite popular in the zero-fault software world: Several verification
>> tools do exist and Leroy et al. are writing a certified compiler for
>> C to plug the hole between the verified source code and the generated
>> assembly.
> 
> C is pretty poor as a compiler target: how would you translate Python
> generators into C, for example?

Depends. If you have CPython available, that'd be a straight forward
extension type. Otherwise, I guess you'd either have a class for them in
C++ or a struct in C. Not exactly complex.

For the yielding, you can use labels and goto. Given that you generate the
code, that's pretty straight forward as well.


> How would you handle garbage collection?

CPython does it automatically for us at least. Lacking that, you'd use one
of the available garbage collection implementations, or provide none at all.


> Haskell doesn't sound all that great as a translation target for Python
> either, unfortunately, because its execution semantics are so different.
> GHC is a very powerful compiler but it was made to compile Haskell code
> that people actually write, and may do less good of a job with compiler
> output from an imperative language like Python.  Compiling Python to
> Scheme and then using a Scheme compiler might be a more natural fit.
> But, compiling to Haskell was probably pretty convenient for that
> particular project.

You'd have some kind of emulation layer that does most of the translation
at runtime. That's why I said that you shouldn't expect too much of a
performance gain from what the platform gives you for the underlying
implementation. It can optimise the emulator, but it won't see enough of
the Python code to make anything efficient out of it. Jython is an example
for that.


> Finally, Python itself isn't all that well suited for compilation, given
> its high dynamicity.

You can get pretty far with static code analysis, optimistic optimisations
and code specialisation. We've decided against whole program analysis in
Cython not only for compiler complexity reasons but also because it would
let the normal compilation time explode for gains that you can much easier
get by manual optimisation. Obviously, optimising JIT compilers can do much
more here (because they actually have to do less), although they won't
always be able to figure out the right thing to do either. That's where
manual optimisation wins again.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Paul Rubin]

Stefan Krah  writes:
> In the free software world, apparently many people like C. C is also
> quite popular in the zero-fault software world: Several verification
> tools do exist and Leroy et al. are writing a certified compiler for
> C to plug the hole between the verified source code and the generated
> assembly.

C is pretty poor as a compiler target: how would you translate Python
generators into C, for example?  How would you handle garbage
collection?

C isn't so great for high-assurance stuff either, compared to (say) Ada.
People do use it in critical apps, but that's just because it is (or
anyway used to be) so ubiquitous.  I'm wondering what you mean about
verification tools, other than analyzers like Coverity that mainly check
for bugs that in a safer language would be caught by the compiler.
Compcert is not all that C-specific it has been adapted to compile a
Haskell-derived language (Habit).

Haskell doesn't sound all that great as a translation target for Python
either, unfortunately, because its execution semantics are so different.
GHC is a very powerful compiler but it was made to compile Haskell code
that people actually write, and may do less good of a job with compiler
output from an imperative language like Python.  Compiling Python to
Scheme and then using a Scheme compiler might be a more natural fit.
But, compiling to Haskell was probably pretty convenient for that
particular project.

Finally, Python itself isn't all that well suited for compilation, given
its high dynamicity.  It will be interesting to see if the language
evolves due to PyPy.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Stefan Behnel, 04.08.2012 15:53:
> Thomas Rachel, 04.08.2012 14:51:
>> Am 04.08.2012 11:10 schrieb Stefan Behnel:
>>> As long as you don't use any features of the Cython language, it's plain
>>> Python. That makes it a Python compiler in my eyes.
>>
>> Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C
>> and C++ are distinct languages and so are Python and Cython.
> 
> So, if a C++ compiler takes a .c file and compiles it with C language
> semantics, it doesn't qualify as a C compiler? That implies a rather weird
> definition of a C compiler, I'd say.

Ah, sorry. Got it. You were again talking about Cython the language. Sure,
Cython the language is different from Python the language. Cython the
compiler can compile both.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Thomas Rachel, 04.08.2012 14:51:
> Am 04.08.2012 11:10 schrieb Stefan Behnel:
>> As long as you don't use any features of the Cython language, it's plain
>> Python. That makes it a Python compiler in my eyes.
> 
> Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C
> and C++ are distinct languages and so are Python and Cython.

So, if a C++ compiler takes a .c file and compiles it with C language
semantics, it doesn't qualify as a C compiler? That implies a rather weird
definition of a C compiler, I'd say.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Ramchandra Apte]

The first time I did reply not 'reply all', so I'm posting again. ;-)
I think Cython is a Python implementation because you can only use the
Python features, not the extra features.
C++ is different because of the different rules (C was in a time of
assembly and costly computers, C++ was made in the time of (relatively)
cheaper computers and computers doing more things)

On 4 August 2012 18:21, Thomas Rachel <
nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa...@spamschutz.glglgl.de> wrote:

> Am 04.08.2012 11:10 schrieb Stefan Behnel:
>
>
>  As long as you don't use any features of the Cython language, it's plain
>> Python. That makes it a Python compiler in my eyes.
>>
>
> Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C
> and C++ are distinct languages and so are Python and Cython.
>
>
> Thomas
> --
> http://mail.python.org/**mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Thomas Rachel]

Am 04.08.2012 11:10 schrieb Stefan Behnel:


As long as you don't use any features of the Cython language, it's plain
Python. That makes it a Python compiler in my eyes.


Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, 
C and C++ are distinct languages and so are Python and Cython.



Thomas
--
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Steven D'Aprano, 04.08.2012 12:54:
> Berp is based on the Glasgow Haskell Compiler, which is a modern, 
> efficient, optimizing compiler capable of producing excellent quality 
> machine code on Windows, Mac, Linux and many Unixes. It gives you all the 
> advantages of a high-level language with high-level data structures, type 
> inference, and a compiler capable of generating optimized, fast, machine 
> code.

Although all those optimisations don't mean that Python code would run fast
on top of it. Just because you translate Python to another language and
platform doesn't mean that there's any benefit from the underlying platform
optimisations. Both PyPy and Cython run Python code faster than CPython,
but not because they eventually translate it into machine code but because
they optimise and specialise it along the way, based on its high-level code
constructs. One big success of the Unladen Swallow project was to show that
bare JIT compilation is mostly worthless for high level languages.


> Who would want to deal with C's idiosyncrasies, low-powered explicit type 
> system, difficult syntax, and core-dumps, when you could use something 
> better?

The core developers of both CPython and Cython aim for exactly that. They
write C so you don't have to. But keep in mind that C is still *the* lingua
franca of software development. A major reason why Python is (slowly)
catching up these days is that the main implementation is written in C and
makes it easy to interface with C code.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Krah]

Steven D'Aprano  wrote:
> Who would want to deal with C's idiosyncrasies, low-powered explicit type 
> system, difficult syntax, and core-dumps, when you could use something 
> better?

In the free software world, apparently many people like C. C is also
quite popular in the zero-fault software world: Several verification
tools do exist and Leroy et al. are writing a certified compiler for
C to plug the hole between the verified source code and the generated
assembly.


Stefan Krah



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Mark Lawrence, 04.08.2012 12:05:
> I agree so it's off topic and can't be discussed here.  Isn't that right,
> Stefan?

Hmm, in case you are referring to a recent friendly and diplomatic request
of mine regarding a couple of people who were burdening a public high
volume mailing list with a purely private back-and-forth chat about having
beer and getting drunk - then, no, I don't think the discussion in this
thread qualifies as yet another example for that so far.

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Steven D'Aprano]

On Sat, 04 Aug 2012 16:34:17 +1000, Chris Angelico wrote:

> On Sat, Aug 4, 2012 at 4:15 PM, Steven D'Aprano
>  wrote:
>> CLPython, an implementation of Python written in Common Lisp.
>>
>> Berp - a compiler which works by translating Python to Haskell and
>> compiling that.
> 
> Okay. WHY? CLPython gives some reason, but how often do you need to
> bridge that particular pair of languages? And why compile Python via
> Haskell, when C is available as a "high level assembly language"?

For much the same reason that PyPy uses RPython when C is available. 
Because Haskell is available as a high level non-assembly language.

Berp is based on the Glasgow Haskell Compiler, which is a modern, 
efficient, optimizing compiler capable of producing excellent quality 
machine code on Windows, Mac, Linux and many Unixes. It gives you all the 
advantages of a high-level language with high-level data structures, type 
inference, and a compiler capable of generating optimized, fast, machine 
code.

Who would want to deal with C's idiosyncrasies, low-powered explicit type 
system, difficult syntax, and core-dumps, when you could use something 
better? Apart from C programmers, of course.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Mark Lawrence]

On 04/08/2012 08:49, Steven D'Aprano wrote:

On Sat, 04 Aug 2012 08:40:16 +0200, Stefan Behnel wrote:


And not to forget Cython, which is the only static Python compiler that
is widely used. Compiles and optimises Python to C code that uses the
CPython runtime and allows for easy manual optimisations to get C-like
performance out of it.

http://cython.org/


Cython is great, but I question that it is a *Python* implementation.
That's not a criticism of Cython, but it is different from Python. Take
this example code from the tutorial:

from libc.math cimport sin

cdef double f(double x):
 return sin(x*x)

If that's Python code, then I'm Ethel the Aardvark.

Cython is very Python-like, but there is no doubt in my mind that it is a
superset of Python and therefore a different language.




I agree so it's off topic and can't be discussed here.  Isn't that 
right, Stefan?


--
Cheers.

Mark Lawrence.

--
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Steven D'Aprano, 04.08.2012 09:49:
> On Sat, 04 Aug 2012 08:40:16 +0200, Stefan Behnel wrote:
>> And not to forget Cython, which is the only static Python compiler that
>> is widely used. Compiles and optimises Python to C code that uses the
>> CPython runtime and allows for easy manual optimisations to get C-like
>> performance out of it.
>>
>> http://cython.org/
> 
> Cython is great, but I question that it is a *Python* implementation. 
> That's not a criticism of Cython, but it is different from Python. Take 
> this example code from the tutorial:
> 
> from libc.math cimport sin
> 
> cdef double f(double x):
> return sin(x*x)
> 
> If that's Python code, then I'm Ethel the Aardvark.

We never met in person, so I can't comment on the last part.

However, the above is Cython code and, yes, that's a different language.
Note that it uses a different file extension: ".pyx". Try putting the above
code into a .py file and compiling that. Cython will reject it and tell you
that "cimport" is not valid Python syntax.


> Cython is very Python-like, but there is no doubt in my mind that it is a 
> superset of Python and therefore a different language.

As long as you don't use any features of the Cython language, it's plain
Python. That makes it a Python compiler in my eyes. The fact that you can
easily use C features to optimise your code (also in Python syntax, BTW)
doesn't impact that.

You mentioned a couple of other implementations and I'm sure they have
features (and bugs) that CPython doesn't have. Interaction with Lisp code
in CLPython, for example. I don't think additional features or language
implementation bugs make a Python implementation non-Python per se.

Also note that most of the less widely known "alternative Python
implementations" do not even publish their results of running the CPython
test suite, so how do you actually know they can run Python code?

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Steven D'Aprano]

On Sat, 04 Aug 2012 08:40:16 +0200, Stefan Behnel wrote:

> And not to forget Cython, which is the only static Python compiler that
> is widely used. Compiles and optimises Python to C code that uses the
> CPython runtime and allows for easy manual optimisations to get C-like
> performance out of it.
> 
> http://cython.org/

Cython is great, but I question that it is a *Python* implementation. 
That's not a criticism of Cython, but it is different from Python. Take 
this example code from the tutorial:

from libc.math cimport sin

cdef double f(double x):
return sin(x*x)

If that's Python code, then I'm Ethel the Aardvark.

Cython is very Python-like, but there is no doubt in my mind that it is a 
superset of Python and therefore a different language.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Stefan Behnel]

Steven D'Aprano, 04.08.2012 08:15:
> Most people are aware, if only vaguely, of the big Four Python 
> implementations:
> 
> CPython, or just Python, the reference implementation written in C.
> IronPython, written in .NET.
> Jython, written in Java.
> PyPy, the optimizing implementation written in Python (actually, it's 
> written in a subset of Python, RPython).
> 
> But the Python ecosystem is a lot bigger than just those four. Here are 
> just a few other implementations that you might be interested in:
> 
> 
> Stackless - the "forgetten Python", Stackless is, I believe, the oldest 
> implementation behind only CPython itself. It's a fork of CPython with 
> the calling stack removed and fast and lightweight microthreads, and is 
> used extensively in EVE Online.
> 
> http://www.stackless.com/
> 
> 
> Nuitka - optimising Python compiler written in C++, supports Python 2.6 
> and 2.7, claims to be up to twice as fast as CPython.
> 
> http://nuitka.net/pages/overview.html
> 
> 
> WPython - another optimizing version of Python with wordcodes instead of 
> bytecodes.
> 
> http://code.google.com/p/wpython/
> 
> 
> CLPython, an implementation of Python written in Common Lisp.
> 
> http://common-lisp.net/project/clpython/
> 
> 
> CapPython is an experimental restricted version of Python with 
> capabilities.
> 
> http://plash.beasts.org/wiki/CapPython
> http://en.wikipedia.org/wiki/Object-capability_model
> 
> 
> Berp - a compiler which works by translating Python to Haskell and 
> compiling that.
> 
> https://github.com/bjpop/berp/wiki

And not to forget Cython, which is the only static Python compiler that is
widely used. Compiles and optimises Python to C code that uses the CPython
runtime and allows for easy manual optimisations to get C-like performance
out of it.

http://cython.org/

Stefan


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: On-topic: alternate Python implementations

[Chris Angelico]

On Sat, Aug 4, 2012 at 4:15 PM, Steven D'Aprano
 wrote:
> CLPython, an implementation of Python written in Common Lisp.
>
> Berp - a compiler which works by translating Python to Haskell and
> compiling that.

Okay. WHY? CLPython gives some reason, but how often do you need to
bridge that particular pair of languages? And why compile Python via
Haskell, when C is available as a "high level assembly language"?

The mind boggles...

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list