Re: Python PMC's
Sam Ruby wrote: Let me try again to move the discussion from subjective adjectives to objective code. Consider: [ example code ] If you run this, you will get 1,2,3. When called as a function, f will return the value of the second parameter. When called as a method, the same code will need to return the value of the first parameter. The calls to f() work more or less w/o problems in branches/leo-ctx5. I'm open to suggestions as to what PIR code should be emitted by Pirate to correspond to the above code. A stripped down PIR-only, pythonless translation is below. The way this currently works with Pirate and the existing Py*.pmc closely follows the Python definition, which you can find here: http://users.rcn.com/python/download/Descriptor.htm#functions-and-methods Good explanation, thanks. To illustrate how this works today, lets consider the call to foo.f(2) above. This involves both a find_method and an invoke. Lets consider each in turn: * pyclass.find_method first calls getprop on f. * pyclass.find_method then calls __get__ on the pyfunc returned, passing the object on which this method was called on. * pyfunc.__get__ creates a new PyBoundMeth object, saving both the original function and the object. * This PyBoundMeth object is then returned then: * pyboundmeth.invoke shifts all registers right, inserts the original object as the first parameter, and then calls invoke on the original function. Needless to say, this doesn't necessarily show off Parrot to its best advantage from a performance perspective. I tried a number of times to argue for a combined find_and_invoke VTABLE entry as much of the above can be optimized if you know that result of the find_method will only be used a single time for an invoke. If you like, you can scan the archives to see what reception this suggestion received. Well I think that the find_and_invoke is just the callmethodcc opcode as used in my translation below. The call to __get__ and the BoundMethod shouldn't be necessary, *if* their is no userdefined descriptor. That is, when getattribute checks, if the attribute provides __get__ then call it. The yet unsopported thing is g = foo.g ... g(3) which really needs a BoundSub object. But this is just a special case of currying, or a special case of Perl6's .assuming(). Therefore I think that Parrot should support this natively. - Sam Ruby leo # # def f(x,y): #return y .sub f .param pmc x .param pmc y .return (y) .end # # class Foo: .sub create_Foo .local pmc self, Foo self = subclass Py, Foo addattribute self, f addattribute self, g Foo = find_global Foo, Foo store_global Foo, Foo .end .namespace [Foo] .sub Foo .local pmc o o = new Foo .return (o) .end #f = f #def g(self,y): # return y .sub __init method .local pmc f, g f = find_name f setattribute self, f, f g = find_name g setattribute self, g, g .end .sub g .param pmc self .param pmc y .return (y) .end .namespace [] .sub main @MAIN .local pmc foo, g init_python() create_Foo() # # foo = Foo() # foo = Foo() # g=foo.g # g = getattribute foo, g # TODO create bound Sub inside gettattribute like so ## $I0 = isa g, Sub ## unless $I0 goto no_sub ## $P0 = new .BoundSub, g ## assign $P0, foo ## g = $P0 ## no_sub: # print f(0,1) $P0 = f(0, 1) print_item $P0 print_newline # print foo.f(2) # emulate python find_name, which checks attributes too push_eh m_nf $P0 = foo.f(2) clear_eh goto m_f m_nf: # getattribute would also check if __get__ is there $P1 = getattribute foo, f $P0 = foo.$P1(2) m_f: print_item $P0 print_newline # print g(3) #$P0 = g(3) #print_item $P0 #print_newline .end # .sub init_python .local pmc py, bs py = newclass Py .end
Re: Python PMC's
Leopold Toetsch wrote: Sam Ruby wrote: Let me try again to move the discussion from subjective adjectives to objective code. Consider: [ example code ] If you run this, you will get 1,2,3. When called as a function, f will return the value of the second parameter. When called as a method, the same code will need to return the value of the first parameter. The calls to f() work more or less w/o problems in branches/leo-ctx5. I'm open to suggestions as to what PIR code should be emitted by Pirate to correspond to the above code. A stripped down PIR-only, pythonless translation is below. Thanks for doing this. Keeping the conversation anchored with code is very helpful. I have a number of unimportant to the topic at hand comments on that code (example: classes aren't global in Python), but for now, I'll focus on comments that are relevant to the topic at hand. The way this currently works with Pirate and the existing Py*.pmc closely follows the Python definition, which you can find here: http://users.rcn.com/python/download/Descriptor.htm#functions-and-methods Good explanation, thanks. To illustrate how this works today, lets consider the call to foo.f(2) above. This involves both a find_method and an invoke. Lets consider each in turn: * pyclass.find_method first calls getprop on f. * pyclass.find_method then calls __get__ on the pyfunc returned, passing the object on which this method was called on. * pyfunc.__get__ creates a new PyBoundMeth object, saving both the original function and the object. * This PyBoundMeth object is then returned then: * pyboundmeth.invoke shifts all registers right, inserts the original object as the first parameter, and then calls invoke on the original function. Needless to say, this doesn't necessarily show off Parrot to its best advantage from a performance perspective. I tried a number of times to argue for a combined find_and_invoke VTABLE entry as much of the above can be optimized if you know that result of the find_method will only be used a single time for an invoke. If you like, you can scan the archives to see what reception this suggestion received. Well I think that the find_and_invoke is just the callmethodcc opcode as used in my translation below. The call to __get__ and the BoundMethod shouldn't be necessary, *if* their is no userdefined descriptor. That is, when getattribute checks, if the attribute provides __get__ then call it. Take a look at the definition of callmethodcc in ops/object.ops. It is a find_method followed by an invoke. It is the VTABLE operations that I would like to see combined. Note that Python provides __get__ methods for you on every function and method, i.e., it is *not* optional, getattribute will always succeed. Observe: [EMAIL PROTECTED]:~$ python Python 2.4.1 (#2, Mar 30 2005, 21:51:10) [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2 Type help, copyright, credits or license for more information. def f(): pass ... f.__get__ method-wrapper object at 0xb7e0404c class c: ... def m(self): pass ... c.m.__get__ method-wrapper object at 0xb7e0440c The yet unsopported thing is g = foo.g ... g(3) which really needs a BoundSub object. But this is just a special case of currying, or a special case of Perl6's .assuming(). Therefore I think that Parrot should support this natively. I agree that BoundSub is a special case of currying, or what Perl6 calls assuming. I also agree that Parrot should support this natively. - Sam Ruby [snip] Below is from the sample that Leo provided. # print foo.f(2) # emulate python find_name, which checks attributes too push_eh m_nf $P0 = foo.f(2) clear_eh goto m_f m_nf: # getattribute would also check if __get__ is there $P1 = getattribute foo, f $P0 = foo.$P1(2) m_f: print_item $P0 print_newline Note that the code above would need to be emitted for *every* method call, as there is no way of knowing at compile time what property you will get and how it is defined. In addition to increasing code size, it is error prone: for example, the above only appears to check for __get__ if the call fails, so the pie-thon b0 test will fail as hooking __get__ is integral to the way it provides a trace. Also, there are other reasons that exceptions may be thrown, and we are incurring the overhead of setting up exception blocks, etc again, this would apply to *every* method call. An alternate solution without these problems would be # print foo.f(2) $P0 = foo.f(2) print_item $P0 print_newline Ideally, callmethodcc would result in one vtable call to enable interesting optimizations for those that care to provide it. The default implementation for this vtable entry could be a call to find_method and invoke. - Sam Ruby
Re: [pirate] Re: Python PMC's
I agree this following would be cool. However in the general case this type of code inference is HARD to do. I believe that the optimizations you are looking for would require a combination of type inference and graph reduction. PyPy may be the eventual answer. Don't get me wrong, I think it is great and the long term goal. As soon as we compile and run correctly, I'm all about optimizations. Good Insight Michal, Kevin, Michal Wallace wrote: Hey Sam, I agree with what you're saying in this thread. This is slightly off topic, but I wanted to point something out. In general, python has so many places where things have to be dynamic that you really can't know this kind of thing at compile time, especially if you allow for eval/exec, or if you allow the code to be used as a module. However if you treat the code as a closed system, *and* you have access to python at compile time, then we can optimize away a lot of these questions. For example, in your original code: def f(x,y): return y class Foo: f = f def g(self,y): return y foo = Foo() g=foo.g print f(0,1) print foo.f(2) print g(3) Once you know how python works, it's *obvious* that this prints 1,2,3. I see no reason why the compiler couldn't figure this out up front just by walking the tree. In fact, a good optimizing compiler would see the return y lines and just get rid of those methods completely. I'd like to allow for the ability to do certain optimizations like this up front, sacrificing flexibility for speed. I know there are many programs that won't allow for this, but for the ones that do, I'd like to be able to do a sort of static compile like this. In other words, sometimes a python-like language is a desirable thing. (But of course this should all be optional so that we can also be 100% python compatible) Sincerely, Michal J Wallace Sabren Enterprises, Inc. - contact: [EMAIL PROTECTED] hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ - ___ pirate mailing list [EMAIL PROTECTED] http://cornerhost.com/mailman/listinfo/pirate
Re: Python PMC's
Sam Ruby wrote: Leopold Toetsch wrote: A stripped down PIR-only, pythonless translation is below. (example: classes aren't global in Python), Yes, of course. The stripped down means essential the absence of any lexical handlings. But as you say, this doesn't matter for these sub and method calls. Note that Python provides __get__ methods for you on every function and method, i.e., it is *not* optional, getattribute will always succeed. That's fine. But if it's not overriden by user code, you exactly know what it is doing. Therefore you can just emulate it, I think. # print foo.f(2) # emulate python find_name, which checks attributes too push_eh m_nf $P0 = foo.f(2) clear_eh goto m_f m_nf: # getattribute would also check if __get__ is there $P1 = getattribute foo, f $P0 = foo.$P1(2) m_f: Note that the code above would need to be emitted for *every* method call, Above snippet should not be emitted for a method call, it just emulates it in PIR. It should demonstrate how Python's find_method() could be implemented: - try to find a method, else - check attributes - call __get__, if it is user provided (or whatever order CPython actually uses). The return value is a callable sub. Ideally, callmethodcc would result in one vtable call to enable interesting optimizations for those that care to provide it. The default implementation for this vtable entry could be a call to find_method and invoke. The interesting optimizations are: - cache the find_method in the global method cache. This happens already, if the method string is constant. - use a PIC [1] (which is tied to the opcode) and cache the final resut of find_method (or any similar lookup) - run the invoked Sub inside the same run loop - specifically not within Parrot_run_meth_fromc_args [1] polymorphic inline cache The important thing is that the method lookup and the subroutine invocation happens from inside the runloop. Run cores that allow rewriting of opcodes (prederefed and JIT) can insert faster equivalent opcodes in these cases. Have a look at src/pic.c (which doesn't implement a lot - it's more a proof of concept now). Anyway here is an example that is implemented: new P0, PyInt# new_p_sc When this opcodes gets execute the first time, the type number of the class is looked up and then the opcode is replaced with the faster variant: new P0, 89 # new_p_ic - arbitrary number for PyInt The same scheme can be applied for all these opcodes that consist of any kind of a lookup (MMD, methods, attributes, ...) and some further action. The lookup is done once at runtime (and again only after cache invalidation). - Sam Ruby leo
Re: Python PMC's
Leopold Toetsch wrote: Sam Ruby wrote: Leopold Toetsch wrote: A stripped down PIR-only, pythonless translation is below. (example: classes aren't global in Python), Yes, of course. The stripped down means essential the absence of any lexical handlings. But as you say, this doesn't matter for these sub and method calls. Agreed. Note that Python provides __get__ methods for you on every function and method, i.e., it is *not* optional, getattribute will always succeed. That's fine. But if it's not overriden by user code, you exactly know what it is doing. Therefore you can just emulate it, I think. I'm not as sure as you appear to be, but we can worry about this later. # print foo.f(2) # emulate python find_name, which checks attributes too push_eh m_nf $P0 = foo.f(2) clear_eh goto m_f m_nf: # getattribute would also check if __get__ is there $P1 = getattribute foo, f $P0 = foo.$P1(2) m_f: Note that the code above would need to be emitted for *every* method call, Above snippet should not be emitted for a method call, it just emulates it in PIR. It should demonstrate how Python's find_method() could be implemented: With that clarification, I agree in principle. - try to find a method, else - check attributes - call __get__, if it is user provided (or whatever order CPython actually uses). What's in dynclass/pyclass.pmc matches Python's semantics closer. The return value is a callable sub. More precisely: a curried function call. This is an important distinction; to see why, see below. Ideally, callmethodcc would result in one vtable call to enable interesting optimizations for those that care to provide it. The default implementation for this vtable entry could be a call to find_method and invoke. The interesting optimizations are: - cache the find_method in the global method cache. This happens already, if the method string is constant. Note that you would then be caching the results of a curried function call. This result depends not only on the method string, but also on the particular object upon which it was invoked. - use a PIC [1] (which is tied to the opcode) and cache the final resut of find_method (or any similar lookup) Again, the results will depends on the object. - run the invoked Sub inside the same run loop - specifically not within Parrot_run_meth_fromc_args I don't understand this. (Note: it may not be important to this discussion that I do understand this - all that is important to me is that it works, and somehow I doubt that Parrot_run_meth_fromc_args cares whether a given function is curried or not). [1] polymorphic inline cache The important thing is that the method lookup and the subroutine invocation happens from inside the runloop. Run cores that allow rewriting of opcodes (prederefed and JIT) can insert faster equivalent opcodes in these cases. Have a look at src/pic.c (which doesn't implement a lot - it's more a proof of concept now). Anyway here is an example that is implemented: new P0, PyInt# new_p_sc When this opcodes gets execute the first time, the type number of the class is looked up and then the opcode is replaced with the faster variant: new P0, 89 # new_p_ic - arbitrary number for PyInt The same scheme can be applied for all these opcodes that consist of any kind of a lookup (MMD, methods, attributes, ...) and some further action. The lookup is done once at runtime (and again only after cache invalidation). The above works because PyInt is a constant. It probably can be extended to handle things that seem unlikely to change very rapidly. But the combination of decisions on how to handle the passing of the self parameter to a method, keeping find_method and invoke separated at the VTABLE level, and the semantics of Python make the notion of caching the results of find_method problematic. - Sam Ruby leo - Sam Ruby
Re: Python PMC's
On Aug 24, 2005, at 19:45, Sam Ruby wrote:
Leopold Toetsch wrote:
Sam Ruby wrote:
The return value is a callable sub.
More precisely: a curried function call. This is an important
distinction; to see why, see below.
A callable sub may be of course a curried one - yes.
The interesting optimizations are:
- cache the find_method in the global method cache.
This happens already, if the method string is constant.
Note that you would then be caching the results of a curried function
call. This result depends not only on the method string, but also on
the particular object upon which it was invoked.
No the inner Parrot_find_method_with_cache just caches the method for
a specific class (obeying C3 method resolution order as in Python).
There is no curried function at that stage.
- use a PIC [1] (which is tied to the opcode) and cache
the final resut of find_method (or any similar lookup)
Again, the results will depends on the object.
Yes. The nice thing with a PIC is that it is per bytecode location. You
have a different PIC and a different cache for every call site. The
prologue of PIC opcode is basically:
if cache.version == interpreter.version:
(cache.function)(args)
else:
# do dynamic lookup
# update cache then repeat
The 'version' compare depends on the cached thingy, and is more
explicit in individual implementations. But the principle remains
always the same: you create a unique id that depends on the variables
of the lookup and remember it. Before invoking the cached result you
compare actual with cached ids. If there is a cache miss, there are 2
or 3 more cache slots to consult before doing just the dynamic original
scheme again (and maybe rewrite the opcode again to just the dynamic
one in case of too many cache misses).
src/pic.c and ops/pic.ops have an implementation for sub Px, Py - just
for fun and profit to show the performance in the MOPS benchmark ;-)
The important part in ops/pic.ops is:
lr_types = (left-vtable-base_type 16) |
right-vtable-base_type;
if (lru-lr_type == lr_types) {
runit_v_pp:
((mmd_f_v_pp)lru-f.real_function)(interpreter, left, right);
(the lru is part of the cache structure, above code will be only run,
when both types are = 0x)
MMD depends on the two involved types. This is compared before calling
the cached function directly. As the cache is per bytecode location,
there is a fair chance (95 %) that the involved types for this very
code location are matching.
The same is true for plain method calls. The callee depends on the
invocant and the method name, which is usually a constant. Therefore
you can compare the cached version with the actual invocant type and
normally, with a match, just run the cached function immediately.
Currying is only important for placing the 'self' into the arguments -
the actual lookup was already done earlier and doesn't influence the
called function. Actually a curried subroutine isa 'Sub' object already
and invoked directly withouth any further method lookup. There is no
caching involved in the call of a curried sub.
- run the invoked Sub inside the same run loop - specifically
not within Parrot_run_meth_fromc_args
I don't understand this. (Note: it may not be important to this
discussion that I do understand this - all that is important to me is
that it works, and somehow I doubt that Parrot_run_meth_fromc_args
cares
whether a given function is curried or not).
There are two points to be considered:
- currying: the effect is that some call arguments (in this special
case the object) are already fixed. The argument passing code has
therefore the duty to insert these known (and remembered) arguments
into the params for the callee. For the BoundMethod this is of course,
shift all arguments up by one, and make the object 'self' the first
param of he sub.
- the second point is only related to call speed aka optimization. It's
just faster to run a PIR sub in the same run loop, then to create a new
run loop.
The above works because PyInt is a constant. It probably can be
extended to handle things that seem unlikely to change very rapidly.
Yes. That's the 'trick' behind PIC. It works best the more constant the
items are. But as said above, method names and invocants usually don't
vary *per byecode location*. Literature states ~95 % of method calls
are monomorphic (one type of invocant), and 99,5 % are cached within 4
cache slots. Look at some typical code
a.'foo'(x)
...
b.'bar'(y)
Both method calls have a distinct cache. The method names are constant.
Therefore the callee depends only on the invocant. The types of 'a' or
'b' are typically the same (except maybe inside compilers AST visit
methods or some such). The same schme applies to plain method or
attribute lookups.
But the combination of decisions on how to handle the passing of the
self parameter to a method, keeping find_method and
Re: Python PMC's
Leopold Toetsch wrote: Note that you would then be caching the results of a curried function call. This result depends not only on the method string, but also on the particular object upon which it was invoked. No the inner Parrot_find_method_with_cache just caches the method for a specific class (obeying C3 method resolution order as in Python). There is no curried function at that stage. Where does Parrot_find_method_with_cache get this data? Remember, in Python, there are no methods, there are only properties. Of course, there is a find_method VTABLE entry, and the implementation of this function calls __get__ which performs the curry function, per the Python specifications. Does Parrot_find_method_with_cache cache the results of the previous call to find_method? - Sam Ruby
Re: Python PMC's
On Aug 24, 2005, at 23:34, Sam Ruby wrote: Leopold Toetsch wrote: Note that you would then be caching the results of a curried function call. This result depends not only on the method string, but also on the particular object upon which it was invoked. No the inner Parrot_find_method_with_cache just caches the method for a specific class (obeying C3 method resolution order as in Python). There is no curried function at that stage. Where does Parrot_find_method_with_cache get this data? Remember, in Python, there are no methods, there are only properties. Above Parrot interface function tries to locate Sub objects in the namespace of the invocant's class and in the MRO of it. When you go back to the PIR translation of your code there is e.g. class Foo: def g(self,y): I have translated this to: .namespace [Foo] .sub g .param pmc self Therefore all the static / default / common methods inside Python code would be callable with the plain Parrot method call syntax. The classname contributes the namespace. The sub declaration creates an entry in that namespace, which is retrievable as: func = findglobal Foo, g And as the namespace is exactly the classname this is exactly what find_method for a Foo object does, when trying to locate a method named g (I'm omitting here further lookups according to MRO due to other parents). The actual namespace that the classes find_method looks into can further be finetuned with the vtable function VTABLE_namespace_name(interp, class). Other methods, like the one defined by f = f, would of course need a fallback to attribute lookup (please can we keep the term attribute, which is also used in CPython) - and not property). Therefore the 'find_method' inside Py code should also consult the attributes of the object (and yes properties for per object overrides). Well, this is at least my understanding how it could work. Of course, there is a find_method VTABLE entry, and the implementation of this function calls __get__ which performs the curry function, per the Python specifications. Yes. But currying isn't needed for a plain method call, when the __get__ isn't user defined. Why first curry the function in the first place, create a new curried sub PMC, then during invocation shift arguments, and eventually run it. This isn't the common case. Even if CPython does it like this now (and Python folks are discussing an optimized opcode on pthon-dev), it's not better or more correct - just the result is important. And that works as is now for Parrot method calls. Does Parrot_find_method_with_cache cache the results of the previous call to find_method? It does one dynamic lookup and caches per class/method the result. The cache is invalidated by a store_global (which could influence the dynamic search result). - Sam Ruby leo
Re: [pirate] Re: Python PMC's
On Wed, 24 Aug 2005, Sam Ruby wrote: [huge cut] Below is from the sample that Leo provided. # print foo.f(2) # emulate python find_name, which checks attributes too push_eh m_nf $P0 = foo.f(2) clear_eh goto m_f m_nf: # getattribute would also check if __get__ is there $P1 = getattribute foo, f $P0 = foo.$P1(2) m_f: print_item $P0 print_newline Note that the code above would need to be emitted for *every* method call, as there is no way of knowing at compile time what property you will get and how it is defined. Hey Sam, I agree with what you're saying in this thread. This is slightly off topic, but I wanted to point something out. In general, python has so many places where things have to be dynamic that you really can't know this kind of thing at compile time, especially if you allow for eval/exec, or if you allow the code to be used as a module. However if you treat the code as a closed system, *and* you have access to python at compile time, then we can optimize away a lot of these questions. For example, in your original code: def f(x,y): return y class Foo: f = f def g(self,y): return y foo = Foo() g=foo.g print f(0,1) print foo.f(2) print g(3) Once you know how python works, it's *obvious* that this prints 1,2,3. I see no reason why the compiler couldn't figure this out up front just by walking the tree. In fact, a good optimizing compiler would see the return y lines and just get rid of those methods completely. I'd like to allow for the ability to do certain optimizations like this up front, sacrificing flexibility for speed. I know there are many programs that won't allow for this, but for the ones that do, I'd like to be able to do a sort of static compile like this. In other words, sometimes a python-like language is a desirable thing. (But of course this should all be optional so that we can also be 100% python compatible) Sincerely, Michal J Wallace Sabren Enterprises, Inc. - contact: [EMAIL PROTECTED] hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ -
Re: Python PMC's
Leopold Toetsch wrote: Above Parrot interface function tries to locate Sub objects in the namespace of the invocant's class and in the MRO of it. When you go back to the PIR translation of your code there is e.g. class Foo: def g(self,y): I have translated this to: .namespace [Foo] .sub g .param pmc self Therefore all the static / default / common methods inside Python code would be callable with the plain Parrot method call syntax. The classname contributes the namespace. The sub declaration creates an entry in that namespace, which is retrievable as: func = findglobal Foo, g Honestly, I don't believe that that is workable for Python. Modules are global in Python. Classes are lexically scoped. All subs emitted by Pirate are @anon. Modules are namespaces and can contain classes, functions, and variables. The way to retrieve a method from a Python class defined in module __main__ would be: $P1 = findglobal __main__, Foo getattribute $P2, $P1, 'g' - Sam Ruby
Python PMC's
Problem: Python PMC's just don't work in the leo-cxt5 branch which will become head/trunk at some time in the hopefully not to distant future. What I've done up time now: I've ported pyint.pmc, pystring.pmc to pass all tests in leo-cxt5 I've written t/dynclasses/pystring.t Note that there are something like 15-20 python.pmc's or which only 4 have corresponding test in I've written t/dynclasses/ I've ripped out a lot of the explicit passing of self as the first argument of pmc methods. - We don't have to pass self around, parrot makes it available to us automatically. Sam's implementation of Python PMC is old and outdated. He did a lot of pioneering work in the realm of the python object system on parrot. Probably the first real attempt at a full object system on top of parrot. Parrot has changed drastically since then. Sam's PMC's were also incomplete, they lacked support for a lot of the __add__ style python methods. This is quite understandable, Sam was trying to get something working. Proposal: I propose to gut all of PyObject, PyFunc, PyClass, PyType PMC's etc. Old New PyObject PyObject - will represent a instantiation of a python class, will inherit from ParrotObject PyClassPyClass - will represent a python class, will inherit from ParrotClass PyType PyMetaClass - will represent the meta-class of a class, this one is still open for debate. I plan on making these changes via test driven development. The plan is to reduce python object system pmc's code content and functionality to zero, and then rebuild it up Mainly because every time I try to morph Sam's code, Leo tells me the whole approach is wrong :) We will rebuild the python object system based on test cases, If functionality is missing, and you want it implemented, write and submit a test case first. One last design note. I had a conversation a long time ago with Dan, in which he agreed that given Python's property bag style object system. That class methods and data members should be stored in parrot properties and the vtable getattribute/setattribute methods for python pmcs should be overridden to use parrot properties instead parrot attributes. Sam's PMC appeared to follow the same idea though not fully flushed out, and I plan on building the new Python PMC in the same manner. Comments welcome. Kevin Tew
Re: [pirate] Python PMC's
Kevin Tew wrote: Problem: Python PMC's just don't work in the leo-cxt5 branch which will become head/trunk at some time in the hopefully not to distant future. What I've done up time now: I've ported pyint.pmc, pystring.pmc to pass all tests in leo-cxt5 I've written t/dynclasses/pystring.t Note that there are something like 15-20 python.pmc's or which only 4 have corresponding test in I've written t/dynclasses/ I've ripped out a lot of the explicit passing of self as the first argument of pmc methods. - We don't have to pass self around, parrot makes it available to us automatically. I added self on Leo's request. Now it is unneccessary. *shrug* Sam's implementation of Python PMC is old and outdated. He did a lot of pioneering work in the realm of the python object system on parrot. Probably the first real attempt at a full object system on top of parrot. Parrot has changed drastically since then. Sam's PMC's were also incomplete, they lacked support for a lot of the __add__ style python methods. This is quite understandable, Sam was trying to get something working. Check out parrot/t/dynclass/pyint_2.pmc. __add__ style methods were working, and tested. Proposal: I propose to gut all of PyObject, PyFunc, PyClass, PyType PMC's etc. Old New PyObject PyObject - will represent a instantiation of a python class, will inherit from ParrotObject PyClassPyClass - will represent a python class, will inherit from ParrotClass PyType PyMetaClass - will represent the meta-class of a class, this one is still open for debate. I plan on making these changes via test driven development. The plan is to reduce python object system pmc's code content and functionality to zero, and then rebuild it up Mainly because every time I try to morph Sam's code, Leo tells me the whole approach is wrong :) We will rebuild the python object system based on test cases, If functionality is missing, and you want it implemented, write and submit a test case first. No matter what you do, Leo will tell you that the whole approach is wrong. I talked this past weekend with Larry Wall, Chip Saltzberg, and chromatic. Hopefully we will get some more people involved. Leo is great at details, but has a tendency to set a new direction every week. I say that knowing that Leo follows this list. (Hi Leo!) I do agree with test-driven development. That is exactly the approach I took. I will also state that I learned a lot *ABOUT* *PYTHON* in running the pie-thon tests. I'd recommend that you don't gut until you try the following: * PirateTest.py is a good sanity test to say that you have something resembling python to work with. Until that test passes, there isn't much point. * parrot/t/dynclasses give you a similar level of confidence in the pmcs. * Once you get that far, cd to parrot/languages/python and issue make. That will run the various tests (culiminating in the pie-thon tests themselves). To run that test, you will need a small batch file in your search PATH named pirate. Mine looks like: export LD_LIBRARY_PATH=/home/rubys/runtime/parrot/dynext python /home/rubys/pirate/pirate.py $* One last design note. I had a conversation a long time ago with Dan, in which he agreed that given Python's property bag style object system. That class methods and data members should be stored in parrot properties and the vtable getattribute/setattribute methods for python pmcs should be overridden to use parrot properties instead parrot attributes. Sam's PMC appeared to follow the same idea though not fully flushed out, and I plan on building the new Python PMC in the same manner. It just might be more fully flushed out than you appreciate. Comments welcome. I'm following this list, and questions are welcome in return. Again, my suggestion is to try to get the existing tests to work again - modifying the tests as necessary to remove now unnecessary self parameters. And, again, resist the temptation to follow Leo's advice du jour. Focus on what is actually implemented - while that does change over time, running code has a tendency to change more slowly than the advice you may receive. Kevin Tew ___ pirate mailing list [EMAIL PROTECTED] http://cornerhost.com/mailman/listinfo/pirate - Sam Ruby
Re: [pirate] Python PMC's
On Aug 23, 2005, at 17:09, Kevin Tew wrote: Problem: Python PMC's just don't work in the leo-cxt5 branch which will become head/trunk at some time in the hopefully not to distant future. Err, I hope to merge RSN ;-) I had a conversation a long time ago with Dan, in which he agreed that given Python's property bag style object system. That class methods and data members should be stored in parrot properties and the vtable getattribute/setattribute methods for python pmcs should be overridden to use parrot properties instead parrot attributes. This isn't quite right - or only partially I think. The main problem with properties is: props dont' inherit. It of course depends a lot on the Pyton - PIR translator. But I think the following strategy should work: - use parrot object system with a PyClass base class - make the compiler smarter about attribute access, it can create a list of attribute names - let these static attributes be handled by the parrot code - provide an additional hash slot (__dict__) as another internal attribute for dynamic attributes and per object attributes - Python __slots__ translates to the arrayish Parrot layout w/o changes probably - you have to check for __xxx__ attributes anyway to get e.g. overloading or other special attribute handling done correctly. Comments welcome. Kevin Tew leo
Re: [pirate] Python PMC's
On Tue, Aug 23, 2005 at 12:21:42PM -0400, Sam Ruby wrote: Kevin Tew wrote: I've ripped out a lot of the explicit passing of self as the first argument of pmc methods. - We don't have to pass self around, parrot makes it available to us automatically. I added self on Leo's request. Now it is unneccessary. *shrug* Parrot's calling conventions having updated, everybody must adjust, it's not about Pirate. I like to think that this most recent change in calling convention is an improvement, by and large. Sam's implementation of Python PMC is old and outdated. He did a lot of pioneering work in the realm of the python object system on parrot. Probably the first real attempt at a full object system on top of parrot. Parrot has changed drastically since then. Sam's PMC's were also incomplete, they lacked support for a lot of the __add__ style python methods. This is quite understandable, Sam was trying to get something working. Check out parrot/t/dynclass/pyint_2.pmc. __add__ style methods were working, and tested. I remember seeing the implementation of __*__. It looked fine to me in design, though I didn't review every particular one. At OSCON I learned that IronPython contrives to make CLR classes look just like native Python classes even under introspection, and that's just plain nifty. The Parrot equivalent would be to make the class look like it has an __add__ method iff the underlying Parrot class implements __add. Interesting? The plan is to reduce python object system pmc's code content and functionality to zero, and then rebuild it up Mainly because every time I try to morph Sam's code, Leo tells me the whole approach is wrong :) No matter what you do, Leo will tell you that the whole approach is wrong. It's a fair cop. Leo's always has strong ideas about how things should evolve. But in language-specific code like the Python classes, I don't see why Leo -- or I, for that matter -- should have standing to tell you what to do. If Python users are happy with the results, whatever they may be, and if the core functionality of Parrot isn't impacted for other languages, more power to you. If we got all unreasonable on you, you'd just take Pirate development out of the Parrot tree or else just quit, and neither of those alternatives is good for Parrot. As Larry says: Live and let learn. Therefore, if you _want_ to rebuild from zero, please by all means go ahead. Just be prepared to discover that the original structure was actually better than you thought. -- Chip Salzenberg [EMAIL PROTECTED]
Re: [pirate] Python PMC's
On Tue, Aug 23, 2005 at 06:35:39PM +0200, Leopold Toetsch wrote: On Aug 23, 2005, at 17:09, Kevin Tew wrote: Problem: Python PMC's just don't work in the leo-cxt5 branch which will become head/trunk at some time in the hopefully not to distant future. Err, I hope to merge RSN ;-) That's my Parrot priority this week: the leo-ctx5 branch. I know I like the feature set, and it's just a matter of reviewing the interface and implementation details. And, since the collective You are reading this message, which therefore must exist, it can be deduced that I have more Parrot bandwidth this week than I have for the past $TIMEPERIOD. :-) -- Chip Salzenberg [EMAIL PROTECTED]
Re: [pirate] Python PMC's
On Aug 23, 2005, at 18:21, Sam Ruby wrote:
Kevin Tew wrote:
We don't have to pass self around, parrot
makes it available to us automatically.
I added self on Leo's request. Now it is unneccessary. *shrug*
Parrot's new calling conventions are passing 'self' as the first
argument of a method. Therefore all the old workarounds like:
METHOD PMC* __abs__(Interp* interp, PMC* SELF, PMC* self) {}
aren't needed any more.
Sam's PMC's were also incomplete, they lacked support for a lot of the
__add__ style python methods. This is quite understandable, Sam was
trying to get something working.
Check out parrot/t/dynclass/pyint_2.pmc. __add__ style methods were
working, and tested.
PMC* __add__(PMC *self, PMC *value)
Please note that this signature includes like above already the
interpreter and, SELF i.e. the invocant PMC. This is for sure not
correct. The implementation is in pyobject.pmc and redispatches via MMD
to the actual multi sub.
So yes, the pyint.t test worked - overloading didn't.
No matter what you do, Leo will tell you that the whole approach is
wrong. I talked this past weekend with Larry Wall, Chip Saltzberg, and
chromatic. Hopefully we will get some more people involved. Leo is
great at details, but has a tendency to set a new direction every week.
I say that knowing that Leo follows this list. (Hi Leo!)
One year ago we had some discussion, where e.g. we both stated that the
object should be passed as first argument to methods. Now it's
eventually done in my branch.
I do agree with test-driven development. That is exactly the approach
I
took.
Well, I agree - until I see such tests (pyint_1):
$P2 = $P1.__abs__($P1)
- Sam Ruby
leo
Re: [pirate] Python PMC's
Leopold Toetsch wrote: I do agree with test-driven development. That is exactly the approach I took. Well, I agree - until I see such tests (pyint_1): $P2 = $P1.__abs__($P1) Take a look at the difference between r7254 and r7312. The tests originally passed without a self argument. Then I was told that that was all wrong, so I changed it (both the test and the code). Now Kevin is being told that passing self is all wrong. Reverting that particular change is not all that difficult. And it can be done without gutting. - Sam Ruby
Re: [pirate] Python PMC's
Sam, Thanks for the comments, They were very much appreciated. Sam Ruby wrote: I added self on Leo's request. Now it is unneccessary. *shrug* I understand completely. Check out parrot/t/dynclass/pyint_2.pmc. __add__ style methods were working, and tested. Yes many are working, I should clarify, there are more to be added. No matter what you do, Leo will tell you that the whole approach is wrong. I talked this past weekend with Larry Wall, Chip Saltzberg, and chromatic. Hopefully we will get some more people involved. Leo is great at details, but has a tendency to set a new direction every week. I say that knowing that Leo follows this list. (Hi Leo!) I do agree with test-driven development. That is exactly the approach I took. I will also state that I learned a lot *ABOUT* *PYTHON* in running the pie-thon tests. I'd recommend that you don't gut until you try the following: * PirateTest.py is a good sanity test to say that you have something resembling python to work with. Until that test passes, there isn't much point. * parrot/t/dynclasses give you a similar level of confidence in the pmcs. * Once you get that far, cd to parrot/languages/python and issue make. That will run the various tests (culiminating in the pie-thon tests themselves). To run that test, you will need a small batch file in your search PATH named pirate. Mine looks like: export LD_LIBRARY_PATH=/home/rubys/runtime/parrot/dynext python /home/rubys/pirate/pirate.py $* Point well taken, I've been leaning back between, rewrite and evolving your work. I have a current change set that is passing most of your original tests that are in t/dynclasses/py*.t It is more of an evolution than a rewrite, which I favor. Some test fail due to remaining uunimplemented features in leo-cxt5(default args), or my lack of understanding. But a lot of the tests pass. I wrote the original email, mainly because I didn't agree with all of leo's comment. I do however value his insight and like to hear his ideas. I have learned what I know about parrot in many cases due to pestering Leo with questions. Leo++. Without Leo, I'd still probably be more lost than I am now. That being said I will continue with the evolution path rather than a gut and rewrite. One last design note. I had a conversation a long time ago with Dan, in which he agreed that given Python's property bag style object system. That class methods and data members should be stored in parrot properties and the vtable getattribute/setattribute methods for python pmcs should be overridden to use parrot properties instead parrot attributes. Sam's PMC appeared to follow the same idea though not fully flushed out, and I plan on building the new Python PMC in the same manner. It just might be more fully flushed out than you appreciate. Your completely right. My apologies. Bad last sentence of the paragraph on my part. I was trying to get some explanation and/or debate on attributes vs properties, which Leo seems to have done in his reply. Comments welcome. I'm following this list, and questions are welcome in return. Again, my suggestion is to try to get the existing tests to work again - modifying the tests as necessary to remove now unnecessary self parameters. And, again, resist the temptation to follow Leo's advice du jour. Focus on what is actually implemented - while that does change over time, running code has a tendency to change more slowly than the advice you may receive. Will do, attached is my work in progress for those interested. Comments appreciated and welcome.
Re: [pirate] Python PMC's Missed attachment
My current work. Python PMC Patch to leo5-cxt5
Re: [pirate] Python PMC's
Kevin Tew wrote: Point well taken, I've been leaning back between, rewrite and evolving your work. I have a current change set that is passing most of your original tests that are in t/dynclasses/py*.t It is more of an evolution than a rewrite, which I favor. Some test fail due to remaining uunimplemented features in leo-cxt5(default args), or my lack of understanding. But a lot of the tests pass. Take a look at the following: Python 2.4.1 (#2, Mar 30 2005, 21:51:10) [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2 Type help, copyright, credits or license for more information. def f(a,b=1,c=2): ... pass ... f.func_defaults (1, 2) In December, this was fully implemented and tested. In pirate.py, search for self.append(setprop %s, 'func_defaults', %s % (ref, defaults)) and search for func_defaults in pyfunc.pmc, pyboundcall.pmc, and pybuiltin.pmc. Finally, take a look at the third test in parrot/languages/python/t/basic/func.t - Sam Ruby
defaults (was: Python PMC's)
How will Perl6 evaluate defaults? Like Python: global x x=1 def f(p1=x): return p1 x=2 print f() or like Ruby: $x=1 def f(p1=$x) return p1 end $x=2 puts f() - Sam Ruby P.S. The former prints 1, the latter, 2.
Re: [pirate] Python PMC's
On Aug 23, 2005, at 20:28, Sam Ruby wrote: Leopold Toetsch wrote: I do agree with test-driven development. That is exactly the approach I took. Well, I agree - until I see such tests (pyint_1): $P2 = $P1.__abs__($P1) Take a look at the difference between r7254 and r7312. I just find a log message about the change w/o any reason. I have always proposed (against Dan's opinion) the the object should be passed as the first argument to methods. This is done now. I've stated that the old MMD syntax, which needed to pass the existing destination PMC is inappropriate, suboptimal and whatever for almost all HLLs. Now you have a complete opcode subset that returns *new* destination PMCs. http://xrl.us/g9dp .pragma n_operators 1 ... $P0 = $P1 + $P2 # return new $P0 PMC Here is a test I've written for pycomplex: .pragma n_operators 1 .HLL Python, python_group .sub main @MAIN .local pmc d, c c = new .PyComplex c = 3+4j d = abs c print d print \n $S0 = typeof d print $S0 print \n .end And *yes* that creates a new PyFloat. Please also not that python_group is autoloaded. Sam, please follow Parrot dev (or stop spreading FUD) - thanks. The tests originally passed without a self argument. Then I was told that that was all wrong, so I changed it (both the test and the code). Please show me one message where you was told that abs has a signature of that one in the test or that another 'self' should be passed in. - Sam Ruby leo
Re: [pirate] Python PMC's
On Tue, Aug 23, 2005 at 09:58:21PM +0200, Leopold Toetsch wrote: Sam, please follow Parrot dev (or stop spreading FUD) - thanks. Be gentle, please. Parrot needs language developers. I'm not saying you're right or wrong. I'm just asking you to be a little more diplomatic. -- Chip Salzenberg [EMAIL PROTECTED]
Re: defaults (was: Python PMC's)
On Tue, Aug 23, 2005 at 03:48:03PM -0400, Sam Ruby wrote: How will Perl6 evaluate defaults? I would like to help with this question, but I can't, and in general p6i won't be the right place to ask. Nobody knows the final form of Perl 6; nobody knows the currently understood form of Perl 6 except for @Larry. You're likely to get a provisionally correct answer if you ask the perl6-language list. I can talk about Parrot, though. Parrot has no concept of default values. It does have a concept of optional parameters. How they're filled in is entirely an HLL decision. -- Chip Salzenberg [EMAIL PROTECTED]
Re: Python PMC's
On Aug 23, 2005, at 22:48, Sam Ruby wrote: From December 16, 2004: http://tinyurl.com/8smmq Sounds like a really ugly misunderstanding, the more that I've proposed not to pass the object (P2 in old parlance) out of band. I've stated several times that calling conventions need changes to properly support HLLs with minor success at these times. What I've always said (and what is actually implemented now in the branch) is that methods gets the object as its first argument. A typical snippet in PIR code is now .sub __abs .param pmc self or if denoted as method: .sub __abs method # 'self' param automagically available The first argument of NCI or vtable methods is and was always SELF, which usually is the invocant and always is the object (the current only excpetion is an invocation through super()). I'm sorry if this caused confusion. - Sam Ruby leo
Re: Python PMC's
Leopold Toetsch wrote: On Aug 23, 2005, at 22:48, Sam Ruby wrote: From December 16, 2004: http://tinyurl.com/8smmq Sounds like a really ugly misunderstanding, the more that I've proposed not to pass the object (P2 in old parlance) out of band. I've stated several times that calling conventions need changes to properly support HLLs with minor success at these times. With the diversity of HLLs out there, I'm not certain that it is wise to declare one set of semantics proper, thereby implicitly declaring all others improper. What I've always said (and what is actually implemented now in the branch) is that methods gets the object as its first argument. A typical snippet in PIR code is now .sub __abs .param pmc self or if denoted as method: .sub __abs method # 'self' param automagically available The first argument of NCI or vtable methods is and was always SELF, which usually is the invocant and always is the object (the current only excpetion is an invocation through super()). It might help to anchor this discussion if we focus on a tangible test case. In particular, lets look at the first few lines of the first pie-thon test, which can be found here: http://svn.perl.org/parrot/trunk/languages/python/t/pie/b0.t What code should be generated for the declaration of proto___new__ or proto___repr__ functions? What code should be generated for the various assignment statements in the declaration of the method __new__ inside the MetaToken class? Particularly, the one involving a staticmethod? The conclusion I came to is that there is no way to tell in advance how a given function or method in Python is ever going to be called as there effectively is no difference between the two. - Sam Ruby
Re: Python PMC's
Chip Salzenberg wrote: On Tue, Aug 23, 2005 at 07:15:41PM -0400, Sam Ruby wrote: Leopold Toetsch wrote: I've stated several times that calling conventions need changes to properly support HLLs with minor success at these times. With the diversity of HLLs out there, I'm not certain that it is wise to declare one set of semantics proper, thereby implicitly declaring all others improper. Well, let's try more coherent and less confusing, then, or maybe just better. HLLs don't have to expose the 'self' parameter if they don't want to. Just extract it into a PMC register that's never used. Let me try again to move the discussion from subjective adjectives to objective code. Consider: def f(x,y): return y class Foo: f = f def g(self,y): return y foo = Foo() g=foo.g print f(0,1) print foo.f(2) print g(3) If you run this, you will get 1,2,3. When called as a function, f will return the value of the second parameter. When called as a method, the same code will need to return the value of the first parameter. I'm open to suggestions as to what PIR code should be emitted by Pirate to correspond to the above code. The way this currently works with Pirate and the existing Py*.pmc closely follows the Python definition, which you can find here: http://users.rcn.com/python/download/Descriptor.htm#functions-and-methods To illustrate how this works today, lets consider the call to foo.f(2) above. This involves both a find_method and an invoke. Lets consider each in turn: * pyclass.find_method first calls getprop on f. * pyclass.find_method then calls __get__ on the pyfunc returned, passing the object on which this method was called on. * pyfunc.__get__ creates a new PyBoundMeth object, saving both the original function and the object. * This PyBoundMeth object is then returned then: * pyboundmeth.invoke shifts all registers right, inserts the original object as the first parameter, and then calls invoke on the original function. Needless to say, this doesn't necessarily show off Parrot to its best advantage from a performance perspective. I tried a number of times to argue for a combined find_and_invoke VTABLE entry as much of the above can be optimized if you know that result of the find_method will only be used a single time for an invoke. If you like, you can scan the archives to see what reception this suggestion received. Precisely emulating the behavior above is an important part of distinguishing a Python-like language from a true Python language, and is necessary to pass the b0 pie-thon test (grep for __get__ to see what I mean). As to whether this is more coherent and less confusing, then, or maybe just better... well, lets just say that I'll leave the adjectives to others. - Sam Ruby
Re: Python PMC's
On Tue, Aug 23, 2005 at 07:15:41PM -0400, Sam Ruby wrote: Leopold Toetsch wrote: I've stated several times that calling conventions need changes to properly support HLLs with minor success at these times. With the diversity of HLLs out there, I'm not certain that it is wise to declare one set of semantics proper, thereby implicitly declaring all others improper. Well, let's try more coherent and less confusing, then, or maybe just better. HLLs don't have to expose the 'self' parameter if they don't want to. Just extract it into a PMC register that's never used. -- Chip Salzenberg [EMAIL PROTECTED]
Re: Python PMC's
I apologize to Leo for accidentally making this reply to the list. It was supposed to be private mail, but I hit 'y' just a _little_ too soon. I had no intention of embarassing anyone. Sorry. On Tue, Aug 23, 2005 at 01:04:58PM -0700, Chip Salzenberg wrote: On Tue, Aug 23, 2005 at 09:58:21PM +0200, Leopold Toetsch wrote: Sam, please follow Parrot dev (or stop spreading FUD) - thanks. Be gentle, please. Parrot needs language developers. I'm not saying you're right or wrong. I'm just asking you to be a little more diplomatic. -- Chip Salzenberg [EMAIL PROTECTED] -- Chip Salzenberg [EMAIL PROTECTED]
Re: defaults (was: Python PMC's)
On Tue, Aug 23, 2005 at 03:48:03PM -0400, Sam Ruby wrote: : How will Perl6 evaluate defaults? : : Like Python: : : global x : x=1 : def f(p1=x): : return p1 : x=2 : print f() : : or like Ruby: : : $x=1 : def f(p1=$x) : return p1 : end : $x=2 : puts f() By default, delayed like Ruby, but you can easily force earlier evaluation if that's what you want. In general, pseudo-assignment to declarators is special-cased to happen at the time appropriate to the scope and lifetime of the declarator, and formal parameters are no exception. Basically, the container constructor is passed the closure of what's on the right of the =, and it can choose to evaluate that closure whenever it wants. For instance, state $s = $x; sets $s to the current value of $x only the first time it's executed. So while formal parameter defaults execute as late as possible, and state variable initializations run just in time, object attribute defaults actually run right away, that is, class declaration time (aka BEGIN time), and you have to add braces to delay evaluation till object init time. Arguably that's a little inconsistent, but the inconsistency can be viewed as implicit to the declarator, not the pseudo-assignment syntax, which is just syntactic sugar for a closure. Larry
Re: Python PMC's
Chip Salzenberg wrote: I apologize to Leo for accidentally making this reply to the list. It was supposed to be private mail, but I hit 'y' just a _little_ too soon. I had no intention of embarassing anyone. Sorry. You did, however, cause me to cancel the email I was composing. If people only respond in private to emails such as the one that you responded to, the public perception is that emails such as Leo's are acceptable. In any case, here is a shorter version of what I said in the email I cancelled: From December 16, 2004: http://tinyurl.com/8smmq The commit message for December 17, 2004: r7312 | rubys | 2004-12-17 22:51:48 -0500 (Fri, 17 Dec 2004) | 2 lines Pass 'self' as the first argument on method calls - Sam Ruby
