bound methods (was: Calling conventions, invocations, and suchlike things)
Dan Sugalski [EMAIL PROTECTED] wrote: At 5:04 PM -0500 1/18/05, Sam Ruby wrote: f = Parrot.find print f(r) Note that I referenced the method as an attribute, and then called it as a function. Mmm, syntax! :) Luckily it makes no difference to us at the parrot level. What that should translate to is something like: $P0 = find_method Parrot_string, find # Elided check for failed lookup and fallback to attribute fetch $P1 = make_bound_method(Parrot_string, $P0) Not quite. It's just: f = getattribute Parrot_string, find nothing more. The Cget_attr_str vtable has to do the right thing, i.e. if the attribute is a callable, it has to return a bound method object. Furthermore, the function remembers what object it is bound to. This is accomplished by VTABLE_find_method creating a new PyBoundMeth PMC which contains two references, one to the object, and one to the method. While a good idea, I think it's not the right way to handle this. Binding objects to methods to create invokable subs is going to be something we're going to need for a lot of the languages, so I think we'd be better served providing a general facility to do it rather than leaving it to each individual language designer to do it. Should save some work all around too. Yeah. When this came up last, I've proposed two ways to handle it: 1) inside the Sub/NCI PMC 2) by a distinct Bound_Meth PMC class derived from 1) The latter is probably cleaner. Binding the object to the callable could be done e.g. by the Cset_pmc vtable. leo
Re: Calling conventions, invocations, and suchlike things
At 5:04 PM -0500 1/18/05, Sam Ruby wrote: Dan Sugalski wrote: Hi folks. Welcome back! Parrot's got the interesting, and somewhat unfortunate, requirement of having to allow all subroutines behave as methods and all methods behave as subroutines. (This is a perl 5 thing, but we have to make it work) That is, an invokable PMC may be invoked as a method call and passed in an object, or as a plain subroutine and not have an object passed in. As far as perl 5 is concerned the object is the first parameter in the argument list, but for everyone else the object is a very distinct and separate thing. Python essentially has the same requirement, with a few twists. Specifically, methods come in be static, class, and regular flavors. But first, a simple example. Strings in python have a find method, so and can do the following: f = Parrot.find print f(r) Note that I referenced the method as an attribute, and then called it as a function. Mmm, syntax! :) Luckily it makes no difference to us at the parrot level. What that should translate to is something like: $P0 = find_method Parrot_string, find # Elided check for failed lookup and fallback to attribute fetch $P1 = make_bound_method(Parrot_string, $P0) $P1(r) Furthermore, the function remembers what object it is bound to. This is accomplished by VTABLE_find_method creating a new PyBoundMeth PMC which contains two references, one to the object, and one to the method. While a good idea, I think it's not the right way to handle this. Binding objects to methods to create invokable subs is going to be something we're going to need for a lot of the languages, so I think we'd be better served providing a general facility to do it rather than leaving it to each individual language designer to do it. Should save some work all around too. Static methods differ in that the object is not passed. How is this different from a subroutine, then? Class methods differ in that the object passed is actually the class of the object in question. I'm assuming this is different from just a method on the class somehow? Note: all this is determined by the callee. It is all transparent to the caller. This is the part I'm not so sure about. It looks like, rather than having two sides (caller and calle) we have three, caller, callee, and the code that fetches the invokable in the first place. I fully agree that the caller shouldn't know about this stuff, since it may well have been handed the invokable thing as part of a function call or pulled it out of a variable or something. I don't think the callee should have to know anything special here, though -- it doesn't seem at all unreasonable to have the callee *not* have to do anything special, nor play any magic games. (And I think I'd be a bit peeved if I was writing code which passed in object A as the object being invoked on, but the method decided it wanted to use object B instead) This is especially true in a mixed-language environment when you've got a class with methods written in different languages -- setting up any conventions that'll actually be followed seems like an exercise in futility. :) That leaves the code that actually fetches the invokable thing in the first place, and that seems like the right place for this to happen. The language the code is written in knows what should happen based on what it gets back when querying the object, so as long as we provide a standard means to do all the binding stuff, we shoul dbe fine. First, observe that I don't have any control over the exception that is raised when a method is not found (fix: raise the exception within find_method). Right. There's going to be one generic method-not-found exception -- there really has to be only one, otherwise we're going to run into all sorts of cross-language problems. Exception unification (and, more likely, aliasing) is going to be one of the tricky issues. My one minor request here is P2 be made available on entry to the invoked method. This would remove some special case logic for me requiring the use of interpinfo. I don't expect any guarantees that this is preserved or restored across sub calls. The one thing that leaving it in the interpreter structure and not explicitly passing it in gets us is we get notice if its actually extracted and used. Which is going to be fairly common, so I'm not sure what it buys us. I think we'll leave things as-is, but I'm not sure for how much longer. Not having objects handle their own method dispatch is less clear-cut, but I do have some reasons, so here they are. Just be aware that in order to preserve Python semantics, find_method will need to return a bound method. That can't happen. find_method has to return an unbound method, since there are just too many cases where that's what we need. If the method then needs to be bound then the fetching code can do the binding. This involves creating an object on the heap,
Re: Calling conventions, invocations, and suchlike things
Dan Sugalski wrote: At 5:04 PM -0500 1/18/05, Sam Ruby wrote: Dan Sugalski wrote: Hi folks. Welcome back! Parrot's got the interesting, and somewhat unfortunate, requirement of having to allow all subroutines behave as methods and all methods behave as subroutines. (This is a perl 5 thing, but we have to make it work) That is, an invokable PMC may be invoked as a method call and passed in an object, or as a plain subroutine and not have an object passed in. As far as perl 5 is concerned the object is the first parameter in the argument list, but for everyone else the object is a very distinct and separate thing. Python essentially has the same requirement, with a few twists. Specifically, methods come in be static, class, and regular flavors. But first, a simple example. Strings in python have a find method, so and can do the following: f = Parrot.find print f(r) Note that I referenced the method as an attribute, and then called it as a function. Mmm, syntax! :) Luckily it makes no difference to us at the parrot level. What that should translate to is something like: $P0 = find_method Parrot_string, find # Elided check for failed lookup and fallback to attribute fetch $P1 = make_bound_method(Parrot_string, $P0) $P1(r) This will be a recurring theme in my replies. Any thing which presumes a bit of knowledge at compile time will ultimately not work. Consider the following: class c: find = 7 def f(x): return x.find print f(c()) print f(Parrot)(r) Now, what should the code for function f look like? The only reasonable answer is something along the lines of: getattribute $P0, P5, 'find' This has to work. In both of the two calls to f(). Furthermore, the function remembers what object it is bound to. This is accomplished by VTABLE_find_method creating a new PyBoundMeth PMC which contains two references, one to the object, and one to the method. While a good idea, I think it's not the right way to handle this. Binding objects to methods to create invokable subs is going to be something we're going to need for a lot of the languages, so I think we'd be better served providing a general facility to do it rather than leaving it to each individual language designer to do it. Should save some work all around too. This would not be necessary, but the current implementation of the callmethodcc opcode unfortunately decouples the find_method (which is subject to the not at compile time restrictions alluded to above), and invoke. If these weren't decoupled (i.e., there was a VTABLE_find_method or equivalent entry), then this would not be necessary. Static methods differ in that the object is not passed. How is this different from a subroutine, then? From a callee-perspective: not at all. What is important to note is that from a caller-perspective, they will invoke such subroutines with the callmethcc opcode. Class methods differ in that the object passed is actually the class of the object in question. I'm assuming this is different from just a method on the class somehow? From a callee perspective, this appears to be a method on the instance. Note: all this is determined by the callee. It is all transparent to the caller. This is the part I'm not so sure about. It looks like, rather than having two sides (caller and calle) we have three, caller, callee, and the code that fetches the invokable in the first place. I fully agree that the caller shouldn't know about this stuff, since it may well have been handed the invokable thing as part of a function call or pulled it out of a variable or something. I don't think the callee should have to know anything special here, though -- it doesn't seem at all unreasonable to have the callee *not* have to do anything special, nor play any magic games. (And I think I'd be a bit peeved if I was writing code which passed in object A as the object being invoked on, but the method decided it wanted to use object B instead) This is especially true in a mixed-language environment when you've got a class with methods written in different languages -- setting up any conventions that'll actually be followed seems like an exercise in futility. :) Perhaps you might not find Python to your liking. That's OK. But the more general question is whether or not Parrot will implement the above as a policy, and thereby preclude langages like Python from being implemented on top of Parrot. That leaves the code that actually fetches the invokable thing in the first place, and that seems like the right place for this to happen. The language the code is written in knows what should happen based on what it gets back when querying the object, so as long as we provide a standard means to do all the binding stuff, we shoul dbe fine. I'm not sure what you mean by the code that actually fetches the invokable thing in this instance. If you mean an at compile time translation approach like you alluded above,
Re: Calling conventions, invocations, and suchlike things
Luke Palmer wrote: Sam Ruby writes: Mmm, syntax! :) Luckily it makes no difference to us at the parrot level. What that should translate to is something like: $P0 = find_method Parrot_string, find # Elided check for failed lookup and fallback to attribute fetch $P1 = make_bound_method(Parrot_string, $P0) $P1(r) This will be a recurring theme in my replies. Any thing which presumes a bit of knowledge at compile time will ultimately not work. Consider the following: class c: find = 7 def f(x): return x.find print f(c()) print f(Parrot)(r) Now, what should the code for function f look like? The only reasonable answer is something along the lines of: getattribute $P0, P5, 'find' I doubt that. All languages have different semantics, and we can't implement them all, because they are conflicting. You, as a compiler designer, have the opportunity to design things so that they work. And you definitely have to be clever if you're looking for language features that Parrot doesn't natively support. I disagree. As Dan once said, as long as we have a proper protocol that everyone can conform to, we should be OK. I don't care if Perl, Python, Ruby, TCL, and others each implement different semantics. I do care that we adopt a common protocol. The current set of VTABLE entries is a excellent first order approximation for the common protocol. My first tendency here is to echo all attributes as methods (like Perl 6 does), and then always call the method when you see a dot. ParrotString has a find method, and it knows how to curry itself. Instances of c also have a find method, and that method always returns 7. The distinction between attributes and methods is a bit subtle. Any object may implement the VTABLE_invoke method (or in Python terms, a __call__ method), so all attributes may already *BE* a method. But largely, I could change the current implementation of Python on Parrot to follow such a protocol in a matter of minutes. But my original comment (Any approach 'which presumes a bit of knowledge at compile time will ultimately not work.') still stands. Although I agree that we should come up with a general, bare-bones object model that allows all of our current target languages to operate smoothly. Parrot's current model makes far too many assumptions. But that doesn't mean that anything you're trying to do is impossible; it just means it's harder. I don't believe that Parrot should impose an object model. I return to Dan's as long as we have a proper protocol that everyone can conform to, we should be OK. - Sam Ruby
Re: Calling conventions, invocations, and suchlike things
Sam Ruby writes: Mmm, syntax! :) Luckily it makes no difference to us at the parrot level. What that should translate to is something like: $P0 = find_method Parrot_string, find # Elided check for failed lookup and fallback to attribute fetch $P1 = make_bound_method(Parrot_string, $P0) $P1(r) This will be a recurring theme in my replies. Any thing which presumes a bit of knowledge at compile time will ultimately not work. Consider the following: class c: find = 7 def f(x): return x.find print f(c()) print f(Parrot)(r) Now, what should the code for function f look like? The only reasonable answer is something along the lines of: getattribute $P0, P5, 'find' I doubt that. All languages have different semantics, and we can't implement them all, because they are conflicting. You, as a compiler designer, have the opportunity to design things so that they work. And you definitely have to be clever if you're looking for language features that Parrot doesn't natively support. My first tendency here is to echo all attributes as methods (like Perl 6 does), and then always call the method when you see a dot. ParrotString has a find method, and it knows how to curry itself. Instances of c also have a find method, and that method always returns 7. Although I agree that we should come up with a general, bare-bones object model that allows all of our current target languages to operate smoothly. Parrot's current model makes far too many assumptions. But that doesn't mean that anything you're trying to do is impossible; it just means it's harder. Luke
Re: Calling conventions, invocations, and suchlike things
Sam Ruby wrote: Now, what should the code for function f look like? The only reasonable answer is something along the lines of: getattribute $P0, P5, 'find' I doubt that. All languages have different semantics, and we can't implement them all, because they are conflicting. You, as a compiler designer, have the opportunity to design things so that they work. And you definitely have to be clever if you're looking for language features that Parrot doesn't natively support. I disagree. As Dan once said, as long as we have a proper protocol that everyone can conform to, we should be OK. I don't care if Perl, Python, Ruby, TCL, and others each implement different semantics. I do care that we adopt a common protocol. The current set of VTABLE entries is a excellent first order approximation for the common protocol. In case that isn't perfectly clear, let me illustrate this with actual source code: inline op getattribute(out PMC, in PMC, in STR) :object_classes { $1 = VTABLE_get_attr_str(interpreter, $2, $3); goto NEXT(); } Note that the opcode doesn't actually implement any semantics. It merely delegates the request to the $2 PMC. Languages which share semantics for this operation can chose to inherit a common implementation. Those with unique semantics can override this (or can find other languages with similar requirements and pool their implementation). It seems rather likely to me that this was the original intent to providing a VTABLE_get_attr_str VTABLE entry in the first place. This seems to me to be a rather good design pattern. - Sam Ruby
Re: Calling conventions, invocations, and suchlike things
Dan Sugalski [EMAIL PROTECTED] wrote: Hi folks. The lost son is back, welcome. The easy one first -- why the object is out-of-band, rather than one of the parameters. Parrot's got the interesting, and somewhat unfortunate, requirement of having to allow all subroutines behave as methods and all methods behave as subroutines. (This is a perl 5 thing, *and* a Python thing *and* a Perl6 thing, when it comes to Perl6 multi subs. And finally *all* Parrot infix operators are multi subs too. For all these the current scheme does not match. multi sub infix:+(Int $left, Int $right) {...} The only remaining methods, where your prerequisite is true is for Perl6 single methods, which the Perl6 code generator can translate either way, because they are declared with the method keyword. Please reconsider this decision. ... Regardless invokable things need to know whether they were called as a method or a sub. Where currently the first argument (object) is passed in out-of-bounds, the proposal is to pass the invocant. Nothing would change WRT all the advantages of the current solution. Not having objects handle their own method dispatch is less clear-cut, but I do have some reasons, so here they are. This is ok. See also Proposed vtable changes WRT method lookup, where the dispatch and the find_method functionality is split. leo
Calling conventions, invocations, and suchlike things
Hi folks. Sorry I've been gone so long. Non-p6i stuff's been well past monopolizing my time. Not much of an excuse, I know, but the Real World intrudes at the most inconvenient times. Things are, I hope, easing up a little, though I apologize in advance if I get a little cranky while I get back into things. Having (very lightly) skimmed the past month of list mail, I'm thinking the best place to start is with the things that've come up about objects and method calls. I want to explain why things are designed the way they are so (hopefully) everyone's on the same page. (And, hopefully, to forestall grumbling when I say things aren't going to change :) The setup, for those following along at home, is that when we make a method call the object is passed out-of-band (that is, not as part of the regular parameter list), and that objects don't actually handle method dispatch -- we split it into a two step affair where we request an invokable method PMC for a named method from an object, and then invoke it as a separate step. The easy one first -- why the object is out-of-band, rather than one of the parameters. (Something that I doubt anyone's that worked up over, and I think everyone's OK with things as they stand, but here are the reasons anyway) Parrot's got the interesting, and somewhat unfortunate, requirement of having to allow all subroutines behave as methods and all methods behave as subroutines. (This is a perl 5 thing, but we have to make it work) That is, an invokable PMC may be invoked as a method call and passed in an object, or as a plain subroutine and not have an object passed in. As far as perl 5 is concerned the object is the first parameter in the argument list, but for everyone else the object is a very distinct and separate thing. Regardless invokable things need to know whether they were called as a method or a sub. We *could* set a flag and have them check, then have some convention where the first parameter is an object if the I'm a method flag is set, but... yech. Having the object be separate and standalone seems cleaner, while still giving us a way to distinguish method/sub invocation. (You check to see if there's an object) This does make things a little tricker for the perl 5 code generator, but not that much trickier and, let's face it, we're below the layer where things are easy. This *also* makes building signature checking into parrot a lot simpler (something we should do), since the signature checking stuff doesn't have to deal with possible parameter shifting based on whether we've a sub or method invocation. Not having objects handle their own method dispatch is less clear-cut, but I do have some reasons, so here they are. First off, one of the things I'm very much concerned about is C stack usage, both because we don't have all that much we can count on (joys of threads -- we'll be lucky to scrape together 10k some places) and because continuations can't cross C stack level boundaries. We're pretty careful about that one (it's the big reason for the limitation that continuations taken from within vtable functions can't escape). I realize we can continue to be careful with it, mandating that the invoke_method vtable function behaves the same as the plain invoke does (that is, returning the address to jump to) but that brings up a separate problem -- transfer of control is a relatively heavyweight thing for us. Method and sub calls can potentially cross bytecode and security boundaries. Doing that right requires (potentially) a fair amount of screwing around inside the interpreter, as well as the invokable thing carrying around enough metadata to properly do the transfer. I'd really prefer to limit the number of PMCs that have that amount of intimate knowledge. Since all methods and subs have the appropriate bits attached to them, I'd as soon just use them. There's also the potential issue of curried methods, where we need to create a new invokable thing and bind some parameters to it. We can certainly do that now with the current scheme so adding an invoke_method to the mix won't get in the way as such, but it does mean we have two near-identical ways of doing the same thing (find_method invoke, and invoke_method) and since we can't toss the find_method way, it doesn't feel like adding invoke_method to the mix will get us anywhere. Anyway, there we go. (I fully expect to find that both topics are dead about an hour after this goes out, but there you go :) -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
Re: Calling conventions, invocations, and suchlike things
Dan Sugalski wrote: Hi folks. Welcome back! Parrot's got the interesting, and somewhat unfortunate, requirement of having to allow all subroutines behave as methods and all methods behave as subroutines. (This is a perl 5 thing, but we have to make it work) That is, an invokable PMC may be invoked as a method call and passed in an object, or as a plain subroutine and not have an object passed in. As far as perl 5 is concerned the object is the first parameter in the argument list, but for everyone else the object is a very distinct and separate thing. Python essentially has the same requirement, with a few twists. Specifically, methods come in be static, class, and regular flavors. But first, a simple example. Strings in python have a find method, so and can do the following: f = Parrot.find print f(r) Note that I referenced the method as an attribute, and then called it as a function. Furthermore, the function remembers what object it is bound to. This is accomplished by VTABLE_find_method creating a new PyBoundMeth PMC which contains two references, one to the object, and one to the method. The sole responsibility of a bound method is to insert the first parameter into the call, and then to invoke the desired method. Static methods differ in that the object is not passed. Class methods differ in that the object passed is actually the class of the object in question. Note: all this is determined by the callee. It is all transparent to the caller. Now, compare that to callmethod or callmethodcc in ops: object = REG_PMC(2); method_pmc = VTABLE_find_method(interpreter, object, REG_STR(0)); if (!method_pmc) { real_exception(interpreter, next, METH_NOT_FOUND, Method '%Ss' not found, REG_STR(0)); } REG_PMC(0) = method_pmc; interpreter-ctx.current_object = object; dest = (opcode_t *)VTABLE_invoke(interpreter, method_pmc, next); First, observe that I don't have any control over the exception that is raised when a method is not found (fix: raise the exception within find_method). Second, observe that this assumes that current object is the object in all cases. This doesn't work well for static or class methods (fix: create a PyBoundMeth object which overrides this, and then call the real method). This does make things a little tricker for the perl 5 code generator, but not that much trickier and, let's face it, we're below the layer where things are easy. This *also* makes building signature checking into parrot a lot simpler (something we should do), since the signature checking stuff doesn't have to deal with possible parameter shifting based on whether we've a sub or method invocation. Parameter shifting actually is made easier by placing the object in question in a separate register... no shifting is required. (Note: in cvs, I actually do the shifting, partially because I'm in the process of reverting a change, and partially do to the issue raised in the next paragraph). My one minor request here is P2 be made available on entry to the invoked method. This would remove some special case logic for me requiring the use of interpinfo. I don't expect any guarantees that this is preserved or restored across sub calls. Not having objects handle their own method dispatch is less clear-cut, but I do have some reasons, so here they are. Just be aware that in order to preserve Python semantics, find_method will need to return a bound method. This involves creating an object on the heap, garbage collection, and a minor addition to the number of instructions executed on invoke (including a nested C stack). This could all be avoided if there was a VTABLE_callmethod interface as the code would know that the intent was to only use this found method exactly once. *shrug* Do you plan to choose banana cream again at OSCON 2005? - Sam Ruby