Mark Wooding <m...@distorted.org.uk> wrote: > Ethan Furman <et...@stoneleaf.us> writes: > > > Mark Wooding wrote: > >> Here's what I think is the defining property of pass-by-value [...]: > >> > >> The callee's parameters are /new variables/, initialized /as if by > >> assignment/ from the values of caller's argument expressions. > >> > >> My soundbite definition for pass-by-reference is this: > >> > >> The callee's parameters are merely /new names/ for the caller's > >> argument variables -- as far as that makes sense. > > > > Greetings, Mark! > > You posted to the whole group -- probably using a wide-reply option > while reading the mailing list. Still, I'll give an answer here in > order to help any other readers of the list or newsgroup. However, as > far as I'm concerned, this discussion is basically at an end and I'm not > really interested in starting it up all over again. To that end, I've > set followups to `poster'. I shall continue reply to private email > which seems interested in a sensible discussion.
I just popped in to this thread, and read the whole thing in a marathon session (why, I'm not quite sure, but somehow I found it interesting :). I'm going to follow up here at the risk of annoying Mark, because I think there may be a way to reconcile the language in a way that is helpful in explaining things to Python beginners. > > I was hoping you might be able to clarify those last two sound bites > > for me -- I think I understand what you are saying, but I'm confused > > about how they relate to Python... I think this is the key point. I am an experienced Python programmer, and I had to think long and hard about what you were saying, Mark, in order to understand how it applied to Python. I think this means that your model and/or your level of abstraction is not "natural" when trying to understand Python programs, and I think I may have figured out why. > > Specifically, how is a new name (pbr) different, in Python, from a new > > name initialized as if by assignment (pbv)? It seems to me than you > > end up with the same thing in either case (in Python, at least), > > making the distinction non-existent. > > You've missed a level of indirection. In particular, `names' aren't > things that you initialize. They're things that you /bind/ to > variables. The crucial difference is that, in pass-by-value, new > variables are created as an intrinsic part of the process, whereas in > pass-by-reference, new variables are not (usually) created, and instead > the formal parameter names are bound to the caller's pre-existing > argument variables. I think that the reason Ethan missed that level of indirection is that Python hides that level of indirection from the programmer. And I think that this is where the conversation between you and Steven got hung up, Mark. You are firmly grounded in how languages work in general, so your conceptual model naturally includes details that Steven's conceptual model can ignore, because he's dealing only with the model used by Python. He can (and wants to!) ignore the level of indirection that Python doesn't even give him access to. In short, there is a reason why you almost never hear the word 'variable' from an experienced Python programmer when talking about the Python model. That level detail is abstracted into something else by the Python conceptual model: it becomes a namespace mapping names to objects. > It's worth noting that I use the terms `name' and `binding' in different > ways from most of the Python community. This is unfortunate. The > discrepancy is actually because the Python meanings of these words are > not the same as the meanings in the wider computer science and > mathematical communities. For example, many Python users seem to use > `binding' to mean `assignment', which is a shame because it leaves the > concept that is usually called `binding' without a name. So I'll stick > with the wider meanings. It seems to me from reading the thread that everyone is actually in pretty good agreement that 'name' is the symbol one types in the program source to denote...something. It's the something where things get tricky. So let's leave name itself alone. But let's replace 'assignment' and 'binding' with the concept that is more natural to Python's model, that of a "mapping". A namespace maps names to objects. If I'm understanding you correctly, Mark, in your model what I'm calling a namespace is the set of variable bindings in a particular environment. Because Python does not allow the programmer to actually rebind a variable (Python handles the association between name and variable completely behind the scenes), Python programmers naturally skip past that name-to-variable binding step, and conceptually think about the name being bound to the _value_ instead. After all, that's the only thing they can change. This may be unfortunate from a precisionist viewpoint, but it is both natural and, IMO, inevitable, because it makes thinking about Python programs more efficient. > A while ago, I posted an essay -- to this group -- which may help > explain the concepts: > > Message-ID: <8763k14nc6.fsf....@metalzone.distorted.org.uk> > http://groups.google.com/group/comp.lang.python/msg/f6f1a321f819d02b > > On with the question. > > > def func(bar): > > bar.pop() > > > > Pass-by-reference: > > foo = ['Ethan','Furman'] > > func(foo) # bar = foo > > > > Pass-by-value: > > foo = ['Python','Rocks!'] > > func(foo) # bar is new name for foo > > # is this any different from above? > > > > If I have this right, in both cases foo will be reduced to a > > single-item list after func. > > You're correct. So: we can conclude that the above test is not > sufficient to distinguish the two cases. Suppose we recast this by saying that the function has a namespace, and in that namespace 'bar' maps to something. What that something is depends on what is passed in that argument position when func is called. The calling code's namespace maps 'foo' to something. When func is called, a mapping to that same object is set up in func's namespace, under the name 'bar'. In Python's conceptual model it doesn't make any sense to ask if the argument is passed by value or by reference. In reality, if I'm understanding you correctly Mark, you are absolutely correct that the 'variable' (the thing actually pointed to by 'name', which is a storage location that contains a pointer to an object) is passed by value. That is, the contents of the storage location that the caller's 'foo' is pointing to is copied into a new storage location that the callee's 'bar' points to. Conceptually, however, that is an implementation detail. Conceptually, the important thing is that a _new mapping_ in a _different namespace_ is set up to the _same object_. That is the semantics that Python the language defines. As Steven said, talking about variables seems to be drilling down into a level of abstraction that is more confusing than it is helpful when you are actually _programming_ in Python. It can be a useful level of abstraction to talk about when trying to explain what is happening to someone with previous experience of other languages, but it isn't useful (in my experience) for everyday Python programming. The reason that is true is because in Python there is _no way_ to get a second reference to the 'variable' that is pointed to by 'foo' or by 'bar'. (Unless you walk into that secret back room, of course, and even then it is hard, I think.) As I said, Python hides that level of detail from the programmer. This means that it effectively does not exist when you program Python, and _this_ is why we get all tangled up in language when we try to discuss it. You, Mark, had to go to great lengths to define your terms, and still a lot of people didn't get it, because in actual, on-the-ground Python programming, variables just don't enter in to it, because they can not be directly manipulated from the Python program. I suspect that the longer one has been a Python programmer, the harder it is to even see this blind spot, and thus the harder it is to respond sensibly to someone new to Python whose conceptual model includes variables that _can_ be accessed and manipulated from the program. I know _I_ had to read and re-read your posts before I finally figured it out (assuming I have!). (FYI, my programming learning path was roughly Basic->FORTRAN ->LISP->assembler->C->Python, and I've been programming Python since 1993 or thereabouts). (BTW, I am not claiming Python is unique in this regard, I'm just talking about how it looks from inside the Python programming mindset.) > > Any further explanation you care to provide will be greatly > > appreciated! > > This test is sufficient to distinguish: > > def test(x): > x = 'clobbered' > > y = 'virgin' > test(y) > print y > > If it prints `virgin' then you have call-by-value. If it prints > `clobbered' then you have call-by-reference. Or if you think in terms of namespace mappings, 'test' cannot change what 'y' is mapped to in the caller's namespace, because (in this function) it has no mapping to that namespace object. It _can_ make changes to the thing that y is mapped _to_, if that thing is mutable. That's the most useful way to think about it that I've found. Your reasoning about variables and call-by-reference and call-by-value, while correct and enlightening at the computer science level, confuses me and makes me less efficient if I think about it while writing a Python program. > Let's examine the two cases, as I did in the essay I cited above. I'll > do call-by-value first. First, we define a function `test'. Then, we > initialize `y'. It's worth examining this process in detail. The name > `y' is initially unbound. so it is implicitly bound to a fresh > variable. Then, (a reference to) the string object 'virgin' is stored > in this variable. We can show this diagrammatically as follows. > > y (in global env.) ====> [VAR] ---> 'virgin' > > (In the diagrams, ===> denotes a binding relationship, between names and > variables; and ---> denotes a reference relationship, between variables > and values.) This diagram is IMO accurate and correct. However, in Python we can never actually touch [VAR] except insofar as we change the right hand side element of the diagram. And we can only do _that_ if we have access to the object that is the global name space, and we mutate that object. So for reasoning about Python programs, I think like this: y (in global namespace) ---> 'virgin' > Next, we call the `test' function. Call-by-value says that we must > evaluate the argument expressions. There's only one: `x'. The value of > a name is obtained by (a) finding which variable is bound to the name, > and (b) extracting the value from this variable. Well, the variable is > the one we just bound, and the value stored is (the reference to) the > string 'virgin'. So the result of evaluating the argument expressions > is simply (the reference to) that string. > > The function has one parameter, `y'. A new environment is constructed > by extending the global environment. In this new environment, the name > `y' is bound to a fresh variable -- distinct from all others, and > especially from the variable bound to `x' -- and in that variable we > store the value of the corresponding argument expression. Result: the > function body is executed in an environment which is like the global > environment except that `y' is bound to a fresh variable containing > 'virgin'. > > y (in global env.) ====> [VAR] ---> 'virgin' > ^ > | > x (in function `test') ====> [VAR] -------' I would describe this (in Python) as follows: we call the test function, passing it x. This means we find the object that x is mapped to in the caller's namespace, and we map 'y' in the function's namespace to the same object: y (in global env.) ---> 'virgin' ^ | x (in function 'test') ---' > Now there's an assignment > > x = 'clobbered' > > The name `x' is already bound to a variable. So we modify that variable > so that it stores (a reference to) the string 'clobbered'. > > y (in global env.) ====> [VAR] ---> 'virgin' > > x (in function `test') ====> [VAR] ---> 'clobbered' Assignment in Python changes a mapping (mutates a namespace object). In this case the mapping being changed is the local function's namespace mapping. We can't affect the caller's namespace mapping, because (in this function) we don't have access to the caller's namespace object. So I would diagram it like this: y (in global env.) -----> 'virgin' x (in function 'test') ----> 'clobbered' > And then the function ends. The environment we constructed is > forgotten. The variable bound to `x' is lost forever, since it wasn't > bound to any other name. Since modifying that variable was the only > action carried out in the function, and the variable is now lost, there > is no externally observable effect. When we finally print `y', we see > `virgin', because the variable bound to `y' was unchanged. > > y (in global env.) ====> [VAR] ---> 'virgin' The function's local namespace is not accessible to the caller, so the caller can't access the object 'clobbered'. Its own namespace mapping is unchanged. The object 'virgin' is unchanged. So the caller sees no change as a result of calling the function. Now, to demonstrate that this model of namespaces-as-mappings (and first level objects themselves) is more useful to the Python programmer than the accurate but too-low-level description in terms of variables, consider this: >>> def test(x): ... x = 'clobbered' ... return locals() >>> y = 'virgin' >>> ns = test(y) >>> print y virgin >>> print ns['x'] clobbered In other words, while thinking about 'variables' and 'pass by value' is _accurate_, it does not add anything to the conceptual model a programmer needs in order to understand and write Python code. Namespaces as first level objects that map names to values, however, is an extremely useful and powerful mental model for programming in Python. The two concepts are conceptually equivalent regardless of implementation details (assuming we say a namespace immutably associates names with variables, which in turn mutably point to values), but the namespace, IMO, is the more useful of the two when reasoning about Python programs. > So much for call-by-value. How about call-by-reference? Well, > everything is the same until the actual call. But then everything > changes. > > Firstly, call-by-reference /doesn't/ evaluate the argument expressions. > Instead, we just note that `y' is bound to a particular variable in the > global environment. The function has a single parameter `x'. A new > environment is constructed by extending the global environment (again): > in this new environment, the name `x' is bound to /the same variable/ > that `y' is bound to in the global environment. > > y (in global env.) ====> [VAR] <===== x (in function `test') > | > | > v > 'virgin' And this is something that you can _never_ do in Python (short of mindbogglingly bad hackery). I know you know that, but the fact that you _cannot_ do it is why people start getting confused when talking about call-by-reference vs call-by-value in a Python context. > Now we assign to `x'. The detailed rules are the same: `x' is bound, so > we modify the variable it's bound to, so that it stores 'clobbered'. > But this time, the variable being clobbered is the /same/ variable that > `y' is bound to. So finally, when we print `y', we see the string > 'clobbered. > > > y (in global env.) ====> [VAR] <===== x (in function `test') > | > | > v > 'clobbered' > > Now, let's look at your example. > > > def func(bar): > > bar.pop() > > > > foo = ['Ethan','Furman'] > > func(foo) # bar = foo > > I won't describe this in the same excruciating detail as I did for the > one above; but I will make some observations. > > In call-by-reference, the environment in which the body of `func' is > executed is constructed by extending the global environment with a > binding of the name `bar' to the same variable as is bound to `foo' in > the global environment. There is only the one variable, so obviously > both names (considered as expressions) must evaluate to the same value. > I won't go into the details of method invocation, which in Python is > quite complicated; but `bar.pop()' mutates this value: it modifies it in > place. So, when the function returns, `foo' can be seen to print > differently. The value is, in some sense, the /same/ value as it was > before, in that it occupies the same storage locations, but the contents > of those storage locations has been altered. > > foo =====> [VAR] <===== bar > (in global env) | (in function `func') > | > v > ['Ethan', 'Furman'] And just to be crystal clear, the above case is _not_ the one Python does. > In call-by-value, `func' executes in an environment constructed by > extending the global environment with a binding of `bar' to a /fresh/ > variable. This fresh variable is then initialized: we store the value > of the argument expression `foo' into it. This value is a reference to > the list ['Ethan', 'Furman']. (See, I've stopped parenthesizing the > reference stuff, because here it really matters.) So, we have two > variables, bound to `foo' and `bar', but both variables refer to the > same list object. The body of `func' doesn't modify any variables; > rather, it mutates the list object in place (as above). Since both > variables refer to the /same/ list object, this mutation is still > observable outside of the function. > > foo =====> [VAR] ---> ['Ethan', 'Furman'] > (in global env) ^ > | > bar =====> [VAR'] ------------' > (in function `func') And for reasoning about Python programs, I would write this diagram (which _is_ the way Python does it) as: foo ---> ['Ethan', 'Furman'] (in global namespace) ^ | bar ------------' (in 'func' namespace) > So the reason that your example doesn't distinguish the two cases is > because, in both cases, there's still only one value, which is mutated. > But there is a conceptual difference, unobservable in this instance, Unobservable in Python because Python gives the programmer no way to create two pointers to the same '[VAR]'. > because call-by-reference has two names bound to the same variable, > while call-by-value has two names, bound to /distinct/ variables but > which both refer to the same value. > > Finally, it may be instructive to remove the issue of function calling > altogether. Consider: > > foo = ['Ethan', 'Furman'] > bar = foo > bar.pop() > > What is the final value of `foo'? > > Now: > > x = 'virgin' > y = x > y = 'clobbered' > > What is the final value of `x'? > > Are you enlightened? Hopefully so! Your explanation of the underlying mechanics of call by reference and call by value is the best I've ever seen, and all that effort you put in to defining your terms earlier in this thread has ultimately paid off. Although this has been long enough already (especially since I quoted your excellent article in full), I still want to go on a bit more. I want to demonstrate the explanatory power of the namespace model, which was one of your arguments with Steven: that his model of Python working directly with values couldn't explain certain things. On of the examples you gave (and I'm doing this from memory) was this: >>> l = [ lambda: x for x in [1,2,3] ] >>> [ f() for f in l ] [3, 3, 3] Your explanation was in terms of variables and environments, and frankly I had a hard time following it (though I get it now). Here's how I explain it in terms of namespaces: lambda creates a closure, and part of the closure is a reference to the locals namespace object that exists when the closure is formed. The (top level of the) list comprehension is operating in the local namespace, so it is this namespace that gets captured inside the closure. When we execute the functions later, what happens is that 'x' is looked up in this namespace, and the object that _it maps to at that time_ is returned. Since the last assignment (mapping) to X was to the object '3', '3' is what is returned by all three functions calls. The code above is equivalent to the following code, which makes is clearer what is happening: >>> l2 = [] >>> for x in 1,2,3: ... l2.append(locals()) >>> [ ns['x'] for ns in l2 ] [3, 3, 3] So in terms of namespaces it is crystal clear why we get three 3s, and why the following produces three 5s: >>> x = 5 >>> [ ns['x'] for ns in l2 ] [5, 5, 5] >>> [ f() for f in l ] [5, 5, 5] This is conceptually equivalent to your variables and environments model, but I think it is easier to understand and reason about, when talking only about Python code, because it elides that level of indirection to which the Python programmer has no access. At least, I find it so! To take the final step toward unifying your view with Steven's view, we have to admit that Python objects that reference other objects can be viewed either as having [VAR] elements containing pointers to objects, or as namespaces referencing other objects, and that these two views are conceptually equivalent. Thus, take your circular list example: >>> a [1, 2, 3] >>> a[1] = a >>> a [1, [...], 3] Your view was that this list must contain a pointer variable which points to the list itself (a reference), while Steven held that Python lists reference other objects directly, and had no problem with this Tardis-like construct. You are obviously correct that what a list is is a sequence of variable slots that can point to any object, including the list itself. But we can also view the list as a namespace whose name elements are a zero-origined, monotonically increasing set of integers, and that within this name space these names map directly to python objects. In Python (or at least in CPython), the two views are equivalent. But I argue, with Steven, that the latter view is more _useful_ for a Python programmer, and that it is in fact Python's conceptual model. I would also argue that the latter view is the one that it is more helpful to explain to newcomers to Python, with due acknowledgement being made to the reality of the behind-the-scenes variables for those coming from a background where that will be helpful. The trick is to get the long-time Python programmers to remember, when talking to such a newcomer, that those variables even exist! --RDM -- http://mail.python.org/mailman/listinfo/python-list