Re: exec() an locals() puzzle

[george trojan]

Thanks. That cdef-locals concept is consistent with the following example:

def f():
i = 1
def g(): print('i' in globals(), 'i' in locals())
def h(): print('i' in globals(), 'i' in locals()); i
g()
h()
f()

False False
False True

It is a mystery, which may be why the documentation for globals() and
locals() is 2-line long.


Le mer. 20 juill. 2022, à 19 h 31, Martin Di Paola <
martinp.dipa...@gmail.com> a écrit :

> I did a few tests
>
> # test 1
> def f():
>  i = 1
>  print(locals())
>  exec('y = i; print(y); print(locals())')
>  print(locals())
>  a = eval('y')
>  print(locals())
>  u = a
>  print(u)
> f()
>
> {'i': 1}
> 1
> {'i': 1, 'y': 1}
> {'i': 1, 'y': 1}
> {'i': 1, 'y': 1, 'a': 1}
> 1
>
> # test 2
> def f():
>  i = 1
>  print(locals())
>  exec('y = i; print(y); print(locals())')
>  print(locals())
>  a = eval('y')
>  print(locals())
>  y = a
>  print(y)
> f()
> {'i': 1}
> 1
> {'i': 1, 'y': 1}
> {'i': 1}
> Traceback (most recent call last):
> NameError: name 'y' is not defined
>
>
> So test 1 and 2 are the same except that the variable 'y' is not
> present/present in the f's code.
>
> When it is not present, exec() modifies the f's locals and adds an 'y'
> to it but when the variable 'y' is present in the code (even if not
> present in the locals()), exec() does not add any 'y' (and the next
> eval() then fails)
>
> The interesting part is that if the 'y' variable is in the f's code
> *and* it is defined in the f's locals, no error occur but once again the
> exec() does not modify f's locals:
>
> # test 3
> def f():
>  i = 1
>  y = 42
>  print(locals())
>  exec('y = i; print(y); print(locals())')
>  print(locals())
>  a = eval('y')
>  print(locals())
>  y = a
>  print(y)
> f()
> {'i': 1, 'y': 42}
> 1
> {'i': 1, 'y': 1}
> {'i': 1, 'y': 42}
> {'i': 1, 'y': 42, 'a': 42}
> 42
>
> Why does this happen? No idea.
>
> I may be related with this:
>
> # test 4
> def f():
>  i = 1
>  print(locals())
>  exec('y = i; print(y); print(locals())')
>  print(locals())
>  print(y)
> f()
> Traceback (most recent call last):
> NameError: name 'y' is not defined
>
> Despite exec() adds the 'y' variable to f's locals, the variable is not
> accessible/visible from f's code.
>
> So, a few observations (by no means this is how the vm works):
>
> 1) each function has a set of variables defined by the code (let's call
> this "code-defined locals" or "cdef-locals").
> 2) each function also has a set of "runtime locals" accessible from
> locals().
> 3) exec() can add variables to locals() (runtime) set but it cannot add
> any to cdef-locals.
> 4) locals() may be a superset of cdef-locals (but entries in cdef-locals
> which value is still undefined are not shown in locals())
> 5) due rule 4, exec() cannot add a variable to locals() if it is already
>   present in the in cdef-locals.
> 6) when eval() runs, it uses locals() set for lookup
>
> Perhaps rule 5 is to prevent exec() to modify any arbitrary variable of
> the caller...
>
> Anyways, nice food for our brains.
>
> On Wed, Jul 20, 2022 at 04:56:02PM +, george trojan wrote:
> >I wish I could understand the following behaviour:
> >
> >1. This works as I expect it to work:
> >
> >def f():
> >i = 1
> >print(locals())
> >exec('y = i; print(y); print(locals())')
> >print(locals())
> >exec('y *= 2')
> >print('ok:', eval('y'))
> >f()
> >
> >{'i': 1}
> >1
> >{'i': 1, 'y': 1}
> >{'i': 1, 'y': 1}
> >ok: 2
> >
> >2. I can access the value of y with eval() too:
> >
> >def f():
> >i = 1
> >print(locals())
> >exec('y = i; print(y); print(locals())')
> >print(locals())
> >u = eval('y')
> >print(u)
> >f()
> >
> >{'i': 1}
> >1
> >{'i': 1, 'y': 1}
> >{'i': 1, 'y': 1}
> >1
> >
> >3. When I change variable name u -> y, somehow locals() in the body of
> >the function loses an entry:
> >
> >def f():
> >i = 1
> >print(locals())
> >exec('y = i; print(y); print(locals())')
> >print(locals())
> >y = eval('y')
> >print(y)
> >f()
> >
> >{'i': 1}
> >1
> >{'i': 1, 'y': 1}
> >{'i': 1}
> >
>
> >---NameError
> >Traceback (most recent call last)
> >Input In [1], in ()  7 print(y)  8 # y
> >= eval('y')  9 #print('ok:', eval('y'))---> 10 f()
> >
> >Input In [1], in f()  4 exec('y = i; print(y); print(locals())')
> >   5 print(locals())> 6 y = eval('y')  7 print(y)
> >
> >File :1, in 
> >NameError: name 'y' is not defined1.
> >
> >Another thing: within the first exec(), the print order seems
> >reversed. What is going on?
> >
> >BTW, I am using python 3.8.13.
> >
> >George
> >--
> >https://mail.python.org/mailman/listinfo/python-list
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec() an locals() puzzle

[Martin Di Paola]

I did a few tests

# test 1
def f():
i = 1
print(locals())
exec('y = i; print(y); print(locals())')
print(locals())
a = eval('y')
print(locals())
u = a
print(u)
f()

{'i': 1}
1
{'i': 1, 'y': 1}
{'i': 1, 'y': 1}
{'i': 1, 'y': 1, 'a': 1}
1

# test 2
def f():
i = 1
print(locals())
exec('y = i; print(y); print(locals())')
print(locals())
a = eval('y')
print(locals())
y = a
print(y)
f()
{'i': 1}
1
{'i': 1, 'y': 1}
{'i': 1}
Traceback (most recent call last):
NameError: name 'y' is not defined


So test 1 and 2 are the same except that the variable 'y' is not
present/present in the f's code.

When it is not present, exec() modifies the f's locals and adds an 'y'
to it but when the variable 'y' is present in the code (even if not
present in the locals()), exec() does not add any 'y' (and the next
eval() then fails)

The interesting part is that if the 'y' variable is in the f's code
*and* it is defined in the f's locals, no error occur but once again the
exec() does not modify f's locals:

# test 3
def f():
i = 1
y = 42
print(locals())
exec('y = i; print(y); print(locals())')
print(locals())
a = eval('y')
print(locals())
y = a
print(y)
f()
{'i': 1, 'y': 42}
1
{'i': 1, 'y': 1}
{'i': 1, 'y': 42}
{'i': 1, 'y': 42, 'a': 42}
42

Why does this happen? No idea.

I may be related with this:

# test 4
def f():
i = 1
print(locals())
exec('y = i; print(y); print(locals())')
print(locals())
print(y)
f()
Traceback (most recent call last):
NameError: name 'y' is not defined

Despite exec() adds the 'y' variable to f's locals, the variable is not
accessible/visible from f's code.

So, a few observations (by no means this is how the vm works):

1) each function has a set of variables defined by the code (let's call
this "code-defined locals" or "cdef-locals").
2) each function also has a set of "runtime locals" accessible from
locals().
3) exec() can add variables to locals() (runtime) set but it cannot add
any to cdef-locals.
4) locals() may be a superset of cdef-locals (but entries in cdef-locals
which value is still undefined are not shown in locals())
5) due rule 4, exec() cannot add a variable to locals() if it is already
 present in the in cdef-locals.
6) when eval() runs, it uses locals() set for lookup

Perhaps rule 5 is to prevent exec() to modify any arbitrary variable of
the caller...

Anyways, nice food for our brains.

On Wed, Jul 20, 2022 at 04:56:02PM +, george trojan wrote:

I wish I could understand the following behaviour:

1. This works as I expect it to work:

def f():
   i = 1
   print(locals())
   exec('y = i; print(y); print(locals())')
   print(locals())
   exec('y *= 2')
   print('ok:', eval('y'))
f()

{'i': 1}
1
{'i': 1, 'y': 1}
{'i': 1, 'y': 1}
ok: 2

2. I can access the value of y with eval() too:

def f():
   i = 1
   print(locals())
   exec('y = i; print(y); print(locals())')
   print(locals())
   u = eval('y')
   print(u)
f()

{'i': 1}
1
{'i': 1, 'y': 1}
{'i': 1, 'y': 1}
1

3. When I change variable name u -> y, somehow locals() in the body of
the function loses an entry:

def f():
   i = 1
   print(locals())
   exec('y = i; print(y); print(locals())')
   print(locals())
   y = eval('y')
   print(y)
f()

{'i': 1}
1
{'i': 1, 'y': 1}
{'i': 1}

---NameError
   Traceback (most recent call last)
Input In [1], in ()  7 print(y)  8 # y
= eval('y')  9 #print('ok:', eval('y'))---> 10 f()

Input In [1], in f()  4 exec('y = i; print(y); print(locals())')
  5 print(locals())> 6 y = eval('y')  7 print(y)

File :1, in 
NameError: name 'y' is not defined1.

Another thing: within the first exec(), the print order seems
reversed. What is going on?

BTW, I am using python 3.8.13.

George
--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

Re: exec() an locals() puzzle

[Eryk Sun]

On 7/20/22, george trojan  wrote:
>
> 1. This works as I expect it to work:
>
> def f():
> i = 1
> print(locals())
> exec('y = i; print(y); print(locals())')
>     print(locals())
> exec('y *= 2')
> print('ok:', eval('y'))
> f()

In CPython, the locals of a function scope (as opposed to a class
scope or module scope) are optimized by storing them in an array in
the current frame. locals(), however, always returns the non-optimized
locals mapping of the current frame, i.e. the value of
sys._getframe(0).f_locals. In the case of optimized locals, the
locals() call first updates this dict with a snapshot of the current
values of local variables. This is useful for introspection, but
modifying the snapshot cannot extend or modify the optimized local
variables of the function scope.

exec() and eval() default to using the global and local variable
mappings that are returned by globals() and locals(). This allows them
to access the snapshot of the containing function's optimized locals,
but they can't extend or modify the function's optimized locals. At
most they can add dynamic locals to the snapshot, as used by
subsequent calls to locals(), exec() and eval().
-- 
https://mail.python.org/mailman/listinfo/python-list

exec() an locals() puzzle

[george trojan]

I wish I could understand the following behaviour:

1. This works as I expect it to work:

def f():
i = 1
print(locals())
exec('y = i; print(y); print(locals())')
print(locals())
exec('y *= 2')
print('ok:', eval('y'))
f()

{'i': 1}
1
{'i': 1, 'y': 1}
{'i': 1, 'y': 1}
ok: 2

2. I can access the value of y with eval() too:

def f():
i = 1
print(locals())
exec('y = i; print(y); print(locals())')
print(locals())
u = eval('y')
print(u)
f()

{'i': 1}
1
{'i': 1, 'y': 1}
{'i': 1, 'y': 1}
1

3. When I change variable name u -> y, somehow locals() in the body of
the function loses an entry:

def f():
i = 1
print(locals())
    exec('y = i; print(y); print(locals())')
print(locals())
y = eval('y')
print(y)
f()

{'i': 1}
1
{'i': 1, 'y': 1}
{'i': 1}

---NameError
Traceback (most recent call last)
Input In [1], in ()  7 print(y)  8 # y
= eval('y')  9 #print('ok:', eval('y'))---> 10 f()

Input In [1], in f()  4 exec('y = i; print(y); print(locals())')
   5 print(locals())> 6 y = eval('y')  7 print(y)

File :1, in 
NameError: name 'y' is not defined1.

Another thing: within the first exec(), the print order seems
reversed. What is going on?

BTW, I am using python 3.8.13.

George
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue46153] function fails in exec when locals is given

[Eryk Sun]

Eryk Sun  added the comment:

> You seem to be arguing that a description in the docs is "misleading", 
> not because it misleads, but because it don't describe a situation 
> which has nothing to do with the situation that the docs are describing.

To me it's misleading to say "the code will be executed as if it were embedded 
in a class definition" because that is not always the case. The example with 
print(a) shows that. One can take it another level to compare function 
definitions in a class definition compared to exec(). A function defined in an 
exec() is not compiled to bind its free variables to the outer lexical scope in 
the context of the exec() call, while a function defined in a class definition 
does. For example:

class:

def f():
   a = 2
   class C:
   def g(): print(a)
   return C.g

>>> a = 1
>>> g = f()
>>> g()
2

exec():

def f():
   a = 2
   l = {}
   exec('def g(): print(a)', globals(), l)
   return l['g']

>>> a = 1
>>> g = f()
>>> g()
1

You asked what I would say in its place, but I don't have a simple answer that 
can take the place of the one-liner in the docs. Here's something, but I'm sure 
you won't be happy with it:

The code will be executed in a manner that's similar to a class definition with 
regard to the use of separate locals and globals scopes. However, there can be 
significant differences in certain contexts with regard to how the same code is 
compiled for an exec() call compared to a class definition. In particular, code 
in a class definition is compiled to bind its free variables to the lexical 
scopes of outer function calls in the defining context, which isn't possible 
with exec(). Also, the top-level code in a class definition supports `nonlocal` 
declarations, which is a syntax error with exec().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Quentin Peter]

Quentin Peter  added the comment:

Maybe a note could be added to 
https://docs.python.org/3/library/functions.html#exec

Something along the lines of:

Note: If exec gets two separate objects as `globals` and `locals`, the code 
will not be executed as if it were embedded in a function definition. For 
example, any function or comprehension defined at the top level will not have 
access to the `locals` scope.

PS: It would be nice for my usecase to have a way around this, maybe a flag in 
`compile` or `exec` that would produce "function code" instead of "module 
code". My workaround for this problem consist in wrapping my code in a function 
definition.

I think this means https://bugs.python.org/issue41918 should be closed as well?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Steven D'Aprano]

Steven D'Aprano  added the comment:

On Thu, Dec 23, 2021 at 05:47:33AM +, Eryk Sun wrote:
> 
> Eryk Sun  added the comment:
> 
> > That's taken straight out of the documentation.
> 
> Yes, but it's still a misleading comparison.

I asked how you would re-word the docs, but you haven't responded.

The description given in the docs exactly explains the observed 
behaviour. Without recognising that, the observed behaviour is 
perplexing to the point that it suggested to at least one person that it 
was a bug in the language.

If you're not prepared to suggest an improvement to the documentation, 
then I don't think that this conversation is going anywhere and maybe 
we should just let the discussion die.

But for the record, in case you, or anyone else, does want to continue 
the discussion in the hope of reaching additional insight to the 
problem, my further comments are below.

[...]
> Saying that code will be "executed as if it were embedded in a class 
> definition" is correct only so far as the fact that globals and locals 
> are different in this case. 

So it's correct in all the ways that matter:

- different globals and locals;
- and the behaviour is different.

and incorrect in no ways at all (see below). I don't think that supports 
a charge of "misleading".

The bottom line here is that the description in the docs that you call 
"misleading" did not mislead me, but lead me directly to the solution of 
why the code behaved as it did, and why that was the intentional 
behaviour rather than a bug.

So un-misleading, if you will.

> But it's also misleading because the code 
> gets compiled as module-level code, not as class code.

Obviously there is no actual "class code" involved. That is why the 
description says that it is executed *as if* it were embedded inside a 
class statement, rather than by having an actual class statement added 
to your source string.

I don't understand your comment about "compiled as module-level ... not 
as class code". What's class code? (Aside from the actual class 
statement itself, which is a red herring.)

If you look at the disassembly of the following two snippets:

dis.dis("""
a = 1
def f():
return a
print(f())
""")

and 

dis.dis("""
class C:
a = 1
def f():
return a
print(f())
""")

the generated bytecode for the lines `a = 1` etc is the same, putting 
aside the code for the actual class statement part. You get the same 
code for `a = 1`

LOAD_CONST(1)
STORE_NAME(a)

the same code for both the body of the function:

LOAD_GLOBAL   (a)
RETURN_VALUE

and the `def f()` statement:

LOAD_CONST()
LOAD_CONST('f')
MAKE_FUNCTION
STORE_NAME

and the same code for the call to print:

 LOAD_NAME(print)
 LOAD_NAME(f)
 CALL_FUNCTION
 CALL_FUNCTION
 POP_TOP
 LOAD_CONST   (None)
 RETURN_VALUE

Obviously the offsets and locations of constants will be different, but 
aside from those incidental details, the code generated for the block is 
the same whether it is inside a class statement or not.

So I don't understand what you consider to be the difference between 
code compiled at module-level and code compiled at class-level. They 
seem to me to be identical (aside from the incidentals).

The visible difference in behaviour relates to the *execution* of the 
code, not to whether (quote):

"the code gets compiled as module-level code [or] as class code".

There is no distinct "class code". The difference in behaviour is in the 
execution, not to the compilation.

> It should be pretty obvious why the following fails:
> 
> exec("a = 1\ndef f(): return a\nprint(f())", {}, {})

Yes, it is obvious why it fails, in the same sense as the maths joke 
about the professor who stares at the equations on the blackboard for 
twenty minutes before exclaiming "Yes, it is obvious!".

It takes a sophisticated understanding of Python's scoping rules to 
understand why that fails when the other cases succeed.

> Assignment is local by default, unless otherwise declared. Function 
> f() has no access to the local scope where `a` is defined

With the same dict used for globals and locals, execution runs the 
statements `a = 1`, the `def f` and the print in the same scope, which 
is *both* global and local. This is what happens when you run code at 
the module level: locals is globals.

Consequently, the statement `a = 1` assigns a to the local namespace, 
which is the global namespace. And the call to f() retrieves a from the 
global namespace, which is the local namespace.

This is what happens when you execute the code at module-level.

With different dicts, the three statements still run in the same scope, 
the local scope, but the call to f() attempts to retrieve a from the 
global namespace, which is distinct from local namespace.

[issue46153] function fails in exec when locals is given

[Eryk Sun]

Eryk Sun  added the comment:

> That's taken straight out of the documentation.

Yes, but it's still a misleading comparison.

> Until I understood that exec with two different mapping objects as 
> globals and locals behaves as if the code where embedded inside a 
> class, I found the reported behaviour totally perplexing.

The basic execution model of Python is that a frame that executes with 
non-optimized locals -- in module and class definitions -- can use the same 
mapping for globals and locals. Indeed, that's how the interpreter executes 
modules. However, exec() is generalized to allow executing module code with 
separate globals and locals. 

Saying that code will be "executed as if it were embedded in a class 
definition" is correct only so far as the fact that globals and locals are 
different in this case. But it's also misleading because the code gets compiled 
as module-level code, not as class code.

It should be pretty obvious why the following fails:

exec("a = 1\ndef f(): return a\nprint(f())", {}, {})

Assignment is local by default, unless otherwise declared. Function f() has no 
access to the local scope where `a` is defined because Python doesn't support 
closures over non-optimized locals, particularly because we emphatically do not 
want that behavior for class definitions. 

It should be equally clear why the following succeeds:

exec("global a\na = 1\ndef f(): return a\nprint(f())", {}, {})

> because a class definition intentionally supports nonlocal closures, 
>
>I don't know what you mean by that. Classes are never closures. Only 
>functions can be closures.

I didn't say that a class can be a closure. That's never the case because a 
class uses non-optimized locals. But a class definition does support free 
variables that are bound to an enclosing scope. exec() does not support this, 
so the exact same code can execute differently in the context of a class 
definition.

> It is equivalent to code executed inside a class scope.

That depends on the code and the context. Please refer to my first example in 
comparison to the following:

a = 1
def f():
a = 2
exec('print(a)', globals(), {})

>>> f()
1

It's different behavior for print(a) because both exec() and compile(source, 
filename, 'exec') produce module code, not class code. The free variable `a` 
gets bound to the global scope for the exec() example, while for the class 
definition free variable `a` is bound to the local `a` in the frame of the 
function call.

To implement this different behavior, the code object for a class definition 
uses bytecode operations such as COPY_FREE_VARS and LOAD_CLASSDEREF, which are 
never used for module-level code. For example, from the original example, 
here's the class definition code:

>>> dis.dis(f.__code__.co_consts[2])
  0 COPY_FREE_VARS   1
  2 LOAD_NAME0 (__name__)
  4 STORE_NAME   1 (__module__)
  6 LOAD_CONST   0 ('f..C')
  8 STORE_NAME   2 (__qualname__)

  4  10 LOAD_NAME3 (print)
 12 LOAD_CLASSDEREF  0 (a)
 14 CALL_FUNCTION1
 16 POP_TOP
 18 LOAD_CONST   1 (None)
 20 RETURN_VALUE

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Steven D'Aprano]

Steven D'Aprano  added the comment:

On Thu, Dec 23, 2021 at 12:15:29AM +, Eryk Sun wrote:
> 
> Eryk Sun  added the comment:
> 
> > If exec gets two separate objects as globals and locals, 
> > the code will be executed as if it were embedded in a 
> > class definition.
> 
> That's a misleading comparison 

That's taken straight out of the documentation.

I don't think it is misleading, it is the opposite of misleading. Until 
I understood that exec with two different mapping objects as globals and 
locals behaves as if the code where embedded inside a class, I found the 
reported behaviour totally perplexing.

If you think it is wrong, how would you explain the observed behaviour, 
and how would you word the documentation?

> because a class definition intentionally supports nonlocal closures, 

I don't know what you mean by that. Classes are never closures. Only 
functions can be closures. (*Be* closures? *Have* a closure? The 
terminology is ambiguous.)

>>> def f():
... a = 1
... class C:
... nonlocal a
... a = 999
... print(a)
... return C
... 
>>> C = f()
999
>>> C.__closure__
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: type object 'C' has no attribute '__closure__'. Did you mean: 
'__module__'?

I don't know what terminology is appropriate here, but "closure" is 
surely not it.

> which exec() doesn't support and shouldn't support. For example:
[snip examples]

Neither of those cases are relevant to the example here.

> exec() executes as module code. Using separate globals and locals 
> mappings doesn't magically change how the code is compiled and 
> executed to make it equivalent to a class definition.

Neither I nor the documentation said it was equivalent to a class 
definition. It is equivalent to code executed inside a class scope.

> To understand 
> the case of separate globals and locals, just remember that assigning 
> to a variable by default makes it a local variable, unless it's 
> declared as a global. Also, class and function definitions are 
> implicitly an assignment, which by default will be local.

Neither of those facts explain why the example code

"""a = 1
def f():
return a
print(f())
"""

behaves differently when given two distinct dicts as the globals and 
locals parameters, versus all the other cases (no arguments provided, or 
one argument, or the same dict repeated twice).

Only the case where the provided globals and locals dicts are distinct 
behaves differently, and it behaves exactly the same as if you embedded 
that chunk of code inside a class definition and then executed it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Eryk Sun]

Eryk Sun  added the comment:

> If exec gets two separate objects as globals and locals, 
> the code will be executed as if it were embedded in a 
> class definition.

That's a misleading comparison because a class definition intentionally 
supports nonlocal closures, which exec() doesn't support and shouldn't support. 
For example:

a = 1

def f():
a = 2
class C:
print(a)

def g():
a = 2
class C:
nonlocal a
a = 3
print(a)

>>> f()
2
>>> g()
3

exec() executes as module code. Using separate globals and locals mappings 
doesn't magically change how the code is compiled and executed to make it 
equivalent to a class definition. To understand the case of separate globals 
and locals, just remember that assigning to a variable by default makes it a 
local variable, unless it's declared as a global. Also, class and function 
definitions are implicitly an assignment, which by default will be local.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Steven D'Aprano]

Steven D'Aprano  added the comment:

"Expected" is a strong word. It took me a lot of careful reading of the 
documentation and experimentation to decide that, yes, I expect the 
second case to fail when the first case succeeds.

Which reminds me of a common anecdote from mathematics:

https://hsm.stackexchange.com/questions/7247/in-a-popular-anecdote-who-took-20-minutes-to-decide-that-a-thing-was-obvious

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Quentin Peter]

Quentin Peter  added the comment:

Thank you for your explaination. Just to be sure, it is expected that:

exec("a = 1\ndef f(): return a\nprint(f())", {})

Runs successfully but

exec("a = 1\ndef f(): return a\nprint(f())", {}, {})

Doesn't?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Steven D'Aprano]

Steven D'Aprano  added the comment:

> I now want to define a closure with exec. I might want to do something like:
> exec("def f(): return a", globals(), locals())

That doesn't create a closure.


> I would expect f() to look for a in the locals().

I'm sorry, but your expectation that f() will look for a in the locals dict is 
not correct. That's not how name resolution in Python works. a is looked up as 
a global. You can't turn it into a local variable just by providing locals.

The names of the parameters are unfortunately confusing. The globals parameter 
is always the global namespace. But locals is *never* the function's local 
namespace. Nor is it a surrounding scope (nested functions), but it may be 
treated as a surrounding *class* scope.

I agree that the behaviour is surprising and complex, but if you work through 
the documentation carefully, it is behaving as designed.

What we need to realise is that locals describes the namespace where the *def 
statement* runs, not the namespace used by the body of the function. The 
function body's locals is always created when the function is called, it is 
inaccessible from outside the function, and it most certainly does not use the 
so-called "locals" parameter given to exec().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Steven D'Aprano]

Steven D'Aprano  added the comment:

Here is the key phrase in the docs:

"If exec gets two separate objects as globals and locals, the code will be 
executed as if it were embedded in a class definition."

https://docs.python.org/3/library/functions.html#exec

And sure enough:

>>> class C:
... a = 1
... def f():
... return a  # This looks for global a, not C.a
... print(f())
... 
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 5, in C
  File "", line 4, in f
NameError: name 'a' is not defined

which is intentional behaviour. Functions defined inside a class do not have 
direct access to the variables inside the class. I thought there was a FAQ 
about this but I can't find it now.

So there is no bug here. By passing two distinct dicts as the globals and 
locals to exec, the interpreter treats the code as if it were being executed 
inside the body of a class statement. Both the a and the f get created in the 
locals dict, not the globals dict:

>>> g = {'__builtins__': None}
>>> l = {}
>>> exec("""a = 1
... def f():
... return a
... """, g, l)
>>> g
{'__builtins__': None}
>>> l
{'a': 1, 'f': }

But when you call f(), it is looking for a in the globals dict.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue46153>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Quentin Peter]

Quentin Peter  added the comment:

The reason I am asking is that I am working on a debugger. The debugger stops 
on a frame which is inside a function. Let's say the locals is:
locals() == {"a": 1}
I now want to define a closure with exec. I might want to do something like:
exec("def f(): return a", globals(), locals())
But this doesn't work because of the issue I describe.I would expect f() to 
look for a in the locals().

Even more surprising is that if I use the second argument of exec, the code in 
the above comment starts to fail.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] function fails in exec when locals is given

[Steven D'Aprano]

Steven D'Aprano  added the comment:

The function you use in exec is not a closure. The function:

def f():
return a

does not capture the top-level variable "a", it does a normal name lookup for 
a. You can check this yourself by looking at f.__closure__ which you will see 
is None. Or you can use the dis module to look at the disassembled bytecode.

To be a closure, you have to insert both the "a" and the `def f()` inside 
another function, and then run that:

code = """
def outer():
a = 1
def f():
return a
return f

f = outer()
print(f())
"""
exec(code, {}, {})


prints 1 as expected.

--
components: +Interpreter Core
nosy: +steven.daprano
title: closure fails in exec when locals is given -> function fails in exec 
when locals is given
type: crash -> behavior

___
Python tracker 
<https://bugs.python.org/issue46153>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] closure fails in exec when locals is given

[Quentin Peter]

Quentin Peter  added the comment:

This might be related to https://bugs.python.org/issue41918

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46153] closure fails in exec when locals is given

[Quentin Peter]

New submission from Quentin Peter :

When both namespace arguments are given to exec, function definitions fail to 
capture closure. See below:
```
Python 3.8.6 (default, Oct  8 2020, 14:06:32) 
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exec("a = 1\ndef f(): return a\nprint(f())")
1
>>> exec("a = 1\ndef f(): return a\nprint(f())", {})
1
>>> exec("a = 1\ndef f(): return a\nprint(f())", {}, {})
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in 
  File "", line 2, in f
NameError: name 'a' is not defined
>>> 
```

--
messages: 409038
nosy: qpeter
priority: normal
severity: normal
status: open
title: closure fails in exec when locals is given
type: crash
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue46153>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: exec and locals

[Alister]

On Thu, 27 Feb 2014 00:31:56 +, Steven D'Aprano wrote:

 On Wed, 26 Feb 2014 14:00:59 +, Alister wrote:
 
 On Wed, 26 Feb 2014 13:15:25 +, Steven D'Aprano wrote:
 
 I have to dynamically generate some code inside a function using exec,
 but I'm not sure if it is working by accident or if I can rely on it.
 [...]
 
 I have no idea but as exec is generally considered to be a bad idea are
 you absolutely sure this is the correct way to achieve your end goal?
 
 perhaps if you detailed your requirement someone may be able to suggest
 a safer solution.
 
 Thanks for your concern, but what I'm doing is perfectly safe. The
 string being exec'ed is a string literal known at compile-time and
 written by me (see my previous email for details) and the only reason
 I'm running it with exec at runtime rather than treating it as normal
 source code is that it relies on a feature that may not be available
 (with statement).
 
 The joys of writing code that has to run under multiple incompatible
 versions of Python.

I hadn't noticed who had made the original post or a I wouldn't have 
bothered, you know Python  what is safe far better than I ever will




-- 
The world is not octal despite DEC.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Gregory Ewing]

Steven D'Aprano wrote:

On Thu, 27 Feb 2014 16:34:33 +1300, Gregory Ewing wrote:


Why not just use this version all the time? It should work in both 2.x
and 3.x.


Because that's yucky. It's an aesthetic thing: when supported, I want the 
Python interpreter to manage the context manager.


More yucky than wrapping the Py3 version in an
exec? To my way of thinking, that cancels out any
elegance that might have been gained from using
a with-statement.

Do you really need to use the context manager
at all? Could you just write the try-statement
that you would have written in Py2 if you
didn't have a context manager?

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Chris Angelico]

On Thu, Feb 27, 2014 at 10:29 PM, Gregory Ewing
greg.ew...@canterbury.ac.nz wrote:
 Steven D'Aprano wrote:

 On Thu, 27 Feb 2014 16:34:33 +1300, Gregory Ewing wrote:

 Why not just use this version all the time? It should work in both 2.x
 and 3.x.


 Because that's yucky. It's an aesthetic thing: when supported, I want the
 Python interpreter to manage the context manager.


 More yucky than wrapping the Py3 version in an
 exec? To my way of thinking, that cancels out any
 elegance that might have been gained from using
 a with-statement.

 Do you really need to use the context manager
 at all? Could you just write the try-statement
 that you would have written in Py2 if you
 didn't have a context manager?

If I have to support two vastly different versions, I would prefer
(when possible) to write the code so that dropping the old version's
support is simply a matter of deleting stuff. Write the code for the
new version, then warp it as little as possible to support the old
version as well, and keep it clear which bits are for the old. Writing
code that avoids 'with' altogether goes against that.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Steven D'Aprano]

On Fri, 28 Feb 2014 00:29:35 +1300, Gregory Ewing wrote:

 Steven D'Aprano wrote:
 On Thu, 27 Feb 2014 16:34:33 +1300, Gregory Ewing wrote:
 
Why not just use this version all the time? It should work in both 2.x
and 3.x.
 
 Because that's yucky. It's an aesthetic thing: when supported, I want
 the Python interpreter to manage the context manager.
 
 More yucky than wrapping the Py3 version in an exec? To my way of
 thinking, that cancels out any elegance that might have been gained from
 using a with-statement.
 
 Do you really need to use the context manager at all? Could you just
 write the try-statement that you would have written in Py2 if you didn't
 have a context manager?

I don't *have* to do it any particular way, but the way it works now, the 
version of the inner function (which does eventually get exposed to the 
caller) is simple with statement wrapping a function call. So for seven 
of the eight versions of Python supported (2.5 through 3.4) the function 
is the simplest it can be. Even if the code creating that function is a 
tiny bit more complex, since it is wrapped inside an exec. For the other 
two versions (2.4 and 2.5), I have to fall back on a more complex chunk 
of code. (And yes, it's deliberate that 2.5 gets counted in both groups.)

I'm not saying that I have objective reasons for preferring this way over 
the manual alternative, or at least not *strong* objective reasons. It's 
mostly subjective. I don't expect to convince you my way is better, and I 
doubt that you'll convince me your way is better. But if I were to try, 
I'd put it this way:

At a cost of six (by memory) extra lines of code, including one call to 
exec, I have an inner function which is *guaranteed* to use the exact 
same semantics and efficiency of a with-statement when possible, because 
it *is* a with-statement. Any differences between with-statement and my 
manual handling will only affect 2.4 and sometimes 2.5, rather than 
everything. The only differences that I know of are insignificant -- the 
byte-code will be different, there may be trivial performance differences 
-- but if it turns out to be some meaningful difference, I have three 
choices:

  1) deny that the difference is meaningful;

  2) accept that the difference is meaningful, and fix it; or

  3) accept that the difference is meaningful, but say that it 
 won't be fixed for 2.4 and 2.5.


If I use the same manual handling for all versions, then I don't have 
that last option, no matter how unlikely it is that I will need it.

But really, it's a subjective choice of what feels right to me in this 
instance. If the body of the with-statement was bigger, or if the feature 
in question was something else, I might choose a different approach.



-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

exec and locals

[Steven D'Aprano]

I have to dynamically generate some code inside a function using exec, 
but I'm not sure if it is working by accident or if I can rely on it.

Here is a trivial example:


py def spam():
... exec( x = 23 )
... return x
...
py spam()
23


(My real example is more complex than this.)

According to the documentation of exec, I don't think this should 
actually work, and yet it appears to. The documentation says:

The default locals act as described for function locals() 
below: modifications to the default locals dictionary should 
not be attempted. Pass an explicit locals dictionary if you 
need to see effects of the code on locals after function 
exec() returns.

http://docs.python.org/3.4/library/functions.html#exec


I *think* this means that if I want to guarantee that a local variable x 
is created by exec, I need to do this instead:

py def eggs():
... mylocals = {}
... exec( x = 23, globals(), mylocals)
... x = mylocals['x']
... return x
...
py eggs()
23

The fact that it works in spam() above is perhaps an accident of 
implementation? Yes no maybe?



-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Chris Angelico]

On Thu, Feb 27, 2014 at 12:15 AM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
 py def spam():
 ... exec( x = 23 )
 ... return x
 ...
 py spam()
 23


 (My real example is more complex than this.)

 According to the documentation of exec, I don't think this should
 actually work, and yet it appears to.

Doesn't work for me, in IDLE in 3.4.0b2, nor in command-line Python on
3.4.0rc1+ (from hg a couple of weeks ago). But it did (happen to?)
work in 2.7. What version did you use?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Peter Otten]

Steven D'Aprano wrote:

 I have to dynamically generate some code inside a function using exec,
 but I'm not sure if it is working by accident or if I can rely on it.
 
 Here is a trivial example:
 
 
 py def spam():
 ... exec( x = 23 )
 ... return x
 ...
 py spam()
 23
 
 
 (My real example is more complex than this.)
 
 According to the documentation of exec, I don't think this should
 actually work, and yet it appears to. The documentation says:
 
 The default locals act as described for function locals()
 below: modifications to the default locals dictionary should
 not be attempted. Pass an explicit locals dictionary if you
 need to see effects of the code on locals after function
 exec() returns.
 
 http://docs.python.org/3.4/library/functions.html#exec
 
 
 I *think* this means that if I want to guarantee that a local variable x
 is created by exec, I need to do this instead:
 
 py def eggs():
 ... mylocals = {}
 ... exec( x = 23, globals(), mylocals)
 ... x = mylocals['x']
 ... return x
 ...
 py eggs()
 23
 
 The fact that it works in spam() above is perhaps an accident of
 implementation? Yes no maybe?

eggs() should work in Python 2 and 3,
spam() should work in Python 2, but not in Python 3.

Fun fact: Python 2 tweaks the bytecode (LOAD_NAME instead of LOAD_GLOBAL) to 
make spam() work:

 def spam():
... return x
... 
 dis.dis(spam)
  2   0 LOAD_GLOBAL  0 (x)
  3 RETURN_VALUE
 def spam():
... exec 
... return x
... 
 dis.dis(spam)
  2   0 LOAD_CONST   1 ('')
  3 LOAD_CONST   0 (None)
  6 DUP_TOP 
  7 EXEC_STMT   

  3   8 LOAD_NAME0 (x)
 11 RETURN_VALUE


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Peter Otten]

Peter Otten wrote:

 Steven D'Aprano wrote:
 
 I have to dynamically generate some code inside a function using exec,
 but I'm not sure if it is working by accident or if I can rely on it.
 
 Here is a trivial example:
 
 
 py def spam():
 ... exec( x = 23 )
 ... return x
 ...
 py spam()
 23
 
 
 (My real example is more complex than this.)
 
 According to the documentation of exec, I don't think this should
 actually work, and yet it appears to. The documentation says:
 
 The default locals act as described for function locals()
 below: modifications to the default locals dictionary should
 not be attempted. Pass an explicit locals dictionary if you
 need to see effects of the code on locals after function
 exec() returns.
 
 http://docs.python.org/3.4/library/functions.html#exec
 
 
 I *think* this means that if I want to guarantee that a local variable x
 is created by exec, I need to do this instead:
 
 py def eggs():
 ... mylocals = {}
 ... exec( x = 23, globals(), mylocals)
 ... x = mylocals['x']
 ... return x
 ...
 py eggs()
 23
 
 The fact that it works in spam() above is perhaps an accident of
 implementation? Yes no maybe?
 
 eggs() should work in Python 2 and 3,
 spam() should work in Python 2, but not in Python 3.
 
 Fun fact: Python 2 tweaks the bytecode (LOAD_NAME instead of LOAD_GLOBAL)
 to make spam() work:
 
 def spam():
 ... return x
 ...
 dis.dis(spam)
   2   0 LOAD_GLOBAL  0 (x)
   3 RETURN_VALUE
 def spam():
 ... exec 
 ... return x
 ...
 dis.dis(spam)
   2   0 LOAD_CONST   1 ('')
   3 LOAD_CONST   0 (None)
   6 DUP_TOP
   7 EXEC_STMT
 
   3   8 LOAD_NAME0 (x)
  11 RETURN_VALUE

Some more bytcode fun, because it just occured to me that you can optimize 
away the code that triggered the modification:

 def spam():
... return x
... if 0: exec 
... 
 dis.dis(spam)
  2   0 LOAD_NAME0 (x)
  3 RETURN_VALUE


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Alister]

On Wed, 26 Feb 2014 13:15:25 +, Steven D'Aprano wrote:

 I have to dynamically generate some code inside a function using exec,
 but I'm not sure if it is working by accident or if I can rely on it.
 
 Here is a trivial example:
 
 
 py def spam():
 ... exec( x = 23 )
 ... return x ...
 py spam()
 23
 
 
 (My real example is more complex than this.)
 
 According to the documentation of exec, I don't think this should
 actually work, and yet it appears to. The documentation says:
 
 The default locals act as described for function locals() below:
 modifications to the default locals dictionary should not be
 attempted. Pass an explicit locals dictionary if you need to see
 effects of the code on locals after function exec() returns.
 
 http://docs.python.org/3.4/library/functions.html#exec
 
 
 I *think* this means that if I want to guarantee that a local variable x
 is created by exec, I need to do this instead:
 
 py def eggs():
 ... mylocals = {}
 ... exec( x = 23, globals(), mylocals)
 ... x = mylocals['x']
 ... return x ...
 py eggs()
 23
 
 The fact that it works in spam() above is perhaps an accident of
 implementation? Yes no maybe?

I have no idea but as exec is generally considered to be a bad idea are 
you absolutely sure this is the correct way to achieve your end goal?

perhaps if you detailed your requirement someone may be able to suggest a 
safer solution.


-- 
Regardless of the legal speed limit, your Buick must be operated at
speeds faster than 85 MPH (140kph).
-- 1987 Buick Grand National owners manual.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Steven D'Aprano]

On Wed, 26 Feb 2014 14:46:39 +0100, Peter Otten wrote:

 Steven D'Aprano wrote:
 
 I have to dynamically generate some code inside a function using exec,
 but I'm not sure if it is working by accident or if I can rely on it.
 
 Here is a trivial example:
 
 
 py def spam():
 ... exec( x = 23 )
 ... return x
 ...
 py spam()
 23
 
 
 (My real example is more complex than this.)
 
 According to the documentation of exec, I don't think this should
 actually work, and yet it appears to. The documentation says:
 
 The default locals act as described for function locals() below:
 modifications to the default locals dictionary should not be
 attempted. Pass an explicit locals dictionary if you need to see
 effects of the code on locals after function exec() returns.
 
 http://docs.python.org/3.4/library/functions.html#exec
 
 
 I *think* this means that if I want to guarantee that a local variable
 x is created by exec, I need to do this instead:
 
 py def eggs():
 ... mylocals = {}
 ... exec( x = 23, globals(), mylocals) ... x =
 mylocals['x']
 ... return x
 ...
 py eggs()
 23
 
 The fact that it works in spam() above is perhaps an accident of
 implementation? Yes no maybe?
 
 eggs() should work in Python 2 and 3, spam() should work in Python 2,
 but not in Python 3.

Aha! That explains it -- I was reading the 3.x docs and testing in Python 
2.7.

Thanks everyone for answering.

By the way, if anyone cares what my actual use-case is, I have a function 
that needs to work under Python 2.4 through 3.4, and it uses a with 
statement. With statements are not available in 2.4 (or 2.5, unless you 
give a from __future__ import). So after messing about for a while with 
circular imports and dependency injections, I eventually settled on some 
code that works something like this:


def factory():
blah blah blah
try:
exec(def inner():
with something:
return something
, globals(), mylocals)
inner = mylocals['inner']
except SyntaxError:
def inner():
# manually operate the context manager
call context manager __enter__
try:
try:
return something
except:  # Yes, a bare except. Catch EVERYTHING.
blah blah blah
finally:
call context manager __exit__
blah blah blah
return inner


(By the way, yes, I have to use a bare except, not just except 
BaseException. Python 2.4 and 2.5 still have string exceptions.)


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Steven D'Aprano]

On Wed, 26 Feb 2014 14:00:59 +, Alister wrote:

 On Wed, 26 Feb 2014 13:15:25 +, Steven D'Aprano wrote:
 
 I have to dynamically generate some code inside a function using exec,
 but I'm not sure if it is working by accident or if I can rely on it.
[...]

 I have no idea but as exec is generally considered to be a bad idea are
 you absolutely sure this is the correct way to achieve your end goal?
 
 perhaps if you detailed your requirement someone may be able to suggest
 a safer solution.

Thanks for your concern, but what I'm doing is perfectly safe. The string 
being exec'ed is a string literal known at compile-time and written by me 
(see my previous email for details) and the only reason I'm running it 
with exec at runtime rather than treating it as normal source code is 
that it relies on a feature that may not be available (with statement).

The joys of writing code that has to run under multiple incompatible 
versions of Python.


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Gregory Ewing]

Steven D'Aprano wrote:

except SyntaxError:
def inner():
# manually operate the context manager
call context manager __enter__
try:
try:
return something
except:  # Yes, a bare except. Catch EVERYTHING.
blah blah blah
finally:
call context manager __exit__


Why not just use this version all the time? It should
work in both 2.x and 3.x.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Dan Sommers]

On Thu, 27 Feb 2014 00:25:45 +, Steven D'Aprano wrote:

 By the way, if anyone cares what my actual use-case is, I have a
 function that needs to work under Python 2.4 through 3.4, and it uses
 a with statement. With statements are not available in 2.4 (or 2.5,
 unless you give a from __future__ import). So after messing about for
 a while with circular imports and dependency injections, I eventually
 settled on some code that works something like this:

 def factory():
 blah blah blah
 try:
 exec(def inner():
 with something:
 return something
 , globals(), mylocals)
 inner = mylocals['inner']
 except SyntaxError:
 def inner():
 # manually operate the context manager
 call context manager __enter__
 try:
 try:
 return something
 except:  # Yes, a bare except. Catch EVERYTHING.
 blah blah blah
 finally:
 call context manager __exit__
 blah blah blah
 return inner

So why not something simpler?

def factory():

def inner():
'''Manually operate the context manager in order to maintain
compatibility with Python 2.4 through 3.4.'''
call context manager __enter__
try:
try:
return something
except:  # Yes, a bare except. Catch EVERYTHING.
blah blah blah
finally:
call context manager __exit__
blah blah blah
return inner

I claim that the less unnecessary code you write, the fewer bugs you
will have.

Does my code misbehave under any of your target versions?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Dave Angel]

 Steven D'Aprano st...@pearwood.info Wrote in message:
 On Wed, 26 Feb 2014 14:46:39 +0100, Peter Otten wrote:
 
 Steven D'Aprano wrote:
 
 I have to dynamically generate some code inside a function using exec,
 but I'm not sure if it is working by accident or if I can rely on it.
 
 I eventually settled on some 
 code that works something like this:
 
 
 def factory():
 blah blah blah
 try:
 exec(def inner():
 

Before I would use exec,  I'd look hard at either generating a
 source file to import,  or using a preprocessor.  And if this
 code was to be installed,  make the version choice or the
 preprocess step happen at install time.

I once implemented a system that generated 20k lines of C++ header
 and sources. And the generated code was properly indented and
 fairly well commented. 



-- 
DaveA

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Steven D'Aprano]

On Thu, 27 Feb 2014 16:34:33 +1300, Gregory Ewing wrote:

 Steven D'Aprano wrote:
 except SyntaxError:
 def inner():
 # manually operate the context manager call context manager
 __enter__
 try:
 try:
 return something
 except:  # Yes, a bare except. Catch EVERYTHING.
 blah blah blah
 finally:
 call context manager __exit__
 
 Why not just use this version all the time? It should work in both 2.x
 and 3.x.

Because that's yucky. It's an aesthetic thing: when supported, I want the 
Python interpreter to manage the context manager.

The exec part is only half a dozen lines, only three lines of source 
code. It's no burden to keep it for the cases where it works (that is, at 
least 2.6 onwards).


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Steven D'Aprano]

On Wed, 26 Feb 2014 23:20:10 -0500, Dave Angel wrote:

 Before I would use exec,  I'd look hard at either generating a
  source file to import,  

Yes, I went through the process of pulling out the code into a separate 
module, but that just made more complexity and was pretty nasty. If the 
function was stand-alone, it might have worked, but it needed access to 
other code in the module, so there were circular dependencies.


 or using a preprocessor. 

I don't think that it's easier/better to write a custom Python 
preprocessor and run the entire module through it, just to avoid a three-
line call to exec.

Guys, I know that exec is kinda dangerous and newbies should be 
discouraged from throwing every string they see at it, but this isn't my 
second day Python programming, and it's not an accident that Python 
supports the dynamic compilation and execution of source code at runtime. 
It's a deliberate language feature. We're allowed to use it :-)


 And if this code was
  to be installed,  make the version choice or the preprocess step happen
  at install time.

Completely inappropriate in my case. This is a module which can be called 
from multiple versions of Python from a single installation.


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and locals

[Chris Angelico]

On Thu, Feb 27, 2014 at 3:47 PM, Steven D'Aprano st...@pearwood.info wrote:
 Guys, I know that exec is kinda dangerous and newbies should be
 discouraged from throwing every string they see at it, but this isn't my
 second day Python programming, and it's not an accident that Python
 supports the dynamic compilation and execution of source code at runtime.
 It's a deliberate language feature. We're allowed to use it :-)

Code smell means look at this. It doesn't mean don't use this
feature ever. :) Steven's looked into this thoroughly, I'm sure, and
exec is important.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: problem with exec and locals()

[Uwe Schmitt]

On 1 Jul., 15:15, Mel [EMAIL PROTECTED] wrote:
 rocksportrockerwrote:

  Hi,

  the following code does not work until I ommit the a=0 statement.

     def test():
         exec a=3 in locals()
         print a
         a=0

      test()

  print raises:
       UnboundLocalError: local variable 'a' referenced before
  assignment

  Can anybody explain what is going wrong here ?

 AFAIK, local variables are implemented rather like __slots__ in new-style
 classes.  This is a very valuable efficiency measure, but it can cause this
 kind of trouble.  Without `a=0`, the bytecode compiler makes no slot for a,
 and dis.dis shows the following bytecode for test: dis.dis (test)

   2           0 LOAD_CONST               1 ('a=3')
               3 LOAD_NAME                0 (locals)
               6 CALL_FUNCTION            0
               9 DUP_TOP
              10 EXEC_STMT

   3          11 LOAD_NAME                1 (a)
              14 PRINT_ITEM
              15 PRINT_NEWLINE
              16 LOAD_CONST               0 (None)
              19 RETURN_VALUE

 At address 11, LOAD_NAME 1(a) gets the value that was set by exec.

 With a=0, the code is dis.dis(test2)

   2           0 LOAD_CONST               1 ('a=4')
               3 LOAD_NAME                0 (locals)
               6 CALL_FUNCTION            0
               9 DUP_TOP
              10 EXEC_STMT

   3          11 LOAD_FAST                0 (a)
              14 PRINT_ITEM
              15 PRINT_NEWLINE

   4          16 LOAD_CONST               2 (0)
              19 STORE_FAST               0 (a)
              22 LOAD_CONST               0 (None)
              25 RETURN_VALUE

 and here, the value of a is found in slot 0 via LOAD_FAST.  Slot 0 is used
 because a=0 forced a to be a local variable.

 Apparently, exec in locals() knows nothing about slots (because locals() is
 the only dictionary in the universe where slots would be involved ? --
 perhaps not, but close).

         Mel.

Thanks for your answer. I wonder if this is a bug, or did I miss
something
in the docs ???

Greetings, Uwe
--
http://mail.python.org/mailman/listinfo/python-list

Re: problem with exec and locals()

[Peter Otten]

Uwe Schmitt wrote:

 Apparently, exec in locals() knows nothing about slots (because locals()
 is the only dictionary in the universe where slots would be involved ? --
 perhaps not, but close).

 Mel.
 
 Thanks for your answer. I wonder if this is a bug, or did I miss
 something in the docs ???
 
Hm, the documentation has an explicit warning:

http://docs.python.org/lib/built-in-funcs.html#l2h-47


locals(  )
 Update and return a dictionary representing the current local symbol table.
Warning: The contents of this dictionary should not be modified; changes
may not affect the values of local variables used by the interpreter.


By the way, the local namespace is affected if you don't provide it
explicitly:

 def f():
... exec a=42
... print a
... a = whatever
...
 f()
42

Peter

--
http://mail.python.org/mailman/listinfo/python-list

Re: problem with exec and locals()

[Gabriel Genellina]

En Fri, 11 Jul 2008 03:51:39 -0300, Uwe Schmitt  
[EMAIL PROTECTED] escribi�:



On 1 Jul., 15:15, Mel [EMAIL PROTECTED] wrote:

rocksportrockerwrote:

 the following code does not work until I ommit the a=0 statement.

    def test():
        exec a=3 in locals()
        print a
        a=0

     test()

 print raises:
      UnboundLocalError: local variable 'a' referenced before
 assignment

 Can anybody explain what is going wrong here ?

AFAIK, local variables are implemented rather like __slots__ in  
new-style
classes.  This is a very valuable efficiency measure, but it can cause  
this
kind of trouble.  Without `a=0`, the bytecode compiler makes no slot  
for a,


Thanks for your answer. I wonder if this is a bug, or did I miss
something
in the docs ???


Read the warnings in the docs for the locals() builtin function:
http://docs.python.org/lib/built-in-funcs.html#l2h-47
and the execfile function:
http://docs.python.org/lib/built-in-funcs.html#l2h-26

--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

problem with exec and locals()

[rocksportrocker]

Hi,

the following code does not work until I ommit the a=0 statement.


   def test():
   exec a=3 in locals()
   print a
   a=0

test()

print raises:
 UnboundLocalError: local variable 'a' referenced before
assignment

Can anybody explain what is going wrong here ?

Greetings, Uwe
--
http://mail.python.org/mailman/listinfo/python-list

Re: problem with exec and locals()

[Mel]

rocksportrocker wrote:

 
 Hi,
 
 the following code does not work until I ommit the a=0 statement.
 
 
def test():
exec a=3 in locals()
print a
a=0
 
 test()
 
 print raises:
  UnboundLocalError: local variable 'a' referenced before
 assignment
 
 Can anybody explain what is going wrong here ?

AFAIK, local variables are implemented rather like __slots__ in new-style
classes.  This is a very valuable efficiency measure, but it can cause this
kind of trouble.  Without `a=0`, the bytecode compiler makes no slot for a,
and dis.dis shows the following bytecode for test:
 dis.dis (test)
  2   0 LOAD_CONST   1 ('a=3')
  3 LOAD_NAME0 (locals)
  6 CALL_FUNCTION0
  9 DUP_TOP
 10 EXEC_STMT

  3  11 LOAD_NAME1 (a)
 14 PRINT_ITEM
 15 PRINT_NEWLINE
 16 LOAD_CONST   0 (None)
 19 RETURN_VALUE

At address 11, LOAD_NAME 1(a) gets the value that was set by exec.

With a=0, the code is
 dis.dis(test2)
  2   0 LOAD_CONST   1 ('a=4')
  3 LOAD_NAME0 (locals)
  6 CALL_FUNCTION0
  9 DUP_TOP
 10 EXEC_STMT

  3  11 LOAD_FAST0 (a)
 14 PRINT_ITEM
 15 PRINT_NEWLINE

  4  16 LOAD_CONST   2 (0)
 19 STORE_FAST   0 (a)
 22 LOAD_CONST   0 (None)
 25 RETURN_VALUE

and here, the value of a is found in slot 0 via LOAD_FAST.  Slot 0 is used
because a=0 forced a to be a local variable.

Apparently, exec in locals() knows nothing about slots (because locals() is
the only dictionary in the universe where slots would be involved ? --
perhaps not, but close).

Mel.

--
http://mail.python.org/mailman/listinfo/python-list