Re: Local variable definition in Python list comprehension

[Dan Stromberg]

On Thu, Sep 1, 2022 at 9:16 AM Chris Angelico  wrote:

> On Fri, 2 Sept 2022 at 02:10, James Tsai  wrote:
> >
> > Hello,
> >
> > I find it very useful if I am allowed to define new local variables in a
> list comprehension. For example, I wish to have something like
> > [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
> > [(x, y) for x in range(10) with y := x ** 2 if x + y < 80].
> >
> > For now this functionality can be achieved by writing
> > [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].
> >
> > Is it worthwhile to add a new feature like this in Python? If so, how
> can I propose this to PEP?
>
> Not everything has to be a one-liner.
>
So true!

I like list comprehensions and generator expressions, but sometimes I end
up regretting their use when there's a bug, and I have to convert one to a
for loop + list.append in order to debug.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[James Tsai]

在 2022年9月2日星期五 UTC+2 00:17:23， 写道：
> On 02Sep2022 07:01, Chris Angelico  wrote: 
> >On Fri, 2 Sept 2022 at 06:55, James Tsai  wrote: 
> >> No but very often when I have written a neat list/dict/set 
> >> comprehension, I find it very necessary 
> >> to define local variable(s) to make it more clear and concise. Otherwise I 
> >> have to break it down 
> >> to several incrementally indented lines of for loops, if statements, and 
> >> variable assignments, 
> >> which I think look less nice. 
> > 
> >Well, if it's outgrown a list comp, write it on multiple lines. Like I 
> >said, not everything has to be a one-liner.
> True, but a comprehension can be more expressive than a less 
> "functional" expression (series of statements). 
> 
> James, can you provide (a) a real world example where you needed to 
> write a series of statements or loops and (b) a corresponding example of 
> how you would have preferred to have written that code, possibly 
> inventing some syntax or misusing ":=" as if it workeed they way you'd 
> like it to work? 
> 
> Cheers, 
> Cameron Simpson 

Yeah, I think list comprehension is particularly useful to construct a deeply 
nested list/dict. For example, I am now using Plotly to visualize a cellular 
network including several base stations and users. Here is the function I have 
written:

def plot_network(area, base_stations, users):
bs_poses = np.array([bs.pos for bs in base_stations])
ue_poses = np.array([ue.pos for ue in users])
fig = px.scatter(x=bs_poses[:, 0], y=bs_poses[:, 1])
fig.add_scatter(x=ue_poses[:, 0], y=ue_poses[:, 1])
fig.update_layout(
xaxis=dict(range=[0, area[0]], nticks=5),
yaxis=dict(range=[0, area[1]], nticks=5),
shapes=[dict(
type="circle",
fillcolor="PaleTurquoise",
x0=x-r, y0=y-r, x1=x+r, y1=y+r,
hovertext=f"({x:.2f}, {y:.2f})",
opacity=0.3
) for bs in base_stations for x, y in [bs.pos]
for r in [bs.cell_radius]],
)
return fig

Simply put, I want to scatter the BSs and users, and additionally I want to 
draw a big circle around each BS to represent its cell coverage. I can choose 
to write 'x0=bs.pos[0]-bs.cell_radius, y0=...' instead, but it becomes less 
concise, and if x, y, or r is the return value of a function instead of a 
property, it becomes more computationally expensive to repeat calling the 
function as well. I also can create the list of 'shapes' by appending to a 
list, like

shapes = []
for bs in base_stations:
   x, y = bs.pos
   r = bs.cell_radius
   shapes.append(dict(...))
fig.update_layout(
xaxis=dict(range=[0, area[0]], nticks=5),
yaxis=dict(range=[0, area[1]], nticks=5),
shapes=shapes
)

But in my opinion this is much less concise. I think it looks better to create 
the list within the nested structure. So I totally agree that list 
comprehension adds much expressiveness in Python. I only wonder whether it is a 
good idea to introduce a specific syntax for local variable assignment in list 
comprehensions, instead of using "for r in [bs.cell_radius]".
I am also surprised to know that the assignment operator ":=" in a list 
comprehension will assign a variable outside of the scope of the comprehension. 
I think it does not make sense since a list comprehension without a ":=" will 
never change name bindings outside itself.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[Avi Gross]

Dumb question. Your y is purely a function of x. So create an f(x) where
you want your y. It probably can even be anonymous inline. I mean your
return values of (x, y) would be (x, f(x)) ...

On Thu, Sep 1, 2022, 5:04 PM Chris Angelico  wrote:

> On Fri, 2 Sept 2022 at 06:55, James Tsai  wrote:
> >
> > 在 2022年9月1日星期四 UTC+2 18:34:36， 写道：
> > > On 9/1/22, James Tsai  wrote:
> > > >
> > > > I find it very useful if I am allowed to define new local variables
> in a
> > > > list comprehension. For example, I wish to have something like
> > > > [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
> > > > [(x, y) for x in range(10) with y := x ** 2 if x + y < 80].
> > > >
> > > > For now this functionality can be achieved by writing
> > > > [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].
> > > You can assign a local variable in the `if` expression. For example:
> > >
> > > >>> [(x, y) for x in range(10) if x + (y := x**2) < 30]
> > > [(0, 0), (1, 1), (2, 4), (3, 9), (4, 16)]
> >
> > Yeah this works great but like [(x, y) for x in range(10) for y in
> [x**2]] I written before, is kind of a hack. And if initially I do not need
> an "if" condition in the list comprehension, this becomes less convenient.
> I still can write
> > >>> [(x, y) for x in range(10) if (y := x**2) or True]
> >
> > But I wonder if Python could have a specific syntax to support this.
> >
>
> But why would you need to assign to y in that example? If you're using
> it more than once, you can use :=, and if you aren't, you don't need
> to. But do be aware that := does not create a comprehension-local name
> binding, but a nonlocal instead.
>
> > No but very often when I have written a neat list/dict/set
> comprehension, I find it very necessary
> > to define local variable(s) to make it more clear and concise. Otherwise
> I have to break it down
> > to several incrementally indented lines of for loops, if statements, and
> variable assignments,
> > which I think look less nice.
>
> Well, if it's outgrown a list comp, write it on multiple lines. Like I
> said, not everything has to be a one-liner.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[Cameron Simpson]

On 02Sep2022 07:01, Chris Angelico  wrote:
>On Fri, 2 Sept 2022 at 06:55, James Tsai  wrote:
>> No but very often when I have written a neat list/dict/set 
>> comprehension, I find it very necessary
>> to define local variable(s) to make it more clear and concise. Otherwise I 
>> have to break it down
>> to several incrementally indented lines of for loops, if statements, and 
>> variable assignments,
>> which I think look less nice.
>
>Well, if it's outgrown a list comp, write it on multiple lines. Like I
>said, not everything has to be a one-liner.

True, but a comprehension can be more expressive than a less 
"functional" expression (series of statements).

James, can you provide (a) a real world example where you needed to 
write a series of statements or loops and (b) a corresponding example of 
how you would have preferred to have written that code, possibly 
inventing some syntax or misusing ":=" as if it workeed they way you'd 
like it to work?

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[Peter J. Holzer]

On 2022-09-01 13:33:16 -0700, James Tsai wrote:
> 在 2022年9月1日星期四 UTC+2 18:34:36， 写道：
> > On 9/1/22, James Tsai  wrote: 
> > > 
> > > I find it very useful if I am allowed to define new local variables in a 
> > > list comprehension. For example, I wish to have something like 
> > > [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or 
> > > [(x, y) for x in range(10) with y := x ** 2 if x + y < 80]. 
> > > 
> > > For now this functionality can be achieved by writing 
> > > [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].
> > You can assign a local variable in the `if` expression. For example: 
> > 
> > >>> [(x, y) for x in range(10) if x + (y := x**2) < 30] 
> > [(0, 0), (1, 1), (2, 4), (3, 9), (4, 16)]
> 
> Yeah this works great but like [(x, y) for x in range(10) for y in
> [x**2]] I written before, is kind of a hack. And if initially I do not
> need an "if" condition in the list comprehension, this becomes less
> convenient. I still can write 
> >>> [(x, y) for x in range(10) if (y := x**2) or True]

In that case 
[(x, x**2) for x in range(10)]
seems to be somewhat more readable.

I would also say that
[(x, x**2) for x in range(10) if x + x**2 < 80]
doesn't really seem worse than any of the variants with y in it.
(Yes, I get that your real duplicated expression is probably a bit more
complex than `x**2`, but by the time a temporary variable really
improves readability it's probably a good time to split that across
multiple lines, too.)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[Chris Angelico]

On Fri, 2 Sept 2022 at 06:55, James Tsai  wrote:
>
> 在 2022年9月1日星期四 UTC+2 18:34:36， 写道：
> > On 9/1/22, James Tsai  wrote:
> > >
> > > I find it very useful if I am allowed to define new local variables in a
> > > list comprehension. For example, I wish to have something like
> > > [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
> > > [(x, y) for x in range(10) with y := x ** 2 if x + y < 80].
> > >
> > > For now this functionality can be achieved by writing
> > > [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].
> > You can assign a local variable in the `if` expression. For example:
> >
> > >>> [(x, y) for x in range(10) if x + (y := x**2) < 30]
> > [(0, 0), (1, 1), (2, 4), (3, 9), (4, 16)]
>
> Yeah this works great but like [(x, y) for x in range(10) for y in [x**2]] I 
> written before, is kind of a hack. And if initially I do not need an "if" 
> condition in the list comprehension, this becomes less convenient. I still 
> can write
> >>> [(x, y) for x in range(10) if (y := x**2) or True]
>
> But I wonder if Python could have a specific syntax to support this.
>

But why would you need to assign to y in that example? If you're using
it more than once, you can use :=, and if you aren't, you don't need
to. But do be aware that := does not create a comprehension-local name
binding, but a nonlocal instead.

> No but very often when I have written a neat list/dict/set comprehension, I 
> find it very necessary
> to define local variable(s) to make it more clear and concise. Otherwise I 
> have to break it down
> to several incrementally indented lines of for loops, if statements, and 
> variable assignments,
> which I think look less nice.

Well, if it's outgrown a list comp, write it on multiple lines. Like I
said, not everything has to be a one-liner.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[James Tsai]

在 2022年9月1日星期四 UTC+2 18:16:03， 写道：
> On Fri, 2 Sept 2022 at 02:10, James Tsai  wrote: 
> > 
> > Hello, 
> > 
> > I find it very useful if I am allowed to define new local variables in a 
> > list comprehension. For example, I wish to have something like 
> > [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or 
> > [(x, y) for x in range(10) with y := x ** 2 if x + y < 80]. 
> > 
> > For now this functionality can be achieved by writing 
> > [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80]. 
> > 
> > Is it worthwhile to add a new feature like this in Python? If so, how can I 
> > propose this to PEP?
> Not everything has to be a one-liner. 
> 
> ChrisA

No but very often when I have written a neat list/dict/set comprehension, I 
find it very necessary to define local variable(s) to make it more clear and 
concise. Otherwise I have to break it down to several incrementally indented 
lines of for loops, if statements, and variable assignments, which I think look 
less nice.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[James Tsai]

在 2022年9月1日星期四 UTC+2 18:34:36， 写道：
> On 9/1/22, James Tsai  wrote: 
> > 
> > I find it very useful if I am allowed to define new local variables in a 
> > list comprehension. For example, I wish to have something like 
> > [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or 
> > [(x, y) for x in range(10) with y := x ** 2 if x + y < 80]. 
> > 
> > For now this functionality can be achieved by writing 
> > [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].
> You can assign a local variable in the `if` expression. For example: 
> 
> >>> [(x, y) for x in range(10) if x + (y := x**2) < 30] 
> [(0, 0), (1, 1), (2, 4), (3, 9), (4, 16)]

Yeah this works great but like [(x, y) for x in range(10) for y in [x**2]] I 
written before, is kind of a hack. And if initially I do not need an "if" 
condition in the list comprehension, this becomes less convenient. I still can 
write 
>>> [(x, y) for x in range(10) if (y := x**2) or True]

But I wonder if Python could have a specific syntax to support this.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[James Tsai]

在 2022年9月1日星期四 UTC+2 16:15:17， 写道：
> James Tsai  writes: 
> 
> > I find it very useful if I am allowed to define new local variables in 
> > a list comprehension. For example, I wish to have something like 
> > [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or 
> > [(x, y) for x in range(10) with y := x ** 2 if x + y < 80]. 
> > 
> > For now this functionality can be achieved by writing 
> > [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].
> x and y are, to a first approximation, new local variables defined in a 
> list comprehension. I think you need to restate what it is you want.
> > Is it worthwhile to add a new feature like this in Python? If so, how 
> > can I propose this to PEP?
> To make any sort of case you'd need to give an example that does not 
> have a clearer way to write it already. Your working version is, to me, 
> clearer that the ones you want to be able to write. 
> 
> -- 
> Ben.

By local variable definition I mean binding a variable to a single value, so it 
doesn't include giving an iterable that a variable can take values iteratively, 
e.g. 'for x in range(10)'. Does it not worth introducing a specific syntax to 
do this, instead of creating a new list ad hoc to define the variable like 'for 
y in [1]'?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[Eryk Sun]

On 9/1/22, James Tsai  wrote:
>
> I find it very useful if I am allowed to define new local variables in a
> list comprehension. For example, I wish to have something like
> [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
> [(x, y) for x in range(10) with y := x ** 2 if x + y < 80].
>
> For now this functionality can be achieved by writing
> [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].

You can assign a local variable in the `if` expression. For example:

>>> [(x, y) for x in range(10) if x + (y := x**2) < 30]
[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16)]
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[Chris Angelico]

On Fri, 2 Sept 2022 at 02:10, James Tsai  wrote:
>
> Hello,
>
> I find it very useful if I am allowed to define new local variables in a list 
> comprehension. For example, I wish to have something like
> [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
> [(x, y) for x in range(10) with y := x ** 2 if x + y < 80].
>
> For now this functionality can be achieved by writing
> [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].
>
> Is it worthwhile to add a new feature like this in Python? If so, how can I 
> propose this to PEP?

Not everything has to be a one-liner.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Local variable definition in Python list comprehension

[Ben Bacarisse]

James Tsai  writes:

> I find it very useful if I am allowed to define new local variables in
> a list comprehension. For example, I wish to have something like
> [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
> [(x, y) for x in range(10) with y := x ** 2 if x + y < 80].
>
> For now this functionality can be achieved by writing
> [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].

x and y are, to a first approximation, new local variables defined in a
list comprehension.  I think you need to restate what it is you want.

> Is it worthwhile to add a new feature like this in Python? If so, how
> can I propose this to PEP?

To make any sort of case you'd need to give an example that does not
have a clearer way to write it already.  Your working version is, to me,
clearer that the ones you want to be able to write.

-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

Local variable definition in Python list comprehension

[James Tsai]

Hello,

I find it very useful if I am allowed to define new local variables in a list 
comprehension. For example, I wish to have something like
[(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
[(x, y) for x in range(10) with y := x ** 2 if x + y < 80].

For now this functionality can be achieved by writing
[(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].

Is it worthwhile to add a new feature like this in Python? If so, how can I 
propose this to PEP?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Exclude 'None' from list comprehension of dicts

[Antoon Pardon]

Op 16/08/2022 om 00:20 schreef dn:

On 16/08/2022 00.56, Antoon Pardon wrote:

Op 5/08/2022 om 07:50 schreef Loris Bennett:

Antoon Pardon   writes:


Op 4/08/2022 om 13:51 schreef Loris Bennett:

Hi,

I am constructing a list of dictionaries via the following list
comprehension:

     data = [get_job_efficiency_dict(job_id) for job_id in job_ids]

However,

     get_job_efficiency_dict(job_id)

uses 'subprocess.Popen' to run an external program and this can fail.
In this case, the dict should just be omitted from 'data'.

I can have 'get_job_efficiency_dict' return 'None' and then run

     filtered_data = list(filter(None, data))

but is there a more elegant way?

Just wondering, why don't you return an empty dictionary in case of a
failure?
In that case your list will be all dictionaries and empty ones will
be processed
fast enough.

When the list of dictionaries is processed, I would have to check each
element to see if it is empty.  That strikes me as being less efficient
than filtering out the empty dictionaries in one go, although obviously
one would need to benchmark that.

I may be missing something but why would you have to check each element
to see if it is empty? What would go wrong if you just treated empty
dictionaries the same as non-empty directories?

'Truthiness':-


In what way is that relevant in this case?

--
Antoon Pardon
--
https://mail.python.org/mailman/listinfo/python-list

Re: Exclude 'None' from list comprehension of dicts

[dn]

On 16/08/2022 00.56, Antoon Pardon wrote:
> Op 5/08/2022 om 07:50 schreef Loris Bennett:
>> Antoon Pardon  writes:
>>
>>> Op 4/08/2022 om 13:51 schreef Loris Bennett:
>>>> Hi,
>>>>
>>>> I am constructing a list of dictionaries via the following list
>>>> comprehension:
>>>>
>>>>     data = [get_job_efficiency_dict(job_id) for job_id in job_ids]
>>>>
>>>> However,
>>>>
>>>>     get_job_efficiency_dict(job_id)
>>>>
>>>> uses 'subprocess.Popen' to run an external program and this can fail.
>>>> In this case, the dict should just be omitted from 'data'.
>>>>
>>>> I can have 'get_job_efficiency_dict' return 'None' and then run
>>>>
>>>>     filtered_data = list(filter(None, data))
>>>>
>>>> but is there a more elegant way?
>>> Just wondering, why don't you return an empty dictionary in case of a
>>> failure?
>>> In that case your list will be all dictionaries and empty ones will
>>> be processed
>>> fast enough.
>> When the list of dictionaries is processed, I would have to check each
>> element to see if it is empty.  That strikes me as being less efficient
>> than filtering out the empty dictionaries in one go, although obviously
>> one would need to benchmark that.
> 
> I may be missing something but why would you have to check each element
> to see if it is empty? What would go wrong if you just treated empty
> dictionaries the same as non-empty directories?

'Truthiness':-

>>> bool( {} )
False
>>> bool( { "a":1 } )
True

-- 
Regards,
=dn
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Exclude 'None' from list comprehension of dicts

[Antoon Pardon]

Op 5/08/2022 om 07:50 schreef Loris Bennett:

Antoon Pardon  writes:


Op 4/08/2022 om 13:51 schreef Loris Bennett:

Hi,

I am constructing a list of dictionaries via the following list
comprehension:

data = [get_job_efficiency_dict(job_id) for job_id in job_ids]

However,

get_job_efficiency_dict(job_id)

uses 'subprocess.Popen' to run an external program and this can fail.
In this case, the dict should just be omitted from 'data'.

I can have 'get_job_efficiency_dict' return 'None' and then run

filtered_data = list(filter(None, data))

but is there a more elegant way?

Just wondering, why don't you return an empty dictionary in case of a failure?
In that case your list will be all dictionaries and empty ones will be processed
fast enough.

When the list of dictionaries is processed, I would have to check each
element to see if it is empty.  That strikes me as being less efficient
than filtering out the empty dictionaries in one go, although obviously
one would need to benchmark that.


I may be missing something but why would you have to check each element
to see if it is empty? What would go wrong if you just treated empty
dictionaries the same as non-empty directories?

--
Antoon Pardon.
--
https://mail.python.org/mailman/listinfo/python-list

RE: Exclude 'None' from list comprehension of dicts

[avi.e.gross]

Benchmarking aside, Lori, there are some ideas about such things.

You are describing a case, in abstract terms, where an algorithm grinds away
and produces results that may include an occasional or a common unwanted
result. The question is when to eliminate the unwanted. Do you eliminate
them immediately at the expense of some extra code at that point, or do you
want till much later or even at the end?

The answer is it DEPENDS and let me point out that many problems can start
multi-dimensional (as in processing a 5-D matrix) and produce a linear
output (as in a 1-D list) or it can be the other way around. Sometimes what
you want eliminated is something like duplicates. Is it easier to remove
duplicates as they happen, or later when you have some huge data structure
containing oodles of copies of each duplicate?

You can imagine many scenarios and sometimes you need to also look at costs.
What does it cost to check if a token is valid, as in can the word be found
in a dictionary? Is it cheaper to wait till you have lots of words including
duplicates and do one lookup to find a bad word then mark it so future
occurrences are removed without that kind of lookup? Or is it better to read
I the dictionary once and hash it so later access is easy?

In your case, you have a single simple criterion for recognizing an item to
leave out. So the above may not apply. But I note we often use pre-created
software that simply returns a result and then the only reasonable way to
remove things  is after calling it. Empty or unwanted items may take up some
room, though, so a long-running process may be better off pruning as it
goes.

-Original Message-
From: Python-list  On
Behalf Of Loris Bennett
Sent: Friday, August 5, 2022 1:50 AM
To: python-list@python.org
Subject: Re: Exclude 'None' from list comprehension of dicts

Antoon Pardon  writes:

> Op 4/08/2022 om 13:51 schreef Loris Bennett:
>> Hi,
>>
>> I am constructing a list of dictionaries via the following list
>> comprehension:
>>
>>data = [get_job_efficiency_dict(job_id) for job_id in job_ids]
>>
>> However,
>>
>>get_job_efficiency_dict(job_id)
>>
>> uses 'subprocess.Popen' to run an external program and this can fail.
>> In this case, the dict should just be omitted from 'data'.
>>
>> I can have 'get_job_efficiency_dict' return 'None' and then run
>>
>>filtered_data = list(filter(None, data))
>>
>> but is there a more elegant way?
>
> Just wondering, why don't you return an empty dictionary in case of a
failure?
> In that case your list will be all dictionaries and empty ones will be 
> processed fast enough.

When the list of dictionaries is processed, I would have to check each
element to see if it is empty.  That strikes me as being less efficient than
filtering out the empty dictionaries in one go, although obviously one would
need to benchmark that.

Cheers,

Loris


--
This signature is currently under construction.
--
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Exclude 'None' from list comprehension of dicts

[Loris Bennett]

Antoon Pardon  writes:

> Op 4/08/2022 om 13:51 schreef Loris Bennett:
>> Hi,
>>
>> I am constructing a list of dictionaries via the following list
>> comprehension:
>>
>>data = [get_job_efficiency_dict(job_id) for job_id in job_ids]
>>
>> However,
>>
>>get_job_efficiency_dict(job_id)
>>
>> uses 'subprocess.Popen' to run an external program and this can fail.
>> In this case, the dict should just be omitted from 'data'.
>>
>> I can have 'get_job_efficiency_dict' return 'None' and then run
>>
>>filtered_data = list(filter(None, data))
>>
>> but is there a more elegant way?
>
> Just wondering, why don't you return an empty dictionary in case of a failure?
> In that case your list will be all dictionaries and empty ones will be 
> processed
> fast enough.

When the list of dictionaries is processed, I would have to check each
element to see if it is empty.  That strikes me as being less efficient
than filtering out the empty dictionaries in one go, although obviously
one would need to benchmark that.

Cheers,

Loris


-- 
This signature is currently under construction.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Exclude 'None' from list comprehension of dicts

[Weatherby,Gerard]

Or:

data = [d for d in [get_job_efficiency_dict(job_id) for job_id in job_ids] if d 
is not None]

or

for job_id in job_ids:
if (d := get_job_efficiency_dict(job_id)) is not None:
  data.append(d)


Personally, I’d got with the latter in my own code.

—
Gerard Weatherby | Application Architect NMRbox | NAN | Department of Molecular 
Biology and Biophysics
 UConn Health 263 Farmington Avenue, Farmington, CT 06030-6406 uchc.edu
On Aug 4, 2022, 2:52 PM -0400, MRAB , wrote:
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

On 2022-08-04 12:51, Loris Bennett wrote:
Hi,

I am constructing a list of dictionaries via the following list
comprehension:

data = [get_job_efficiency_dict(job_id) for job_id in job_ids]

However,

get_job_efficiency_dict(job_id)

uses 'subprocess.Popen' to run an external program and this can fail.
In this case, the dict should just be omitted from 'data'.

I can have 'get_job_efficiency_dict' return 'None' and then run

filtered_data = list(filter(None, data))

but is there a more elegant way?

I'm not sure how elegant it is, but:

data = [result for job_id in job_ids if (result :=
get_job_efficiency_dict(job_id)) is not None]
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!iqxhYMoHcYQY1xohGCpafpBKZIUcGEV6Zj1-RLzOCF61TUXGr-8oh9HLuL-H8w4gxgDCypcOYOYkqNXLJxUIqhWd$
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Exclude 'None' from list comprehension of dicts

[MRAB]

On 2022-08-04 12:51, Loris Bennett wrote:

Hi,

I am constructing a list of dictionaries via the following list
comprehension:

   data = [get_job_efficiency_dict(job_id) for job_id in job_ids]

However,

   get_job_efficiency_dict(job_id)

uses 'subprocess.Popen' to run an external program and this can fail.
In this case, the dict should just be omitted from 'data'.

I can have 'get_job_efficiency_dict' return 'None' and then run

   filtered_data = list(filter(None, data))

but is there a more elegant way?


I'm not sure how elegant it is, but:

data = [result for job_id in job_ids if (result := 
get_job_efficiency_dict(job_id)) is not None]

--
https://mail.python.org/mailman/listinfo/python-list

Re: Exclude 'None' from list comprehension of dicts

[Antoon Pardon]

Op 4/08/2022 om 13:51 schreef Loris Bennett:

Hi,

I am constructing a list of dictionaries via the following list
comprehension:

   data = [get_job_efficiency_dict(job_id) for job_id in job_ids]

However,

   get_job_efficiency_dict(job_id)

uses 'subprocess.Popen' to run an external program and this can fail.
In this case, the dict should just be omitted from 'data'.

I can have 'get_job_efficiency_dict' return 'None' and then run

   filtered_data = list(filter(None, data))

but is there a more elegant way?


Just wondering, why don't you return an empty dictionary in case of a failure?
In that case your list will be all dictionaries and empty ones will be processed
fast enough.

--
Antoon Pardon.

--
https://mail.python.org/mailman/listinfo/python-list

Exclude 'None' from list comprehension of dicts

[Loris Bennett]

Hi,

I am constructing a list of dictionaries via the following list
comprehension:

  data = [get_job_efficiency_dict(job_id) for job_id in job_ids]

However,

  get_job_efficiency_dict(job_id)

uses 'subprocess.Popen' to run an external program and this can fail.
In this case, the dict should just be omitted from 'data'.

I can have 'get_job_efficiency_dict' return 'None' and then run

  filtered_data = list(filter(None, data))

but is there a more elegant way?

Cheers,

Loris

-- 
This signature is currently under construction.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Exclude 'None' from list comprehension of dicts

[Loris Bennett]

r...@zedat.fu-berlin.de (Stefan Ram) writes:

> "Loris Bennett"  writes:
>>data = [get_job_efficiency_dict(job_id) for job_id in job_ids]
> ...
>>filtered_data = list(filter(None, data))
>
>   You could have "get_job_efficiency_dict" return an iterable
>   that yields either zero dictionaries or one dictionary.
>   For example, a list with either zero entries or one entry.
>
>   Then, use "itertools.chain.from_iterable" to merge all those
>   lists with empty lists effectively removed. E.g.,
>
> print( list( itertools.chain.from_iterable( [[ 1 ], [], [ 2 ], [ 3 ]])))
>
>   will print
>
> [1, 2, 3]

'itertool' is a bit of a blind-spot of mine, so thanks for pointing that
out. 

>   . Or, consider a boring old "for" loop:
>
> data = []
> for job_id in job_ids:
> dictionary = get_job_efficiency_dict( job_id )
> if dictionary:
> data.append( dictionary )
>
>   . It might not be "elegant", but it's quite readable to me.

To me to.  However, 'data' can occasionally consist of many 10,000s of
elements.  Would there be a potential performance problem here?  Even if
there is, it wouldn't be so bad, as the aggregation of the data is not
time-critical and only occurs once a month.  Still, I wouldn't want the
program to be unnecessarily inefficient.

Cheers,

Loris

-- 
This signature is currently under construction.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

This was fixed in 3.10.0 if I am not mistaken. Could you provide a reproducer, 
please?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Peter Roelants]

Peter Roelants  added the comment:

If I understand correctly this should be fixed? In which 3.10.* version should 
this be fixed?

The reason why I'm asking is that I ran into this issue when using Dask 
(2022.02.0) with multithreading on Python 3.10.2:

Exception in thread Profile:
Traceback (most recent call last):
  File "./lib/python3.10/site-packages/distributed/profile.py", line 115, in 
process
d = state["children"][ident]
KeyError: '_all_objs;./lib/python3.10/site-packages/bokeh/embed/bundle.py;357'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./lib/python3.10/threading.py", line 1009, in _bootstrap_inner
self.run()
  File "./lib/python3.10/threading.py", line 946, in run
self._target(*self._args, **self._kwargs)
  File "./lib/python3.10/site-packages/distributed/profile.py", line 274, in 
_watch
process(frame, None, recent, omit=omit)
  File "./lib/python3.10/site-packages/distributed/profile.py", line 119, in 
process
"description": info_frame(frame),
  File "./lib/python3.10/site-packages/distributed/profile.py", line 72, in 
info_frame
line = linecache.getline(co.co_filename, frame.f_lineno, 
frame.f_globals).lstrip()
  File "./lib/python3.10/linecache.py", line 31, in getline
if 1 <= lineno <= len(lines):
TypeError: '<=' not supported between instances of 'int' and 'NoneType'

--
nosy: +peter.roelants
versions: +Python 3.10 -Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46549] Error in list comprehension conditionals when used in class attributes

[Dennis Sweeney]

Dennis Sweeney  added the comment:

See also 
https://stackoverflow.com/questions/13905741/accessing-class-variables-from-a-list-comprehension-in-the-class-definition

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue46549>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46549] Error in list comprehension conditionals when used in class attributes

[Dennis Sweeney]

Dennis Sweeney  added the comment:

I believe this is a duplicate of bpo-45899

--
nosy: +Dennis Sweeney

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46549] Error in list comprehension conditionals when used in class attributes

[din14970]

New submission from din14970 :

I discovered this one by accident. When using a conditional inside a list 
comprehension in class attributes one can get some unexpected behavior. The 
following does not work:

```
class TestClass:
list_1 = [1, 2, 3, 4, 5]
exclude = [2, 4]
list_2 = [i for i in list_1 if i not in exclude]
```

It throws a `NameError` saying exclude isn't defined. The following snippets do 
work:

```
exclude = [2, 4]

class TestClass:
list_1 = [1, 2, 3, 4, 5]
list_2 = [i for i in list_1 if i not in exclude]
```

```
class TestClass:
list_1 = [1, 2, 3, 4, 5]
exclude = [2, 4]
list_2 = []
for i in list_1:
 if i not in exclude:
   list_2.append(i)
```

```
class TestClass:
list_1 = [1, 2, 3, 4, 5]
exclude = [2, 4]
list_2 = [i for i in list_1]
```


So it seems that only when a class attribute is used in the conditional part of 
another class attribute a `NameError` is raised.

--
components: Interpreter Core
messages: 411869
nosy: din14970
priority: normal
severity: normal
status: open
title: Error in list comprehension conditionals when used in class attributes
type: behavior
versions: Python 3.10, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue46549>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46302] IndexError inside list comprehension + workaround

[Pierre Fortin]

Pierre Fortin  added the comment:

[Thanks for the replies! I was trying to post this before seeing them.]

Major egg on face...
The more complex the code becomes, the more likely you will be burned by a 
rookie mistake...
var = ''
var[0]  WILL give IndexError  -- Duh!
It was buried in the each.split('=') returning an empty string -- that's what 
you get for making things easier for the user. 
The easier code is to use, the more complex it must be...

Sorry for the noise.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46302] IndexError inside list comprehension + workaround

[Steven D'Aprano]

Steven D'Aprano  added the comment:

Your functions test1 and test2 are irrelevant to the bug report. The driver 
code using eval() to pick which function to call is unneeded. The business of 
simulating a file is complexity for no purpose.

Those ignore, count, unique functions are also irrelevant.

Removing all the irrelevant and extraneous complexity that obfuscates the 
problem, we get to two lines of simplified code sufficient to demonstrate the 
issue:

each = "name = "
[code.strip()[0] for func, code in [each.split('=')]]

which sure enough raises IndexError.

The hint was looking at your fix() function, which made it clear that you are 
getting an IndexError because the string is an empty string. Well of course you 
do, that is expected behaviour.

Why do you get an empty string? Because you split the line "name = " on the 
equals sign, giving

func = "name "
code = " "

then you strip the whitespace from code giving:

code = ""

then you try to extract the first character of code using code[0], which raises 
IndexError, because that's what it is supposed to do.

So there is no bug here, Python is working correctly.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46302] IndexError inside list comprehension + workaround

[Stefan Pochmann]

Stefan Pochmann  added the comment:

The error occurs when you do code.strip()[0] when code is " ", not "u2".

--
nosy: +Stefan Pochmann

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46302] IndexError inside list comprehension + workaround

[Steven D'Aprano]

Steven D'Aprano  added the comment:

> it returns IndexError (instead of "u") if used in a list comprehension

Works as expected inside a list comprehension:

>>> var = "u2"
>>> [var.strip()[0] for i in range(2)]
['u', 'u']

>>> ["ok" for i in range(2) if var.strip()[0] == "u"]
['ok', 'ok']


I am 99.99% certain that you have a bug in your code, but your code is so 
complicated that it is not obvious at a glance where the bug is. I am strongly 
tempted to just close this as "Works for me" and tell you to come back and 
re-open the bug report when you have isolated the issue to a simpler case, but 
I will resist the temptation for now.

--
nosy: +steven.daprano

___
Python tracker 
<https://bugs.python.org/issue46302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46302] IndexError inside list comprehension + workaround

[Pierre Fortin]

New submission from Pierre Fortin :

var = "u2"
var.strip()[0] 
Works as expected, except that it returns IndexError (instead of "u") if used 
in a list comprehension -- at least, that's where I found it.  Attached example 
script illustrates the issue.
Call it with "mytest N" where N is 1 (fails), 2 (works), 3 (fails).
mytest 1 -- KeyError expected; this was due to infile design change
adding a digit to previously single char code
mytest 2 -- workaround to actual issue in next test
mytest 3 -- adding [0] fails when used in list comprehension

--
components: Interpreter Core
files: mytest
messages: 410054
nosy: NetAlien
priority: normal
severity: normal
status: open
title: IndexError inside list comprehension + workaround
type: behavior
versions: Python 3.8
Added file: https://bugs.python.org/file50548/mytest

___
Python tracker 
<https://bugs.python.org/issue46302>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45899] NameError on if clause of class-level list comprehension

[Matthew Barnett]

Matthew Barnett  added the comment:

It's not just in the 'if' clause:

>>> class Foo:
... a = ['a', 'b']
... b = ['b', 'c']
... c = [b for x in a]
...
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 4, in Foo
  File "", line 4, in 
NameError: name 'b' is not defined

--
nosy: +mrabarnett

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45899] NameError on if clause of class-level list comprehension

[Mark Dickinson]

Mark Dickinson  added the comment:

This is expected behaviour. See the docs here: 
https://docs.python.org/3.9/reference/executionmodel.html#resolution-of-names

> The scope of names defined in a class block is limited to the class block; it 
> does not extend to the code blocks of methods – this includes comprehensions 
> and generator expressions since they are implemented using a function scope.

--
nosy: +mark.dickinson
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45899] NameError on if clause of class-level list comprehension

[ThiefMaster]

Change by ThiefMaster :


--
nosy: +ThiefMaster

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45899] NameError on if clause of class-level list comprehension

[jhpfjyne]

New submission from jhpfjyne :

Accessing an attribute defined at class-level in the if clause of a list 
comprehension at class-level throws a NameError.

>>> class Foo:
... a = ['a', 'b']
... b = ['b', 'c']
... c = [x for x in a if x not in b]
...
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 4, in Foo
  File "", line 4, in 
NameError: name 'b' is not defined

--
components: Interpreter Core
messages: 407002
nosy: jhpfjyne
priority: normal
severity: normal
status: open
title: NameError on if clause of class-level list comprehension
type: behavior
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue45899>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45862] Anomaly of eval() of list comprehension

[Mark Dickinson]

Mark Dickinson  added the comment:

True: there's another detail here that's needed to explain the behaviour. The 
first "for" clause in a list comprehension is special: it's evaluated in the 
enclosing scope, rather than in the local function scope that the list 
comprehension creates. See the docs here: 
https://docs.python.org/3.9/reference/expressions.html?highlight=list%20comprehension#displays-for-lists-sets-and-dictionaries

--

___
Python tracker 
<https://bugs.python.org/issue45862>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45862] Anomaly of eval() of list comprehension

[David Pratten]

David Pratten  added the comment:

Hi Mark,

Thanks.  

The anomaly is that the print("eg def2", ...)  works.  Should it not fail in 
the same way that print("eg def4", ...) does.

David

On 22/11/2021 7:36:31 PM, Mark Dickinson  wrote:

Mark Dickinson added the comment:

Thanks for the report. The behaviour is by design: see #5242 (especially 
msg81898) for an explanation.

Closing this issue as a duplicate of #5242.

--
nosy: +mark.dickinson
resolution: -> duplicate
stage: -> resolved
status: open -> closed
superseder: -> eval() function in List Comprehension doesn't work

___
Python tracker

___

[755e9508-5fde-465a-ac8e-d82585c103f3]

--

___
Python tracker 
<https://bugs.python.org/issue45862>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45862] Anomaly of eval() of list comprehension

[Mark Dickinson]

Mark Dickinson  added the comment:

See also #41216

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45862] Anomaly of eval() of list comprehension

[Mark Dickinson]

Mark Dickinson  added the comment:

Thanks for the report. The behaviour is by design: see #5242 (especially 
msg81898) for an explanation.

Closing this issue as a duplicate of #5242.

--
nosy: +mark.dickinson
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> eval() function in List Comprehension doesn't work

___
Python tracker 
<https://bugs.python.org/issue45862>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45862] Anomaly of eval() of list comprehension

[David Pratten]

New submission from David Pratten :

Hi

Example "eg def2" works but "eg def4" gives an error?

David

```
emp = [
{
"empno": 7839,
"mgr": 0,
"ename": "KING"
},
{
"empno": 7566,
"mgr": 7839,
"ename": "JONES"
},
{
"empno": 7698,
"mgr": 7839,
"ename": "BLAKE"
}
]

a = [e for e in emp if e["mgr"] == 0]
print('eg 1', [b for b in a])
print('eg 2', eval('[b for b in a]'))
print('eg 3', [e for e in emp for b in a if e["mgr"] == b["empno"]])
print('eg 4', eval('[e for e in emp for b in a if e["mgr"] == b["empno"]]'))


def eval_anomaly():
a_anomaly = [e for e in emp if e["mgr"] == 0]
print('eg def1', [b for b in a_anomaly])
print('eg def2', eval('[b for b in a_anomaly]'))
print('eg def3', [e for e in emp for b in a_anomaly if e["mgr"] == 
b["empno"]])
print('eg def4', eval('[e for e in emp for b in a_anomaly if e["mgr"] == 
b["empno"]]'))

eval_anomaly()
```

--
messages: 406743
nosy: david2
priority: normal
severity: normal
status: open
title: Anomaly of eval() of list comprehension
type: behavior
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue45862>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Feel free to reopen if we are missing anything

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
priority: release blocker -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 61eb9b5dfd919ba5d1ec9f7df0137f2e6d196972 by Pablo Galindo in 
branch '3.10':
[3.10] bpo-6: support lineno being None in traceback.FrameSummary 
(GH-26781) (GH-27072)
https://github.com/python/cpython/commit/61eb9b5dfd919ba5d1ec9f7df0137f2e6d196972


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +25623
pull_request: https://github.com/python/cpython/pull/27072

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 91a8f8c16ca9a7e2466a8241d9b41769ef97d094 by Filipe Laíns in 
branch 'main':
bpo-6: support lineno being None in traceback.FrameSummary (GH-26781)
https://github.com/python/cpython/commit/91a8f8c16ca9a7e2466a8241d9b41769ef97d094


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Filipe Laíns]

Filipe Laíns  added the comment:

The issue that made the line number be missing is fixed, but GH-26781 is needed 
to account for 088a15c49d99ecb4c3bef93f8f40dd513c6cae3b, even though it is no 
longer triggered in this situation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Beta 4 is in a few days. Can someone confirm of this is fixed or if it still 
needs a patch?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Mark Shannon]

Mark Shannon  added the comment:

With the latest 3.10, I get:

  File "/home/mark/test/test.py", line 13, in 
next(bar().__await__(), None)
  File "/home/mark/test/test.py", line 10, in bar
return [chunk async for chunk in foo()]
  File "/home/mark/test/test.py", line 10, in 
return [chunk async for chunk in foo()]
  File "/home/mark/test/test.py", line 6, in foo
traceback.print_stack()
working!


Thomas, can you confirm?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Mark Shannon]

Mark Shannon  added the comment:

This appears to be a duplicate of https://bugs.python.org/issue44297

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Mark, can you take a look at this?

--
nosy: +pablogsal
versions: +Python 3.11 -Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Filipe Laíns]

Filipe Laíns  added the comment:

Upon further investigation, there are actually two issues here. The first would 
be the one I identified already, traceback.FrameSummary not being prepared for 
lineno being None, but there is also a regression in lineno being invalid in 
this situation in the first place.

With only GH-26781, the traceback will look like the following:


  File "/home/anubis/git/cpython/rep.py", line 13, in 
next(bar().__await__(), None)
  File "/home/anubis/git/cpython/rep.py", line 10, in bar
return [chunk async for chunk in foo()]
  File "/home/anubis/git/cpython/rep.py", line None, in 
  File "/home/anubis/git/cpython/rep.py", line 6, in foo
traceback.print_stack()
working!


which is different from 3.9



  File "/home/anubis/git/cpython/rep.py", line 13, in 
next(bar().__await__(), None)
  File "/home/anubis/git/cpython/rep.py", line 10, in bar
return [chunk async for chunk in foo()]
  File "/home/anubis/git/cpython/rep.py", line 10, in 
return [chunk async for chunk in foo()]
  File "/home/anubis/git/cpython/rep.py", line 6, in foo
traceback.print_stack()
working!


I bisected the second issue to b37181e69209746adc2119c471599a1ea5faa6c8 which 
moves generators to bytecode, and when doing so changes the behavior to set 
lineno to -1. I have opened a GH-26782 to fixing this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Filipe Laíns]

Change by Filipe Laíns :


--
pull_requests: +25367
pull_request: https://github.com/python/cpython/pull/26782

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Filipe Laíns]

Filipe Laíns  added the comment:

I bissected this to 088a15c49d99ecb4c3bef93f8f40dd513c6cae3b and submitted a 
patch making traceback.FrameSummary take into consideration that lineno might 
be None, I believe this is probably the correct fix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Filipe Laíns]

Change by Filipe Laíns :


--
keywords: +patch
nosy: +FFY00
nosy_count: 4.0 -> 5.0
pull_requests: +25366
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/26781

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Filipe Laíns]

Change by Filipe Laíns :


--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Thomas Grainger]

Change by Thomas Grainger :


--
nosy: +nedbat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Thomas Grainger]

Change by Thomas Grainger :


--
nosy: +Mark.Shannon -nedbat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Ned Batchelder]

Change by Ned Batchelder :


--
nosy: +nedbat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44446] linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[Thomas Grainger]

New submission from Thomas Grainger :

demo:

import traceback
import io

async def foo():
yield 1
traceback.print_stack(file=io.StringIO())
yield 2

async def bar():
return [chunk async for chunk in foo()]


next(bar().__await__(), None)
print("working!")


Traceback (most recent call last):
  File "/home/graingert/projects/anyio/foo.py", line 13, in 
next(bar().__await__(), None)
  File "/home/graingert/projects/anyio/foo.py", line 10, in bar
return [chunk async for chunk in foo()]
  File "/home/graingert/projects/anyio/foo.py", line -1, in 
  File "/home/graingert/projects/anyio/foo.py", line 6, in foo
traceback.print_stack(file=io.StringIO())
  File "/usr/lib/python3.10/traceback.py", line 203, in print_stack
print_list(extract_stack(f, limit=limit), file=file)
  File "/usr/lib/python3.10/traceback.py", line 224, in extract_stack
stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/usr/lib/python3.10/traceback.py", line 379, in extract
f.line
  File "/usr/lib/python3.10/traceback.py", line 301, in line
self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/lib/python3.10/linecache.py", line 31, in getline
if 1 <= lineno <= len(lines):
TypeError: '<=' not supported between instances of 'int' and 'NoneType'

--
messages: 396014
nosy: graingert
priority: normal
severity: normal
status: open
title: linecache.getline TypeError when formatting tracebacks in stacks 
containing an async list comprehension
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue6>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43161] Taking sum of massive list comprehension results in total system crash.

[Mark Dickinson]

Mark Dickinson  added the comment:

> 28 bytes for the actual object

Gah. I should proof-read before hitting "Submit Changes". That should say "28 
bytes for each int object".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43161] Taking sum of massive list comprehension results in total system crash.

[Mark Dickinson]

Mark Dickinson  added the comment:

The list you're attempting to create needs around 41GB of RAM (28 bytes for the 
actual object, rounded up to 32 bytes per object for alignment reasons, plus 8 
bytes for each pointer in the list). Assuming you don't have that much memory 
on your system, it'll likely start swapping at that point.

> Should I submit this as a bug to the kernel project then?

That's up to you. The effect is well known and well documented - see for 
example https://bugzilla.redhat.com/show_bug.cgi?id=1577528 .

You may want to consider using `ulimit` to limit the amount of RAM that the 
Python process can use.

In any case, closing here, since this isn't a Python bug.

--
nosy: +mark.dickinson
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43161] Taking sum of massive list comprehension results in total system crash.

[George]

George  added the comment:

Thanks Christian for the solutions. I am guessing that since it is my entire 
machine that crashes, and python is not simply killed for requesting so much 
memory, that this is an operating system problem. Should I submit this as a bug 
to the kernel project then?

George

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43161] Taking sum of massive list comprehension results in total system crash.

[Christian Heimes]

Christian Heimes  added the comment:

You are using a list comprehension that consumes a LOT of memory very fast. The 
line requires more physical RAM than available on a typical user system. This 
causes your computer to become unresponsive to input.

You can rewrite tie list comprehension as generator expression:

sum(i for i in range(10**8 + 10**9))

It will consume far less memory, but it's slow and inefficient. You could use 
the triangular number algorithm instead:

n = 10**8 + 10**9
n * (n-1) // 2

--
nosy: +christian.heimes

___
Python tracker 
<https://bugs.python.org/issue43161>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43161] Taking sum of massive list comprehension results in total system crash.

[George]

New submission from George :

I tried running the following command in the interpreter, and without fail it 
will completely crash my entire computer.

Python 3.8.7 (default, Jan 20 2021, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> sum([i for i in range(10**8 + 10**9)])

My kernel is 5.10.12-100

I am running on Fedora 32 with Cinnamon Desktop if it helps. Let me know if you 
need any other information. 

I eagerly await resolution of this bug, as I want to know the result of the 
computation. Unless, of course it is harebrained, then we should ignore because 
Guido already discussed such computations.

George

--
messages: 386615
nosy: U9dB37BPZx
priority: normal
severity: normal
status: open
title: Taking sum of massive list comprehension results in total system crash.
type: crash
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue43161>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40438] Python 3.9 eval on list comprehension sometimes returns coroutines

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> This can be closed, but for completeness, the test you ran didn't verify that 
> the bug was fixed. This is because the hard coded compile flags I gave in my 
> example seem to have changed in Python 3.9 (is this documented?). 

Yes, this is documented on What's New.

> Which does produce the correct output as expected. So, the issue can remain 
> closed. I am curious what the bug in 3.9.0a5 was though if you have any 
> speculations.

See issue 39562

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40438] Python 3.9 eval on list comprehension sometimes returns coroutines

[Jonathan Crall]

Jonathan Crall  added the comment:

This can be closed, but for completeness, the test you ran didn't verify that 
the bug was fixed. This is because the hard coded compile flags I gave in my 
example seem to have changed in Python 3.9 (is this documented?). 

In python3.8 the compile flags I specified correspond to division, 
print_function, unicode_literals, and absolute_import. 

python3.8 -c "import __future__; print(__future__.print_function.compiler_flag 
| __future__.division.compiler_flag | __future__.unicode_literals.compiler_flag 
| __future__.absolute_import.compiler_flag)"


Results in: 221184


In Python 3.9 the same code results in: 3538944


I can modify the MWE to accommodate these changes: 

./python -c "import __future__; print(eval(compile('[i for i in range(3)]', 
mode='eval', filename='fo', flags=__future__.print_function.compiler_flag | 
__future__.division.compiler_flag | __future__.unicode_literals.compiler_flag | 
__future__.absolute_import.compiler_flag)))"


Which does produce the correct output as expected. So, the issue can remain 
closed. I am curious what the bug in 3.9.0a5 was though if you have any 
speculations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40438] Python 3.9 eval on list comprehension sometimes returns coroutines

[STINNER Victor]

STINNER Victor  added the comment:

> I first noticed this when testing xdoctest on Python 3.9, and then again when 
> using IPython.

What is your Python 3.9 exact version number?

I cannot reproduce your issue with Python 3.9.0a6:

vstinner@apu$ ./python -c "print(eval(compile('[i for i in range(3)]', 
mode='eval', filename='foo', flags=221184)))"
Traceback (most recent call last):
  File "", line 1, in 
ValueError: compile(): unrecognised flags

vstinner@apu$ ./python -VV
Python 3.9.0a6+ (heads/master:9a8c1315c3, Apr 29 2020, 17:03:41) 
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40438] Python 3.9 eval on list comprehension sometimes returns coroutines

[STINNER Victor]

STINNER Victor  added the comment:

I close the issue: it's already fixed in 3.9.0a6. If it's not the case, feel 
free to reopen the issue ;-)

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40438] Python 3.9 eval on list comprehension sometimes returns coroutines

[Jonathan Crall]

Jonathan Crall  added the comment:

Ah, sorry. I neglected all the important information. 

I tested this using: 

Python 3.9.0a5 (default, Apr 23 2020, 14:11:34) 
[GCC 8.3.0]


Specifically, I ran in a docker container: 

DOCKER_IMAGE=circleci/python:3.9-rc
docker pull $DOCKER_IMAGE
docker run --rm -it $DOCKER_IMAGE bash

And then in the bash shell in the docker image I ran:

python -c "print(eval(compile('[i for i in range(3)]', mode='eval', 
filename='foo', flags=221184)))"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40438] Python 3.9 eval on list comprehension sometimes returns coroutines

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
nosy: +BTaskaya

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40438] Python 3.9 eval on list comprehension sometimes returns coroutines

[Jonathan Crall]

New submission from Jonathan Crall :

I first noticed this when testing xdoctest on Python 3.9, and then again when 
using IPython.

I was finally able to generate a minimal working example in Python itself. The 
following code:

python -c "print(eval(compile('[i for i in range(3)]', mode='eval', 
filename='foo', flags=221184)))"

produces 

[0, 1, 2]
 
in Python <= 3.8, but in 3.9 it produces: 

 at 0x7fa336d40ec0>
:1: RuntimeWarning: coroutine '' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback


Is this an intended change? I can't find any notes in the CHANGELOG that seem 
to correspond to it.

--
components: Interpreter Core
messages: 367651
nosy: Jonathan Crall
priority: normal
severity: normal
status: open
title: Python 3.9 eval on list comprehension sometimes returns coroutines
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue40438>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38970] [PDB] NameError in list comprehension in PDB

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I think this is same as https://bugs.python.org/issue26072#msg277578 and is 
reproducible with below sample script without depending on turbogears. The 
attached patch in the issue fixes this. Try using interact command from pdb as 
a workaround.

import pdb

def foo():
  pdb.set_trace()

foo()

--
nosy: +xdegaye, xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38970] [PDB] NameError in list comprehension in PDB

[castix]

New submission from castix :

Related to https://bugs.python.org/issue27316

This code works from the repl:
Python 3.7.4 (default, Oct  4 2019, 06:57:26) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb; pdb.set_trace()
--Return--
> (1)()->None
(Pdb) z = True
(Pdb) [x for x in [1,2] if z]
[1, 2]
(Pdb) 

However in my (turbogears2) wsgi application it raises:
(Pdb) z = True
(Pdb) [x for x in [1,2] if z]
*** NameError: name 'z' is not defined
(Pdb) z
True
(Pdb) 

I don't know how to report the issue in a reproducible way.
Thanks

--
components: Library (Lib)
messages: 357807
nosy: castix
priority: normal
severity: normal
status: open
title: [PDB] NameError in list comprehension in PDB
type: behavior
versions: Python 3.7

___
Python tracker 
<https://bugs.python.org/issue38970>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38937] NameError in list comprehension within .pth file

[Chris Billington]

Chris Billington  added the comment:

Sorry for the spamming, realised I misunderstood further.

The original behaviour isn't because the exec'd code can't create new local 
variables - it can - it's because of the documented behaviour of exec when it 
gets different dicts for globals and locals:

"If exec gets two separate objects as globals and locals, the code will be 
executed as if it were embedded in a class definition"

So the new scope made by the list comprehension can't access the enclosing 
scope in which the new variable was defined, because that's how scoping works 
in class definitions.

--

___
Python tracker 
<https://bugs.python.org/issue38937>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38937] NameError in list comprehension within .pth file

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
pull_requests: +16895
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17414

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38937] NameError in list comprehension within .pth file

[Chris Billington]

Chris Billington  added the comment:

I see. site.py calls exec() from within a function, and therefore the code is 
executed in the context of that function's locals and the site module globals. 
This means the code in .pth files can access (but not add new) local names from 
the site.addpackage() function:

$ echo 'import sys; f.close()' | sudo tee 
/usr/lib/python3.8/site-packages/test.pth
import sys; f.close()
$ python
Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/usr/lib/python3.8/site.py", line 580, in 
main()
  File "/usr/lib/python3.8/site.py", line 567, in main
known_paths = addsitepackages(known_paths)
  File "/usr/lib/python3.8/site.py", line 350, in addsitepackages
addsitedir(sitedir, known_paths)
  File "/usr/lib/python3.8/site.py", line 208, in addsitedir
addpackage(sitedir, name, known_paths)
  File "/usr/lib/python3.8/site.py", line 164, in addpackage
for n, line in enumerate(f):
ValueError: I/O operation on closed file.

The example with the sys module worked because sys is in the globals the site 
module already.

Probably site.addpackage() should exec() code it its own environment:

if line.startswith(("import ", "import\t")):
exec(line, {})
continue

(added empty dict for exec() call)

or for backward compatibility for .pth files that are using globals from the 
site module without importing them (such as sys or os):

if line.startswith(("import ", "import\t")):
exec(line, globals().copy())
continue

This resolves the original issue

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38937] NameError in list comprehension within .pth file

[Chris Billington]

New submission from Chris Billington :

The following one-liner works fine in a regular Python interpreter:

$ python -c 'import sys; x = 5; [print(x + i) for i in range(5)]'
5
6
7
8
9

But in a .pth file, it raises a NameError:

$ echo 'import sys; x = 5; [print(x + i) for i in range(5)]' | sudo tee 
/usr/lib/python3.8/site-packages/test.pth
$ python
Error processing line 1 of /usr/lib/python3.8/site-packages/test.pth:

  Traceback (most recent call last):
File "/usr/lib/python3.8/site.py", line 169, in addpackage
  exec(line)
File "", line 1, in 
File "", line 1, in 
  NameError: name 'x' is not defined

Remainder of file ignored
Python 3.8.0 (default, Oct 23 2019, 18:51:26) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Since site.py uses exec() to exec each line of a .pth file, I thought I'd 
compare with that. It also works fine:

$ python -c 'exec("import sys; x = 5; [print(x + i) for i in range(5)]")'
5
6
7
8
9

This slight modification (the variable being used in the next statement still, 
but not within the loop of the comprehension) does not raise a NameError:

$ echo 'import sys; x = 5; [print(i) for i in range(x)]' | sudo tee 
/usr/lib/python3.8/site-packages/test.pth
import sys; x = 5; [print(i) for i in range(x)]
$ python
0
1
2
3
4
Python 3.8.0 (default, Oct 23 2019, 18:51:26) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

I know .pth file processing is very early in interpreter startup such that many 
things aren't working yet, but I wouldn't expect using a name defined outside a 
list comprehension within the loop body of said list comprehension not to work.

The following is fine also, showing that using names from outside the list 
comprehension doesn't always break:

$ echo 'import sys; [print(sys) for i in range(5)]' | sudo tee 
/usr/lib/python3.8/site-packages/test.pth
import sys; [print(sys) for i in range(5)]
$ python





Python 3.8.0 (default, Oct 23 2019, 18:51:26) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

This is fine too:

$ echo 'import sys; [print(str(sys) * i) for i in range(5)]' | sudo tee 
/usr/lib/python3.8/site-packages/test.pth
import sys; [print(str(sys) * i) for i in range(5)]
$ python





Python 3.8.0 (default, Oct 23 2019, 18:51:26) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 


My use case is looping over subdirs of a directory and adding them all to 
sys.path to provide similar functionality to python setup.py develop, with all 
python vcs repositories within a specific directory being prepended to 
sys.path, rather than having to add them one-by-one. I probably won't end up 
doing what I'm doing this way, but in any case the above seems like it's a bug, 
unless I'm grossly misunderstanding something.

--
components: Interpreter Core
messages: 357627
nosy: Chris Billington
priority: normal
severity: normal
status: open
title: NameError in list comprehension within .pth file
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue38937>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38847] AST Optimization for Single Target List Comprehension

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I concur with Pablo.

Thank you for the suggestion though.

--
nosy: +rhettinger
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38847] AST Optimization for Single Target List Comprehension

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

IMHO, I think is such a specific use case that is not worth the maintenance 
cost. Especially because of such list comprehension is very uncommon as you can 
simply do `list(it)` which more readable and anecdotally faster.

--
nosy: +pablogsal

___
Python tracker 
<https://bugs.python.org/issue38847>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38847] AST Optimization for Single Target List Comprehension

[Batuhan]

Change by Batuhan :


--
components: +Interpreter Core
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38847] AST Optimization for Single Target List Comprehension

[Batuhan]

New submission from Batuhan :

I was going through old issues and I found @inada.naoki's comment on issue 36551

> How about converting `[x for x in it]` to `[*it]` in AST?

Is this feature still wanted? I can try to work on an implementation for this 
if there is a need. Also should this cover set (`{x for x in it}`) 
comprehensions?

--
messages: 356965
nosy: BTaskaya, inada.naoki
priority: normal
severity: normal
status: open
title: AST Optimization for Single Target List Comprehension

___
Python tracker 
<https://bugs.python.org/issue38847>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38847] AST Optimization for Single Target List Comprehension

[Batuhan]

Change by Batuhan :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset 9341dcb4b9520ab92df10d4256e93a50e1e7d19f by Raymond Hettinger 
(Miss Islington (bot)) in branch '3.8':
bpo-37646:  Document that eval() cannot access nested scopes (GH-15117) 
(GH-15155)
https://github.com/python/cpython/commit/9341dcb4b9520ab92df10d4256e93a50e1e7d19f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Raymond Hettinger]

Change by Raymond Hettinger :


--
assignee:  -> docs@python
components: +Documentation -Interpreter Core
nosy: +docs@python
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[miss-islington]

Change by miss-islington :


--
pull_requests: +14888
pull_request: https://github.com/python/cpython/pull/15155

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset 610a4823cc0a3c2380ad0dfe64ae483ced4e5304 by Raymond Hettinger in 
branch 'master':
bpo-37646:  Document that eval() cannot access nested scopes (GH-15117)
https://github.com/python/cpython/commit/610a4823cc0a3c2380ad0dfe64ae483ced4e5304


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Raymond Hettinger]

Raymond Hettinger  added the comment:

After more thought, I think the existing behavior is probably what we want.  
There may not be a clean way to allow access and updates to non-locals.  Even 
if a way was found, it may tie our hands and preclude other implementation 
changes down the road.  Also, such a feature may be at odds with the current 
API which allows the execution environment to be retargeted.  There is also a 
risk of introducing new security issues.

I've attached a PR to update the eval() docs to reflect the actual behavior.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Raymond Hettinger]

Change by Raymond Hettinger :


--
keywords: +patch
pull_requests: +14858
stage: resolved -> patch review
pull_request: https://github.com/python/cpython/pull/15117

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Grzegorz Krasoń]

Grzegorz Krasoń  added the comment:

I re-opened the issue.

Dear core developers, can we ask you to confirm if described behavior of eval() 
is expected?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Grzegorz Krasoń]

Change by Grzegorz Krasoń :


--
resolution: not a bug -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Steven D'Aprano]

Steven D'Aprano  added the comment:

I'm not sure we should be so quick to close this. At the very least, I 
think the documentation could be improved.

It does seem desirable to have the invariant:

`expression` == `eval("expression")`

apply in any environment. Was the change in behaviour between 2 and 3 
intentional, or just a side-effect of the change in implementation?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Grzegorz Krasoń]

Grzegorz Krasoń  added the comment:

Steven, I believed that any `` replaced by `eval('')` 
will not change behaviour of the code. Now I understand that my assumption was 
incorrect.

Raymond, thanks for helping me understand this.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Raymond Hettinger]

Raymond Hettinger  added the comment:

This used to work as you expected in Python 2.

In Python 3, list comprehensions create their own inner scope just like 
generator expressions.  

Per the eval() docs, "if both dictionaries are omitted, the expression is 
executed in the environment where eval() is called." 

In your code example, the inner scope doesn't have a local variable "x", so the 
global variable "x" is retrieved.  

That said, I would have expected the inner "x" to be found as a non-local.  So 
yes, this does seem odd an it isn't really true that "the expression is 
executed in the environment where eval() is called."  Instead, it uses the 
globals() and locals() of the environment where it is called but not the nested 
scope.  Perhaps this should be clarified in the docs if it is in fact the 
intended behavior.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Steven D'Aprano]

Steven D'Aprano  added the comment:

What leads you to believe that eval *shouldn't* work in the global scope in a 
comprehension?

If not the global scope, which scope should it be, local or nonlocal? Is the 
behaviour documented differently?

For reference, the current docs for eval are here:
https://docs.python.org/3.5/library/functions.html#eval

--
nosy: +steven.daprano

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37646] eval() in a list comprehension

[Grzegorz Krasoń]

New submission from Grzegorz Krasoń :

eval() works in a global scope when invoked in a list comprehension.

--
components: Interpreter Core
files: demo.py
messages: 348271
nosy: Grzegorz Krasoń
priority: normal
severity: normal
status: open
title: eval() in a list comprehension
type: behavior
versions: Python 3.7
Added file: https://bugs.python.org/file48495/demo.py

___
Python tracker 
<https://bugs.python.org/issue37646>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36158] Regex search behaves differently in list comprehension

[Josh Rosenberg]

Change by Josh Rosenberg :


--
resolution:  -> not a bug
stage:  -> resolved
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com