[issue36051] Drop the GIL during large bytes.join operations?

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Please provide benchmarks that demonstrate the benefit of this change.

We also need to add a test for join() which covers the new code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39170] Sqlite3 row_factory for attribute access: NamedRow

[Clinton James]

Change by Clinton James :


--
keywords: +patch
pull_requests: +17204
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17768

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39170] Sqlite3 row_factory for attribute access: NamedRow

[Clinton James]

New submission from Clinton James :

Currently, sqlite3 returns rows by tuple or sqlite3.Row for dict-style, index 
access.  I constantly find myself wanting attribute access like namedtuple for 
rows.  I find attribute access cleaner
without the brackets and quoting field names.  However, unlike previous 
discussions (https://bugs.python.org/issue13299), I don't want to use the 
namedtuple object.  I appreciate the simple API and minimal memory consumption 
of sqlite3.Row and used it as my guide in creating sqlite3.NamedRow to allow 
access by index and attribute.

A pull request is ready

Why a new object instead of adding attribute access to the existing sqlite3.Row?
There is an existing member method `keys` and any table with the field "keys" 
would cause a hard to debug, easily avoidable, collision.

Features:

+ Optimized in C, so it will be faster than any python implementation.
+ Access columns by attribute for all valid names and by index for all names.
+ Iterate over fields by name/value pairs.
+ Works with standard functions `len` and `contains`.
+ Identical memory consumption to sqlite3.Row with two references: the data 
tuple and the cursor description.
+ Identical speed to sqlite3.Row if not faster.  Timing usually has it slightly 
faster for index by name or attribute, but it is almost identical.

Examples:

>>> import sqlite3
>>> c = sqlite3.Connection(":memory:").cursor()
>>> c.row_factory = sqlite3.NamedRow
>>> named_row = c.execute("SELECT 'A' AS letter, '.-' AS morse, 65 AS 
ord").fetchone()

>>> len(named_row)
3
>>> 'letter' in named_row
true
>>> named_row == named_row
true
>>> hash(named_row)
5512444875192833987

Index by number and range.
>>> named_row[0]
'A'
>>> named_row[1:]
('.-', 65)

Index by column name.
>>> named_row["ord"]
65

Access by attribute.
>>> named_row.morse
'.-'

Iterate row for name/value pairs.
>>> dict(named_row)
{'letter': 'A', 'morse': '.-', 'ord': 65}
>>> tuple(named_row)
(('letter', 'A'), ('morse', '.-'), ('ord', 65))


How sqlite3.NamedRow differs from sqlite3.Row
--

The class only has class dunder methods to allow any valid field name. When the 
field name would be an invalid attribute name, you have two options: either use 
the SQL `AS` in the select statement or index by name.
To get the field names use the iterator `[x[0] for x in row]` or do the same 
from the
`cursor.description`.

```python
titles = [x[0] for x in row]
titles = [x[0] for x in cursor.description]
titles = dict(row).keys()
```

Attribute and dict access are no longer case-insensitive.  There are three 
reasons for this.
1. Case-insensitive comparison only works well for ASCII characters.  In a 
Unicode world, case-insensitive edge cases create unnecessary errors. Looking 
at a several existing codebases,
   this feature of Row is almost never used and I believe is not needed in 
NamedRow.
2. Case-insensitivity is not allowed for attribute access.  This "feature" 
would treat attribute access differently from the rest of Python and "special 
cases aren't special enough to break the rules". Where `row.name`, `row.Name`, 
and `row.NAME` are all the same it gives off the faint code smell of something 
wrong.  When case-insensitively is needed and the query SELECT can not be 
modified, sqlite3.Row is still there.
3. Code is simpler and easier to maintain.
4. It is faster.

Timing Results
--

NamedRow is faster than sqlite3.Row for index-by-name access.
I have published a graph and the methodology of my testing.  In the worst-case 
scenario, it is just as fast as sqlite3.Row without any extra memory.  In most 
cases, it is faster.
For more information, see the post at 
http://jidn.com/2019/10/namedrow-better-python-sqlite3-row-factory/

--
components: Library (Lib)
messages: 359104
nosy: jidn
priority: normal
severity: normal
status: open
title: Sqlite3 row_factory for attribute access: NamedRow
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Nick Coghlan]

Nick Coghlan  added the comment:

Thinking about that idea further, I don't think that change would help much, 
since the relevant operations should already be checking for thread termination 
when they attempt to reacquire the GIL.

That means what we're missing is:

1. When daemon threads still exist after the non-daemon threads terminate, 
deliberately giving them additional time to run (and hence terminate)
2. Explicitly attempting to kick daemon threads out of blocking system calls by 
sending them signals to provoke EINTR (I have no idea if there's a windows 
equivalent for this, but we should be able to use pthread_kill on POSIX 
systems. However, choosing *which* wakeup signal to send could be fraught with 
compatibility problems)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36051] Drop the GIL during large bytes.join operations?

[Bruce Merry]

Bruce Merry  added the comment:

If we want to be conservative, we could only drop the GIL if all the buffers 
pass the PyBytes_CheckExact test. Presumably that won't encounter any of these 
problems because bytes objects are immutable?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27973] urllib.urlretrieve() fails on second ftp transfer

[Senthil Kumaran]

Change by Senthil Kumaran :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Nick Coghlan]

Nick Coghlan  added the comment:

Perhaps we need a threading.throw() API, similar to the one we have for 
generators and coroutines?

If we had that, then Py_FinalizeEx() could gain a few new features:

* throw SystemExit into all daemon threads and then give them a chance to 
terminate before calling any atexit handlers (printing a warning if some of the 
threads don't exit)
* throw SystemExit into all daemon and non-daemon threads after running atexit 
handlers (printing a warning if any such threads exist at all, along with 
another warning if some of the threads don't exit)

Adding that would require an enhancement to the PendingCall machinery, though, 
since most pending calls are only processed in the main thread (there's no way 
to route them to specific child threads).

A simpler alternative would be to have an atomic "terminate_threads" counter in 
the ceval runtime state that was incremented to 1 to request that SystemExit be 
raised in daemon threads, and then to 2 to request that SystemExit be raised in 
all still running threads. When a thread received that request to exit, it 
would set a new flag in the thread state to indicate it was terminating, and 
then raise SystemExit. (The thread running Py_FinalizeEx would set that flag in 
advance so it wasn't affected, and other threads would use it to ensure they 
only raised SystemExit once). The runtime cost of this would just be another 
_Py_atomic_load_relaxed call in the eval_breaker branch. (Probably inside 
`make_pending_calls`, so it gets triggered both by the eval_breaker logic, and 
by explicit calls to `Py_MakePendingCalls`).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35411] FTP tests of test_urllib2net fail on Travis CI: 425 Security: Bad IP connecting.

[Senthil Kumaran]

Senthil Kumaran  added the comment:


New changeset f82e59ac4020a64c262a925230a8eb190b652e87 by Senthil Kumaran in 
branch '2.7':
[2.7] bpo-27973 - Fix for urllib.urlretrieve() failing on second ftp transfer 
(#1040)
https://github.com/python/cpython/commit/f82e59ac4020a64c262a925230a8eb190b652e87


--
nosy: +orsenthil

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27973] urllib.urlretrieve() fails on second ftp transfer

[Senthil Kumaran]

Senthil Kumaran  added the comment:


New changeset f82e59ac4020a64c262a925230a8eb190b652e87 by Senthil Kumaran in 
branch '2.7':
[2.7] bpo-27973 - Fix for urllib.urlretrieve() failing on second ftp transfer 
(#1040)
https://github.com/python/cpython/commit/f82e59ac4020a64c262a925230a8eb190b652e87


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20443] __code__. co_filename should always be an absolute path

[Nick Coghlan]

Nick Coghlan  added the comment:

With the sys.argv[0] change reverted, I think this overall issue is fixed now - 
code objects will get absolute paths, while sys.argv[0] will continue to 
reflect how __main__ was identified.

--
priority:  -> normal
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36051] Drop the GIL during large bytes.join operations?

[Inada Naoki]

Inada Naoki  added the comment:

> 2. If the thread tries to change the size of the bytearrays during the join 
> (ba1 += b'123'), it'll die with a BufferError that wasn't previously possible

Makes sense.  We shouldn't drop GIL while having buffer of arbitrary objects.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Closing this for now, let's open another issue if we plan to discuss calling 
Py_INCREF and Py_DECREF in PyObject_RichCompare or do_richcompare in the future.

Thanks to everyone involved!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 53f11ba7b1498133ce3ff8173d5ae2e0182a3603 by Pablo Galindo 
(Dong-hee Na) in branch '3.7':
[3.7] bpo-38588: Fix possible crashes in dict and list when calling P… 
(GH-17765)
https://github.com/python/cpython/commit/53f11ba7b1498133ce3ff8173d5ae2e0182a3603


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 2ee87913dde038436a25f1db13ee3fddd2bcc983 by Pablo Galindo 
(Dong-hee Na) in branch '3.8':
[3.8] bpo-38588: Fix possible crashes in dict and list when calling P… 
(GH-17764)
https://github.com/python/cpython/commit/2ee87913dde038436a25f1db13ee3fddd2bcc983


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37446] Undefined behavior in Python/hamt.c

[miss-islington]

miss-islington  added the comment:


New changeset a278ae19b4daa1deb11e2a8eed38838027e90ece by Miss Islington (bot) 
in branch '3.8':
closes bpo-37446: resolve undefined behavior in Python/hamt.c (GH-17727)
https://github.com/python/cpython/commit/a278ae19b4daa1deb11e2a8eed38838027e90ece


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37446] Undefined behavior in Python/hamt.c

[miss-islington]

Change by miss-islington :


--
pull_requests: +17203
pull_request: https://github.com/python/cpython/pull/17767

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37446] Undefined behavior in Python/hamt.c

[Benjamin Peterson]

Benjamin Peterson  added the comment:


New changeset d0c92e81aa2171228a23cb2bed36f7dab975257d by Benjamin Peterson 
(Batuhan Taşkaya) in branch 'master':
closes bpo-37446: resolve undefined behavior in Python/hamt.c (GH-17727)
https://github.com/python/cpython/commit/d0c92e81aa2171228a23cb2bed36f7dab975257d


--
nosy: +benjamin.peterson
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32615] Inconsistent behavior if globals is a dict subclass

[Guido van Rossum]

Guido van Rossum  added the comment:

This is probably an issue for the new Steering Council.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Inada Naoki]

Inada Naoki  added the comment:


New changeset dfef986f12dd92bd6434117bba0db3cbb4e65243 by Inada Naoki in branch 
'master':
bpo-38588: Optimize list comparison. (GH-17766)
https://github.com/python/cpython/commit/dfef986f12dd92bd6434117bba0db3cbb4e65243


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Sorry, I meant that I reviewed PR 17766.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> Moving INCREF and DECREF is a huge change.  It is just a future idea to 
> prevent same type of bugs.  I think it can not be backported.

Now I am wondering how many other APIs are affected by the same pattern other 
than PyObject_RichCompareBool

> To fix this issue with minimum performance regression, I think PR 17766 is 
> the best solution.  So no need to revert PR 17734.

Thanks for the quick fix and the analysis. I reviewed PR 17734.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32615] Inconsistent behavior if globals is a dict subclass

[Terry J. Reedy]

Terry J. Reedy  added the comment:

There are more examples on duplicate #36220.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32615] Inconsistent behavior if globals is a dict subclass

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Guido, some people would like manipulation and use of globals to work with dict 
subclasses as well as dicts themselves.  I believe that the current 
restriction/optimization is at least partly your work.  Do you have any quick 
opinion, or do you think discussion should go to python-ideas?

--
nosy: +gvanrossum
stage:  -> test needed
type: behavior -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39146] too much memory consumption in re.compile unicode

[Zhipeng Xie]

Zhipeng Xie <775350...@qq.com> added the comment:

Hi, I tracked it down and found that this problem was introduced in python2.7.9 
by following commit:

https://hg.python.org/cpython/rev/ebd48b4f650d

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36051] Drop the GIL during large bytes.join operations?

[Josh Rosenberg]

Josh Rosenberg  added the comment:

This will introduce a risk of data races that didn't previously exist. If you 
do:

ba1 = bytearray(b'\x00') * 5
ba2 = bytearray(b'\x00') * 5
... pass references to thread that mutates them ...
ba3 = b''.join((ba1, ba2))

then two things will change from the existing behavior:

1. If the thread in question attempts to write to the bytearrays in place, then 
it could conceivably write data that is only partially picked up (ba1[0], 
ba1[4] = 2, 3 could end up copying the results of the second write without 
the first; at present, it could only copy the first without the second)

2. If the thread tries to change the size of the bytearrays during the join 
(ba1 += b'123'), it'll die with a BufferError that wasn't previously possible

#1 isn't terrible (as noted, data races in that case already existed, this just 
lets them happen in more ways), but #2 is a little unpleasant; code that 
previously had simple data races (the data might be inconsistent, but the code 
ran and produced some valid output) can now fail hard, nowhere near the actual 
call to join that introduced the behavioral change.

I don't think this sinks the patch (loudly breaking code that was silently 
broken before isn't awful), but I feel like a warning of some kind in the 
documentation (if only a simple compatibility note in What's New) might be 
appropriate.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Inada Naoki]

Inada Naoki  added the comment:

>> This makes list comparison 2x slower.
>
> This is affected by PR 17734? or PyObject_RichCompare patched?

Caused by PR 17734.


> Would you like to revert PR 17734? Calling Py_INCREF and Py_DECREF in 
> PyObject_RichCompare or do_richcompare will take the same effect, no?

Moving INCREF and DECREF is a huge change.  It is just a future idea to prevent 
same type of bugs.  I think it can not be backported.

To fix this issue with minimum performance regression, I think PR 17766 is the 
best solution.  So no need to revert PR 17734.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Dong-hee Na]

Dong-hee Na  added the comment:

Master
Mean +- std dev: 1.08 us +- 0.02 us

Before PR-17734
Mean +- std dev: 584 ns +- 12 ns

New suggested
.
Mean +- std dev: 578 ns +- 14 ns

diff --git a/Objects/object.c b/Objects/object.c
index 6fc1146..b42f41a 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -865,6 +865,8 @@ PyObject_RichCompareBool(PyObject *v, PyObject *w, int op)
 return 0;
 }

+Py_INCREF(v);
+Py_INCREF(w);
 res = PyObject_RichCompare(v, w, op);
 if (res == NULL)
 return -1;
@@ -873,6 +875,8 @@ PyObject_RichCompareBool(PyObject *v, PyObject *w, int op)
 else
 ok = PyObject_IsTrue(res);
 Py_DECREF(res);
+Py_DECREF(v);
+Py_DECREF(w);
 return ok;
 }

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Inada Naoki]

Change by Inada Naoki :


--
pull_requests: +17202
pull_request: https://github.com/python/cpython/pull/17766

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39168] Generic type subscription is a huge toll on Python performance

[Guido van Rossum]

Guido van Rossum  added the comment:

Hm, here's what I measure in Python 3.8.1. (I don't use IPython or notebooks so 
this looks a little different.)


>>> timeit.timeit('Foo()', 'class Foo: pass')
0.3763025619934


>>> timeit.timeit('Foo()', 'class Foo:\n  def __new__(cls): return 
>>> super().__new__(cls)')
1.575319603864


>>> timeit.timeit('Foo()', 'from typing import Generic, TypeVar\nT = 
>>> TypeVar("T")\nclass Foo(Generic[T]): pass')
3.874873715068


>From this I conclude that adding a minimal __new__() method is responsible for 
>about 4x slowdown, and the functionality in typing.py for another factor 2.5.


While this isn't great I don't see an easy way to improve upon this without 
rewriting the entire typing module in C.  (Some of this may or may not happen 
for PEP 604.)

PS. I just realized my Python binary was built with debug options, so absolute 
numbers will look different (better) for you -- but relative numbers will look 
the same, and I get essentially the same factors with 3.9.0a1+.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39167] argparse boolean type bug

[Josh Rosenberg]

Change by Josh Rosenberg :


--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> ArgumentParser should support bool type according to truth 
values

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Dong-hee Na]

Dong-hee Na  added the comment:

> This makes list comparison 2x slower.

This is affected by PR 17734? or PyObject_RichCompare patched?

--
nosy: +corona10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> This makes list comparison 2x slower.

Would you like to revert PR 17734? Calling Py_INCREF and Py_DECREF in 
PyObject_RichCompare or do_richcompare will take the same effect, no?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Inada Naoki]

Inada Naoki  added the comment:

$ ./python -m pyperf timeit -s 'a = ["a"]*100; b = ["a"]*100;' -- 'a == b'

master : Mean +- std dev: 276 ns +- 1 ns
patched: Mean +- std dev: 572 ns +- 3 ns

This makes list comparison 2x slower.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Dong-hee Na]

Change by Dong-hee Na :


--
pull_requests: +17201
pull_request: https://github.com/python/cpython/pull/17765

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> How about calling Py_INCREF and Py_DECREF in PyObject_RichCompare or 
> do_richcompare?

Apologies, I had missed this suggestion before merging the PR :(

If we decide to add the check to PyObject_RichCompare or do_richcompare we 
should also adapt the fix for https://bugs.python.org/issue38610

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Dong-hee Na]

Change by Dong-hee Na :


--
pull_requests: +17200
pull_request: https://github.com/python/cpython/pull/17764

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38588] Use-after-free in dict/list

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 2d5bf568eaa5059402ccce9ba5a366986ba27c8a by Pablo Galindo 
(Dong-hee Na) in branch 'master':
bpo-38588: Fix possible crashes in dict and list when calling 
PyObject_RichCompareBool (GH-17734)
https://github.com/python/cpython/commit/2d5bf568eaa5059402ccce9ba5a366986ba27c8a


--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32615] Inconsistent behavior if globals is a dict subclass

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> exec() params are already checked, as a seconds param, only dict or dict 
> subclasses are accepted. Seems like good enough contract.

As I said, the documentation is considered the public API contract and again, 
the documentation says that dict subclasses are not accepted:

>> it must be a dictionary (and not a subclass of dictionary)

My opinion is that as Terry said, this is a violation of the contract and 
supporting dict subclasses is a case unusual enough to not sacrifice 
performance for the general case. 

> I am not, and please stop your accusations, that violates CoC.

A respectful indication to keep the discussion civilized does not violate the 
CoC. If you are not being sarcastic, then everything is fine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39168] Generic type subscription is a huge toll on Python performance

[Guido van Rossum]

Guido van Rossum  added the comment:

Sorry, you already said 3.6 and 3.8 give the same effect. But what if you add a 
minimal __new__() to Foo?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32615] Inconsistent behavior if globals is a dict subclass

[Paul Sokolovsky]

Paul Sokolovsky  added the comment:

> I agree with Terry, the moment you pass a dict subclass to exec you are out 
> of contract. If any, we may need to sanitize the input to exec, although I 
> don't think is worth paying the performance price for that.

exec() params are already checked, as a seconds param, only dict or dict 
subclasses are accepted. Seems like good enough contract.

> Paul, I don't know if you are being sarcastic here so I will assume that you 
> are not but in case you are, I suggest you stop as this violates our CoC.

I am not, and please stop your accusations, that violates CoC.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39168] Generic type subscription is a huge toll on Python performance

[Guido van Rossum]

Guido van Rossum  added the comment:

What Python version was used for the timings? If not 3.8, please do over in 3.8.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39169] TypeError: 'int' object is not callable if the signal handler is SIG_IGN

[Ronald Li]

New submission from Ronald Li :

The attached script ign2ndsig.py demonstrates an attempted way to handle 
signals at most once and ignore any subsequent signals. SIGINT and SIGTERM are 
used in the demo.


In Python 3.5, the subprocess would go into one of the "except 
KeyboardInterrupt:" or "except SystemExit:" blocks before doing "finally:" and 
exiting zero:


# execute the script with no args:
# ./ign2ndsig.py

subproc: sys.version_info(major=3, minor=5, micro=9, releaselevel='final', 
serial=0)
raising KeyboardInterrupt
handling KeyboardInterrupt
doing finally
rc: 0



In Python 3.6, 3.7 or 3.8, the subprocess would go into neither the "except 
KeyboardInterrupt:" nor "except SystemExit:" blocks before doing "finally:" and 
exiting non-zero, with a traceback like this:


subproc: sys.version_info(major=3, minor=6, micro=10, releaselevel='final', 
serial=0)
raising KeyboardInterrupt
doing finally
Traceback (most recent call last):
  File "./ign2ndsig.py", line 30, in subproc
input()
  File "./ign2ndsig.py", line 18, in handler
raise KeyboardInterrupt
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./ign2ndsig.py", line 58, in 
subproc()
  File "./ign2ndsig.py", line 32, in subproc
printef('handling KeyboardInterrupt')
  File "./ign2ndsig.py", line 10, in printef
print(*objects, file=sys.stderr, flush=True)
TypeError: 'int' object is not callable
rc: 1



More on the behaviors of ign2ndsig.py :
1. Replacing SIG_IGN at line 14 and 15 with a no-op Python function "fixes" the 
problem - such that in Python 3.6, 3.7 or 3.8, it'd behave similarly to the 
Python 3.5 output above.
2. Sending a SIGTERM then a SIGINT (instead of a SIGINT followed by a SIGTERM) 
(i.e. reversing the order of SIGINT and SIGTERM) at line 49-50 results in a 
similar behavior (that the TypeError is raised).
3. Sending 2 SIGINTs or 2 SIGTERMs (instead of 1 SIGINT + 1 SIGTERM) (i.e. 
sending the same signal twice) at line 49-50 does NOT result in the TypeError.


Potentially related issues:

_thread.interrupt_main() errors if SIGINT handler in SIG_DFL, SIG_IGN
https://bugs.python.org/issue23395

Turn SIG_DFL and SIG_IGN into functions
https://bugs.python.org/issue23325

--
files: ign2ndsig.py
messages: 359071
nosy: Ronald Li
priority: normal
severity: normal
status: open
title: TypeError: 'int' object is not callable if the signal handler is SIG_IGN
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file48810/ign2ndsig.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34938] Fix mimetype.init() to account for from import

[Ashley Whetter]

Ashley Whetter  added the comment:

Yes I'm happy to make those changes as part of this.

So clarify what needs to change in PR 16567:

1) Include a note in the docs for `inited` that outlines that if it is imported 
into the local scope, it will not be updated by calls to `init()`. Only 
`mimetypes.inited` will.

2) Add a clarification to the `add_type()` docs that changes made with 
`strict=True` means that they will persist after subsequent calls to `init()`.

3) Remove `_default_mime_types()` in favour of public global constants. The 
documentation currently refers to these globals as the "official MIME types". 
Should that wording change to use "default" instead of "official", or maybe 
should the names of the constants change to `OFFICIAL_SUFFIX_MAP` for example?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32615] Inconsistent behavior if globals is a dict subclass

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> The doc for exec says globals "must be a dictionary (and not a subclass of 
> dictionary)"

I agree with Terry, the moment you pass a dict subclass to exec you are out of 
contract. If any, we may need to sanitize the input to exec, although I don't 
think is worth paying the performance price for that.

> Some smart maintainer closed https://bugs.python.org/issue36220 as a 
> duplicate of this one.

I closed issue36220 as a duplicate. Paul, I don't know if you are being 
sarcastic here so I will assume that you are not but in case you are, I suggest 
you stop as this violates our CoC.

> Docs are full of mistakes and outdated information.

They are also full of contracts and as of today, what the docs document is what 
is considered the public API. 

In any case, my opinion is that LOAD_NAME and LOAD_GLOBAL should not pay the 
performance price to handle less-specific APIs just to support a very uncommon 
case (using dict subclasses as globals) that are also out of contract. For 
STORE_GLOBAL I feel less strongly about that, but I am still -1 on it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32615] Inconsistent behavior if globals is a dict subclass

[Paul Sokolovsky]

Paul Sokolovsky  added the comment:

> The doc for exec says globals "must be a dictionary (and not a subclass of 
> dictionary)"

Docs are full of mistakes and outdated information.

Fixing STORE_GLOBAL case from https://bugs.python.org/issue36220#msg359046 
would be trivial and cheap re: overheads. And useful, yeah.

And the gentleman who submitted the other ticket argues that:

> but the inconsistency between LOAD_NAME and LOAD_GLOBAL definitely seems 
> wrong.

Which is easy to agree with, though there're more concerns re: runtime overhead.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32615] Inconsistent behavior if globals is a dict subclass

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Victor, Pablo, I added you two since you commented on the duplicate.

I believe this should be closed as 'not a bug'.  The doc for globals() says is 
returns a 'dictionary', meaning an instance of dict.  The doc for exec says 
globals "must be a dictionary (and not a subclass of dictionary)".  (This is 
mere implied when locals is also given, but it *is* implied.).  The behavior 
when globals is otherwise is undefined.

--
nosy: +pablogsal, vstinner
versions: +Python 3.9 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34118] Fix some class entries in 'Built-in Functions'

[miss-islington]

miss-islington  added the comment:


New changeset ec941568bdb25e164a87a23cf1b8870ac047b4e3 by Miss Islington (bot) 
in branch '3.8':
bpo-34118: memoryview, range, and tuple are classes  (GH-17761)
https://github.com/python/cpython/commit/ec941568bdb25e164a87a23cf1b8870ac047b4e3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32615] Inconsistent behavior if globals is a dict subclass

[Paul Sokolovsky]

Paul Sokolovsky  added the comment:

Some smart maintainer closed https://bugs.python.org/issue36220 as a duplicate 
of this one. That ticket might have more details of the underlying issues.

--
nosy: +pfalcon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34118] Fix some class entries in 'Built-in Functions'

[miss-islington]

miss-islington  added the comment:


New changeset c9c17cc933dcffb9ed7f03e3f791d8cfd7acc54a by Miss Islington (bot) 
in branch '3.7':
bpo-34118: memoryview, range, and tuple are classes  (GH-17761)
https://github.com/python/cpython/commit/c9c17cc933dcffb9ed7f03e3f791d8cfd7acc54a


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34118] Fix some class entries in 'Built-in Functions'

[miss-islington]

Change by miss-islington :


--
pull_requests: +17199
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17763

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34118] Fix some class entries in 'Built-in Functions'

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I started with the agreed-on 'function' => 'class' changes and intend to follow 
with a PR for text changes.

--
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34118] Fix some class entries in 'Built-in Functions'

[Terry J. Reedy]

Terry J. Reedy  added the comment:


New changeset ee9ff05ec22ecd47dbffdd361967ccd55963dad2 by Terry Jan Reedy in 
branch 'master':
 bpo-34118: memoryview, range, and tuple are classes  (GH-17761)
https://github.com/python/cpython/commit/ee9ff05ec22ecd47dbffdd361967ccd55963dad2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34118] Fix some class entries in 'Built-in Functions'

[miss-islington]

Change by miss-islington :


--
pull_requests: +17198
pull_request: https://github.com/python/cpython/pull/17762

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39168] Generic type subscription is a huge toll on Python performance

[Ruslan Dautkhanov]

Ruslan Dautkhanov  added the comment:

In [12]: cProfile.run("for _ in range(100_000): Bar()") 

   
 23 function calls in 0.136 seconds 

   


   
   Ordered by: standard name

   


   
   ncalls  tottime  percall  cumtime  percall filename:lineno(function) 

   
10.0470.0470.1360.136 :1()  

   
   100.0790.0000.0890.000 typing.py:865(__new__)

   
   100.0100.0000.0100.000 {built-in method __new__ of type 
object at 0x55ab65861ac0}   

10.0000.0000.1360.136 {built-in method builtins.exec}   

   
10.0000.0000.0000.000 {method 'disable' of 
'_lsprof.Profiler' objects} 



   


   


   
In [13]: # And typing.py:865 points to  

   


   
In [14]: inspect.getsourcelines(Generic.__new__)

   
Out[14]:

   
(['def __new__(cls, *args, **kwds):\n', 

   
  'if cls in (Generic, Protocol):\n',   

   
  'raise TypeError(f"Type {cls.__name__} cannot be instantiated; 
"\n',   
  
  '"it can be used only as a base class")\n',   

   
  'if super().__new__ is object.__new__ and cls.__init__ is not 
object.__init__:\n',
   
  'obj = super().__new__(cls)\n',   

[issue34118] Fix some class entries in 'Built-in Functions'

[Terry J. Reedy]

Change by Terry J. Reedy :


--
pull_requests: +17197
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17761

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39167] argparse boolean type bug

[Mark Dickinson]

Change by Mark Dickinson :


--
Removed message: https://bugs.python.org/msg359060

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39167] argparse boolean type bug

[Mark Dickinson]

Mark Dickinson  added the comment:

This looks like a duplicate of #37564.

--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38967] Improve error message in enum for member name surrounded by underscore.

[Ethan Furman]

Ethan Furman  added the comment:

Rubén, good idea.

I wonder, though, if we should say:

  _sunder_ names, such as "%s", are reserved for future Enum use

since `_sunder_` is used in the Enum docs and is unlikely to be confused with 
one's own variable name.

--
assignee:  -> ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39167] argparse boolean type bug

[paul j3]

paul j3  added the comment:

The rejected boolean type proposal:

https://bugs.python.org/issue37564

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38870] Expose ast.unparse in the ast module

[Batuhan]

Change by Batuhan :


--
pull_requests: +17196
pull_request: https://github.com/python/cpython/pull/17760

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38610] use-after-free in list object function

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 296d45ec10fb55532bc3fac2311a3f91299ecf59 by Pablo Galindo in 
branch '3.7':
[3.7] bpo-38610: Fix possible crashes in several list methods (GH-17022) 
(GH-17759)
https://github.com/python/cpython/commit/296d45ec10fb55532bc3fac2311a3f91299ecf59


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38610] use-after-free in list object function

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38610] use-after-free in list object function

[miss-islington]

miss-islington  added the comment:


New changeset fcaf14cd9179bb48850f8f81ce8d5cee28129745 by Miss Islington (bot) 
in branch '3.8':
bpo-38610: Fix possible crashes in several list methods (GH-17022)
https://github.com/python/cpython/commit/fcaf14cd9179bb48850f8f81ce8d5cee28129745


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38610] use-after-free in list object function

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +17195
pull_request: https://github.com/python/cpython/pull/17759

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38610] use-after-free in list object function

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset d9e561d23d994e3ed15f4fcbd7ee5c8fe50f190b by Pablo Galindo 
(Zackery Spytz) in branch 'master':
bpo-38610: Fix possible crashes in several list methods (GH-17022)
https://github.com/python/cpython/commit/d9e561d23d994e3ed15f4fcbd7ee5c8fe50f190b


--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38610] use-after-free in list object function

[miss-islington]

Change by miss-islington :


--
pull_requests: +17194
pull_request: https://github.com/python/cpython/pull/17758

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39167] argparse boolean type bug

[paul j3]

paul j3  added the comment:

Despite the name, the 'type' parameter specifies a function, not a Python 
class.  

The only string that produces False is the empty one: bool('').  So 'type=bool' 
is valid Python, even if it isn't useful.

With `nargs='+'` there's no problem with providing strings like 'False', 
'true', 'no', 'oui', 'niet', but if you want to convert those to boolean 
True/False values, you need to write your own 'type' function.


There was a recent bug/issue that proposed providing such a function (or 
importing it from another module), and shadowing the existing 'bool' function, 
but that has been rejected (I think).  It isn't really needed, and the proposed 
solution was too language specific.

Seems to me that expecting your user to provide an open ended list of 'True 
False False True' strings would be rather confusing, or at least require a 
complicated 'help' string.  In any case it's not a common enough case to 
require any changes to the core argparse functionality.

In sum, it isn't clear what the bug is, or what patch you expect.  This sounds 
more like a StackOverflow question, than a bug report.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36220] LOAD_NAME and LOAD_GLOBAL, STORE_GLOBAL handle dict subclasses for globals() differently

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Closed as duplicate of issue32615.

--
nosy: +pablogsal
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24955] webbrowser broken on Mac OS X when using the BROWSER variable

[Eitan Adler]

Change by Eitan Adler :


--
nosy: +eitan.adler

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36220] LOAD_NAME and LOAD_GLOBAL, STORE_GLOBAL handle dict subclasses for globals() differently

[ppperry]

ppperry  added the comment:

Duplicate of issue32615

--
nosy: +ppperry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39168] Generic type subscription is a huge toll on Python performance

[Ned Deily]

Change by Ned Deily :


--
nosy: +gvanrossum, levkivskyi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39167] argparse boolean type bug

[Trenton Bricken]

Trenton Bricken  added the comment:

Thank you for your quick and helpful reply. 

The problem with your solution is twofold: 
1. it adds some cognitive load in needing to remember whether or not the flag 
defaults to True or False and thus whether or not you need to add it. It is 
easier for me to have everything as a flag with a value that follows. 
2. More importantly, I am using argparse for trying lots of different 
combinations of inputs to a function so I pass them in as a list and then 
permute the possible values for each input. 

For example: --flag1 a b
--flag2 c d
 
I want to then run a command with the parameter combinations: 
a, c; a, d; b, c; b, d;

And I don't think store_true store_false would allow me to pass in having 
combinations of True and False like --flag True False would. 

I am fine now with my current work around, but was very confused for some time 
why my flag would be true when I set it to false. And think this is a 
counterintuitive pitfall other developers can fall into.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39168] Generic type subscription is a huge toll on Python performance

[Ruslan Dautkhanov]

Ruslan Dautkhanov  added the comment:

Python typing gives an order of magnitude slow down in this case

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39168] Generic type subscription is a huge toll on Python performance

[Ruslan Dautkhanov]

New submission from Ruslan Dautkhanov :

Reported originally here - 
https://twitter.com/__zero323__/status/1210911632953692162

See details here
https://asciinema.org/a/290643

In [4]: class Foo: pass
In [5]: %timeit -n1_000_000 Foo()
88.5 ns ± 3.44 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [6]: T = TypeVar("T")
In [7]: class Bar(Generic[T]): pass
In [8]: %timeit -n1_000_000 Bar()
883 ns ± 3.46 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)

Same effect in Python 3.6 and 3.8

--
messages: 359049
nosy: Ruslan Dautkhanov
priority: normal
severity: normal
status: open
title: Generic type subscription is a huge toll on Python performance
type: performance
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39167] argparse boolean type bug

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

It seems like this is a common problem : 
https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse
 . I guess you want to store verbose=True when --verbose is passed and 
verbose=False when --verbose is not passed where store_true might help.

--
nosy: +paul.j3, rhettinger, xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39167] argparse boolean type bug

[Trenton Bricken]

Trenton Bricken  added the comment:

Update: 

I was being dumb before, the problem still remains but my work around 
previously was wrong. This is the new workaround: 

def buildBool(arg):
if arg == 'False':
return False
else:
return True

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39167] argparse boolean type bug

[Trenton Bricken]

New submission from Trenton Bricken :

This is a bug with argparse. 

Say I have:

parser.add_argument('--verbose', type=bool, action='store', nargs='+',
default = [False],
help='turns on verbosity')

If in the command line I have "--verbose False' the default will then evaluate 
to True and return the value "True" rather than the value of "False" that I 
tried to set it to!

When a developer has lots of arguments to pass in, they may not remember if an 
argument defaults to False or True. Setting the value to False and having it 
then return True is very confusing and should not occur. 

Right now I have a work-around where I have a new type=buildBool where:

def buildBool(arg):
   return bool(arg)

and do:

parser.add_argument('--verbose', type=buildBool, action='store', nargs='+',
default = ['False'],
help='turns on verbosity')

but this means I have to have this type and have my default value be a string 
which is suboptimal and this bug remains a trap for other developers.

--
messages: 359045
nosy: Trenton Bricken
priority: normal
severity: normal
status: open
title: argparse boolean type bug
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36220] LOAD_NAME and LOAD_GLOBAL, STORE_GLOBAL handle dict subclasses for globals() differently

[Paul Sokolovsky]

Paul Sokolovsky  added the comment:

> I wanted to write a sandbox for Python.

Sandbox indeed, it is.

class NS(dict):

def __setitem__(self, k, v):
if not isinstance(v, type(lambda: 0)):
raise RuntimeError("Global variables considered harmful")

globals = NS()

exec("foo = 1", globals)

But then:

exec("""
global foo
foo = "watch me escaping your sandboxes"
""", globals)

This is due to STORE_GLOBAL not handling dict subclasses, unlike STORE_NAME.


While @Kevin Shweh's issue with LOAD_NAME, and @vstinner's concerns of adding 
yet another branch to LOAD_NAME implementation may be one matter, issue with 
STORE_GLOBAL is both related and somewhat different. STORE_GLOBAL's should be 
relatively infrequent, so adding another branch to it unlikely will be 
quantifiable in performance. But lack of its support disallows to write to 
tools which police/constrain Python's overdynamicity, which is useful in the 
age.

Anyway, I decided to not open a separate ticket for this, but add here. Fairly 
speaking, as long as work to support dict subclasses as globals/locals started, 
the only reasonable way forward seems to implement it completely.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39019] Missing class getitems in standard library classes

[Ivan Levkivskyi]

Ivan Levkivskyi  added the comment:


New changeset 09c482fad11c769be38b2449f1056e264b701bb7 by Ivan Levkivskyi 
(Batuhan Taşkaya) in branch 'master':
bpo-39019: Implement missing __class_getitem__ for SpooledTemporaryFile 
(GH-17560)
https://github.com/python/cpython/commit/09c482fad11c769be38b2449f1056e264b701bb7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39019] Missing class getitems in standard library classes

[Ivan Levkivskyi]

Ivan Levkivskyi  added the comment:


New changeset 4dc5a9df59837446ec1dc5b7a0e6ce95ae5b5cec by Ivan Levkivskyi 
(Batuhan Taşkaya) in branch 'master':
bpo-39019: Implement missing __class_getitem__ for subprocess classes (GH-17558)
https://github.com/python/cpython/commit/4dc5a9df59837446ec1dc5b7a0e6ce95ae5b5cec


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36220] LOAD_NAME and LOAD_GLOBAL, STORE_GLOBAL handle dict subclasses for globals() differently

[Paul Sokolovsky]

Change by Paul Sokolovsky :


--
nosy: +pfalcon
title: LOAD_NAME and LOAD_GLOBAL handle dict subclasses for globals() 
differently -> LOAD_NAME and LOAD_GLOBAL, STORE_GLOBAL handle dict subclasses 
for globals() differently

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36051] Drop the GIL during large bytes.join operations?

[Bruce Merry]

Change by Bruce Merry :


--
keywords: +patch
pull_requests: +17193
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17757

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39119] email/_header_value_parser.py:parse_message_id: UnblondLocalError

[Drew DeVault]

Drew DeVault  added the comment:

Sorry for the delay, was travelling for the holidays. I'll check that this is 
not an issue with 3.8.1. Thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39125] Type signature of @property not shown in help()

[Nguyễn Gia Phong]

Nguyễn Gia Phong  added the comment:

Relating to this, should there also be indication about the mode (get, set, 
del) the property?  Currently there is no way to tell if the *attribute* is 
read-only, read-write or write-only.

--
versions:  -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38314] Implement unix read_pipe.is_reading() method

[Callum Ward]

Change by Callum Ward :


--
pull_requests: +17192
pull_request: https://github.com/python/cpython/pull/17755

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39165] Completeness and symmetry in RE, avoid `findall(...)[0]`

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +ezio.melotti, mrabarnett, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39166] Python 3.9.0a2 changed how "async for" traces its final iteration

[Ned Batchelder]

New submission from Ned Batchelder :

3.9.0a2 changed how the final iteration of "async for" is traced.  The body of 
the loop is traced when the body is not executed.  Standard "for" loops don't 
show the same effect.

In the output below, notice that 3.9.0a2 and 3.9.0a2+ both show one last 
execution of line 32, but that line is not actually executed (there is no 
output).  The standard for loop doesn't show line 27 doing that in any version.

--- 8< 
import linecache, sys

def trace(frame, event, arg):
# The weird globals here is to avoid a NameError on shutdown...
if frame.f_code.co_filename == globals().get("__file__"):
lineno = frame.f_lineno
print("{} {}: {}".format(event[:4], lineno, linecache.getline(__file__, 
lineno).rstrip()))
return trace

import asyncio

class AsyncIteratorWrapper:
def __init__(self, obj):
self._it = iter(obj)

def __aiter__(self):
return self

async def __anext__(self):
try:
return next(self._it)
except StopIteration:
raise StopAsyncIteration

def doit_sync():
for letter in "ab":
print(letter)
print(".")

async def doit_async():
async for letter in AsyncIteratorWrapper("ab"):
print(letter)
print(".")

print(sys.version)
sys.settrace(trace)

doit_sync()

loop = asyncio.new_event_loop()
loop.run_until_complete(doit_async())
loop.close()
--- 8< 



$ /usr/local/pythonz/pythons/CPython-3.9.0a1/bin/python3.9 /tmp/bpo2.py
3.9.0a1 (default, Nov 20 2019, 18:52:14)
[Clang 10.0.0 (clang-1000.10.44.4)]
call 25: def doit_sync():
line 26: for letter in "ab":
line 27: print(letter)
a
line 26: for letter in "ab":
line 27: print(letter)
b
line 26: for letter in "ab":
line 28: print(".")
.
retu 28: print(".")
call 30: async def doit_async():
line 31: async for letter in AsyncIteratorWrapper("ab"):
call 13: def __init__(self, obj):
line 14: self._it = iter(obj)
retu 14: self._it = iter(obj)
call 16: def __aiter__(self):
line 17: return self
retu 17: return self
call 19: async def __anext__(self):
line 20: try:
line 21: return next(self._it)
retu 21: return next(self._it)
exce 31: async for letter in AsyncIteratorWrapper("ab"):
line 32: print(letter)
a
line 31: async for letter in AsyncIteratorWrapper("ab"):
call 19: async def __anext__(self):
line 20: try:
line 21: return next(self._it)
retu 21: return next(self._it)
exce 31: async for letter in AsyncIteratorWrapper("ab"):
line 32: print(letter)
b
line 31: async for letter in AsyncIteratorWrapper("ab"):
call 19: async def __anext__(self):
line 20: try:
line 21: return next(self._it)
exce 21: return next(self._it)
line 22: except StopIteration:
line 23: raise StopAsyncIteration
exce 23: raise StopAsyncIteration
retu 23: raise StopAsyncIteration
exce 31: async for letter in AsyncIteratorWrapper("ab"):
line 33: print(".")
.
retu 33: print(".")

$ /usr/local/pythonz/pythons/CPython-3.9.0a2/bin/python3.9 /tmp/bpo2.py
3.9.0a2 (default, Dec 19 2019, 08:42:29)
[Clang 10.0.0 (clang-1000.10.44.4)]
call 25: def doit_sync():
line 26: for letter in "ab":
line 27: print(letter)
a
line 26: for letter in "ab":
line 27: print(letter)
b
line 26: for letter in "ab":
line 28: print(".")
.
retu 28: print(".")
call 30: async def doit_async():
line 31: async for letter in AsyncIteratorWrapper("ab"):
call 13: def __init__(self, obj):
line 14: self._it = iter(obj)
retu 14: self._it = iter(obj)
call 16: def __aiter__(self):
line 17: return self
retu 17: return self
call 19: async def __anext__(self):
line 20: try:
line 21: return next(self._it)
retu 21: return next(self._it)
exce 31: async for letter in AsyncIteratorWrapper("ab"):
line 32: print(letter)
a
line 31: async for letter in AsyncIteratorWrapper("ab"):
call 19: async def __anext__(self):
line 20: try:
line 21: return next(self._it)
retu 21: return next(self._it)
exce 31: async for letter in AsyncIteratorWrapper("ab"):
line 32: print(letter)
b
line 31: async for letter in AsyncIteratorWrapper("ab"):
call 19: async def __anext__(self):
line 20: try:
line 21: return next(self._it)
exce 21: return next(self._it)
line 22: except StopIteration:
line 23: raise StopAsyncIteration
exce 23: raise StopAsyncIteration
retu 23: raise StopAsyncIteration
exce 31: async for letter in AsyncIteratorWrapper("ab"):
line 32: print(letter)
line 33: 

[issue39165] Completeness and symmetry in RE, avoid `findall(...)[0]`

[Juancarlo Añez]

New submission from Juancarlo Añez :

The problematic `findall(...)[0]` is a common anti-pattern in Python programs. 
The reason is lack of symmetry and completeness in the `re` module.

The original proposal in `python-ideas` was to add `re.findfirst(pattern, 
string, flags=0, default=_mark)` with more or less the semantics of 
`next(findall(pattern, string, flags=flags), default=default)`. 

The referenced PR adds `findalliter(pattern, string, flags=0)` with the value 
semantics of `findall()` over a generator, implements `findall()` as `return 
list(findalliter(...))`, and implements `findfirst()`. 

Consistency and correctness are likely because all tests pass with the 
redefined `findall()`.

--
components: Library (Lib)
messages: 359039
nosy: apalala
priority: normal
pull_requests: 17191
severity: normal
status: open
title: Completeness and symmetry in RE, avoid `findall(...)[0]`
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39114] Python 3.9.0a2 changed how finally/return is traced

[Ned Batchelder]

Ned Batchelder  added the comment:

BTW: this is not a regression in your fix. 3.9a2 behaved this way also.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39114] Python 3.9.0a2 changed how finally/return is traced

[Ned Batchelder]

Ned Batchelder  added the comment:

Thanks, that fixes the original case in this issue.

Here is another problem which seems related enough to append here instead of 
opening a new issue:

--- 8< 
import linecache, sys

def trace(frame, event, arg):
lineno = frame.f_lineno
print("{} {}: {}".format(event[:4], lineno, linecache.getline(__file__, 
lineno).rstrip()))
return trace

print(sys.version)
sys.settrace(trace)

def b(exc):
print("exc is {!r}".format(exc))
try:
raise Exception(exc)
except Exception as e:
if exc != 'expected':
raise   # line 17
q = "removing this line changes the behavior"
r = "is this executed?" # line 19
return

b('expected')
try:
b('unexpected')
except:
pass
--- 8< 

With your fix, it reports that the raise on line 17 is followed by line 19, 
which can't be right:

3.9.0a2+ (heads/pr/17737:f2545467fa, Dec 29 2019, 15:38:57)
[Clang 10.0.0 (clang-1000.10.44.4)]
call 11: def b(exc):
line 12: print("exc is {!r}".format(exc))
exc is 'expected'
line 13: try:
line 14: raise Exception(exc)
exce 14: raise Exception(exc)
line 15: except Exception as e:
line 16: if exc != 'expected':
line 18: q = "removing this line changes the behavior"
line 19: r = "is this executed?"
line 20: return
retu 20: return
call 11: def b(exc):
line 12: print("exc is {!r}".format(exc))
exc is 'unexpected'
line 13: try:
line 14: raise Exception(exc)
exce 14: raise Exception(exc)
line 15: except Exception as e:
line 16: if exc != 'expected':
line 17: raise
line 19: r = "is this executed?"
retu 19: r = "is this executed?"


Python 3.8 reported this:

3.8.1 (default, Dec 19 2019, 08:38:38)
[Clang 10.0.0 (clang-1000.10.44.4)]
call 11: def b(exc):
line 12: print("exc is {!r}".format(exc))
exc is 'expected'
line 13: try:
line 14: raise Exception(exc)
exce 14: raise Exception(exc)
line 15: except Exception as e:
line 16: if exc != 'expected':
line 18: q = "removing this line changes the behavior"
line 19: r = "is this executed?"
line 20: return
retu 20: return
call 11: def b(exc):
line 12: print("exc is {!r}".format(exc))
exc is 'unexpected'
line 13: try:
line 14: raise Exception(exc)
exce 14: raise Exception(exc)
line 15: except Exception as e:
line 16: if exc != 'expected':
line 17: raise
retu 17: raise

--
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39162] setup.py not picking up tkinter headers

[Ned Deily]

Change by Ned Deily :


--
assignee:  -> ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34790] Deprecate passing coroutine objects to asyncio.wait()

[Andrew Svetlov]

Andrew Svetlov  added the comment:


New changeset 89aa7f0ede1a11c020e83f24394593c577a61509 by Andrew Svetlov (Kyle 
Stanley) in branch 'master':
bpo-34790: Implement deprecation of passing coroutines to asyncio.wait() 
(GH-16977)
https://github.com/python/cpython/commit/89aa7f0ede1a11c020e83f24394593c577a61509


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39114] Python 3.9.0a2 changed how finally/return is traced

[Mark Shannon]

Mark Shannon  added the comment:

Ned, I think this is fixed. Feel free to re-open if you disagree.

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33387] Simplify bytecodes for try-finally, try-except and with blocks.

[Mark Shannon]

Change by Mark Shannon :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33777] dummy_threading: .is_alive method returns True after execution has completed

[Inada Naoki]

Change by Inada Naoki :


--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39037] Fix the trial order of the __exit__ and __enter__ methods in the with statement documentation

[Géry]

Change by Géry :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39163] spam

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed
title: Perfect Exchange Migration tool -> spam
type: performance -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39163] Perfect Exchange Migration tool

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
Removed message: https://bugs.python.org/msg359028

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39163] Perfect Exchange Migration tool

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


Removed file: https://bugs.python.org/file48809/exchange-migration.jpg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39156] Break up COMPARE_OP into logically distinct operations.

[Mark Shannon]

Change by Mark Shannon :


--
keywords: +patch
pull_requests: +17190
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17754

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com