date:20210324



Change by Raymond Hettinger :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43198] One operation on sets < 1/100th as efficient in 3.9 than before



Raymond Hettinger  added the comment:


New changeset 72789592a3491bab49f144bb292679b1484885d9 by Raymond Hettinger in 
branch 'master':
bpo-43198:  Revert 3dd2157 that removed freeslot tracking. (#25010)
https://github.com/python/cpython/commit/72789592a3491bab49f144bb292679b1484885d9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43618] random.shuffle loses most of the elements



Change by Raymond Hettinger :


--
nosy: +skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43618] random.shuffle loses most of the elements



Change by Raymond Hettinger :


--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43511] tkinter with Tk 8.6.11 is slow on macOS

2021-03-24 Thread Thomas Wamm



Thomas Wamm  added the comment:

tkinter performance on Windows 10 looks okay, no obvious problems.

Attached is a transcript showing bench.py results for Pythons 3.8.2, 3.9.2, and 
3.10.0a6 on Windows 10 on a Dell laptop with an Intel i5-8250U CPU (medium good 
speed, a good comparison to Apple M1 Mac).

I originally raised this Issue because my new M1 Mac Mini performed so poorly 
compared to my Windows laptop, only for tkinter-related code.

--
Added file: 
https://bugs.python.org/file49913/results_tk-bench_Win10_i5-8250U.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43620] os.path.join does not use os.sep as documentation claims

2021-03-24 Thread Jared Sutton



New submission from Jared Sutton :

The behavior of os.path.join() does not match the documentation, in regards to 
the use of os.sep.  From the docs:

"""
The return value is the concatenation of path and any members of *paths with 
exactly one directory separator (os.sep) following each non-empty part except 
the last, meaning that the result will only end in a separator if the last part 
is empty.
"""

The documentation clearly states that the function uses the value of os.sep 
(which differs based on platform).  However, if you review the 2 
implementations (ntpath.py and posixpath.py), the separator character used is 
clearly hard-coded and doesn't reference os.sep at all.

One could say that this is either a doc bug or an implementation bug, depending 
on what the intended behavior is. I submit that this is an implementation bug, 
as one might want to use os.path.join() to construct a path to be used on a 
platform other than the one currently running the application. For example, a 
person might be running Python on Windows, but calling a web API and 
constructing a path for use on a remote posix system.

--
assignee: docs@python
components: Documentation, Library (Lib)
messages: 389489
nosy: docs@python, jpsutton
priority: normal
severity: normal
status: open
title: os.path.join does not use os.sep as documentation claims
type: behavior
versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43614] Search is not beginner friendly

2021-03-24 Thread Kamil Turek



Change by Kamil Turek :


--
nosy: +kamilturek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43619] convenience of using create_datagram_endpoint (and friends)

2021-03-24 Thread Roman Valov



New submission from Roman Valov :

Please check the attached source code.

I have to implement an UDP server listening on all interfaces and able to 
detect what is the local address is used to communicate with remote address. In 
order to do this I'm using a temporary socket connected to exact remote 
endpoint to retrieve it's sock name.

When I implement the solution in a pure `asyncio` fashion I faced pair of 
inconveniences:

ISSUE-1: there is no idiomatic way to sleep forever inside async function. The 
example of using `create_datagram_endpoint` in documentation uses `sleep(3600)` 
which is not so useful. I've used `loop.create_future()` but it's perceived to 
be kind of hack. Also I can't use `loop.run_forever` in this context.

Possible solutions:
 - `serve_forever` for a transport object
 - `asyncio.setup_and_run_forever(main())` -- function to setup file 
descriptors for an event loop and run forever.
 - `asyncio.sleep(None)` or `asyncio.pause()` -- special argument for sleep or 
dedicated `pause` function.


ISSUE-2: callbacks for `Protocol` class are assumed to be sync `def`s. Despite 
the class is designed to be used as a part of `asyncio`. So, in order to invoke 
async code from sync callback I have to add some boilerplate code. Compare with 
`start_server`. It's `client_connected_cb` argument maybe a plain callable or 
co-routine function. So it's proposed to let Protocol callbacks to be `async 
def`s.

--
components: asyncio
files: async.py
messages: 389488
nosy: Roman.Valov, asvetlov, yselivanov
priority: normal
severity: normal
status: open
title: convenience of using create_datagram_endpoint (and friends)
type: enhancement
versions: Python 3.8
Added file: https://bugs.python.org/file49912/async.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

2021-03-24 Thread David Brochart



David Brochart  added the comment:

Regarding the initial message in this issue, and enabling recursive event 
loops, this has proved to be very useful when an event loop is already running 
and some non-async code needs to run async code. This situation is very 
frequent when e.g. a library is designed to be async-first, and also provides a 
blocking API which just wraps the async code by running it until complete.
The nest-asyncio library (https://github.com/erdewit/nest_asyncio) allows that 
by patching asyncio's event loop, but obviously this doesn't work with other 
event loops such as uvloop. I was wondering if things had changed since the 
original post of this issue, and if such a feature had any chance to make it 
into the standard library.

--
nosy: +davidbrochart

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41064] Specialise syntax error of **dict in f-string field

2021-03-24 Thread Pablo Galindo Salgado



Pablo Galindo Salgado  added the comment:


New changeset 8efad61963809d239cac986e3f3bc4cb505ab8fe by Pablo Galindo in 
branch 'master':
bpo-41064: Improve syntax error for invalid usage of '**' in f-strings 
(GH-25006)
https://github.com/python/cpython/commit/8efad61963809d239cac986e3f3bc4cb505ab8fe


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41064] Specialise syntax error of **dict in f-string field

2021-03-24 Thread Pablo Galindo Salgado



Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43618] random.shuffle loses most of the elements

2021-03-24 Thread Tim Peters



Tim Peters  added the comment:

Are you sure it's "a list"? At least print out `type(questions_element)`. 
`random.shuffle()` doesn't contain any code _capable_ of changing a list's 
length. It only does indexed accessing of the list:

...
for i in reversed(range(1, len(x))):
# pick an element in x[:i+1] with which to exchange x[i]
j = randbelow(i + 1)
x[i], x[j] = x[j], x[i]

That's about all there is to it. Note that, for this purpose, it doesn't matter 
want `randbelow()` does, because that function never even sees `x`.

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43606] PySimpleGUI: initial huge window & no widgets visible

2021-03-24 Thread FRANK BENNETT



FRANK BENNETT  added the comment:

Roundup requires submission inplain text...forget how to turn this off in 
thunderbird...sounds like a left over from microsoft emailers...how do I 
see/respond to comments with an issue ?

On 3/24/21 9:37 AM, Serhiy Storchaka wrote:
> Serhiy Storchaka  added the comment:
>
> PySimpleGUI is not a part of the stdlib.
>
> If you have problems with pure Tkinter, look if you have files .Xdefaults or 
> .Xresources in your home directory. They can alter default look of Tk 
> widgets. Try to remove them and test whether the problem is gone.
>
> If the problem is still here, perhaps you have some fonts misconfiguration.
Good suggestion but same results...
  Tkinter ? Python 3.10 wants tkinter
> ___
> Python tracker 
> 

--
components:  -Tkinter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43618] random.shuffle loses most of the elements

2021-03-24 Thread Eric V. Smith



Eric V. Smith  added the comment:

Same advice as issue 43616: please provide an example we can run.

This most likely a problem with how you're using lxml, and not a bug in python. 
But since we can't test it, we can't know for sure.

--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42964] Draft PEP blob etc



Change by Irit Katriel :


--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43613] gzip.compress and gzip.decompress are sub-optimally implemented.



Change by Irit Katriel :


--
components: +Library (Lib)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43616] random.shuffle() crashes with Unhandled exception



Change by Irit Katriel :


--
components: +Library (Lib)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43618] random.shuffle loses most of the elements



Change by Irit Katriel :


--
components: +Library (Lib)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43618] random.shuffle loses most of the elements

2021-03-24 Thread Rowan Sylvester-Bradley



New submission from Rowan Sylvester-Bradley :

This issue is probably related to issue ??? but I have created it as a separate 
issue. When shuffle doesn't crash it sometimes (or maybe always - I haven't 
fully analysed this yet) looses most of the elements in the list that it is 
supposed to be shuffling. Here is an extract of the code that I'm using:

import io
from io import StringIO 
from lxml import etree 
import random  

filename_xml = 'MockExam5.xml'
with io.open(filename_xml, mode="r", encoding="utf-8") as xml_file:
xml_to_check = xml_file.read()
doc = etree.parse(StringIO(xml_to_check))
exams = doc.getroot()
questions_element = exams.find("questions")
logmsg(L_TRACE, "There are now " + 
str(len(questions_element.findall("question"))) + " questions")
logmsg(L_TRACE, "Randomising order of questions in this exam")
random.shuffle(questions_element)
logmsg(L_TRACE, "Finished randomise")
logmsg(L_TRACE, "There are now " + 
str(len(questions_element.findall("question"))) + " questions")

And here is the log produced by this code:

21-03-24 18:10:11.989 line:  2057 file: D:\XPS_8700 Extended 
Files\Users\RowanB\Documents\My_Scripts NEW\mockexam\put_exam.py 2 There are 
now 79 questions
21-03-24 18:10:11.991 line:  2065 file: D:\XPS_8700 Extended 
Files\Users\RowanB\Documents\My_Scripts NEW\mockexam\put_exam.py 2 Randomising 
order of questions in this exam
21-03-24 18:10:11.992 line:  2067 file: D:\XPS_8700 Extended 
Files\Users\RowanB\Documents\My_Scripts NEW\mockexam\put_exam.py 2 Finished 
randomise
21-03-24 18:10:11.993 line:  2068 file: D:\XPS_8700 Extended 
Files\Users\RowanB\Documents\My_Scripts NEW\mockexam\put_exam.py 2 There are 
now 6 questions

How come the shuffle starts off with 79 elements, and finishes with 6?

Thanks - Rowan

--
messages: 389482
nosy: rowan.bradley
priority: normal
severity: normal
status: open
title: random.shuffle loses most of the elements
type: behavior
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43616] random.shuffle() crashes with Unhandled exception



Change by Raymond Hettinger :


--
nosy: +skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43617] Missing definition in configure.ac causing autoreconf to create damaged configure script



Cong Ma  added the comment:

> The extra macros are provided by optional packages. On Fedora and 
> Debian/Ubuntu the package is called autoconf-archive.

Thank you very much. This (and the patch) is clearing things up for me a lot. 
At first I thought I was supposed to copy the m4 files over from the autoconf 
archive into the m4 directory under the repo.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41551] test.support has way too many imports in libregrtest

2021-03-24 Thread hai shi



hai shi  added the comment:

bpo-41718

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41551] test.support has way too many imports in libregrtest

2021-03-24 Thread hai shi



hai shi  added the comment:

There have another related issue and have been fixed now in bpo 41718.
So I suggest to close this bpo.

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43511] tkinter with Tk 8.6.11 is slow on macOS



E. Paine  added the comment:

As this appears to be a Tk issue, I have created 
https://core.tcl-lang.org/tk/tktview/f642d7c0f4.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43511] tkinter with Tk 8.6.11 is slow on macOS



E. Paine  added the comment:

> there appears to be a tk 8.6.11.1 release

This was the version I used when testing Wish on both MacOS and Linux.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43481] PyEval_EvalCode() namespace issue not observed in Python 2.7.

2021-03-24 Thread Terry J. Reedy



Terry J. Reedy  added the comment:

Perhaps both.  If you want more discussion, please post to python-list.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32592] Drop support of Windows Vista and Windows 7

2021-03-24 Thread Alexey Izbyshev



Alexey Izbyshev  added the comment:

> If we had a dedicated maintainer who was supporting Win7 and making releases 
> for it, then we (i.e. they) could support it. But then, there's nothing to 
> stop someone doing that already, and even to stop them charging money for it 
> if they want (which they wouldn't be able to do under the auspices of 
> python-dev). So I suspect nobody is really that motivated ;)

That's totally fair.

> (Also, a "little bit of complaining" here is totally worthwhile, as it helps 
> us gauge community sentiment. Without it, we're limited to trawling forums 
> and Twitter - especially without conferences to actually meet and hear 
> directly from people - and so our inputs are biased. Having polite, informed, 
> discussion on the tracker is a great way for us to see and capture these 
> positions.)

I agree. In my comment, I only intended to contrast "complaining" with stepping 
up to do the work. I didn't mean to imply that it's inappropriate per-se.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43616] random.shuffle() crashes with Unhandled exception



Change by Raymond Hettinger :


--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40601] [C API] Hide static types from the limited C API

2021-03-24 Thread Guido van Rossum



Guido van Rossum  added the comment:

FWIW I have an idea that would allow code using e.g. _Type to continue 
to work, and even ABI compatible (though only in the main interpreter).

// In some header file

PyAPI_FUNC(PyHeapTypeObject *) PyList_GetType();

#define PyList_Type (PyList_GetType()->ht_type)

For the main interpreter we could make this return the address of PyList_Type.

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43617] Missing definition in configure.ac causing autoreconf to create damaged configure script

2021-03-24 Thread Christian Heimes



Change by Christian Heimes :


--
keywords: +patch
pull_requests: +23771
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/25016

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43614] Search is not beginner friendly

2021-03-24 Thread Serhiy Storchaka



Serhiy Storchaka  added the comment:

Ha! I just was going to open the same issue (perhaps we read the same post on 
Quora).

I concur that it is perhaps worth to add index entries for common parameter 
names (args, kwargs, self, cls) and also for *args and **kwargs and add notions 
that such name is commonly used for such parameter.

--
nosy: +rhettinger, serhiy.storchaka
type:  -> enhancement
versions: +Python 3.10, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43617] Missing definition in configure.ac causing autoreconf to create damaged configure script

2021-03-24 Thread Christian Heimes



Christian Heimes  added the comment:

The extra macros are provided by optional packages. On Fedora and Debian/Ubuntu 
the package is called autoconf-archive.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43606] PySimpleGUI: initial huge window & no widgets visible

2021-03-24 Thread Serhiy Storchaka



Serhiy Storchaka  added the comment:

PySimpleGUI is not a part of the stdlib.

If you have problems with pure Tkinter, look if you have files .Xdefaults or 
.Xresources in your home directory. They can alter default look of Tk widgets. 
Try to remove them and test whether the problem is gone.

If the problem is still here, perhaps you have some fonts misconfiguration.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42137] Prefer using spec over module_repr() for ModuleType.repr

2021-03-24 Thread Brett Cannon



Change by Brett Cannon :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42137] Prefer using spec over module_repr() for ModuleType.repr

2021-03-24 Thread Brett Cannon



Brett Cannon  added the comment:


New changeset 9cb31d671646a5ff0901f79d2d61022621447190 by Brett Cannon in 
branch 'master':
bpo-42137: have ModuleType.__repr__ prefer __spec__ over module_repr() 
(GH-24953)
https://github.com/python/cpython/commit/9cb31d671646a5ff0901f79d2d61022621447190


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43605] Issue of scopes unclear in documentation, or wrongly implemented



Cong Ma  added the comment:

I think this is in the same class of behaviours as

```
def func(l):
def get(i):
return l[i]
print(eval("(lambda x: get(x))(0)"))  # Call anonymous lambda with the 
constant 0 as argument
```
 
Calls like ``func(["spam"])`` will not "work", and ``NameError`` is raised.

In this case, inside the lambda's body the name "get" can't be resolved. For 
the lambda body, the name "get" is a nonlocal but there's no way to access a 
nonlocal in a lambda.

The comprehensions, like lambdas, are in their own nested scope.

--
nosy: +congma

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43481] PyEval_EvalCode() namespace issue not observed in Python 2.7.

2021-03-24 Thread Chris Morton



Chris Morton  added the comment:

Is this change in behavior considered to be by design or is this issue an 
unintended consequence? It seems inconsistent that comprehensions have no 
visibility of variable previously defined in the same scoping for this 
particular use-case. Is there a way to work around this other than reworking 
the comprehension?

It seems odd that [ci for ci in c] does not produce the error but [c for i in 
[1]] does produce the error.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43617] Missing definition in configure.ac causing autoreconf to create damaged configure script



STINNER Victor  added the comment:

Ah, maybe the configure.ac comment should be updated to suggest running 
"autoconf" instead ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43617] Missing definition in configure.ac causing autoreconf to create damaged configure script



Cong Ma  added the comment:

>From the configure.ac file:

> dnl ***
> dnl * Please run autoreconf to test your changes! *
> dnl ***

I take it to mean "if configure.ac is changed, run autoreconf first", and 
that's what I did. Could you let me know what is the intended way to modify 
configure.ac and have it take effect?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43617] Missing definition in configure.ac causing autoreconf to create damaged configure script



STINNER Victor  added the comment:

> If ``autoreconf`` is run, ...

I never used autoreconf. If you don't run it, ./configure works fine.

I only use "autoconf" and "autoheader".

$ ./configure --enable-optimizations --enable-shared
(...)
$ grep semantic Makefile
CONFIGURE_CFLAGS_NODIST= -fno-semantic-interposition -std=c99 -Wextra 
-Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers 
-Werror=implicit-function-declaration -fvisibility=hidden
CONFIGURE_LDFLAGS_NODIST= -fno-semantic-interposition

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43615] [PATCH] Properly implement Py_UNREACHABLE macro using autoconf.



Cong Ma  added the comment:

> If you consider that there is a bug, please open a new issue since you closed 
> this one.

Please see the follow up in Issue 43617.

Many thanks for bearing with me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43617] Missing definition in configure.ac causing autoreconf to create damaged configure script



Change by Cong Ma :


--
type:  -> compile error

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43617] Missing definition in configure.ac causing autoreconf to create damaged configure script



New submission from Cong Ma :

The problem
---

In the repository, the definition for ``AX_CHECK_COMPILE_FLAG`` in Python's 
``configure.ac`` file is missing. If ``autoreconf`` is run, an invalid 
``configure`` script is generated. The following is the behaviour of running 
``autoreconf`` followed by ``configure``:

```
# In cpython repository top-level directory
$ autoreconf
$ mkdir build
$ cd build
$ ../configure  # <- using newly generated configure script
[... omitted ...]
checking for --enable-optimizations... no
../configure: line 6498: syntax error near unexpected token 
`-fno-semantic-interposition,'
../configure: line 6498: `  
AX_CHECK_COMPILE_FLAG(-fno-semantic-interposition,'
```


The solution


It appears a file was missing in the m4/ directory. The file matches this one 
from the Autoconf Archive:

https://www.gnu.org/software/autoconf-archive/ax_check_compile_flag.html

Simply adding the correct m4 file to m4/ should make ``autoreconf`` work.

--
components: Build
messages: 389463
nosy: congma
priority: normal
severity: normal
status: open
title: Missing definition in configure.ac causing autoreconf to create damaged 
configure script

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43616] random.shuffle() crashes with Unhandled exception

2021-03-24 Thread Jelle Zijlstra



Jelle Zijlstra  added the comment:

Could you post code that fully reproduces the problem?

I suspect the root cause for the bug is in lxml (a third-party library), not in 
CPython itself.

--
nosy: +Jelle Zijlstra

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43615] [PATCH] Properly implement Py_UNREACHABLE macro using autoconf.



STINNER Victor  added the comment:

> BTW, do we need to fix the missing definition of the AX_CHECK_COMPILE_FLAG 
> macro in configure.ac? This is a separate problem, if a problem at all.

I'm not sure which problem you are trying to solve. If you consider that there 
is a bug, please open a new issue since you closed this one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32592] Drop support of Windows Vista and Windows 7

2021-03-24 Thread Steve Dower



Steve Dower  added the comment:

> not sure about the strategies here but maybe a better approach would be to 
> kill support for unsupported version of windows in a major release

We do, but the "major" release is the second field of the version number (you 
can think of the first field as the "series" or "product" number).

Yes, this is not SemVer. Sorry. We pre-date SemVer.

Python 3.8 and earlier are just as supported for Windows 7 as our volunteers 
can handle, which is all that has ever been offered here. If people need better 
support than that, I'd suggest paying someone.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43615] [PATCH] Properly implement Py_UNREACHABLE macro using autoconf.



STINNER Victor  added the comment:

> The headers Python.h and also the ones pulled in by it were actually from 
> Python 3.8 release, which unconditionally defines the macro as a call to 
> Py_FatalError.

Using __builtin_unreachable() is a recent change (bpo-38249), Python 3.9.0:

* commit eebaa9bfc593d5a46b293c1abd929fbfbfd28199

Fix for old GCC (bpo-41875):

* master: commit 24ba3b0df5e5f2f237d7b23b4017ba12f16320ae
* 3.9.1: commit cca896e13b230a934fdb709b7f10c5451ffc25ba

I prefer to not backport this feature, since, as you wrote, it's tricky to make 
sure that it's available on the C compiler.

I close the issue since Py_UNREACHABLE() works as expected on Python 3.9 and 
newer.

--
stage: resolved -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43615] [PATCH] Properly implement Py_UNREACHABLE macro using autoconf.



Cong Ma  added the comment:

BTW, do we need to fix the missing definition of the AX_CHECK_COMPILE_FLAG 
macro in configure.ac? This is a separate problem, if a problem at all.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43615] [PATCH] Properly implement Py_UNREACHABLE macro using autoconf.



Cong Ma  added the comment:

Hello Victor,

I think you're right. This is bogus on my part. TL;DR: The Python version is 
3.8 but I was trying to understand what's going on using the latest source.

Full version: I was trying to understand why the following C file when compiled 
with -shared using Clang 11 generates a call to Py_FatalError():

```
#define PY_SSIZE_T_CLEAN
#include "Python.h"


void
unreach(void)
{
Py_UNREACHABLE();
}
```

The headers Python.h and also the ones pulled in by it were actually from 
Python 3.8 release, which unconditionally defines the macro as a call to 
Py_FatalError.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43616] random.shuffle() crashes with Unhandled exception

2021-03-24 Thread Rowan Sylvester-Bradley



New submission from Rowan Sylvester-Bradley :

When I do 
  random.shuffle(questions_element) 
(questions_element is an element generated by lxml via the code
  questions_element = exams.find("questions") )
I get a crash:
Unhandled exception at 0x7FFD7AE8EF89 (ntdll.dll) in python.exe: 
0xC374: A heap has been corrupted (parameters: 0x7FFD7AEF77F0).

Is there a way to work around this?

Thanks - Rowan

--
messages: 389456
nosy: rowan.bradley
priority: normal
severity: normal
status: open
title: random.shuffle() crashes with Unhandled exception
type: crash
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43615] [PATCH] Properly implement Py_UNREACHABLE macro using autoconf.



STINNER Victor  added the comment:

> The current implementation tests the ``__GNUC__`` and ``__GNUC_MINOR__`` 
> macros as the logic (GCC version >= 4.5) for determining whether the compiler 
> intrinsic ``__builtin_unreachable()`` is present (see commits eebaa9bf, 
> 24ba3b0d). However, Clang defines these macros too and can cause confusion. 
> Clang 11 pretends to be GCC 4.2.1 in its predefined macros. As a result, 
> Clang won't use the intrinsic even if it's supported. This doesn't seem to 
> match the intent behind the original implementation.

Hum. The current code is:

#elif defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 
5))
#  define Py_UNREACHABLE() __builtin_unreachable()
#elif defined(__clang__) || defined(__INTEL_COMPILER)
#  define Py_UNREACHABLE() __builtin_unreachable()

Even if clang pretends to be GCC 4.2 and the first test fails, the second test 
is true and so Py_UNREACHABLE() is always defined as __builtin_unreachable() on 
clang, no?

You can please better explain your problem? You can write a short C code 
demonstrating the bug?

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43615] [PATCH] Properly implement Py_UNREACHABLE macro using autoconf.

New submission from Cong Ma :

(This is a summarized form of the commit message in the attached patch. I'm
submitting a patch instead of a PR over GitHub, because it seems that the
``autoreconf`` output files are part of the repository. In order for the
changes to take effect in the repo, I may have to run ``autoreconf`` and add
the clobbered output files to the repo, which I don't think is a good idea.
Also on my system the ``autoreconf`` can only work correctly if I add a missing
M4 file "ax_check_compile_flag.m4" from the Autoconf Archive
for
the ``AX_CHECK_COMPILE_FLAG`` macro used in the existing ``configure.ac``. I
don't think it's wise for me to introduce so many changes at once if most
developers don't need to run ``autoreconf`` often.)

The problem
---

Definition of the ``Py_UNREACHABLE()`` macro relied on testing compiler
versions in preprocessor directives. This is unreliable chiefly because
compilers masquerade as each other.

The current implementation tests the ``__GNUC__`` and ``__GNUC_MINOR__`` macros
as the logic (GCC version >= 4.5) for determining whether the compiler
intrinsic ``__builtin_unreachable()`` is present (see commits eebaa9bf,
24ba3b0d). However, Clang defines these macros too and can cause confusion.
Clang 11 pretends to be GCC 4.2.1 in its predefined macros. As a result, Clang
won't use the intrinsic even if it's supported. This doesn't seem to match the
intent behind the original implementation.

The solution

Test the presence of the compiler-builtin ``__builtin_unreachable()`` at
configure-time using Autoconf, and conditionally define the
``Py_UNREACHABLE()`` macro depending on the configuration.

The idea is based on the ``ax_gcc_builtin.m4`` code [0] by Gabriele Svelto.

Alternative ideas
-

Recent versions of Clang and GCC support the ``__has_builtin()`` macro.
However, this may be unreliable before Clang 10 [1], while GCC support is only
available as of GCC 10 and its semantics may not be the same as Clang's [2].
Therefore ``__has_builtin()`` may not be as useful as it seems.

We may attempt to improve the accuracy of version checking in ``#if``
directives, but this could be brittle and difficult to explain, verify, or
maintain.

Links
-

[0] https://www.gnu.org/software/autoconf-archive/ax_gcc_builtin.html
[1] https://clang.llvm.org/docs/LanguageExtensions.html#has-builtin
[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66970#c24

--
components: Build
files: 0001-Properly-implement-Py_UNREACHABLE-macro-using-autoco.patch
keywords: patch
messages: 389454
nosy: congma
priority: normal
severity: normal
status: open
title: [PATCH] Properly implement Py_UNREACHABLE macro using autoconf.
type: enhancement
Added file:
https://bugs.python.org/file49910/0001-Properly-implement-Py_UNREACHABLE-macro-using-autoco.patch

___
Python tracker

___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2001] Pydoc interactive browsing enhancement



STINNER Victor  added the comment:

The "getfile" feature has a directory traversal vulnerability and so I propose 
to remove the feature: see bpo-42988.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42988] [security] CVE-2021-3426: Information disclosure via pydoc -p: /getfile?key=path allows to read arbitrary file on the filesystem



STINNER Victor  added the comment:

The "pydoc -p port" command only listen on the local link ("localhost") by 
default, even if it's possible to listen on another IPv4 address using -n 
HOSTNAME command line option.

While the "getfile" feature is convenient when the pydoc server is accessed 
from a different machine, I don't think that it's worth it, compared to the 
security risks and the complexity of PR 24285 and PR 24337 fixes.

I propose to simply remove the "getfile" feature with PR 25015, but keep links 
using file:// scheme. So we delegate the security to the web browser. If the 
web server is allowed to read sensitive data of a local Python file, it's not 
our problem: pydoc doesn't make things worse.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43606] PySimpleGUI: initial huge window & no widgets visible

2021-03-24 Thread FRANK BENNETT



FRANK BENNETT  added the comment:

fwb@fw:/s/opt/cpython/debug$ ./python -V
Python 3.10.0a6+

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42988] [security] CVE-2021-3426: Information disclosure via pydoc -p: /getfile?key=path allows to read arbitrary file on the filesystem



Change by STINNER Victor :


--
pull_requests: +23770
pull_request: https://github.com/python/cpython/pull/25015

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43511] tkinter with Tk 8.6.11 is slow on macOS



E. Paine  added the comment:

The reduction in speed with later Tk versions does not seem to occur on Linux. 
See the below table (times are microseconds per iteration - tested using wish):

++--+--+
| Tk Version | clearing | not clearing |
++==+==+
| 8.6.8  | 197.7| 183.46   |
++--+--+
| 8.6.11 | 170.3| 201.6|
++--+--+

I have not tested on Windows but I don't think it's unreasonable to conclude 
the performance issue is limited to MacOS.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41100] Support macOS 11 and Apple Silicon Macs

2021-03-24 Thread Roundup Robot



Change by Roundup Robot :


--
pull_requests: +23769
pull_request: https://github.com/python/cpython/pull/25014

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32592] Drop support of Windows Vista and Windows 7

2021-03-24 Thread Mitja kocjančič


Mitja kocjančič  added the comment:

not sure about the strategies here but maybe a better approach would be to kill 
support for unsupported version of windows in a major release
since I guess python 3 was a complete rewrite of python2 (at least the low 
level side I assume it was) 

and it would be easier for people to remeber (eg, if I have to rewrite my app 
because python4 has a major language differences I might as well drop support 
for older OSes in it), eg, python2 works on XP, python3 works on vista and up, 
python4 works on windows 10

for instance, I didn't even know you can run python3 on XP, I always thought 
that python2.7 is the last version that would run there

and since the code to support windows 7 is still present (almost, just the 
installer change would need to be rolleed back and some compiler declaratives: 
https://github.com/python/cpython/commit/0b72ccff56fb47e14f7b1f6590eafff8d104c229
https://github.com/izbyshev/cpython/commit/6a65eba44bfd82ccc8bed4b5c6dd6637549955d5

I see no reason to touch it (it probably just stays there serving its purpose) 
and when python4 comes along (if its a rewrite then windows 7 hacks (would just 
not be written anymore)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43511] tkinter with Tk 8.6.11 is slow on macOS

2021-03-24 Thread Ronald Oussoren



Ronald Oussoren  added the comment:

One thing to look into: there appears to be a tk 8.6.11.1 release, I have no 
idea what changed in this micro release.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43511] tkinter with Tk 8.6.11 is slow on macOS

2021-03-24 Thread Ronald Oussoren



Ronald Oussoren  added the comment:

If this is a Tk issue we should report it in their bug tracker.

I'm definitely against switching to an older version of Tk, the most recent 
release fixes a lot of problems with Tk on macOS.  Correctness before 
performance.

How bad is the performance compared to Tk on other platforms (with similar 
hardware)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43511] tkinter with Tk 8.6.11 is slow on macOS

2021-03-24 Thread Thomas Wamm



Thomas Wamm  added the comment:

I ran the bench.py and bench.tcl programs from E. Paine (epaine) on my M1 Mac 
Mini.  Attached is a commented Terminal transcript.

The tl;dr summary again is that Python 3.10.0a6 arm64 for Apple silicon is 3 to 
100 times SLOWER than earlier pythons compiled for Intel x86-64 running with 
Rosetta2, when doing window operations thru Tcl/Tk.

I have not yet attempted to install any version of Tcl/Tk independent of any 
Python or Apple build.  I'm still learning slowly.  And I am inspired to re-jig 
my little TerraLunar graphics program to run with python2.7 arm64.

--
Added file: https://bugs.python.org/file49908/results-tkbench.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32592] Drop support of Windows Vista and Windows 7

2021-03-24 Thread Steve Dower



Steve Dower  added the comment:

FWIW, the installer change was only to forcibly warn naive users that they need 
to stick with the previous version, so I wouldn't read much into it. It's 
easily rolled back.

The real issue is that we need to decide how to drop support _somehow_, and 
following the company that has the most to gain/lose from declaring support for 
one of their platforms makes the most sense.

If we had a dedicated maintainer who was supporting Win7 and making releases 
for it, then we (i.e. they) could support it. But then, there's nothing to stop 
someone doing that already, and even to stop them charging money for it if they 
want (which they wouldn't be able to do under the auspices of python-dev). So I 
suspect nobody is really that motivated ;)

(Also, a "little bit of complaining" here is totally worthwhile, as it helps us 
gauge community sentiment. Without it, we're limited to trawling forums and 
Twitter - especially without conferences to actually meet and hear directly 
from people - and so our inputs are biased. Having polite, informed, discussion 
on the tracker is a great way for us to see and capture these positions.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43606] PySimpleGUI: initial huge window & no widgets visible



E. Paine  added the comment:

Apologies, you did give the platform and Python version. I presume you are 
compiling the master branch, in which case do you have the 'tk8.6-dev' package 
installed?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43606] PySimpleGUI: initial huge window & no widgets visible



E. Paine  added the comment:

That example also works on my setup (see attached). The only reason I could 
think that your label is not showing is that the font given is not of a valid 
format (see https://www.tcl.tk/man/tcl8.6/TkCmd/font.htm#M13), though I would 
have thought this would raise a TclError.

Please confirm what platform and Python version you are using as well as the 
output of `tkinter.test.support.get_tk_patchlevel()`.

--
Added file: https://bugs.python.org/file49907/working example.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43614] Search is not beginner friendly

2021-03-24 Thread Anthony Flury



New submission from Anthony Flury :

A commonly asked question on Quora is 'What do *args and **kwargs' mean ?

While it is relatively easy for community to answer these questions the search 
tool on the standard documentation doesn't make it easy.

I understand that 'args' and 'kwargs' are both naming conventions, they are 
very common across the documentation, but searching on '*args' or '**kwargs' 
doesn't actually find anything useful - it certainly doesn't place 
'https://docs.python.org/3/tutorial/controlflow.html#arbitrary-argument-lists' 
at or close to the top of the list.

It is my view that the documentation should be beginner friendly, but in this 
case (and many other I guess) you have to know what to search for to find 
something useful.

I note that even common phrases in Computing (such as 'variable arguments' or 
'variable parameters') don't find anything useful. The term 'variadic' does 
find the relevant page, but the link displayed in the search results lands on 
the page (but not the relevant section) - and many beginners wont search for 
'variadic'.

The index and search need to be improved to help beginners - specifically in 
this case
* Search Index should include common conventional names (such as args, kwargs)
* Search Index should include common computing terms ('variable arguments' for 
example - even if the documentation doesn't actually use that terminology).
* Search should link to the relevant section (and not just the page).

--
assignee: docs@python
components: Documentation
messages: 389442
nosy: anthony-flury, docs@python
priority: normal
severity: normal
status: open
title: Search is not beginner friendly
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)

2021-03-24 Thread Hans-Christoph Steiner



Hans-Christoph Steiner  added the comment:

> - For full reproducible builds you may have to write files to zipfiles in a 
> well-defined order.

That already works fine now, we've been doing that with Python for years.  But 
that leaves it up to the implemented to do.  I suppose zipfile could provide a 
method to sort entries, but that's out of scope for this issue IMHO.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43613] gzip.compress and gzip.decompress are sub-optimally implemented.



Change by Ruben Vorderman :


--
type:  -> performance

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43612] zlib.compress should have a wbits argument



Change by Ruben Vorderman :


--
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43612] zlib.compress should have a wbits argument



Change by Ruben Vorderman :


--
keywords: +patch
pull_requests: +23768
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/25011

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42914] pprint numbers with underscore

2021-03-24 Thread Gregory P. Smith


Gregory P. Smith  added the comment:

Thanks for the contribution Stéphane!

I agree that this would be a nice default.  We're just being conservative in 
the pace of default behavior changes.  Changing the default could be considered 
in the future after a few releases with this parameter have shipped.

--
resolution:  -> fixed
stage: patch review -> commit review
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42914] pprint numbers with underscore

2021-03-24 Thread Gregory P. Smith



Gregory P. Smith  added the comment:


New changeset 3ba3d513b1e3c63d09cb798b982a9e6c369cea4c by sblondon in branch 
'master':
bpo-42914: add a pprint underscore_numbers option (GH-24864)
https://github.com/python/cpython/commit/3ba3d513b1e3c63d09cb798b982a9e6c369cea4c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43613] gzip.compress and gzip.decompress are sub-optimally implemented.



Change by Raymond Hettinger :


--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43613] gzip.compress and gzip.decompress are sub-optimally implemented.



New submission from Ruben Vorderman :

When working on python-isal which aims to provide faster drop-in replacements 
for the zlib and gzip modules I found that the gzip.compress and 
gzip.decompress are suboptimally implemented which hurts performance.

gzip.compress and gzip.decompress both do the following things:
- Instantiate a BytesIO object to mimick a file
- Instantiate a GzipFile object to compress or read the file.

That means there is way more Python code involved than strictly necessary. Also 
the 'data' is already fully in memory, but the data is streamed anyway. That is 
quite a waste.

I propose the following:
- The documentation should make it clear that zlib.decompress(... ,wbits=31) 
and zlib.compress(..., wbits=31) (after 43612 has been addressed), are both 
quicker but come with caveats. zlib.compress can not set mtime. zlib.decompress 
does not take multimember gzip into account. 
- For gzip.compress -> The GzipFile._write_gzip_header function should be moved 
to a module wide _gzip_header function that returns a bytes object. 
GzipFile._write_gzip_header can call this function. gzip.compress can also call 
this function to create a header. gzip.compress than calls zlib.compress(data, 
wbits=-15) (after 43612 has been fixed) to create a raw deflate block. A gzip 
trailer can be easily created by calling zlib.crc32(data) and len(data) & 
0x and packing those into a struct. See for an example implementation 
here: 
https://github.com/pycompression/python-isal/blob/v0.8.0/src/isal/igzip.py#L242
-> For gzip.decompress it becomes quite more involved. A read_gzip_header 
function can be created, but the current implementation returns EOFErrors if 
the header is incomplete due to a truncated file instead of BadGzipFile errors. 
This makes it harder to implement something that is not a major break from 
current gzip.decompress. Apart from the header, the implementation is 
straightforward. Do a while true loop. All operations are performed in the 
loop. Validate the header and report the end of the header. Create a 
zlib.decompressobj(wbits=-15). Decompress all the data from the end of header. 
Flush. Extract the crc and length from the first 8 bytes of the unused data. 
data = decompobj.unused_data[8:]. if not data: break. For a reference 
implementation check here: 
https://github.com/pycompression/python-isal/blob/v0.8.0/src/isal/igzip.py#L300.
 Note that the decompress function is quite straightforward. Checking the 
header however while maintaining backwards compatibility with gzip.deco
 mpress is not so simple.

And that brings to another point. Should non-descriptive EOFErrors be raised 
when reading the gzip header? Or throw informative BadGzipFile errors when the 
header is parsed. I tend towards the latter. For example BadGzipFile("Truncated 
header") instead of EOFError. Or at least EOFError("Truncated gzip header"). I 
am aware that confounds this issue with another issue, but these things are 
coupled in the implementation so both need to be solved at the same time.

Given the headaches that gzip.decompress gives it might be easier to solve 
gzip.compress first in a first PR and do gzip.decompress later.

--
messages: 389438
nosy: rhpvorderman
priority: normal
severity: normal
status: open
title: gzip.compress and gzip.decompress are sub-optimally implemented.

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43198] One operation on sets < 1/100th as efficient in 3.9 than before



Change by Raymond Hettinger :


--
keywords: +patch
pull_requests: +23767
stage: resolved -> patch review
pull_request: https://github.com/python/cpython/pull/25010

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43612] zlib.compress should have a wbits argument