[issue36051] (Performance) Drop the GIL during large bytes.join operations?

[Rémi Lapeyre]

Change by Rémi Lapeyre :


--
nosy: +remi.lapeyre

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36051] (Performance) Drop the GIL during large bytes.join operations?

[Bruce Merry]

New submission from Bruce Merry :

A common pattern in libraries doing I/O is to receive data in chunks, put them 
in a list, then join them all together using b"".join(chunks). For example, see 
http.client.HTTPResponse._safe_read. When the output is large, the memory 
copies can block the interpreter for a non-trivial amount of time, and prevent 
multi-threaded scaling. If the GIL could be dropped during the memcpys it could 
improve parallel I/O performance in some high-bandwidth scenarios (36050 
mentions a case where I've run into this serialisation bottleneck in practice).

Obviously it could hurt performance to drop the GIL for small cases. As far as 
I know numpy uses thresholds to decide when it's worth dropping the GIL and it 
seems to work fairly well.

--
components: Interpreter Core
messages: 336082
nosy: bmerry
priority: normal
severity: normal
status: open
title: (Performance) Drop the GIL during large bytes.join operations?
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com