On 19/10/2021 00.06, Chris Angelico wrote:
On Tue, Oct 19, 2021 at 9:00 AM Cameron Simpson <[email protected]> wrote:
The problem with a "download()" method is that it is almost never what
you need. There are too many ways to want to do it, and one almost
_never_ wants to suck the download itself into memory as you do above
with read() because downloads are often large, sometimes very large.
You also don't always want to put it into a file.
OTOH, if you *do* want to put it into a file, it should be possible to
take advantage of zero-copy APIs to reduce unnecessary transfers. I'm
not sure if there's a way to do that with requests. Ideally, what you
want is os.sendfile() but it'd need to be cleanly wrapped by the
library itself.
Splicing APIs like sendfile() require a Kernel socket. You cannot do
sendfile() with userspace sockets like OpenSSL socket for e.g. HTTPS.
Latest Linux Kernel and OpenSSL 3.0.0 have a new feature called kTLS.
Kernel TLS uses OpenSSL to establish the TLS connection and then handles
payload transfer in Kernel to for zero-copy sendfile().
Christian
_______________________________________________
Python-ideas mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/4IWZOPHWKQKKGAZPHYFH3D667W64VSCV/
Code of Conduct: http://python.org/psf/codeofconduct/
