On Wed, Oct 20, 2021 at 1:23 AM Christian Heimes <[email protected]> wrote: > > On 19/10/2021 00.06, Chris Angelico wrote: > > On Tue, Oct 19, 2021 at 9:00 AM Cameron Simpson <[email protected]> wrote: > >> The problem with a "download()" method is that it is almost never what > >> you need. There are too many ways to want to do it, and one almost > >> _never_ wants to suck the download itself into memory as you do above > >> with read() because downloads are often large, sometimes very large. > >> > >> You also don't always want to put it into a file. > >> > > > > OTOH, if you *do* want to put it into a file, it should be possible to > > take advantage of zero-copy APIs to reduce unnecessary transfers. I'm > > not sure if there's a way to do that with requests. Ideally, what you > > want is os.sendfile() but it'd need to be cleanly wrapped by the > > library itself. > > Splicing APIs like sendfile() require a Kernel socket. You cannot do > sendfile() with userspace sockets like OpenSSL socket for e.g. HTTPS. > > Latest Linux Kernel and OpenSSL 3.0.0 have a new feature called kTLS. > Kernel TLS uses OpenSSL to establish the TLS connection and then handles > payload transfer in Kernel to for zero-copy sendfile(). >
Ah, of course, forgot about that. But obviously other people have seen the same problem, and come up with a solution. In any case, it cements the need for an actual API for "download this into a file", despite the limitations of it. Obviously you also need the API of "give me the next lot of bytes as a string", but being able to download directly to a file *is* of significant value. ChrisA _______________________________________________ Python-ideas mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/RAD2GKYFZP5TWM52TA62AMV6VDUHQ3SU/ Code of Conduct: http://python.org/psf/codeofconduct/
