[issue13559] Use sendfile where possible in httplib

[Christian Heimes]

Christian Heimes  added the comment:

sendfile() only works for plain HTTP. For technical reasons it does not work 
for HTTPS (*). These days majority of services use HTTPS. Therefore the 
usefulness of sendfile() patch is minimal.

(*) It is possible to use sendfile() for TLS connections, but the feature 
requires a Kernel module that provides kTLS offloading feature, 
https://www.kernel.org/doc/html/latest/networking/tls-offload.html . In user 
space it requires OpenSSL 3.0.0 with kTLS support. 3.0.0 is currently under 
development.

--
nosy: +christian.heimes
versions: +Python 3.10 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Senthil Kumaran]

Senthil Kumaran  added the comment:

Yes, the point number 5. We will have to evaluate if sendfile side-steps and 
avoids the issues noted in issue23740

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Alex Willmer]

Alex Willmer  added the comment:

To check my understanding

Is the motivation for the closer to

1. using sendfile() will break $X, and we know X
2. there's high probability sendfile() will break something
3. there's unknown probability sendfile() will break something
4. there's low probability sendfile() will break something, but it is still too 
high
5. any non-trivial change here is too risky, regardless of sendfile()
6. something else?

My guess is 5.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Senthil Kumaran]

Senthil Kumaran  added the comment:

Alex, https://bugs.python.org/issue23740 is identified as a dependency on this 
issue. We will have to resolve that first, and come back to this. And yes, if 
you contribute on other's patch, both the contributions will be included and 
appropriately credited.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Alex Willmer]

Alex Willmer  added the comment:

I would like to take a stab at this. Giampaolo, would it be okay if I made a 
pull request updated from your patch? With the appropriate "Co-authored-by: 
Author Name " line.

--
nosy: +Alex.Willmer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[STINNER Victor]

STINNER Victor  added the comment:

This issue is no newcomer friendly, I remove the "easy" keyword.

--
keywords:  -easy
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Martin Panter]

Martin Panter added the comment:

The multiple personalities of HTTPConnection.send() and friends is a bit of a 
can of worms. I suggest working on Issue 23740 to get an idea of what kinds of 
file objects are meant to be supported, and what things may work by accident 
and be used in the real world.

For instance, is it possible to manually set Content-Length, and say supply a 
GzipFile reader, or file object positioned halfway through the file? How does 
this interact with the socket.sendfile() call?

--
dependencies: +http.client request and send method have some datatype issues
nosy: +martin.panter
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Giampaolo Rodola']

Giampaolo Rodola' added the comment:

Patch in attachment uses the newly added socket.sendfile() method (issue 17552).

--
keywords: +patch
type: enhancement -> performance
Added file: http://bugs.python.org/file35569/httplib-sendfile.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Giampaolo Rodola']

Changes by Giampaolo Rodola' :


--
dependencies: +Add a new socket.sendfile() method
versions: +Python 3.5 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Kasun Herath]

Changes by Kasun Herath :


--
nosy: +kasun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Éric Araujo]

Changes by Éric Araujo :


--
nosy: +eric.araujo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Ross Lagerwall]

Changes by Ross Lagerwall :


--
nosy: +rosslagerwall

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

Ops! I meant issue13564.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

ftplib's sendfile support is not tracked as issue13559.
Considerations I made there should apply here as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Benjamin Peterson]

Benjamin Peterson  added the comment:

2011/12/8 Giampaolo Rodola' :
>
> Giampaolo Rodola'  added the comment:
>
> This is not possible for two reasons:
>
> - on most POSIX systems, sendfile() works with mmap-like ("regular") files 
> only, while HTTPConnection.send() accepts any file-like object as long as it 
> provides a read() method
>
> - after read()ing a chunk of data from the file and before send()ing it over 
> the socket, the data can be subject to an intermediate conversion 
> (datablock.encode("iso-8859-1")):
> http://hg.python.org/cpython/file/87c6be1e393a/Lib/http/client.py#l839
> ...whereas sendfile() can only be used to send a binary file "as-is"

I presume you could check for a binary mode, though? Also, you can
catch EINVAl on invalid fds.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

This is not possible for two reasons:

- on most POSIX systems, sendfile() works with mmap-like ("regular") files 
only, while HTTPConnection.send() accepts any file-like object as long as it 
provides a read() method

- after read()ing a chunk of data from the file and before send()ing it over 
the socket, the data can be subject to an intermediate conversion 
(datablock.encode("iso-8859-1")):
http://hg.python.org/cpython/file/87c6be1e393a/Lib/http/client.py#l839
...whereas sendfile() can only be used to send a binary file "as-is"

I think we can use sendfile() in ftplib.py though .
I'll open a ticket for that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +giampaolo.rodola, orsenthil

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13559] Use sendfile where possible in httplib

[Benjamin Peterson]

New submission from Benjamin Peterson :

HTTPConnection.send() should use os.sendfile when possible to avoid copying 
data into userspace and back.

--
components: Library (Lib)
keywords: easy
messages: 149052
nosy: benjamin.peterson
priority: normal
severity: normal
stage: needs patch
status: open
title: Use sendfile where possible in httplib
type: feature request
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com