[issue23740] http.client request and send method have some datatype issues

[Alex Willmer]

Alex Willmer  added the comment:

http_dump.py now covers CPython 3.6-3.10 (via Tox), and HTTPSConnection 
https://github.com/moreati/bpo-23740

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Alex Willmer]

Alex Willmer  added the comment:

First stab at characterising http.client.HTTPConnnection.send().

https://github.com/moreati/bpo-23740

This uses a webserver that returns request details, in the body of the 
response. Raw (TCP level) content is included. It shares a similar purpose to 
HTTP TRACE command. In principal the bytes that HTTPConection.send() writes 
will match to the bytes returned (after they're decapsulated from the JSON). 
I've not tested that aspect yet.

TODO
- further testing (verify round trip, bytes in = bytes out) 
- cover multiple Python versions
- cover cases such client manually setting Content-Length

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Alex Willmer]

Alex Willmer  added the comment:

A data point found while I researched this

MyPy typeshed [1] currently declares

_DataType = Union[bytes, IO[Any], Iterable[bytes], str]
class HTTPConnection:
def send(self, data: _DataType) -> None: ...


[1] 
https://github.com/python/typeshed/blob/e2967a8beee9e079963ea91a67087ba8fded1d0b/stdlib/http/client.pyi#L26

--
nosy: +Alex.Willmer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Mariatta Wijaya]

Changes by Mariatta Wijaya :


--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Martin Panter]

Martin Panter added the comment:

I’ve decided I would prefer deprecating the Latin-1 encoding, rather than 
adding more encoding support for iterables and text files. If not deprecating 
it altogether, at least prefer just ASCII encoding (like Python 2). The 
urlopen() function already rejects text str, and in Issue 26045 people often 
want or expect UTF-8.

Here is a summary I made of the different data types handled for the body 
object, from highest priority to lowest priority.

 HTTPConnection   HTTPConn. default
body _send_output()   Content-Length urlopen() 
===  ===  =  ==
None No body  0 or unset #23539  C-L unset 
Has read()   read() #1065257 * #5038   
- Text file  encode() #5314
str  encode() r56128  len()  TypeError #11082  
Byte seq.sendall()len()  C-L: nbytes   
Bytes-like   * #27340 *  C-L: nbytes #3243 
Iterable iter() #3243C-L required #3243
Sized len()† #23350
fstat()   st_size #1065257 
OtherwiseTypeErrorUnset #1065257 C-L optional† 

* Possible bugs
† Degenerate cases not worth supporting IMO
C-L = Content-Length header field, set by default or required to be specified 
in various cases
Byte seq. = Sequence of bytes, including “bytes” type, bytearray, array("B"), 
etc
Bytes-like = Any C-contiguous buffer, including zero- and multi-dimensional 
arrays, and with items other than bytes
Iterable = Iterable of bytes or bytes-like objects (depending on Issue 27340 
about SSL)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Martin Panter]

Changes by Martin Panter :


--
dependencies: +bytes-like objects with socket.sendall(), SSL, and http.client

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Akshit Khurana]

Changes by Akshit Khurana axitkhur...@gmail.com:


--
nosy: +axitkhurana

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23740
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Martin Panter]

Martin Panter added the comment:

The priority of file-like objects versus bytes-like and iterable objects should 
be well defined. See Issue 5038: mmap objects seem to satisfy all three 
interfaces, but the result may be different depending on the file position.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23740
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Note that for file-like objects we have also the same issue as issue22468. 
Content-Length is not determined correctly for GzipFile and like.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23740
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Demian Brecht]

Demian Brecht added the comment:

FWIW, I've done some additional work to request/send in issue #12319 where I've 
added support for chunked request encoding.

@David
 if an iterable is passed in, it must be an iterable of bytes-like objects

This specific issue is addressed in the patch in #23350.

@Martin
 text files could also be handled more consistently by checking the read() 
 return type

I wouldn't be opposed to this at all. In fact, I was going to initially make 
that change in #12319, but wanted to keep the change surface minimal and 
realistically, peeking at the data type rather than checking for 'b' in mode 
doesn't /really/ make that much of a difference.

--
nosy: +demian.brecht

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23740
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[R. David Murray]

New submission from R. David Murray:

While committing the patch for issue 23539 I decided to rewrite the 'request' 
docs for clarity.  I doing so I found that http.client isn't as consistent as 
it could be about how it handles bytes and strings.  Two points specifically:  
it will only take the length of a bytes-like object (to supply a default 
Content-Length header) if isinstance(x, bytes) is true (that is, it doesn't 
take the length of eg array or memoryview objects), and (2) if an iterable is 
passed in, it must be an iterable of bytes-like objects.  Since it already 
automatically encodes string objects and text files, for consistency it should 
probably also encode strings if they are passed in via an iterator.

--
keywords: easy
messages: 238928
nosy: r.david.murray
priority: normal
severity: normal
stage: needs patch
status: open
title: http.client request and send method have some datatype issues
type: enhancement
versions: Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23740
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

See also issue23350 and issue23360.

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23740
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Martin Panter]

Martin Panter added the comment:

As well as encoding iterables of str(), text files could also be handled more 
consistently by checking the read() return type. That would eliminate the 
complication of checking for a b mode.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23740
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[Martin Panter]

Martin Panter added the comment:

Summary of the main supported types as I see them, whether documented, 
undocumented, or only working by accident:

* None
* Bytes-like sequences, e.g. bytes(), bytearray. I believe Content-Length is 
actually automatically set for all these types.
* Arbitrary bytes-like objects, including array.array(I) and ctypes 
structures, if custom Content-Length provided (HTTP over TCP only)
* str() objects, to be automatically encoded with Latin-1
* File reader objects not supporting stat(), e.g. BytesIO
* File with valid stat().st_size, i.e. not named pipes
* Binary file reader
* Text file reader with mode attribute without a b, automatically encoded 
with Latin-1
* Any iterable of bytes-like sequences
* Any iterable of arbitrary bytes-like objects (HTTP over TCP only)

Arbitrary bytes-like objects are not properly supported by SSLSocket.sendall(). 
After all, the non-SSL socket.sendall() documentation does not explicitly 
mention supporting arbitrary bytes-like objects either, though it does seem to 
support them in practice.

Suggested documentation fixes and additions:
* Clarify Content-Length is added for all sequence objects, not just “string or 
bytes objects”
* Warn that a custom Content-Length should be provided with non-bytes 
sequences, and with special files like named pipes, since the len() or stat() 
call will give the wrong value
* end_headers() should mention byte string rather than just “string”, since it 
does not (yet) accept Latin-1 text strings
* end_headers() should not mention sending in the same packet either, since 
separate sendall() calls are made for all cases; see Issue 23302
* send() should clarify what it accepts
* Either omit mentioning arbitrary bytes-like objects, or add support to 
SSLSocket.sendall() and document support by non-SSL socket.sendall()

I would support encoding iterables of str() objects for consistency. The patch 
for Issue 23350 already does this, although I am starting to question the 
wisdom of special-casing lists and tuples in that patch.

--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23740
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23740] http.client request and send method have some datatype issues

[R. David Murray]

R. David Murray added the comment:

Yeah, if we're going to check the type for iterables to convert strings, we 
might as well check the type for read() as well.

The bit about the len not being set except for str and bytes was me 
mis-remembering what I read in the code.  (The isinstance check is about 
whether _send_output sends the body in the same packet.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23740
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com