Bug#801506: 413 Client Error: Request Entity Too Large on SSL site where curl works
On Sun, May 15, 2016 at 06:54:06PM +0200, Daniele Tricoli wrote: > This is my actual plan: > 1. Update urllib3 and requests (the first package is ready, I'm updating >requests right now) > 2. see if the problem is still present; > 3. forward the bug upstream; upstream is very responsive so we will have >news soon. We worked out more details on #debian-admin and I've already posted the results on the upstream repository. I'll write it also here: https://bz.apache.org/bugzilla/show_bug.cgi?id=39243 is relevant, and they have a rationale and a work-around: But you should really design your site to ensure that the first request to a client-cert-protected area is not a POST request with a large body; make it a GET or something. Any request body has to be buffered into RAM to handle this case, so represents an opportunity to DoS the server. I can change python-debiancontributors to do a GET before a post, the GET gets to negotiate SSL correctly and smoothly, and the POST afterwards should go through. Enrico -- GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini signature.asc Description: PGP signature
Bug#801506: 413 Client Error: Request Entity Too Large on SSL site where curl works
Hello Enrico, On Sunday, May 15, 2016 03:44:39 PM Enrico Zini wrote: > is there any news on this? Sorry for the long delay. I started working on it but I got distracted by real life so I did not finished. > I'd like at some point to be able to have again a toolchain that works > out of the box without workarounds. I see, and it's important to sort this out. This is my actual plan: 1. Update urllib3 and requests (the first package is ready, I'm updating requests right now) 2. see if the problem is still present; 3. forward the bug upstream; upstream is very responsive so we will have news soon. Thanks and sorry again for the delay on handling this. Kind regards, -- Daniele Tricoli 'eriol' https://mornie.org signature.asc Description: This is a digitally signed message part.
Bug#801506: 413 Client Error: Request Entity Too Large on SSL site where curl works
On Thu, Nov 05, 2015 at 10:48:30AM +0100, Enrico Zini wrote: > > It seems that the attachment was lost, please can you send it again? > > I tried with an empty data file but it's not enough to replicate this bug. > Whoops, sorry, I missed both the attachment and your reply :/ > Here it is. Hello, is there any news on this? Most scripts that submit data to contributors.debian.org across the Debian infrastructure have over time hit this bug and had to migrate to a two step approach of saving the JSON data locally and them submitting it with curl. I'd like at some point to be able to have again a toolchain that works out of the box without workarounds. Enrico -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini signature.asc Description: PGP signature
Bug#801506: 413 Client Error: Request Entity Too Large on SSL site where curl works
Hello Enrico,
many thanks for your report!
On Sunday, October 11, 2015 02:24:18 PM Enrico Zini wrote:
> When I run the attached script and test file, using python2 and python3,
> I get:
[CUT traceback and curl example]
It seems that the attachment was lost, please can you send it again?
I tried with an empty data file but it's not enough to replicate this bug.
This is my test script:
-8<
import requests
url = 'https://contributors.debian.org/contributors/test_post'
files = {'data': open('test.json.xz', 'rb')}
payload = {'source': 'bugs.debian.org', 'data_compression': 'xz'}
if __name__ == '__main__':
r = requests.post(url, files=files, data=payload)
print(r.text)
-8<
> I suspect that python-requests is getting confused with ssl
> renegotiation that was introduced when we implemented client certificate
> authentication[1], although it's hard for me to track that down.
Many thanks for the hint, I will also ask to upstream developers: they are
very supportive so I hope we will have a solution soon.
> The best I can do is to provide a test case for reproducing the behaviour,
> that does not require any debian credentials to be run[2].
This is really appreciated, thanks! I have already used it to wrote my test
script. And I will use again for testing.
I have quite completed the packaging requests 2.8.1-1, but I'm waiting for the
upload of python-urllib3 1.12-1: I'm sorry for the delay, but although I'm a
DM since September, my usual sponsor had problems giving me upload permissions
using dcut.
I will try your test case both with requests 2.7.0-3 and 2.8.1-1.
Kind regards,
--
Daniele Tricoli 'eriol'
https://mornie.org
signature.asc
Description: This is a digitally signed message part.
Bug#801506: 413 Client Error: Request Entity Too Large on SSL site where curl works
Package: python-requests
Version: 2.7.0-3
Severity: normal
Hello,
thank you for maintaining python requests.
When I run the attached script and test file, using python2 and python3,
I get:
$ ./test_post Traceback (most recent call last):
File "./test_post", line 33, in
res.raise_for_status()
File "/usr/lib/python3/dist-packages/requests/models.py", line 851, in
raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 413 Client Error: Request Entity Too Large
...but if I post it with curl, it works:
$ curl https://contributors.debian.org/contributors/test_post -F
source=bugs.debian.org -F data=@test.json.xz -F data_compression=xz -k
{
"records_parsed": 3,
"contributions_created": 0,
"code": 200,
"contributions_updated": 0,
"errors": [],
"identifiers_skipped": 0,
"contributions_processed": 0,
"records_skipped": 0,
"contributions_skipped": 0
}
I suspect that python-requests is getting confused with ssl
renegotiation that was introduced when we implemented client certificate
authentication[1], although it's hard for me to track that down. The
best I can do is to provide a test case for reproducing the behaviour,
that does not require any debian credentials to be run[2].
Thank you,
Enrico
[1] https://wiki.debian.org/DebianSingleSignOn#Debian_SSO_documentation
[2] I have just implemented a test submission url on
contributors.debian.org and a random submission generator, so that I
could post this bug report: it was yak shaving morning
-- System Information:
Debian Release: stretch/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.2.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages python-requests depends on:
ii ca-certificates 20150426
ii python-chardet 2.3.0-1
ii python-urllib3 1.11-2
pn python:any
python-requests recommends no packages.
Versions of packages python-requests suggests:
ii python-ndg-httpsclient 0.4.0-1
ii python-openssl 0.15.1-2
ii python-pyasn1 0.1.8-2
-- no debconf information
