Re: httplib continuation packets
[Steve Holden] > It really does seem quite bizarre that a server should respond > differently to the same TCP request when it is split differently into IP > datagrams. > > There really is nothing wrong (from a standards point of view) with > sending FIN with your last data segment. FIN means "I guarantee to send > no more data, and will continue to acknowledge your segments until I see > your FIN". It is the server that sends the FIN. What happens is this (each line corresponds to one packet): client: POST request headers client: POST request body server: FIN + ACK On receiving the FIN + ACK packet, Python gets immediate end-of-file on the POST request. Unless the order of the parameters in the POST request matters (I haven't yet tested this), I have no other explanation than the fragmentation. If Ipswitch bothers to reply to my bug report, I'll look into it. Otherwise, I'm not wasting any more time on this -- it's not that big a deal for me personally, since I have already scripted the stuff I needed with Perl. > Are you planning to report this bug to Ipswitch? It certainly sounds > like someone should. I quickly browsed through ipswitch.com, but couldn't find any good place to submit bugs. I ended up using the product feedback web form. Wrote a one-line summary, and referred to this thread on Google Groups. -- Haakon -- http://mail.python.org/mailman/listinfo/python-list
Re: httplib continuation packets
Haakon Riiser wrote: > [Fredrik Lundh] > >> Haakon Riiser wrote: >> >>> Yes, and it doesn't help. >> then the server is mostly likely broken beyond repair. > > It's not in my power to upgrade the server, unfortunately. > Guess I'll have to use Perl. > >> to see if this really is the problem, you could try moving the call to >> self._send_output() from the end of the endheaders() method to the end >> of the _send_request() method (around line 870 in httplib.py, at least >> in 2.5). > > Tried this, but the tcpdump still looks the same (two packets: one > with the headers, one with the body), and now it fails with > > urllib2.HTTPError: HTTP Error 501: Not Implemented > > Nevertheless, I'm fairly sure that the packet fragmentation is > the culprit. It works perfectly with Perl, even when I make > no effort at all to spoof the browser (no user-agent, referer, > cookies, etc.). > It really does seem quite bizarre that a server should respond differently to the same TCP request when it is split differently into IP datagrams. There really is nothing wrong (from a standards point of view) with sending FIN with your last data segment. FIN means "I guarantee to send no more data, and will continue to acknowledge your segments until I see your FIN". Are you planning to report this bug to Ipswitch? It certainly sounds like someone should. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden -- http://mail.python.org/mailman/listinfo/python-list
Re: httplib continuation packets
[Fredrik Lundh] > Haakon Riiser wrote: > >> Yes, and it doesn't help. > > then the server is mostly likely broken beyond repair. It's not in my power to upgrade the server, unfortunately. Guess I'll have to use Perl. > to see if this really is the problem, you could try moving the call to > self._send_output() from the end of the endheaders() method to the end > of the _send_request() method (around line 870 in httplib.py, at least > in 2.5). Tried this, but the tcpdump still looks the same (two packets: one with the headers, one with the body), and now it fails with urllib2.HTTPError: HTTP Error 501: Not Implemented Nevertheless, I'm fairly sure that the packet fragmentation is the culprit. It works perfectly with Perl, even when I make no effort at all to spoof the browser (no user-agent, referer, cookies, etc.). -- Haakon -- http://mail.python.org/mailman/listinfo/python-list
Re: httplib continuation packets
Haakon Riiser wrote: > Yes, and it doesn't help. then the server is mostly likely broken beyond repair. > By the way, this is the closest thing I've found in the bug tracker: > https://sourceforge.net/tracker/?func=detail&atid=105470&aid=547093&group_id=5470 > The bug was closed in 2002 with this comment: > > "I changed httplib to send requests as a single packet in rev >1.60. The change was made to address a performance problem, >but happens to fix the problem you had with the bogus >server, too." > > Has someone changed it back since then? nope; that change buffers the *header* part of the request to avoid problems with certain TCP/IP mechanisms; see http://svn.python.org/view?rev=27644&view=rev for a discussion. note that there's still no guarantee that the entire header is sent in a single TCP packet. to see if this really is the problem, you could try moving the call to self._send_output() from the end of the endheaders() method to the end of the _send_request() method (around line 870 in httplib.py, at least in 2.5). -- http://mail.python.org/mailman/listinfo/python-list
Re: httplib continuation packets
[Fredrik Lundh] > Haakon Riiser wrote: > >> Is this a bug in httplib or the web server? > > it could be that they're blocking requests from Python's urllib, of > course. have you tried overriding the user-agent string ? Yes, and it doesn't help. By the way, this is the closest thing I've found in the bug tracker: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=547093&group_id=5470 The bug was closed in 2002 with this comment: "I changed httplib to send requests as a single packet in rev 1.60. The change was made to address a performance problem, but happens to fix the problem you had with the bogus server, too." Has someone changed it back since then? -- Haakon -- http://mail.python.org/mailman/listinfo/python-list
Re: httplib continuation packets
Haakon Riiser wrote: > Is this a bug in httplib or the web server? it could be that they're blocking requests from Python's urllib, of course. have you tried overriding the user-agent string ? -- http://mail.python.org/mailman/listinfo/python-list
