On Sun, Mar 22, 2020 at 8:30 PM Frank Millman <fr...@chagford.com> wrote: > > On 2020-03-22 10:45 AM, Chris Angelico wrote: > > On Sun, Mar 22, 2020 at 6:58 PM Frank Millman <fr...@chagford.com> wrote: > >>> I'd look at the network traffic with wireshark to see if there is > >>> anything different between edge and the other browsers. > >>> > >> > >> You are leading me into deep waters here :-) I have never used > >> Wireshark before. I have now downloaded it and am running it - it > >> generates a *lot* of data, most of which I do not understand yet! > >> > >> One thing immediately stands out. When I run it with MS Edge and > >> Python3.8, it shows a lot of lines highlighted in red, with the symbols > >> [RST,ACK]. They do not appear when running Chrome, and they do not > >> appear when running Python3.7. > > > > Interesting. RST means "Reset" and is sent when the connection is > > closed. Which direction were these packets sent (Edge to Python or > > Python to Edge)? You can tell by the source and destination ports - > > one of them is going to be the port Python is listening on (eg 80 or > > 443), so if the destination port is 80, it's being sent *to* Python, > > and if the source port is 80, it's being sent *from* Python. > > > > They are all being sent *from* Python *to* Edge.
Very interesting indeed. What that *might* mean is that Python is misinterpreting something and then believing that the connection has been closed, so it responds "Okay, I'm closing the connection". Or possibly it sees some sort of error condition. > >> I have another data point. I tried putting an asyncio.sleep() after > >> sending each file. A value of 0.01 made no difference, but a value of > >> 0.1 makes the problem go away. > > > > Interesting also. > > > > Can you recreate the problem without Edge? It sounds like something's > > going on with concurrent transfers, so it'd be awesome if you can > > replace Edge with another Python program, and then post both programs. > > > > Do you mean write a program that emulates a browser - make a connection, > receive the HTML page, send a GET request for each file, and receive the > results? > > I will give it a go! Yes - although the HTML page is most likely irrelevant. You could just make a connection and then spam requests. Or make multiple connections. Actually, that's another thing worth checking. Is Edge using a single connection for all requests, or separate connections for each request, or something in between (eg a pool of four connections and spreading requests between them)? You'll be able to recognize different connections by the port numbers Edge uses, which are guaranteed unique among concurrent connections, and are most likely to not be reused for a while even if the other is closed. > > Also of interest: Does the problem go away if you change "Connection: > > Keep-Alive" to "Connection: Close" in your headers? > > > > Yes, the problem does go away. > This makes me think that the answer to the previous question is going to involve some connection reuse. If you can recreate the problem with a single socket and multiple requests, that would be extremely helpful. I also think it's highly likely that this is the case. My theory: Your program is sending a large amount of data to the lower level API functions, which attempt to send a large amount of data to the socket. At some point, something gets told "sorry, can't handle all that data, hold some of it back" at a time when it's not prepared to do so, and it misinterprets it as an error. This error results in the connection being closed. But that's just a theory. ChrisA -- https://mail.python.org/mailman/listinfo/python-list