[issue27973] urllib.urlretrieve() fails on second ftp transfer

Change by Senthil Kumaran : -- pull_requests: +17245 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/17819 ___ Python tracker ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: New changeset 5bba60290b4ac8c95ac46cfdaba5deee37be1fab by Senthil Kumaran in branch '2.7': bpo-27973 - Use test.support.temp_dir instead of NamedTemporaryFile for the (#17774)

[issue27973] urllib.urlretrieve() fails on second ftp transfer

David Bolen added the comment: Ok, I can confirm that the updated PR 17774 test passes under Windows (and still cleans up after itself). -- ___ Python tracker ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: Sorry, I have updated it now: https://github.com/python/cpython/pull/17774 (I had pushed to a different branch earlier and it didn't reflect in my PR) -- ___ Python tracker

[issue27973] urllib.urlretrieve() fails on second ftp transfer

David Bolen added the comment: I'd be happy to test an updated PR 17774 on a Windows builder, but I don't actually see any change yet. It still appears to hold the earlier NamedTemporaryFile with mode='w' change from a few days ago. -- ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: Thanks for the suggestion, David. I have updated the PR 17774 to use temp_support instead of NamedTemporaryFile. Please review this. -- ___ Python tracker

[issue27973] urllib.urlretrieve() fails on second ftp transfer

David Bolen added the comment: The issue appears to be the temporary flag (O_TEMPORARY) that is used under Windows with delete on close temporary files. That appears to prevent any separate access to the file by anyone else including obtaining another reference in the same process: >>>

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: Hi Pablo, Is there a way for us to test https://github.com/python/cpython/pull/17774 on a Windows Builder which displayed the post-commit failure? The CI custom-builders seem to be broken for a different reason:

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: Thanks for the note, Pablo. I am going to check if this patch https://github.com/python/cpython/pull/17774 will solve the Windows buildbot issues. -- stage: patch review -> resolved ___ Python tracker

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Change by Senthil Kumaran : -- pull_requests: +17207 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/17774 ___ Python tracker ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Pablo Galindo Salgado added the comment: This PR1040 is failing in several 2.7 buildbots: Tests result: FAILURE then FAILURE test test_urllibnet failed -- Traceback (most recent call last): File "D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\test\test_urllibnet.py", line 232,

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Change by Senthil Kumaran : -- resolution: -> fixed stage: -> resolved status: open -> closed ___ Python tracker ___ ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: New changeset f82e59ac4020a64c262a925230a8eb190b652e87 by Senthil Kumaran in branch '2.7': [2.7] bpo-27973 - Fix for urllib.urlretrieve() failing on second ftp transfer (#1040)

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Changes by Senthil Kumaran : -- pull_requests: +1193 ___ Python tracker ___ ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: I spent time digging for a proper fix for this issue. I've come to a conclusion that this commit, https://hg.python.org/cpython/rev/44d02a5d59fb (10 May 2016) in 2.7.12, was a mistake and needs to be reverted. The reason this change was made was apparently,

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Stefan Behnel added the comment: Actually, it seems that calling urlcleanup() is sufficient as a work-around. -- ___ Python tracker ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Stefan Behnel added the comment: This bug makes the installation of lxml fail on many systems (especially MacOS) when pip installs from sources, because it tries to download its library dependencies from FTP and crashes due to the FTP "error". I guess the current fix is to not use urllib for

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: Just an update. We missed the last 2.7.13 release train. Sorry for that. It's worth to fix this bug still ASAP. It's intricately related to some other bugs like (issue25458, issue26960). I got to this and realized the dependency and other bugs. All those

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Changes by Senthil Kumaran : -- assignee: -> orsenthil ___ Python tracker ___ ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Sohaib Ahmad added the comment: @Senthil, thanks for looking into this. Looking forward to your commit. Regards. -- ___ Python tracker ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: @Sohaib, Thanks for the ping. Yeah, I will act on it. Your analysis in msg276795 seems plausible, On the patch itself, I will try to reproduce this and see if I can avoid introducing this clear_buffer function. If no other go, then it should just be a

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Sohaib Ahmad added the comment: Can someone please review this patch so that it would be in 2.7.13 when it comes out? -- ___ Python tracker ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Sohaib Ahmad added the comment: I finally found the actual problem causing the failure of second download. urlretrieve() works with FTP in PASV mode, and in PASV mode after sending the file to client, the FTP server sends an ACK that the file has been transferred. After the fix of

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Changes by Sohaib Ahmad : Removed file: http://bugs.python.org/file44692/urllib.patch ___ Python tracker ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Sohaib Ahmad added the comment: Hi Senthil, Thanks for the review. Now that I look at it, even with a default value, an ftp specific parameter sure does break the open() API abstraction. -- ___ Python tracker

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Senthil Kumaran added the comment: Hi Sohaib, I will get the proper fix for this issue. A comment on the patch. Changing the API to `def open(self, fullurl, data=None, ftp_retrieve=False):` just breaks the abstraction of the open method and may not be the way to go for this. Any changes

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Sohaib Ahmad added the comment: The attached patch fixes the problem with multiple ftp downloads while keeping the fix for issue1067702 intact. The fix basically uses a new parameter ftp_retrieve to change the behavior of ftpwrapper.retrfile() if it is being called by urlretrieve(). I am not

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Sohaib Ahmad added the comment: I manually reverted the issue26960 patch which fixed my issue of consecutive downloads but it also caused regression of issue26960. I am looking into what could be causing this hang when voidresp() is called using the demo available in issue26960 and it looks

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Sohaib Ahmad added the comment: I didn't know that urllib.urlopen() retrieves complete object in case of FTP. When getresp() is called for big files (the one in issue26960), RETR command is initiated and server returns code 150 which means "standby for another reply" and there is where the

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Changes by Berker Peksag : -- nosy: +orsenthil ___ Python tracker ___ ___

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Sohaib Ahmad added the comment: Thank you for pointing me towards hg bisect. I got some time to look into it and was able to find the commit that broke this functionality. A fix from Python 3 was backported in issue "urllib hangs when closing connection" which removed a call to

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Andreas Gustasfson added the comment: I am also suffering from this bug, using Python 2.7.12 on NetBSD, and it is blocking my efforts to do automated testing of NetBSD/Xen. I'm attaching a minimal test case (only 3 lines). -- nosy: +gson Added file:

[issue27973] urllib.urlretrieve() fails on second ftp transfer

Sohaib Ahmad added the comment: I am not much familiar with mercurial. I will try to setup the development environment. Traceback is: [Errno ftp error] 200 Switching to Binary mode. Traceback (most recent call last): File "multiple_ftp_download.py", line 49, in main file2_path =

[issue27973] urllib.urlretrieve() fails on second ftp transfer

R. David Murray added the comment: If you want to help out with tracking this down, you could hg bisect the commits and find out which one broke it. -- nosy: +r.david.murray ___ Python tracker

[issue27973] urllib.urlretrieve() fails on second ftp transfer

New submission from Sohaib Ahmad: urllib.urlretrieve() fails on ftp: - start and complete a transfer - immediately start another transfer The second transfer will fail with the following error: [Errno ftp error] 200 Type set to I I am using urllib.urlretrieve(url, filename) to retrieve two