[issue13234] os.listdir breaks with literal paths
Roundup Robot added the comment: New changeset 12aaa2943791 by Tim Golden in branch 'default': Issue13234 Allow listdir to handle extended paths on Windows (Patch by Santoso Wijaya) http://hg.python.org/cpython/rev/12aaa2943791 New changeset 5c187d6162c5 by Tim Golden in branch 'default': Issue13234 Credit Santoso for the patch and add NEWS item http://hg.python.org/cpython/rev/5c187d6162c5 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Tim Golden added the comment: Applied. Thanks for the patch. -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Tim Golden added the comment: Santoso Wijaya: sorry for the delay. If you'd like to retarget your patch against the tip, I'm happy to apply. At this stage, 3.3 and 3.4 seem the appropriate branches. -- assignee: - tim.golden versions: -Python 2.7, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Santoso Wijaya added the comment: Here you go. -- Added file: http://bugs.python.org/file32334/issue13234_tip_refresh.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Santoso Wijaya added the comment: Done. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Serhiy Storchaka added the comment: I added minor comments in Rietveld. Santoso Wijaya, can you please submit a contributor form? http://python.org/psf/contrib/contrib-form/ http://python.org/psf/contrib/ -- nosy: +serhiy.storchaka versions: +Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Manuel de la Pena man...@canonical.com added the comment: Indeed, in our code we had to write a number of wrappers around the os calls to be able to work with long path on Windows. At the moment working with long paths on windows and python is broken in a number of places and is a PITA to work with. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Antoine Pitrou pit...@free.fr added the comment: Thanks for the patch. Is there a reason you don't use shutil.rmtree in tearDown()? I don't know if it's our business to convert forward slashes passed by the user. Also, general support for extended paths is another can of worms ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Manuel de la Pena man...@canonical.com added the comment: In case of my patch (I don't know about santa4nt case) I did not use shutil.remove because it was not used in the other tests and I wanted to be consistent and not add a new import. Certainly if there is not an issue with that we should use it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Santoso Wijaya santoso.wij...@gmail.com added the comment: Even if we decide not to convert any forward slash, listdir() adds u\\*.* when the input is unicode, but it adds /*.* when it is not, before passing it off to Windows API. Hence the inconsistency and the problem Manuel saw. IMO, his patch shouldn't have differentiated if the path starts with r\\?\ and just be consistent with adding \\*.*, unicode or not. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Martin v. Löwis mar...@v.loewis.de added the comment: This issue is getting messy. I declare that this issue is *only* about the original problem reported in msg146031. When that is fixed, this issue will be closed, and any further issues need to be reported separately. As for the original problem, ISTM that the right fix is to replace namebuf[len++] = '/'; with namebuf[len++] = '\\'; No further change should be necessary. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Santoso Wijaya santoso.wij...@gmail.com added the comment: Addressing patch comments. -- Added file: http://bugs.python.org/file23519/issue13234_py33_v3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Santoso Wijaya santoso.wij...@gmail.com added the comment: Fair enough. Simplifying. -- Added file: http://bugs.python.org/file23520/issue13234_py33_v4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
STINNER Victor victor.stin...@haypocalc.com added the comment: -if (ch != SEP ch != ALTSEP ch != ':') +if (ch != '\\' ch != '/' ch != ':') I don't understand this change in issue13234_py33_v4.patch (the change looks to be useless). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Santoso Wijaya santoso.wij...@gmail.com added the comment: I don't understand this change in issue13234_py33_v4.patch (the change looks to be useless). It's pedantic correctness on my part. SEP and ALTSEP are defined as wide strings L'\\' and L'/' respectively. Their usage in the unicode conditional branch and the bytes conditional branch seem to have been reversed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Santoso Wijaya santoso.wij...@gmail.com added the comment: I'd also like to point out that Unicode path is handled correctly in both 2.7.x and 3.x: Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win 32 Type help, copyright, credits or license for more information. import os os.listdir(u'?\\D:\\Temp\\tempdir') [u'sub1', u'test1.txt'] os.listdir('?\\D:\\Temp\\tempdir') Traceback (most recent call last): File stdin, line 1, in module WindowsError: [Error 123] The filename, directory name, or volume label syntax i s incorrect: '?\\D:\\Temp\\tempdir/*.*' Python 3.2 (r32:88445, Feb 20 2011, 21:30:00) [MSC v.1500 64 bit (AMD64)] on win 32 Type help, copyright, credits or license for more information. import os os.listdir('?\\D:\\Temp\\tempdir') ['sub1', 'test1.txt'] os.listdir(b'?\\D:\\Temp\\tempdir') Traceback (most recent call last): File stdin, line 1, in module WindowsError: [Error 123] The filename, directory name, or volume label syntax i s incorrect: '?\\D:\\Temp\\tempdir/*.*' The problem only lies in the code handling narrow string paths. If you look at (py33) posixmodule.c:2545-6, the bare path is appended with L\\*.*. Whereas in the narrow string case, posixmodule.c:2625-6, the bare path is appended with /*.*, instead. To be consistent, we should use '\\'. -- Added file: http://bugs.python.org/file23512/issue13234_py33.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Santoso Wijaya santoso.wij...@gmail.com added the comment: There are also several other edge cases to be taken care of: Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win 32 Type help, copyright, credits or license for more information. import os os.listdir(r'\\?\C:\Python27/') Traceback (most recent call last): File stdin, line 1, in module WindowsError: [Error 123] The filename, directory name, or volume label syntax i s incorrect: '?\\C:\\Python27/*.*' os.listdir(r'\\?\C:/Python27\Lib') Traceback (most recent call last): File stdin, line 1, in module WindowsError: [Error 3] The system cannot find the path specified: '?\\C:/Py thon27\\Lib/*.*' -- Added file: http://bugs.python.org/file23513/issue13234_py33_v2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Santoso Wijaya santoso.wij...@gmail.com added the comment: Additionally, there might be issues in other APIs when handling with extended path lengths: D:\Temp\tempdirdir Volume in drive D is Data Volume Serial Number is 7E3D-EC81 Directory of D:\Temp\tempdir 10/24/2011 04:22 PMDIR . 10/24/2011 04:22 PMDIR .. 10/24/2011 04:28 PMDIR A AA 10/24/2011 01:31 PMDIR sub1 10/24/2011 03:39 PM 262 test1.txt 1 File(s)262 bytes 4 Dir(s) 244,483,321,856 bytes free D:\Temp\tempdircd A AA D:\Temp\tempdir\ AAAdir Volume in drive D is Data Volume Serial Number is 7E3D-EC81 Directory of D:\Temp\tempdir\AA A 10/24/2011 04:28 PMDIR . 10/24/2011 04:28 PMDIR .. 10/24/2011 04:14 PM 0 1234567.txt 10/24/2011 04:28 PMDIR B 1 File(s) 0 bytes 3 Dir(s) 244,483,321,856 bytes free Python 3.2 (r32:88445, Feb 20 2011, 21:30:00) [MSC v.1500 64 bit (AMD64)] on win 32 Type help, copyright, credits or license for more information. import os subdir = 'B'*13 os.path.isdir(subdir) False os.getcwd() 'D:\\Temp\\tempdir\\ AAA' subdir_abs = os.path.join(os.getcwd(), subdir) os.path.isdir(subdir) False subdir_ext = r'\\?\%s' % subdir_abs os.path.isdir(subdir_ext) True In the above example, perhaps a ValueError('path too long') is better than returning False? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Changes by Terry J. Reedy tjre...@udel.edu: -- nosy: +loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Changes by Santoso Wijaya santoso.wij...@gmail.com: -- nosy: +santa4nt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Changes by Santoso Wijaya santoso.wij...@gmail.com: -- components: +Windows type: - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
New submission from Manuel de la Pena man...@canonical.com: During the development of an application that needed to write paths longer than 260 chars we opted to use \\?\ as per http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath. When working with literal paths the following the os.listdir funtion would return the following trace: import os test = r'\\?\C:\Python27' os.listdir(test) Traceback (most recent call last): File stdin, line 1, in module WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: '?\\C:\\Python27/*.*' The reason for this is that the implementation of listdir appends '/' at the end of the path if os.path.sep is not present at the end of it which FindFirstFile does not like. This is a inconsistency from the OS but it can be easily fixed (see attached patch). -- components: Library (Lib) files: listdir.patch keywords: patch messages: 146031 nosy: mandel priority: normal severity: normal status: open title: os.listdir breaks with literal paths versions: Python 2.7 Added file: http://bugs.python.org/file23482/listdir.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +brian.curtin, pitrou, tim.golden stage: - patch review versions: +Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13234] os.listdir breaks with literal paths
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com