[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 26da299ca88e by Ronald Oussoren in branch '3.1':
Fix for issue 10684: Folders get deleted when trying to change case with 
shutil.move (case insensitive file systems only)
http://hg.python.org/cpython/rev/26da299ca88e

New changeset 051190230254 by Ronald Oussoren in branch '2.7':
Backport fix for issue #10684 from 3.x
http://hg.python.org/cpython/rev/051190230254

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I've committed the fix in 2.7, 3.1, 3.2 and 3.3

--
assignee:  -> ronaldoussoren
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Nadeem Vawda]

Nadeem Vawda  added the comment:

test_move_dir_caseinsensitive is failing on some of the XP buildbots:

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/4514
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/2696
http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.2/builds/239

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

That's rather annoying. I'm reopening the issue because of this.

I'm looking into the issue and will revert when I cannot find a solution soon.

--
resolution: fixed -> accepted
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

This seems to be a bug in ntpath.samefile, in particular in this code:


# determine if two files are in fact the same file
try:
# GetFinalPathNameByHandle is available starting with Windows 6.0.
# Windows XP and non-Windows OS'es will mock _getfinalpathname.
if sys.getwindowsversion()[:2] >= (6, 0):
from nt import _getfinalpathname
else:
raise ImportError
except (AttributeError, ImportError):
# On Windows XP and earlier, two files are the same if their absolute
# pathnames are the same.
# Non-Windows operating systems fake this method with an XP
# approximation.
def _getfinalpathname(f):
return abspath(f)

def samefile(f1, f2):
"Test whether two pathnames reference the same actual file"
return _getfinalpathname(f1) == _getfinalpathname(f2)

Python2 doesn't have ntpath.samefile and shutil then falls back to comparing 
"os.path.normcase(os.path.abspath(src))" with the same transformation of dst.

On XP _getfinalpath doesn't call os.path.normcase, hence it doesn't notice that 
"a" and "A" refer to the same file (on all common NT filesystems)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I'm not sure what the right course of action is, revert my patch try to get 
ntpath.samefile fixed and then reapply my patch or something else?

ntpath.samefile is definitely broken on XP:

* Create a file "a.txt"
* Start python3.2
  >>> os.path.samefile("a.txt", "A.TXT")
  False

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Brian Curtin]

Brian Curtin  added the comment:

On XP, os.path.samefile is really "os.path.abspath(x) == os.path.abspath(y)", 
which does not work correctly with different cases. We could add a ".lower()" 
to line 657 of Lib/ntpath.py so the abspath is always returned in lower, so the 
XP version of samefile compares two lower case strings.

On versions after XP, this isn't an issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

The attached patch seems to fix ntpath.samefile (at least for the shutils 
tests, I haven't run a full testsuite and cannot build python on a windows 
machine anyway)

--
Added file: http://bugs.python.org/file21908/npath-fix.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Brian Curtin]

Brian Curtin  added the comment:

I don't have time to test it at the moment, but it seems fine to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I'm currrently running 'python -mtest.regrtest -uall' on an XP machine where 
I've applied my test to a binary install of 3.2.

I'll apply my patch if that testrun indicates that I don't introduce other 
problems, and I'll revert my shutil patch when the change to ntpath.samefile 
isn't fine after call.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 011e4bb8b933 by Ronald Oussoren in branch '3.2':
ntpath.samefile fails to detect that "A.TXT" and "a.txt" refer to the same file 
on Windows XP.
http://hg.python.org/cpython/rev/011e4bb8b933

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I've committed my fix for ntpath.samefile for 3.3 and 3.2.

I've also filed a new issue because ntpath.samefile has no unittests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Changes by Ronald Oussoren :


--
resolution: accepted -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Heikki Toivonen]

Heikki Toivonen  added the comment:

I also noticed this last week. However, this is not Windows specific. It 
happens with file systems that are not case sensitive. Besides Windows (NTFS, 
FAT*) the other common platform is Macintosh (HFS+ with default settings).

What happens is that we copy source into itself, then delete source.

I am not sure what the optimal solution would be but an easy one would be to 
first try os.rename (which works in this case), but if that fails then do the 
stuff that is currently shutil.move.

--
nosy: +heikki
title: Folders get deleted when trying to change case with shutil.move 
(Windows) -> Folders get deleted when trying to change case with shutil.move 
(case insensitive file systems only)
versions: +Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Heikki Toivonen]

Changes by Heikki Toivonen :


--
assignee:  -> ronaldoussoren
components: +Macintosh
nosy: +ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Brian Curtin]

Changes by Brian Curtin :


--
assignee: ronaldoussoren -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Senthil Kumaran]

Senthil Kumaran  added the comment:

Here is a patch (against release27-maint) for to fix this issue. BTW,what is 
the best way to check for case insensitive file-system? The test here merely 
checks if sys.platform returns mac, darwin or win32.

--
keywords: +patch
nosy: +orsenthil
Added file: http://bugs.python.org/file20511/Issue10684-py27.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Senthil Kumaran]

Senthil Kumaran  added the comment:

I would also add 'cygwin' to the list. I am not sure about the behavior of 
OpenVMS or other less prevalent file systems.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Heikki Toivonen]

Heikki Toivonen  added the comment:

You can't solve this by trying to do different things on different operating 
systems. This bug depends on file system properties, not OS.

Also I don't think you can just lower case the path and do a comparison, 
because there are funky characters that don't round trip lower->upper->lower. 
And you certainly can't do this for just the last component of the path name - 
any component of the path could have changed case.

I still think the best avenue would be to first try straight os.rename, and if 
that fails (maybe only if target exists), the logic that is currently in 
shutil.move.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Nadeem Vawda]

Nadeem Vawda  added the comment:

> BTW,what is the best way to check for case insensitive file-system?
> The test here merely checks if sys.platform returns mac, darwin or win32.
I would suggest not checking at all. If the system is case-sensitive, the test 
will pass, so it doesn't really make a difference. You could write a small 
function that creates a dummy file and then tries to access it via a case 
variant of its name, but that seems unnecessary.

> You can't solve this by trying to do different things on different
> operating systems. This bug depends on file system properties, not OS.
It's worth pointing out that it depends on both the FS *and* OS. For example, 
an NTFS filesystem is case-insensitive under Windows, but case-sensitive under 
Linux. This has caused me headaches in the past.

> I still think the best avenue would be to first try straight os.rename,
> and if that fails (maybe only if target exists), the logic
> that is currently in shutil.move.
I agree. If os.rename() succeeds, there is no need to copy the file and then 
delete the original. If it fails because the two paths are on different 
devices, the existing code can safely be used without any further checks. I'm 
not sure if there are any other failure cases that would need to be handled, 
though.

--
nosy: +nvawda

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[nooB]

nooB  added the comment:

Few points that could be useful,

1) I added print statements in `shutil.move` and found that the `real_dst` was 
wrong for the case coz os.path.isdir returns true.

>> import os
>> os.mkdir('test')
>> os.path.isdir('TEst')
True

In shutil.move, when we do shutil.move('test', 'TEst'), os.rename is actually 
applied to 'test' and 'TEst\test'. Thats why os.rename failed for the case in 
shutil.move.

2) os.rename has its own problems.

>> import os
>> os.mkdir('test')
>> os.rename('TEst', 'teST')

os.rename succeeded when the source 'TEst' did not exist.
Is this behaviour correct?.
This applies to shutil.move also.

>> import os,shutil
>> os.mkdir('test')
>> shutil.move('TEst', 'teST')

The folder 'test' gets deleted when trying to move 'TEst' to 'teST'. The case 
check should be done to the source argument also.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

The fallback to copy+remove happens because shutil.move first checks if the 
destination exists and is a directory, if so it moves the source into the 
destination, that is, given:

os.mkdir('foo')
os.mkdir('bar')

Then ``shutil.move('foo', 'bar')`` is the same as ``shutil.move('foo', 
'bar/foo')``.

On filesystems that are case insensitive this triggers for ``shutil.move('foo', 
'FOO')`` as wel, causing a call to ``os.rename('foo', 'FOO/foo')`` and that 
fails because you cannot move a folder inside itself.

The attached patch makes the test unconditional (as it should pass always when 
the filesystem is case sensitive) and checks if src and dst are the same when 
dst is a directory, in that case os.rename is called and we never try to copy.

--
Added file: http://bugs.python.org/file21129/issue10684-py33.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Changes by Ronald Oussoren :


--
keywords: +needs review
stage: needs patch -> patch review
versions: +Python 3.3 -Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

Could someone with access to a windows box test the patch there?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Santoso Wijaya]

Changes by Santoso Wijaya :


--
nosy: +santa4nt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10684] Folders get deleted when trying to change case with shutil.move (case insensitive file systems only)

[Tim Golden]

Tim Golden  added the comment:

Patch fixes the issue and tests run ok on 3.3 Win7; just building a 2.7 branch 
to test

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com