[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: os.replace() committed in 3.3! -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___

[issue8828] Atomic function to rename a file

Roundup Robot added the comment: New changeset 80ddbd87 by Antoine Pitrou in branch 'default': Issue #8828: Add new function os.replace(), for cross-platform renaming with overwriting. http://hg.python.org/cpython/rev/80ddbd87 -- nosy: +python-dev _

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: Here is a patch adding os.replace(). -- keywords: +patch stage: needs patch -> patch review Added file: http://bugs.python.org/file24328/osreplace.patch ___ Python tracker

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: > As a Windows programmer I am quite surprised to read this thread with > information that on Linux os.rename() overwrites files without questions, > so as I Windows programmer I want os.rename() to stop that. As a Windows programmer, you are not really qualifi

[issue8828] Atomic function to rename a file

R. David Murray added the comment: So maybe my warning idea isn't such a bad idea :) As a unix programmer, I was very surprised to read in this thread that Windows doesn't overwrite the file on rename. As a unix programmer, I don't check for errors on a rename, because I expect it to just wo

[issue8828] Atomic function to rename a file

anatoly techtonik added the comment: On Fri, Dec 23, 2011 at 10:35 PM, Antoine Pitrou wrote: > > os.rename(overwrite=False) by default will do less harm than the > opposite, > > Disagreed. > Fine. No arguments == no consensus. > > Then I believe that having a small chance of overwriting file

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: > os.rename(overwrite=False) by default will do less harm than the opposite, Disagreed. > Then I believe that having a small chance of overwriting file just created > at exactly the same moment on a POSIX is a small price to pay for function > that does safety

[issue8828] Atomic function to rename a file

anatoly techtonik added the comment: 2011/12/23 Charles-François Natali > > Charles-François Natali added the comment: > > > I propose quite the opposite. rename() should not overwrite existing > > files by default. > > 1. That's not what I understood from: > > os.rename(overwrite=True) to pr

[issue8828] Atomic function to rename a file

Charles-François Natali added the comment: > I propose quite the opposite. rename() should not overwrite existing > files by default. 1. That's not what I understood from: > os.rename(overwrite=True) to produce consistent cross-platform > behavior. 2. The above argument on backward incompatib

[issue8828] Atomic function to rename a file

anatoly techtonik added the comment: 2011/12/23 Charles-François Natali > > > One of the Python advantages is providing predictable cross-platform > > behavior. If we can't introduce nice API without BC break, it is not > > a reason to introduce ulgy API. > > We cannot make rename() overwrite e

[issue8828] Atomic function to rename a file

R. David Murray added the comment: Ah, you are right about the race of course. So yes, I agree with your proposal. It's a weird API, but probably the best we can do. -- ___ Python tracker ___

[issue8828] Atomic function to rename a file

Charles-François Natali added the comment: > How about overwrite=[None, True] with None meaning "OS default"? +1. > One of the Python advantages is providing predictable cross-platform > behavior. If we can't introduce nice API without BC break, it is not > a reason to introduce ulgy API. We

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: > I'm good with None/True, but that would imply that for posix rename > we'll need to implement the overwrite=False option...which would be a > nice thing (the shell mv command has -i for that). My point was rather to forbid False as a value (on all OSes) :) >

[issue8828] Atomic function to rename a file

R. David Murray added the comment: I'm good with None/True, but that would imply that for posix rename we'll need to implement the overwrite=False option...which would be a nice thing (the shell mv command has -i for that). I think a warning would be good, because a unix programmer will assum

[issue8828] Atomic function to rename a file

anatoly techtonik added the comment: One of the Python advantages is providing predictable cross-platform behavior. If we can't introduce nice API without BC break, it is not a reason to introduce ulgy API. -- ___ Python tracker

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: > An alternative might be a flag on rename: overwrite=['always', 'os_default'] How about overwrite=[None, True] with None meaning "OS default"? > with a warning and a switch of the default in a subsequent release. I think we should be conservative with warnin

[issue8828] Atomic function to rename a file

anatoly techtonik added the comment: os.rename(overwrite=True) to produce consistent cross-platform behavior. -- ___ Python tracker ___ __

[issue8828] Atomic function to rename a file

Charles-François Natali added the comment: I'd prefer an optional flag to rename() too. I really don't like having different functions that achieve the same thing. It's not obvious to infer from 'replace' its real intent, since it doesn't match any standard syscall/library. Ideally, this should

[issue8828] Atomic function to rename a file

R. David Murray added the comment: Ah, I see, people may be depending on rename on Windows not overwriting. I suppose a new function (and eventually deprecating the old?) would be the most straightforward way forward, though I dislike the necessity :) An alternative might be a flag on rename:

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: > What is the motivation for providing a new function? Because changing os.rename would break compatibility. -- ___ Python tracker ___

[issue8828] Atomic function to rename a file

R. David Murray added the comment: What is the motivation for providing a new function? -- ___ Python tracker ___ ___ Python-bugs-list

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: So how about providing a new public `os` module function doing a rename-with-overwrite on all platforms? We could name it e.g. os.replace(). os.rename_overwrite() is another possibility, more explicit but also longer. -- stage: -> needs patch ___

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: I see that Sun/Oracle Java trusts MoveFileEx to do atomic renames: 290 // atomic case 291 if (atomicMove) { 292 try { 293 MoveFileEx(sourcePath, targetPath, MOVEFILE_REPLACE_EXISTING); 294

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: > > "The solution? Let's remember that metadata changes are atomic. Rename is > > such a case." > > > > Hmmm. > Is he referring to the "standard" rename? The blog doesn't evoke a > specific function, but if it was the case, then why bother at all? Standard re

[issue8828] Atomic function to rename a file

Charles-François Natali added the comment: > MoveFileTransacted is only available under Vista or later. You should be able > to use MoveFileEx for the same effect. Nice. > "The solution? Let's remember that metadata changes are atomic. Rename is > such a case." > Hmmm. Is he referring to th

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: "The solution? Let's remember that metadata changes are atomic. Rename is such a case." This is from a MSDN blog, I would hope he knows what he's talking about. http://blogs.msdn.com/b/adioltean/archive/2005/12/28/507866.aspx (MoveFileEx appears in Windows XP

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: MoveFileTransacted is only available under Vista or later. You should be able to use MoveFileEx for the same effect. -- ___ Python tracker ___

[issue8828] Atomic function to rename a file

Charles-François Natali added the comment: The recent issue #13146 renewed my interest, so I'd like to make this move forward, since I think an atomic rename/write API could be quite useful. Issue #8604 (Adding an atomic FS write API) can be achieved relatively easily with the typical (fsync()

[issue8828] Atomic function to rename a file

Charles-François Natali added the comment: > According to the following article, a fsync is also needed on the > directory after a rename. I don't understand if is it always needed for > an atomic rename, or if we only need it for the "atomic write" pattern. It's not needed if you just want a

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: According to the following article, a fsync is also needed on the directory after a rename. I don't understand if is it always needed for an atomic rename, or if we only need it for the "atomic write" pattern. http://lwn.net/Articles/457667/ "The more subtle

[issue8828] Atomic function to rename a file

Changes by R. David Murray : -- type: -> feature request versions: +Python 3.3 -Python 3.2 ___ Python tracker ___ ___ Python-bugs-list

[issue8828] Atomic function to rename a file

Changes by Éric Araujo : -- nosy: +eric.araujo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue8828] Atomic function to rename a file

anatoly techtonik added the comment: Does it work with FAT32 or network filesystem? -- ___ Python tracker ___ ___ Python-bugs-list mai

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: [atomic_move_file-windows.py]: implementation of atomic_move_file() for Windows, depends on CreateTransaction() and MoveFileTransacted(), only available on Windows Vista, Windows Server 2008, or more recent version. This function *is* atomic. This function i

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: About the function names: - shutil.atomic_move_file(): only available on some OS - shutil.move_file(): use shutil.atomic_move_file() if available, or fall back to a best effort implementation Implement an atomic function to rename a directory is more complex

[issue8828] Atomic function to rename a file

anatoly techtonik added the comment: Dulwich bug discussion (closed) - https://bugs.edge.launchpad.net/dulwich/+bug/557585 Trac implementation - http://trac.edgewall.org/browser/trunk/trac/util/__init__.py?#L82 Stackoverflow - http://stackoverflow.com/questions/167414/is-an-atomic-file-rena

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: Implementation of "atomic" rename() for Windows in Mercurial: http://selenic.com/repo/hg/file/tip/mercurial/windows.py (search for "def rename") -- ___ Python tracker _

[issue8828] Atomic function to rename a file

anatoly techtonik added the comment: Atomic file renames are reimplemented by: 1. Mercurial 2. Trac 3. Dulwich (aka HgGit) and essential for many other projects out there, so this should be in standard library. Atomic renames depend on filesystem more that on the OS. -- nosy: +techton

[issue8828] Atomic function to rename a file

R. David Murray added the comment: Victor: you could always name it best_effort_at_atomic_rename :) -- nosy: +r.david.murray ___ Python tracker ___ __

[issue8828] Atomic function to rename a file

Changes by Raghuram Devarakonda : -- nosy: +draghuram ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: > Antoine Pitrou added the comment: > > Would you prefer to not define the function instead of writing a pseudo- > > atomic function? > > Your current implementation is useless, since it doesn't achieve anything > new. Besides, if the function isn't atomic, do

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: > Would you prefer to not define the function instead of writing a pseudo- > atomic function? Your current implementation is useless, since it doesn't achieve anything new. Besides, if the function isn't atomic, don't name it atomic_XXX. -- __

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: @pitrou: Yes, as I wrote: it's not possible to write an atomic function for all OS. The documentation must give a list of the OS on which the function is atomic. Would you prefer to not define the function instead of writing a pseudo-atomic function? -- Jav

[issue8828] Atomic function to rename a file

Antoine Pitrou added the comment: It seems you are proposing to call "atomic" something which isn't atomic: def atomic_rename(src, dst): if os.path.exists(dst): old = _create_old_filename(dst) rename(dst, old) rename(src, dst) unlink(old) --

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: We have to think about network file systems like NFS. Gnome (nautilus) had a bug on rename because NFS emitted a delete notification on a rename: http://linux-nfs.org/pipermail/nfsv4/2009-March/010134.html https://bugzilla.gnome.org/show_bug.cgi?id=575684 It l

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: > This implementation is atomic on POSIX, ... Wrong :-) "On how rename is broken in Mac OS X" http://www.weirdnet.nl/apple/rename.html "Update January 8, 2010: ... the original bug (5398777) was resolved in Snow Leopard but the issue I reported was not fixe

[issue8828] Atomic function to rename a file

Changes by Trundle : -- nosy: +Trundle ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/ma

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: Begin by removing the dest file is maybe not the safer approach :-) Here is a new try: begin by renaming the dest file to a new file. -- # use maybe a PRNG instead of a dummy counter or tempfile def _create_old_filename(filename): old = filename + '.old

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: See issue #8604: proposal of a new "with atomic_write(filename) as fp: ..." context manager. -- ___ Python tracker ___

[issue8828] Atomic function to rename a file

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:/

[issue8828] Atomic function to rename a file

STINNER Victor added the comment: A first implementation can be: if os.name in ('nt', 'ce'): def atomic_rename(a, b): if os.path.exists(b): unlink(b) rename(a, b) else: atomic_rename = os.rename This implementation is atomic on POSIX, and not atomic on Windows. Tell

[issue8828] Atomic function to rename a file

Changes by STINNER Victor : -- nosy: +tarek ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue8828] Atomic function to rename a file

New submission from STINNER Victor : os.rename() is atomic on Linux, but on Windows it raises an error if the destination does already exist. Not atomic pseudo-code for Windows: if exists(b): unlink(b) rename(a, b) Windows offers different functions depending on the version: - MoveFi