[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: It looks like each platform and each filesystem behave differently, with subtle differencies with different kernel versions. I'm not more sure that the Python stdlib is the best place to host an implementation of an "atomic write" API. I now suggest to create

[issue8604] Adding an atomic FS write API

Thomas Kluyver added the comment: For reference, we added our own atomic_writing() context manager in Jupyter/IPython, after someone lost work when save failed on a full disk. We initially went for the same approach - write to a temporary file, fsync, rename over the original file - but it cau

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: This issue is old and different operating systems provide different warranties on rename. Maybe this project should start as a project on PyPI to find the best API and catch compatibilitites issues. For example os.scandir() also started on PyPI. --

[issue8604] Adding an atomic FS write API

Марк Коренберг added the comment: Also, modern kernels allows to replace entire directory! renameat2() + RENAME_EXCHANGE So, there should be more atomic operations -- ___ Python tracker ___

[issue8604] Adding an atomic FS write API

Марк Коренберг added the comment: instead of realpath, os.lstat() may be used in order to detect if target file object is not plain file. Also, os.rename() cannot be used since it follows symlinks by default. -- ___ Python tracker

[issue8604] Adding an atomic FS write API

Марк Коренберг added the comment: You also forgot about two things: 1. set temporary file permissions before rename 2. fsync(open(os.dirname(temporaryfile))) 3. if original file name is symlink, replace will works wrong. os.realpath should be used. So, here are real life function, that we use

[issue8604] Adding an atomic FS write API

Barry A. Warsaw added the comment: Of course, I have my own atomic-rename thing, but I'm not going to post the code here. It's fairly limited to my own use case and I have no interest in making it cross platform. That being said, I personally found that a context manager with a signature ide

[issue8604] Adding an atomic FS write API

Changes by Antoine Pitrou : -- stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue8604] Adding an atomic FS write API

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue8604] Adding an atomic FS write API

Charles-François Natali added the comment: > (Note that the Beaker version would need to be enhanced with the extra API > parameters from Victor's version, as well as updated to use the exclusive > open and close-on-exec flags) I think the API would be nicer if it was just a wrapper around the

[issue8604] Adding an atomic FS write API

Nick Coghlan added the comment: (Note that the Beaker version would need to be enhanced with the extra API parameters from Victor's version, as well as updated to use the exclusive open and close-on-exec flags) -- ___ Python tracker

[issue8604] Adding an atomic FS write API

Nick Coghlan added the comment: We have one of these in Beaker (we didn't need to wait for os.replace, since Beaker only runs on Linux): http://git.beaker-project.org/cgit/beaker/tree/Common/bkr/common/helpers.py?h=develop#n90 It turns out to be beneficial to separate the three operations in o

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: > Here's the part of the code that does the open/close part: > http://bazaar.launchpad.net/~exabyte/blackherd/async-refactor/view/61/blackherd/misc.py#L498 This code contains a bug: hasattr('os', 'fsync') is never True :-) This part is interesting: # f

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: atomic_write.patch calls os.replace(src, dst) whereas the src file is open. It works on Linux, but it sounds a little bit strange to me and may fail on other platforms. Here is another patch (atomic_write_mkstemp.patch) using tempfile.mkstemp() instead of tem

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: "I prefer to use tempfile.NamedTemporaryFile because it is portable and well tested. It also uses a random suffix. On Windows, the O_TEMPORARY flag is passed to os.open(), ..." Oh, and it sets also the close-on-exec safe, which is also more secure. --

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: + # Flush Python buffers to system buffers + fileobj.flush() + + if hasattr(os, 'fsync'): + # Flush system buffers to disk + fd = fileobj.fileno() + os.fsync(fd) A fsync=True paramater may be added if calling os.fs

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: Here is a patch based on the new os.replace() function. I only tested the patch on Linux. My patch must be tested on Windows, I don't know what happens when the file object is closed when the NamedTemporaryFile.delete attribute is set to False. Does close() r

[issue8604] Adding an atomic FS write API

Changes by Barry A. Warsaw : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue8604] Adding an atomic FS write API

Changes by Martin Panter : -- nosy: +vadmium ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue8604] Adding an atomic FS write API

Nick Coghlan added the comment: Since it may not be clear from the rest of the thread, the os.replace() cross-platform atomic renaming building block was added in 3.3 (in #8828, which Victor linked above) Another stdlib use case now also exists in importlib (see http://hg.python.org/cpython/f

[issue8604] Adding an atomic FS write API

Martin v. Löwis added the comment: I'm opposed to adding "atomic" files to the standard library. This has nothing to do with the original issue; anybody contributing to the issue is making up user needs that have not been sufficiently demonstrated. It would be best to release such code on PyP

[issue8604] Adding an atomic FS write API

Charles-François Natali added the comment: > Ah, it's a temporary file indeed. > But does that mean you can't inspect your logs in real time? All log > files I have ever seen are available under their final name while they > are still being written to. > Yes, logging was a poor example :-) > A

[issue8604] Adding an atomic FS write API

Antoine Pitrou added the comment: > There's no point in calling fsync() after each write, since data is > written to a temporary file which won't be visible before rename(): > even if you call fsync() after each write, if the file is not properly > closed, it won't be committed (i.e.renamed). A

[issue8604] Adding an atomic FS write API

Charles-François Natali added the comment: > If you want atomicity to apply to logging, > you must instead guarantee the durability of each write() call, meaning > calling fsync() on each logging call Why so? There's no point in calling fsync() after each write, since data is written to a tempo

[issue8604] Adding an atomic FS write API

Antoine Pitrou added the comment: > > Does it have to be a class? What would be the operations apart from > > write()? > > Well, I thought that making it a file-like object could be useful: > that way, one could pass it to pickle.dump(), logging.StreamHandler or > any method expecting a file-li

[issue8604] Adding an atomic FS write API

Charles-François Natali added the comment: > Does it have to be a class? What would be the operations apart from > write()? Well, I thought that making it a file-like object could be useful: that way, one could pass it to pickle.dump(), logging.StreamHandler or any method expecting a file-like

[issue8604] Adding an atomic FS write API

Antoine Pitrou added the comment: > Anyway, my "thoughts?" wasn't referring to this namespace hierarchy > "proposal" (which really isn't one), but rather to the best module > which could host an AtomicFile class. Does it have to be a class? What would be the operations apart from write()? > St

[issue8604] Adding an atomic FS write API

Charles-François Natali added the comment: > I adhere to "flat is better than nested". e.g. it always irks me to type > "urllib.request" instead of "urllib". Well, that's what (selective) imports are for, no ? from urllib import request from file.path import exists import xml.etree.cElementTree

[issue8604] Adding an atomic FS write API

Antoine Pitrou added the comment: > I wish we had something like: > io.file > io.file.tempfile > io.file.path > io.file.atomicfile I adhere to "flat is better than nested". e.g. it always irks me to type "urllib.request" instead of "urllib". A 3-level deep nesting in the stdlib would certainl

[issue8604] Adding an atomic FS write API

Charles-François Natali added the comment: > I prefer to write a "best-effort" function I disagree. People who explicitely use an atomic file API want atomicity/persistency, otherwise they wouldn't use it. Exposing a function that may, or may not, be atomic is just plain wrong. IMHO, the righ

[issue8604] Adding an atomic FS write API

Antoine Pitrou added the comment: > I'm not sure about the best module to host this, though: os.path ? os.path is mostly about path manipulation functions, although it also performs directory I/O (e.g. realpath()). -- ___ Python tracker

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: > I'm not sure about the best module to host this, though: os.path ? Some OS don't provide atomic rename. If we only define a function when it is atomic (as we do in the posix module, only expose functions available on the OS), programs will have to write two

[issue8604] Adding an atomic FS write API

Charles-François Natali added the comment: I'd like to make this move forward. Milko, I see you have an implementation which looks OK at a first glance. Would like to propose a patch against the default branch? The patch should include documentation and tests. I'm not sure about the best modul

[issue8604] Adding an atomic FS write API

Milko Krachounov added the comment: > Something's missing in all the implementations presented: > to make sure that the new version of the file is available afer > a crash, fsync must be called on the containing directory after > the rename. I upgraded my proposed approach to include dir fsyn

[issue8604] Adding an atomic FS write API

Charles-François Natali added the comment: Something's missing in all the implementations presented: to make sure that the new version of the file is available afer a crash, fsync must be called on the containing directory after the rename. -- nosy: +neologix _

[issue8604] Adding an atomic FS write API

Milko Krachounov added the comment: Well, since I had a typo in the main method which called the Windows implementation on POSIX, the unit test works on the code for Windows when ran on POSIX. Heh, I'm sorry for the noise, but it seems that re-reading the code four times and running the unit

[issue8604] Adding an atomic FS write API

Milko Krachounov added the comment: I have a class for overwriting files "properly" that I use in three of my projects. It aims to be atomic as possible, supports creating backups, but it doesn't have functions to set or retain permissions when requested (which might be desirable if such a fu

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: Trac has an AtomicFile class which copies file mode, owner, etc. http://trac.edgewall.org/browser/trunk/trac/util/__init__.py?#L145 -- ___ Python tracker __

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: We need an atomic rename function to implement atomic_write(): I opened a new issue, #8828. -- dependencies: +Atomic function to rename a file ___ Python tracker _

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: An article about ext3 and fsync: "Solving the ext3 latency problem" http://lwn.net/Articles/328363/ Another good article, written by the maintainer of ext4 (Theodore Ts'o): "Don’t fear the fsync!" http://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/ --

[issue8604] Adding an atomic FS write API

STINNER Victor added the comment: > This sounds silly to me. You can write a file in two lines: > > with open("foo", "wb") as f: >f.write(contents) If the disk is full, write fails and the new file only contains a part of 'contents'. If the file does already exist and the write fails, the

[issue8604] Adding an atomic FS write API

Changes by Éric Araujo : -- nosy: +merwok ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org

[issue8604] Adding an atomic FS write API

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:/

[issue8604] Adding an atomic FS write API

Changes by Tarek Ziadé : -- title: Alias for distutils.file_util.write_file in e.g. shutils -> Adding an atomic FS write API ___ Python tracker ___ __