On Thu, Jan 6, 2011 at 7:59 PM, Enrico Weigelt weig...@metux.de wrote:
* Olaf van der Spek olafvds...@gmail.com schrieb:
A transaction to update multiple files in one atomic go?
Yes. The application first starts an transaction, creates/writes/
removes a bunch of files and then sends a
On Thu, Jan 6, 2011 at 1:54 AM, Ted Ts'o ty...@mit.edu wrote:
I was thinking, doesn't ext have this kind of dependency tracking already?
It has to write the inode after writing the data, otherwise the inode
might point to garbage.
No, it doesn't. We use journaling, and forced data writeouts,
* Ted Ts'o ty...@mit.edu [110105 19:26]:
So one of the questions is how much should be penalizing programs that
are doing things right (i.e., using fsync), versus programs which are
doing things wrong (i.e., using rename and trusting to luck).
Please do not call it wrong. All those programs
On Thu, Jan 6, 2011 at 5:01 AM, Ted Ts'o ty...@mit.edu wrote:
On Thu, Jan 06, 2011 at 12:57:07AM +, Ian Jackson wrote:
Ted Ts'o writes (Re: Safe File Update (atomic)):
Then I invite you to implement it, and start discovering all of the
corner cases for yourself. :-) As I predicted
On Thu, Jan 6, 2011 at 12:39 PM, Olaf van der Spek olafvds...@gmail.com wrote:
On Thu, Jan 6, 2011 at 5:01 AM, Ted Ts'o ty...@mit.edu wrote:
On Thu, Jan 06, 2011 at 12:57:07AM +, Ian Jackson wrote:
Ted Ts'o writes (Re: Safe File Update (atomic)):
Then I invite you to implement
Getting people to believe that you can't square a circle[1]
is very hard,
Just allow an infinite number of steps and it's almost trivial ;-)
It's like trying teaching a pig to sing.
Well, that works, just sounds a bit like vogon poetry ;-o
If you give me a specific approach, I can tell
On Thu, Jan 6, 2011 at 7:33 PM, Enrico Weigelt weig...@metux.de wrote:
To come back to the original question, I'd like to know which concrete
realworld problems should be solved by that. One thing an database-like
transactional filesystem (w/ MVCC) would be nice is package managers:
we still
* Olaf van der Spek olafvds...@gmail.com schrieb:
A transaction to update multiple files in one atomic go?
Yes. The application first starts an transaction, creates/writes/
removes a bunch of files and then sends a commit. The changes
should become visible atomically and the call returns when
On Wed, Jan 5, 2011 at 1:25 AM, Ted Ts'o ty...@mit.edu wrote:
On Wed, Jan 05, 2011 at 01:05:03AM +0100, Olaf van der Spek wrote:
Why is it that you ignore all my responses to technical questions you asked?
In general, because they are either (a) not well-formed, or (b) you
are asking me to
On Wed, Jan 05, 2011 at 12:55:22PM +0100, Olaf van der Spek wrote:
If you give me a specific approach, I can tell you why it won't work,
or why it won't be accepted by the kernel maintainers (for example,
because it involves pouring far too much complexity into the kernel).
Let's consider
On Wed, Jan 5, 2011 at 7:26 PM, Ted Ts'o ty...@mit.edu wrote:
On Wed, Jan 05, 2011 at 12:55:22PM +0100, Olaf van der Spek wrote:
If you give me a specific approach, I can tell you why it won't work,
or why it won't be accepted by the kernel maintainers (for example,
because it involves
On Wed, Jan 05, 2011 at 09:38:30PM +0100, Olaf van der Spek wrote:
Performance is important, I agree.
But you're trading performance for safety here.
... but if the safety is not needed, then you're paying for no good
reason. And if performance is needed, then use fsync().
OK, what about
On Wed, Jan 5, 2011 at 10:37 PM, Ted Ts'o ty...@mit.edu wrote:
Ah. So performance isn't the problem, it's just hard too implement.
Would've been a lot faster if you said that earlier.
Too hard to implement doesn't go far enough. It's also a matter of
near impossibility to add new features
On Wed, Jan 05, 2011 at 10:47:03PM +0100, Olaf van der Spek wrote:
That was about soft updates. I'm not sure this is just as complex.
Then I invite you to implement it, and start discovering all of the
corner cases for yourself. :-) As I predicted, you're not going to
believe me when I tell
Ted Ts'o writes (Re: Safe File Update (atomic)):
Then I invite you to implement it, and start discovering all of the
corner cases for yourself. :-) As I predicted, you're not going to
believe me when I tell you it's too hard.
How about you reimplement all of Unix userland, first, so
On Thu, Jan 06, 2011 at 12:57:07AM +, Ian Jackson wrote:
Ted Ts'o writes (Re: Safe File Update (atomic)):
Then I invite you to implement it, and start discovering all of the
corner cases for yourself. :-) As I predicted, you're not going to
believe me when I tell you it's too hard
On Mon, Jan 3, 2011 at 3:43 PM, Ted Ts'o ty...@mit.edu wrote:
On Mon, Jan 03, 2011 at 12:26:29PM +0100, Olaf van der Spek wrote:
Given that the issue has come up before so often, I expected there to
be a FAQ about it.
Your asking the question over (and over... and over...) doesn't make
it
On Wed, Jan 05, 2011 at 01:05:03AM +0100, Olaf van der Spek wrote:
Why is it that you ignore all my responses to technical questions you asked?
In general, because they are either (a) not well-formed, or (b) you
are asking me to prove a negative. Getting people to believe that you
can't
On 02/01/11 17:37, Olaf van der Spek wrote:
A userspace lib is fine with me. In fact, I've been asking for it
multiple times. Result: no response.
Excuse me?
You (well, Henrique, but you were CCed) said how about a user space
lib? I said I'm working on one, will be ready about this
On Sun, Jan 2, 2011 at 6:14 PM, Henrique de Moraes Holschuh Maybe I
wasn't clear, in that case I'm sorry. To me, O_ATOMIC is
Whether this should map to O_ATOMIC in glibc or be something new, I don't
care. But if it is a flag, I'd highly suggest naming it O_CREATEUNLINKED or
something else
On Sun, Jan 2, 2011 at 7:55 PM, Adam Borowski kilob...@angband.pl wrote:
Note that on the other side of the fence there's something called TxF
Not GA AFAIK.
And what if you're changing one byte inside a 50 GB file?
I see an easy implementation on btrfs/ocfs2 (assuming no other writers),
but
On Mon, Jan 3, 2011 at 4:25 AM, Ted Ts'o ty...@mit.edu wrote:
On Sun, Jan 02, 2011 at 04:14:15PM +0100, Olaf van der Spek wrote:
Last time you ignored my response, but let's try again.
The implementation would be comparable to using a temp file, so
there's no need to keep 2 g in memory.
Ted,
Thanks for the reply and detailed analysis.
Which gets me back to the question of use cases. When are we going to
be using this monster? For many use cases, where the original reason
Where implicit rollbacks are desireable, I suppose. It is incompatible
with edit-in-place, anyway.
On Mon, Jan 03, 2011 at 09:49:40AM -0200, Henrique de Moraes Holschuh wrote:
1) You care about data loss in the case of power failure, but not in
the case of hard drive or storage failure, *AND* you are writing tons
and tons of tiny 3-4 byte files and so you are worried about
performance
On Mon, Jan 3, 2011 at 6:28 AM, Enrico Weigelt weig...@metux.de wrote:
* Ted Ts'o ty...@mit.edu schrieb:
This is possible. It would be specific only to file systems that
support inodes (i.e., ix-nay for NFS, FAT, etc.).
FAT supports inodes ?
ix-nay: no/except
Olaf
--
To UNSUBSCRIBE,
On Mon, Jan 3, 2011 at 11:35 AM, Shachar Shemesh shac...@debian.org wrote:
On 02/01/11 17:37, Olaf van der Spek wrote:
A userspace lib is fine with me. In fact, I've been asking for it
multiple times. Result: no response.
Excuse me?
You (well, Henrique, but you were CCed) said how about
On Mon, Jan 03, 2011 at 12:26:29PM +0100, Olaf van der Spek wrote:
Given that the issue has come up before so often, I expected there to
be a FAQ about it.
Your asking the question over (and over... and over...) doesn't make
it an FAQ. :-)
Aside from your asking over and over, it hasn't
On Mon, Jan 3, 2011 at 3:43 PM, Ted Ts'o ty...@mit.edu wrote:
On Mon, Jan 03, 2011 at 12:26:29PM +0100, Olaf van der Spek wrote:
Given that the issue has come up before so often, I expected there to
be a FAQ about it.
Your asking the question over (and over... and over...) doesn't make
it
Ted Ts'o tytso at mit.edu writes:
actually. The right answer has been known for decades, and it's is
very simple; write a temp file, copy over the xattr's and ACL's if you
care (in many cases, such as an application's private state files, it
won't care, so it can skip this step --- it's only
On Sun, 02 Jan 2011, Ted Ts'o wrote:
And of course, Olaf isn't actually offerring to implement this
hypothetical O_ATOMIC. Oh, no! He's just petulently demanding it,
even though he can't give us any concrete use cases where this would
actually be a huge win over a userspace safe-write
On Sun, Jan 2, 2011 at 8:09 AM, Ted Ts'o ty...@mit.edu wrote:
You could ask for a new (non-POSIX?) API that does not ask of a
POSIX-like filesystem something it cannot provide (i.e. don't ask for
something that requires inode-path reverse mappings). You could ask
for syscalls to copy inodes,
On Sun, Jan 2, 2011 at 1:52 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:
Olaf, O_ATOMIC is difficult in the kernel sense and in the long run. It
is an API that is too hard to implement in a sane way, with too many
boundary conditions.
OTOH, you don't need O_ATOMIC. You need a way
On Sun, 02 Jan 2011, Olaf van der Spek wrote:
On Sun, Jan 2, 2011 at 1:52 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:
Olaf, O_ATOMIC is difficult in the kernel sense and in the long run. It
is an API that is too hard to implement in a sane way, with too many
boundary
On Sun, Jan 02, 2011 at 04:14:15PM +0100, Olaf van der Spek wrote:
On Sun, Jan 2, 2011 at 8:09 AM, Ted Ts'o ty...@mit.edu wrote:
The O_ATOMIC open flag is highly problematic, and it's not fully
specified.
Note that on the other side of the fence there's something called TxF
(Transactional
On Sun, Jan 02, 2011 at 04:14:15PM +0100, Olaf van der Spek wrote:
Last time you ignored my response, but let's try again.
The implementation would be comparable to using a temp file, so
there's no need to keep 2 g in memory.
Write the 2 g to disk, wait one day, append the 1 k, fsync, update
On Sun, Jan 02, 2011 at 03:14:41PM -0200, Henrique de Moraes Holschuh wrote:
1. Create unlinked file fd (benefits from kernel support, but doesn't
require it). If a filesystem cannot support this or the boundary conditions
are unaceptable, fail. Needs to know the destination name to do the
* Ted Ts'o ty...@mit.edu schrieb:
This is possible. It would be specific only to file systems that
support inodes (i.e., ix-nay for NFS, FAT, etc.).
FAT supports inodes ?
IIRC it puts all file information (including attributes and first
data block) directly into the dirent ...
Some file
On Fri, Dec 31, 2010 at 5:08 PM, Enrico Weigelt weig...@metux.de wrote:
Not true. Renaming a running executable works just fine, for example.
Well, has been quite a while since I last used Windows, but IIRC
renaming an running executable was denied.
Maybe on FAT. However, that's OT.
Why
Olaf van der Spek olafvds...@gmail.com (01/01/2011):
Doing it in the kernel would be fine (maybe DLM could be used here),
What's DLM?
CONFIG_DLM.
KiBi.
signature.asc
Description: Digital signature
On Sat, Jan 1, 2011 at 7:13 PM, Cyril Brulebois k...@debian.org wrote:
Olaf van der Spek olafvds...@gmail.com (01/01/2011):
Doing it in the kernel would be fine (maybe DLM could be used here),
What's DLM?
CONFIG_DLM.
DLM seems independent of atomic updates.
Olaf
--
To UNSUBSCRIBE,
* Olaf van der Spek olafvds...@gmail.com schrieb:
Doing it in the kernel would be fine (maybe DLM could be used here),
What's DLM?
Distributed lock manager.
but would be a nonportable solution for quite a long time ;-o
Since it's the only proper solution I don't think that's a
On Sun, Jan 2, 2011 at 12:03 AM, Enrico Weigelt weig...@metux.de wrote:
I doubt that the only proper solution. As said, an (userland)
filesystem could also do fine.
Do you think distros like Debian would install such a setup by default?
Olaf
--
To UNSUBSCRIBE, email to
On Fri, Dec 31, 2010 at 09:51:50AM -0200, Henrique de Moraes Holschuh wrote:
On Fri, 31 Dec 2010, Olaf van der Spek wrote:
Ah, hehe. BTW, care to respond to the mail I send to you?
There is nothing more I can add to this thread. You want O_ATOMIC. It
cannot be implemented for all use
On Fri, Dec 31, 2010 at 3:17 AM, Henrique de Moraes Holschuh
h...@debian.org wrote:
On Thu, 30 Dec 2010, Henrique de Moraes Holschuh wrote:
BTW: safely removing a file is also tricky. AFAIK, one must open it RW,
in exclusive mode. stat it by fd and check whether it is what one
expects
On Fri, 31 Dec 2010, Olaf van der Spek wrote:
Ah, hehe. BTW, care to respond to the mail I send to you?
There is nothing more I can add to this thread. You want O_ATOMIC. It
cannot be implemented for all use cases of the POSIX API, so it will not
be implemented by the kernel. That's all there
On Fri, Dec 31, 2010 at 12:51 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:
On Fri, 31 Dec 2010, Olaf van der Spek wrote:
Ah, hehe. BTW, care to respond to the mail I send to you?
There is nothing more I can add to this thread. You want O_ATOMIC. It
That's a shame. I thought I
* Olaf van der Spek olafvds...@gmail.com schrieb:
something that requires inode-path reverse mappings). You could ask
for syscalls to copy inodes, etc. You could ask for whatever is needed
To me, inodes are an implementation detail that shouldn't be exposed.
Well, they're an fundamental
On Fri, Dec 31, 2010 at 2:57 PM, Enrico Weigelt weig...@metux.de wrote:
To me, inodes are an implementation detail that shouldn't be exposed.
Well, they're an fundamental concept which sometimes *IS* significant
to the applications. It's very different from systems where each
file has exactly
* Olaf van der Spek olafvds...@gmail.com schrieb:
Well, they're an fundamental concept which sometimes *IS* significant
to the applications. It's very different from systems where each
file has exactly one name (eg. DOS/Windows) or where there're just
filesnames that point to opaque
On Fri, Dec 31, 2010 at 03:44:56PM +0100, Enrico Weigelt wrote:
* Olaf van der Spek olafvds...@gmail.com schrieb:
Well, they're an fundamental concept which sometimes *IS* significant
to the applications. It's very different from systems where each
file has exactly one name (eg.
On Fri, Dec 31, 2010 at 3:44 PM, Enrico Weigelt weig...@metux.de wrote:
* Olaf van der Spek olafvds...@gmail.com schrieb:
Well, they're an fundamental concept which sometimes *IS* significant
to the applications. It's very different from systems where each
file has exactly one name (eg.
On Fri, Dec 31, 2010 at 3:58 PM, brian m. carlson
sand...@crustytoothpaste.net wrote:
These cases aren't that rare. Windows, for example, tends to deny
renames on open files, as they're also identified by the filename.
(yes, there're other solutions for this problem, eg. having some
* Olaf van der Spek olafvds...@gmail.com schrieb:
renames on open files, as they're also identified by the filename.
Not true. Renaming a running executable works just fine, for example.
Well, has been quite a while since I last used Windows, but IIRC
renaming an running executable was
On Wed, 29 Dec 2010, Olaf van der Spek wrote:
Writing a temp file, fsync, rename is often proposed. However, the
It is:
write temp file (in same directory as file to be replaced), fsync temp
file[1], rename (atomic), fsync directory[2].
[1] Makes sure file data has been commited to backend
On Thu, Dec 30, 2010 at 12:46 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:
write temp file (in same directory as file to be replaced), fsync temp
What if the target name is actually a symlink? To a different volume?
What if you're not allowed to create a file in that dir.
If we
On 30/12/10 13:46, Henrique de Moraes Holschuh wrote:
Is there a code snippet or lib function that handles this properly?
I don't know. I'd be interested in the answer, though :-)
I'm working on one under the MIT license. Will probably release it by
the end of this week. Will also
On Thu, Dec 30, 2010 at 3:51 PM, Shachar Shemesh shac...@shemesh.biz wrote:
I'm working on one under the MIT license. Will probably release it by the
end of this week. Will also handle copying the permissions over and
following symlinks.
Sounds great!
Got a project page already?
What aboue
On Thu, Dec 30, 2010 at 03:30:29PM +0100, Olaf van der Spek wrote:
name the temp file properly, and teach your program to clean old ones up
*safely* (see vim swap file handling for an example) when it starts.
What about restoring meta-data? File-owner?
owner, permissions, acl, xattrs, and
On 30/12/10 17:02, Olaf van der Spek wrote:
On Thu, Dec 30, 2010 at 3:51 PM, Shachar Shemeshshac...@shemesh.biz wrote:
I'm working on one under the MIT license. Will probably release it by the
end of this week. Will also handle copying the permissions over and
following symlinks.
On Thu, Dec 30, 2010 at 4:12 PM, Shachar Shemesh shac...@debian.org wrote:
No. I was doing it as code to accompany an article on my company's site
about how it should be done. I was originally out to write the article, and
then decided to add code. A good thing, too, as recursively resolving
On Thu, 30 Dec 2010, Olaf van der Spek wrote:
On Thu, Dec 30, 2010 at 12:46 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:
write temp file (in same directory as file to be replaced), fsync temp
What if the target name is actually a symlink? To a different volume?
Indeed. You have
On Thu, 30 Dec 2010, Olaf van der Spek wrote:
The reason I asked for a kernelland solution is because it's hard if
not impossible to do properly in userland. But some kernel devs (Ted
and others) don't agree. They reason that the desire to preserve all
meta-data isn't reasonable by itself.
It
On Thu, Dec 30, 2010 at 4:20 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:
What if the target name is actually a symlink? To a different volume?
Indeed. You have to check that first, of course :-( This is about safe
handling of such functions, symlinks always have to be derreferenced
On Thu, Dec 30, 2010 at 4:24 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:
On Thu, 30 Dec 2010, Olaf van der Spek wrote:
The reason I asked for a kernelland solution is because it's hard if
not impossible to do properly in userland. But some kernel devs (Ted
and others) don't agree.
On 30/12/10 13:46, Henrique de Moraes Holschuh wrote:
Is there a code snippet or lib function that handles this properly?
I don't know. I'd be interested in the answer, though :-)
I'm working on one under the MIT license. Will probably release it by
the end of this week. Will
On Thu, 30 Dec 2010, Olaf van der Spek wrote:
On Thu, Dec 30, 2010 at 4:24 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:
On Thu, 30 Dec 2010, Olaf van der Spek wrote:
The reason I asked for a kernelland solution is because it's hard if
not impossible to do properly in userland.
On Thu, Dec 30, 2010 at 6:48 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:
Why not?
You touched it, it is not the same file/inode anymore.
That's again a regression from the non-atomic case.
How does it handle meta-data you don't know about yet?
It doesn't. You need a copy inode
On 30/12/10 19:48, Henrique de Moraes Holschuh wrote:
It doesn't. You need a copy inode without the file data filesystem
interface to be able to do that in the first place. It might exist, but I
never heard of it.
If my (extremely leaky) memory serves me right, Windows has it. It's
On 30/12/10 17:17, Olaf van der Spek wrote:
On Thu, Dec 30, 2010 at 4:12 PM, Shachar Shemeshshac...@debian.org wrote:
No. I was doing it as code to accompany an article on my company's site
about how it should be done. I was originally out to write the article, and
then decided to add
On Thu, Dec 30, 2010 at 7:15 PM, Shachar Shemesh shac...@debian.org wrote:
If my (extremely leaky) memory serves me right, Windows has it. It's called
delete and then rename. It is not atomic (since when do Windows care about
not breaking stuff), but it does exactly that.
If you delete a file
On Thu, Dec 30, 2010 at 7:20 PM, Shachar Shemesh shac...@debian.org wrote:
Depending on /proc is probably not reasonable.
Are you sure it will be atomic? ;)
open old file, get fd (we'll assume it's 5). Do readlink on /proc/self/fd/5,
and get file's real path. Do everything in said path.
On Thu, 2010-12-30 at 19:29 +0100, Olaf van der Spek wrote:
On Thu, Dec 30, 2010 at 7:15 PM, Shachar Shemesh shac...@debian.org wrote:
If my (extremely leaky) memory serves me right, Windows has it. It's called
delete and then rename. It is not atomic (since when do Windows care about
not
On Thu, Dec 30, 2010 at 7:46 PM, Ben Hutchings b...@decadent.org.uk wrote:
You're kidding me. Got any source to back this up?
http://support.microsoft.com/?kbid=172190
Interesting. Although no longer available on Vista / 7.
Olaf
--
To UNSUBSCRIBE, email to
On Thu, 30 Dec 2010, Henrique de Moraes Holschuh wrote:
BTW: safely removing a file is also tricky. AFAIK, one must open it RW,
in exclusive mode. stat it by fd and check whether it is what one
expects (regular file, ownership). unlink it by fd. close the fd.
Eh, as it was pointed to me by
On 30/12/10 17:02, Olaf van der Spek wrote:
Got a project page already?
Watch this space. Actual code coming soon(tm).
https://github.com/Shachar/safewrite
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com
--
To UNSUBSCRIBE, email to
Since the introduction of ext4, some apps/users have had issues with
file corruption after a system crash. It's not a bug in the FS AFAIK
and it's not exclusive to ext4.
Writing a temp file, fsync, rename is often proposed. However, the
durable aspect of fsync isn't always required and this way
76 matches
Mail list logo
