On Sat, Feb 27, 2010 at 1:00 AM, Mark Hammond <skippy.hamm...@gmail.com> wrote:
> Regarding the file versioning - I'm sorry that we haven't kept you in the > loop a little better, but with Damien's guidance we think we have found a > better strategy for this. nb, it's great that the issue gets attention. > > The root of this strategy comes from a realization that if a file is opened > on windows with FILE_SHARE_DELETE, the file can be deleted *or renamed* > while it is open. One limitation is that a file of the same name can not be > re-created while the old one still has handles open (the 'deleted but still > open' file still appears in directory listings until the handle is closed, > for example) > > Given this, what we can do is something like: > > * Arrange for erlang to be able to open the DB and view files with this > flag. > * Instead of deleting a file before replacing it, we first rename the file > to a unique name (ie, based on a UUID) in a special directory. > * As couch starts up, attempt to delete any old files in this special > directory. In theory, no such files should exist - the OS should take care > of actually removing any such files even if erlang crashes. > > The end result of this is that things can be made to work with a lot less > friction than the 'file versioning' scheme. I've a patch to couchdb that > works when used with a patch to erlang to open *all* files with that flag. > The next step down this path seems to be to create an erlang 'port driver' > for disk IO and use this instead of the builtin erlang file objects. We've > identified a 'bfile' erlang port driver we may be able to fork and adapt to > our needs - using our own port driver would also allow optimizations to the > file IO for non-windows platforms (eg, indicate to the OS that certain files > should not get OS-level caching, etc) > Sounds like a good plan. I noticed Jan just filed a proposal to erlang-patches for adding FILE_SHARE_DELETE flag to CreateFile in erts. > Unfortunately, none of this has been discussed anywhere formal - much of it > was on IRC - so I apologize if this is the first you heard of it. But > whatever strategy we wind up with for this is almost certainly not going to > land and be tested enough to make 0.11. Any idea when it'll land to trunk? I could use a rough estimate to decide if I should temporarily deploy the file versioning patch. cheers, -juhani
