NightOwl888 commented on issue #933: URL: https://github.com/apache/lucenenet/issues/933#issuecomment-2080427474
I have done a bit of research on the fsync issue. The following info is helpful: 1. [Everything You Always Wanted To Know About fsync()](https://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/) 2. [Testing Lucene's index durability after crash or power loss](https://blog.mikemccandless.com/2014/04/testing-lucenes-index-durability-after.html) Here is the fsync approach in Lucene 4.8.0: https://github.com/apache/lucene/blob/8fdf89690404c0e65784b2c5477552b9dec58591/lucene/core/src/java/org/apache/lucene/util/IOUtils.java#L372-L418 It supports either a file name or a directory name (a string) and is called like this: https://github.com/apache/lucene/blob/releases/lucene-solr/4.8.0/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java#L299-L316 Correct me if I am wrong, but it seems that .NET only supports fsync on individual files, but not on all of the files in a directory. So, if that assumption is true, the only way to do fsync is to use native code. And indeed that is what is done in [ravendb](https://github.com/search?q=repo%3Aravendb%2Fravendb%20fsync&type=code). It also looks like the fsync call needs to be synchronized with any writes or flushes to the file buffer, but maybe Lucene already does external synchronization. Do note that our file locking approach depends on `FileStream`, so if for some reason we cannot use `FileStream`, we will also need to redesign our file locking approach. ## Directory Level Fsync Ravendb has a method that accepts a directory path and will do the [fsync on the whole directory](https://github.com/ravendb/ravendb/blob/99fe8133164d9d09f0f7f66891d890ab3c0d5b9e/src/Sparrow.Server/Platform/Posix/Syscall.cs#L321-L443), so this appears to be a suitable replacement for the `FileChannel.force()` call in Java when the `FileChannel` is based on a directory. Unfortunately, it seems that ravendb isn't covered under one of the licenses that are [acceptable by Apache](https://www.apache.org/legal/resolved.html#criteria). They were GNU back on version 3.5 and prior, but it doesn't look like any of those versions have this set up yet on non-Windows OSes. ## File Level Fsync In Lucene, "stale files" were tracked in a synchronized `HashSet<string>`. These are fsynced first before doing the fsync at the directory level. I looked at the source of `FileStream` and indeed when you call `Flush(false)` it will simply flush into the OS cache but not do the FSync. It also does not FSync if you call `Dispose()`. Technically, we could open a new `FileStream` instance on a file string for writing and then call `Flush(true)` to do the file-level FSync call, but there would be some overhead to instantiate the `FileStream` instance. And although there is an option that can be passed that is supposed to indicate not to create a buffer, it doesn't appear to be implemented, so there would need to be at least a 1 byte array allocated as well. So, it appears that a better option might be to use the same low level native calls that Ravendb uses at the file level as well. ## Synchronization This is the tricky bit. In Java, the `FileChannel` instance does blocking to ensure that no write/flush/fsync operations can happen simultaneously according to the [docs in Apache Harmony](https://github.com/apache/harmony/blob/02970cb7227a335edd2c8457ebdde0195a735733/classlib/modules/nio/src/main/java/common/java/nio/channels/FileChannel.java#L64-L68). I haven't dug much into the implementation as to how to allow individual files to be written during normal application use but then be able to gain an exclusive lock on the entire directory to do the directory-level fsync. It also appears that the need to block makes it impossible (or at least extremely difficult) to use `FileOptions.Asynchronous`. It is kind of pointless anyway if you don't explicitly call the `FlushAsync()` and `WriteAsync()` methods, and it takes you down a completely different implementation. Also, when it is enabled on Linux, the FSync happens automatically on `FlushAsync()` and on `Dispose()`. ------------------------------------------------------ It would be great to get some feedback on this to see if anyone has any ideas that we could integrate. And of course setting up the project for deploying the native bits via NuGet on the various operating systems that .NET Core supports is not something I have looked into. If someone with experience doing this could provide some guidance, that would be great. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
