reidkaufmann commented on code in PR #890:
URL:
https://github.com/apache/arrow-rs-object-store/pull/890#discussion_r4518021673
##########
src/local.rs:
##########
@@ -1032,6 +1096,147 @@ fn sync_and_close(file: File, path: &std::path::Path,
fsync: bool) -> Result<()>
Ok(())
}
+/// Directories whose entries changed, or are changing, without an fsync
since. Shared by every
+/// `LocalFileSystem` in the process: a durable write into a directory another
call is still
+/// making durable fsyncs it too, and a failed fsync leaves the directory to
the next write.
+///
+/// A map rather than a queue: changes to one directory collapse into one
entry, and the change
+/// counter tells a sync whether the directory changed again after the sync
started, so the entry
+/// is then kept.
+static UNSYNCED: Mutex<Unsynced> = Mutex::new(Unsynced {
+ dirs: BTreeMap::new(),
+ changes: 0,
+});
Review Comment:
An alternative is to blindly sync parent directories without this level of
tracking. I did some tests on that idea; on xfs there was little overhead but
on ext4 the overhead increases substantially with concurrency and directory
depth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
