alamb commented on PR #643: URL: https://github.com/apache/arrow-rs-object-store/pull/643#issuecomment-4099043293
> Defaulting to false seems a bit reckless to me, users are most certainly expecting the local "object store" to be durable, as are all other object store implementations in this crate (aside from memory...). > > It was definitely a great surprise to me to learn that it was not. I agree that if we were writing this library from scratch, having writes to files automatically call `fsync` would be a very reasonable behavior. Likewise, I am not debating that adding some way to call `fsync` is valuable, I am simply voting against changing the default behavior to avoid causing hard to track down downstream implications (though it enough people vote the other way I will be willing to defer). I recommend we do this in two PRs: 1. Add a new flag, default to false that adds `fsync` on write (there is broad agreement that this is a useful feature) 2. Propose a separate PR to change the default of the new flag to true where we can debate the implications of that change. > Is this true? You need a File to call fsync on, and you can't call fsync after the file is closed. The ObjectStore trait doesn't expose a way to get the file after a write as far as I can tell. This is a good point @adamreeve -- I stand corrected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
