> On Apr 18, 2017, at 1:50 AM, Deon Brewis <de...@outlook.com> wrote: > > Are you by change using memory mapped IO (MMAP_SIZE something other than 0)? > > This does not work on OSX. Not even remotely. I tracked an issue down in > November 2015, and was able to trivially corrupt a database 100% of the time > using the repro steps below. This happens long after our app gets shut down > and SQLITE is flushed. It got fixed when I set MMAP_SIZE to 0.
That is a hugely important piece of information, and while I thank you for letting me know, I’m shocked that this hasn’t been conveyed to the SQLite developers (or, apparently, to Apple.) Did you report it anywhere? I’m the architect of Couchbase Lite and the lead developer for iOS and macOS. I enabled SQLite’s memory-mapped I/O at least two years ago. I never considered it could be problematic since (a) Brendan is IIRC the only iOS/Mac developer who’s reported database corruption, and (b) I assumed scenarios like this would be covered as part of SQLite testing, either by SQLite themselves or by Apple. > “From the OSX documentation: > Note that while fsync() will flush all data from the host to the drive (i.e. > the "permanent storage device"), the drive itself may not physically write > the data to the platters for quite some time and it may be written in an > out-of-order sequence. Specifically, if the drive loses power or the OS > crashes, the application may find that only some or none of their data was > written. The disk drive may also re-order the data so that later writes may > be present, while earlier writes are not. This is a statement about hard disk controller firmware and is true for any OS. (I used to work for Apple, and corresponded with filesystem architect Dominic Giampaolo about this back in the day.) Some disk controllers don’t flush all cached data to the platters when told to flush, apparently to boost benchmark scores. Darwin has a FULLFSYNC command (an option to ioctl) that triggers an expensive full reset of the disk controller, which does produce a full flush. The filesystem periodically performs FULLFSYNCs to ensure durability of changes to filesystem metadata, so that the filesystem itself can’t become corrupted by power loss. So I’m surprised that, in your test, cutting power after macroscopic amounts of time (2 minutes) have passed since closing the SQLite file still resulted in data being lost. This doesn’t seem like it would be specific to memory-mapped I/O, though. Darwin has a universal buffer cache (like many other kernels) so memory-mapped writes and ordinary file writes are treated the same way by the filesystem cache. Of course it’s possible there’s some kind of OS bug involved here; if so, it seems pretty serious. I’ll be glad to file a bug report with Apple if it turns out to be so. —Jens _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
