On Mon, Feb 6, 2012 at 4:38 PM, Simon Slavin <slav...@bigfraud.org> wrote: > On 6 Feb 2012, at 9:49pm, Nico Williams wrote: >> Encryption is not enough. You really need block pointers to carry the >> block checksum/hash/MAC/integrity tag. > > File systems (FAT, NTFS, HTFS) already have block checksums. So do network > file systems (AFP, SMB, NFS). Adding more to SQLite, whether at a row or > page level, would catch only cases where the communication was correct at the > storage level but wrong at the semantic level -- which /should/ mean it can > catch /only/ bugs in SQLite.
I don't know of any filesystems other than ZFS, btrfs, and the new one from Microsoft, that store checksums of blocks *with the pointers*. > Problem is, what would you do if a checksum was wrong ? Signal it in a > newly-invented result code ? Then what ? All apps would have to have some > logic to cope with the situation. What should a simple app do about it ? > How many more lines of code would that add to apps which use SQLite specially > because of its low overheads ? If you have redundancy you can use the checksum to find the best copy of a block, or, in the case of a RAID-5-like storage strategy, to reconstruct the correct version of the block. If you can't do any of that, then you can fail. This is painful for the user, but the alternative is to return garbage without anyone knowing it. > I don't think it's worth doing this in a thin/light/fast system like SQLite. > Not only does it all weight to the API, it also adds weight to any app which > uses it. And the number of bugs that cause corruption that would be spotted > this way seems to be low. Instead perhaps it would be better to look into > logic which does simple monitoring at the structure level, e.g. checking that > if a command should add three rows, it really does add three rows. (Although > some sort of analysis of the corruption bugs previously found in SQLite > should be undertaken first.) These can be enabled/disabled with a > compilation flag, whereas a checksum can't because you always need to > calculate it. I would generally prefer to let the filesystem do this. However, Roger has a point that it will be years before a significant number of SQLite3 databases are hosted on filesystems that provide strong end-to-end integrity protection, and that is a good argument for doing the protection in the DB. (Well, it's also a strong argument for asking the OS vendors just when they're going to get off their butts and ship reliable filesystems. But that's another story, and anyways, it's likely all in the works at this time for all major OS vendors.) Nico -- _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
