Preferences are as the name implies intended for preferences. There is no sane use case for storing data in preferences. I would give any patch I come across doing that an automatic sr- for poor taste and general insanity.
SQLite is definitely not cheap, and we should look at more suitable backends for our storage needs, but done right off the main thread, its definitely the saner way to go than (1). While (2) is a foot-gun, (3) is a guaranteed foot-nuke. While its easy to use sqlite wrong, its almost guaranteed that you get your own atomic storage file use wrong, across our N platforms. Chrome is working on replacing sqlite with leveldb for indexeddb and most their storage needs. Last time we looked it wasn't ready for prime time. Maybe it is now. This might be the best option. Andreas On Apr 26, 2013, at 11:17 AM, Gregory Szorc <g...@mozilla.com> wrote: > I'd like to start a discussion about the state of storage in Gecko. > > Currently when you are writing a feature that needs to store data, you > have roughly 3 choices: > > 1) Preferences > 2) SQLite > 3) Manual file I/O > > Preferences are arguably the easiest. However, they have a number of > setbacks: > > a) Poor durability guarantees. See bugs 864537 and 849947 for real-life > issues. tl;dr writes get dropped! > b) Integers limited to 32 bit (JS dates overflow b/c milliseconds since > Unix epoch). > c) I/O is synchronous. > d) The whole method for saving them to disk is kind of weird. > e) The API is awkward. See Preferences.jsm for what I'd consider a > better API. > f) Doesn't scale for non-trivial data sets. > g) Clutters about:config (all preferences aren't config options). > > We have SQLite. You want durability: it's your answer. However, it too > has setbacks: > > a) It eats I/O operations for breakfast. Multiple threads. Lots of > overhead compared to prefs. (But hard to lose data.) > b) By default it's not configured for optimal performance (you need to > enable the WAL, muck around with other PRAGMA). > c) Poor schemas can lead to poor performance. > d) It's often overkill. > e) Storage API has many footguns (use Sqlite.jsm to protect yourself). > f) Lots of effort to do right. Auditing code for 3rd party extensions > using SQLite, many of them aren't doing it right. > > And if one of those pre-built solutions doesn't offer what you need, you > can roll your own with file I/O. But that also has setbacks: > > a) You need to roll your own. (How often do I flush? Do I use many small > files or fewer large files? Different considerations for mobile (slow > I/O) vs desktop?) > b) You need to roll your own. (Listing it twice because it's *really* > annoying, especially for casual developers that just want to implement > features - think add-on developers.) > c) Easy to do wrong (excessive flushing/fsyncing, too many I/O > operations, inefficient appends, poor choices for mobile, etc). > d) Wheel reinvention. Atomic operations/transactions. Data marshaling. etc. > > I believe there is a massive gap between the > easy-but-not-ready-for-prime-time preferences and > the-massive-hammer-solving-the-problem-you-don't-have-and-introducing-many-new-ones > SQLite. Because this gap is full of unknowns, I'm arguing that > developers tend to avoid it and use one of the extremes instead. And, > the result is features that have poor durability and/or poor > performance. Not good. What's worse is many developers (including > myself) are ignorant of many of these pitfalls. Yes, we have code review > for core features. But code review isn't perfect and add-ons likely > aren't subjected to the same level of scrutiny. The end result is the > same: Firefox isn't as awesome as it could be. > > I think there is an opportunity for Gecko to step in and provide a > storage subsystem that is easy to use, somewhere between preferences and > SQLite in terms of durability and performance, and "just works." I don't > think it matters how it is implemented under the hood. If this were to > be built on top of SQLite, I think that would be fine. But, please don't > make consumers worry about things like SQL, schema design, and PRAGMA > statements. So, maybe I'm advocating a generic key-value store. Maybe > something like DOM Storage? Maybe SQLite 4 (which is emphasizing > key-value storage and speed)? Just... something. Please. > > Anyway, I just wanted to see if others have thought about this. Do > others feel it is a concern? If so, can we formulate a plan to address > it? Who would own this? > > Gregory > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
