On Fri, Apr 26, 2013 at 11:17 AM, Gregory Szorc <g...@mozilla.com> wrote:
> I'd like to start a discussion about the state of storage in Gecko. > > Currently when you are writing a feature that needs to store data, you > have roughly 3 choices: > > 1) Preferences > 2) SQLite > 3) Manual file I/O > > Preferences are arguably the easiest. However, they have a number of > setbacks: > > a) Poor durability guarantees. See bugs 864537 and 849947 for real-life > issues. tl;dr writes get dropped! > b) Integers limited to 32 bit (JS dates overflow b/c milliseconds since > Unix epoch). > c) I/O is synchronous. > d) The whole method for saving them to disk is kind of weird. > e) The API is awkward. See Preferences.jsm for what I'd consider a > better API. > f) Doesn't scale for non-trivial data sets. > g) Clutters about:config (all preferences aren't config options). > > We have SQLite. You want durability: it's your answer. However, it too > has setbacks: > > a) It eats I/O operations for breakfast. Multiple threads. Lots of > overhead compared to prefs. (But hard to lose data.) > b) By default it's not configured for optimal performance (you need to > enable the WAL, muck around with other PRAGMA). > c) Poor schemas can lead to poor performance. > d) It's often overkill. > e) Storage API has many footguns (use Sqlite.jsm to protect yourself). > f) Lots of effort to do right. Auditing code for 3rd party extensions > using SQLite, many of them aren't doing it right. > > And if one of those pre-built solutions doesn't offer what you need, you > can roll your own with file I/O. But that also has setbacks: > > a) You need to roll your own. (How often do I flush? Do I use many small > files or fewer large files? Different considerations for mobile (slow > I/O) vs desktop?) > b) You need to roll your own. (Listing it twice because it's *really* > annoying, especially for casual developers that just want to implement > features - think add-on developers.) > c) Easy to do wrong (excessive flushing/fsyncing, too many I/O > operations, inefficient appends, poor choices for mobile, etc). > d) Wheel reinvention. Atomic operations/transactions. Data marshaling. etc. > > I believe there is a massive gap between the > easy-but-not-ready-for-prime-time preferences and > > the-massive-hammer-solving-the-problem-you-don't-have-and-introducing-many-new-ones > SQLite. Because this gap is full of unknowns, I'm arguing that > developers tend to avoid it and use one of the extremes instead. And, > the result is features that have poor durability and/or poor > performance. Not good. What's worse is many developers (including > myself) are ignorant of many of these pitfalls. Yes, we have code review > for core features. But code review isn't perfect and add-ons likely > aren't subjected to the same level of scrutiny. The end result is the > same: Firefox isn't as awesome as it could be. > > I think there is an opportunity for Gecko to step in and provide a > storage subsystem that is easy to use, somewhere between preferences and > SQLite in terms of durability and performance, and "just works." I don't > think it matters how it is implemented under the hood. If this were to > be built on top of SQLite, I think that would be fine. But, please don't > make consumers worry about things like SQL, schema design, and PRAGMA > statements. So, maybe I'm advocating a generic key-value store. Maybe > something like DOM Storage? Maybe SQLite 4 (which is emphasizing > key-value storage and speed)? Just... something. Please. > > Anyway, I just wanted to see if others have thought about this. Do > others feel it is a concern? If so, can we formulate a plan to address > it? Who would own this? > > Gregory > Have you explored using IndexedDB? - Kyle _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
