On Fri, Apr 26, 2013 at 11:43 AM, Gregory Szorc <g...@mozilla.com> wrote:
> On 4/26/2013 11:26 AM, Kyle Huey wrote: > > On Fri, Apr 26, 2013 at 11:17 AM, Gregory Szorc <g...@mozilla.com> wrote: > >> I'd like to start a discussion about the state of storage in Gecko. >> >> Currently when you are writing a feature that needs to store data, you >> have roughly 3 choices: >> >> 1) Preferences >> 2) SQLite >> 3) Manual file I/O >> >> Preferences are arguably the easiest. However, they have a number of >> setbacks: >> >> a) Poor durability guarantees. See bugs 864537 and 849947 for real-life >> issues. tl;dr writes get dropped! >> b) Integers limited to 32 bit (JS dates overflow b/c milliseconds since >> Unix epoch). >> c) I/O is synchronous. >> d) The whole method for saving them to disk is kind of weird. >> e) The API is awkward. See Preferences.jsm for what I'd consider a >> better API. >> f) Doesn't scale for non-trivial data sets. >> g) Clutters about:config (all preferences aren't config options). >> >> We have SQLite. You want durability: it's your answer. However, it too >> has setbacks: >> >> a) It eats I/O operations for breakfast. Multiple threads. Lots of >> overhead compared to prefs. (But hard to lose data.) >> b) By default it's not configured for optimal performance (you need to >> enable the WAL, muck around with other PRAGMA). >> c) Poor schemas can lead to poor performance. >> d) It's often overkill. >> e) Storage API has many footguns (use Sqlite.jsm to protect yourself). >> f) Lots of effort to do right. Auditing code for 3rd party extensions >> using SQLite, many of them aren't doing it right. >> >> And if one of those pre-built solutions doesn't offer what you need, you >> can roll your own with file I/O. But that also has setbacks: >> >> a) You need to roll your own. (How often do I flush? Do I use many small >> files or fewer large files? Different considerations for mobile (slow >> I/O) vs desktop?) >> b) You need to roll your own. (Listing it twice because it's *really* >> annoying, especially for casual developers that just want to implement >> features - think add-on developers.) >> c) Easy to do wrong (excessive flushing/fsyncing, too many I/O >> operations, inefficient appends, poor choices for mobile, etc). >> d) Wheel reinvention. Atomic operations/transactions. Data marshaling. >> etc. >> >> I believe there is a massive gap between the >> easy-but-not-ready-for-prime-time preferences and >> >> the-massive-hammer-solving-the-problem-you-don't-have-and-introducing-many-new-ones >> SQLite. Because this gap is full of unknowns, I'm arguing that >> developers tend to avoid it and use one of the extremes instead. And, >> the result is features that have poor durability and/or poor >> performance. Not good. What's worse is many developers (including >> myself) are ignorant of many of these pitfalls. Yes, we have code review >> for core features. But code review isn't perfect and add-ons likely >> aren't subjected to the same level of scrutiny. The end result is the >> same: Firefox isn't as awesome as it could be. >> >> I think there is an opportunity for Gecko to step in and provide a >> storage subsystem that is easy to use, somewhere between preferences and >> SQLite in terms of durability and performance, and "just works." I don't >> think it matters how it is implemented under the hood. If this were to >> be built on top of SQLite, I think that would be fine. But, please don't >> make consumers worry about things like SQL, schema design, and PRAGMA >> statements. So, maybe I'm advocating a generic key-value store. Maybe >> something like DOM Storage? Maybe SQLite 4 (which is emphasizing >> key-value storage and speed)? Just... something. Please. >> >> Anyway, I just wanted to see if others have thought about this. Do >> others feel it is a concern? If so, can we formulate a plan to address >> it? Who would own this? >> >> Gregory >> > > Have you explored using IndexedDB? > > > Not seriously. The "this is an experimental technology" warning on MDN is > off-putting. > Yeah that's not accurate. It's pretty solid now. It's the storage backend for everything in b2g for instance ... and it's not going to see any changes that aren't backwards compatible. > Is IndexedDB ready for use by internal Gecko consumers, including add-ons? > Yes. Can you use it from the main thread (for cases where a dedicated worker is > too heavy weight)? > You can only use it from the main thread right now. Worker support is coming (Bug 798875). > What are current risks associated with this "experimental technology?" > It's not experimental. Aside from those concerns, I reckon IndexedDB could fill the gap I > described. I want to learn more. > Always happy to answer questions. - Kyle _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
