Re: Starting work on Indexed DB v2 spec - feedback wanted
Hi, Tim wrote: > Personally, the main thing I want to see is expose simpler and lower > level APIs. For my uses (backend to git server), the leveldb API is > plenty powerful[…] Would it make sense to standardize on a simpler set > of APIs similar to what levelDB offers and expose that in addition to > what indexedDB currently exposes? Or would this make sense as a new > API apart from IDB? I think it would be a mistake to grow the existing IndexedDB API to be more "LevelDB-y". Keep in mind that some IndexedDB implementations are built on top of LevelDB. We don't want to repeat the mistake of Web SQL Database, where the Web-exposed API surface depended on the UA using a particular version of SQLite as its backend. Also, one of the goals of the IndexedDB API was to provide a substrate on which database APIs could be built. Like Joshua wrote: > You may want to try prototyping this on top of Indexed DB as a > library, and see what others think. Apparently this already exists: > The level / node.js community already build a prototype on top of IDB > ( https://github.com/maxogden/level.js ) We should certainly ensure that the IndexedDB API is rich enough to let many such libraries be built on top of it. We should avoid changing it in ways that favor a particular backend implementation strategy. Ted
RE: Starting work on Indexed DB v2 spec - feedback wanted
Speaking from the perspective of an IDB library author Promisification would be high on my list of wants as it was the primary driver behind my libraries existence. The next thing would be function expression filtering, again this is something that my library attempts to solve (https://github.com/aaronpowell/db.js/blob/master/tests/specs/query.js#L543-L552) but has to do it in user space. And to call out specific items from the IndexedDatabaseFeatures [1] it'd be Cancel versionchanged, Database Observers and batch-get's. [1] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures Date: Sat, 19 Apr 2014 02:55:46 -0700 From: rayn...@gmail.com To: jo...@sicking.cc CC: dome...@domenicdenicola.com; jsb...@google.com; t...@creationix.com; public-webapps@w3.org; a...@microsoft.com Subject: Re: Starting work on Indexed DB v2 spec - feedback wanted > especially as your applicationgrows more complex or simply if the user has > your application open in two separate tabs. The two tabs use case is an interesting point. Currently as I understand the leveldb interface avoids this problem by saying "only one process can open a database". Not quite sure what the best way a low level interface can handle this. One valid option is to simply say "a database cant be opened by two tabs". This then requires a developer to use a cross tab communication channel to manually handle synchronization. This enables the default interface to be race free. On Fri, Apr 18, 2014 at 11:30 AM, Jonas Sicking wrote: On Thu, Apr 17, 2014 at 2:04 PM, Domenic Denicola wrote: > From: Joshua Bell >> How much of leveldb's API you consider part of the minimum set - write >> batches? iterators? snapshots? custom comparators? multiple instances per >> application? And are IDB-style keys / serialized script values appropriate, >> or is that extra overhead over e.g. just strings? > > This is my question for Tim as well. My personal hope has always been for > something along the lines of async local storage [1], but that omits a lot > of LevelDB's API and complexity, so presumably it goes too far for LevelDB > proponents. > > [1]: https://github.com/slightlyoff/async-local-storage A big question here is "do you need transactional integrity/atomic mutations?" These things will always make the API more complex and so it gets in the way if you don't need it. But not having them means exposing yourself to race conditions, especially as your application grows more complex or simply if the user has your application open in two separate tabs. My experience is that people need transactional integrity more often than they think they do. The API at [1] punts on transactional integrity entirely. It does not allow you to perform complex operations like "increase the value at 'unreadEmailCount' by one" in a race-free manner. / Jonas
Re: Starting work on Indexed DB v2 spec - feedback wanted
> especially as your application grows more complex or simply if the user has your application open in two separate tabs. The two tabs use case is an interesting point. Currently as I understand the leveldb interface avoids this problem by saying "only one process can open a database". Not quite sure what the best way a low level interface can handle this. One valid option is to simply say "a database cant be opened by two tabs". This then requires a developer to use a cross tab communication channel to manually handle synchronization. This enables the default interface to be race free. On Fri, Apr 18, 2014 at 11:30 AM, Jonas Sicking wrote: > On Thu, Apr 17, 2014 at 2:04 PM, Domenic Denicola > wrote: > > From: Joshua Bell > >> How much of leveldb's API you consider part of the minimum set - write > >> batches? iterators? snapshots? custom comparators? multiple instances > per > >> application? And are IDB-style keys / serialized script values > appropriate, > >> or is that extra overhead over e.g. just strings? > > > > This is my question for Tim as well. My personal hope has always been for > > something along the lines of async local storage [1], but that omits a > lot > > of LevelDB's API and complexity, so presumably it goes too far for > LevelDB > > proponents. > > > > [1]: https://github.com/slightlyoff/async-local-storage > > A big question here is "do you need transactional integrity/atomic > mutations?" These things will always make the API more complex and so > it gets in the way if you don't need it. But not having them means > exposing yourself to race conditions, especially as your application > grows more complex or simply if the user has your application open in > two separate tabs. > > My experience is that people need transactional integrity more often > than they think they do. > > The API at [1] punts on transactional integrity entirely. It does not > allow you to perform complex operations like "increase the value at > 'unreadEmailCount' by one" in a race-free manner. > > / Jonas > >
Re: Starting work on Indexed DB v2 spec - feedback wanted
> A big question here is "do you need transactional integrity/atomic > mutations?" These things will always make the API more complex and so > it gets in the way if you don't need it. But not having them means > exposing yourself to race conditions, especially as your application > grows more complex or simply if the user has your application open in > two separate tabs. > > My experience is that people need transactional integrity more often > than they think they do. > > The API at [1] punts on transactional integrity entirely. It does not > allow you to perform complex operations like "increase the value at > 'unreadEmailCount' by one" in a race-free manner. > > / Jonas Jonas, Totally agree here. However I'd rather have the flexibility to choose between a low level transactionless API and a higher level transaction based API, as each is a valid use case that each comes with tradeoffs. Brendan Eich commented here (https://gist.github.com/maxogden/11031041#comment-1213935) saying "We need atomic transaction building blocks", which I also fully agree with, but to me that means the base layer of functionality should not impose transactions on you, but rather let you opt in to them. Max
Re: Starting work on Indexed DB v2 spec - feedback wanted
On Thu, Apr 17, 2014 at 2:04 PM, Domenic Denicola wrote: > From: Joshua Bell >> How much of leveldb's API you consider part of the minimum set - write >> batches? iterators? snapshots? custom comparators? multiple instances per >> application? And are IDB-style keys / serialized script values appropriate, >> or is that extra overhead over e.g. just strings? > > This is my question for Tim as well. My personal hope has always been for > something along the lines of async local storage [1], but that omits a lot > of LevelDB's API and complexity, so presumably it goes too far for LevelDB > proponents. > > [1]: https://github.com/slightlyoff/async-local-storage A big question here is "do you need transactional integrity/atomic mutations?" These things will always make the API more complex and so it gets in the way if you don't need it. But not having them means exposing yourself to race conditions, especially as your application grows more complex or simply if the user has your application open in two separate tabs. My experience is that people need transactional integrity more often than they think they do. The API at [1] punts on transactional integrity entirely. It does not allow you to perform complex operations like "increase the value at 'unreadEmailCount' by one" in a race-free manner. / Jonas
Re: Starting work on Indexed DB v2 spec - feedback wanted
On Wed, Apr 16, 2014 at 11:49 AM, Joshua Bell wrote: > This is an informal call for feedback to implementers on what is missing > from v1: > > * What features and functionality do you see as important to include? > * How would you prioritize the features? These are somewhat ordered in terms of priority, which is correlated to the value/amount-of-work-to-make-it-happen ratio: 1. At the top of my list is change events. The fact that people end up using localStorage to circumvent the lack of this is sort of ironic given that one of the goals of IDB was to avoid synchronous IO :-) I'm happy to write up a proposal for this. 2. Some low hanging API fruit. Like ObjectStore/Index.getAll, ObjectStore/Index.getAllKeys and ObjectStore.openKeyCursor. Again, happy to write up a proposal for these though they are probably simple enough that a proposal might not be needed. 3. Ability to start and continue an index cursor to a specific primary key. Possibly even create a key range which specifies both index and primary key for both endpoints. 4. Localized sorting. At least the ability to sort an index user-locale order. But probably also the ability to sort an index in a page specified order. 5. Expression indexes. Lots of cool stuff here. Such as creating indexes on the result of a JS expression. And the ability to create an index on the result of a map-reduce of an objectStore. Maybe even being able to get all records that match an expression or update all records using an expression, though here be dragons with footguns. A big missing piece here though is a JS primitive representing a piece of code which can be run side effect free in an efficient way. The only thing we have right now is taking a string which is eval()ed in a new global each time, which is both awkward development-wise and expensive CPU-wise. This is something that we at mozilla will look into, but I can't promise any results given that we might need to add new JS language features in order to make this really good. 6. Promisification. This sounds a lot simpler than it is given that IndexedDB currently relies quite heavily on the exact timing of when result callbacks are executed. This is especially relied on in order to get the auto-comitting transaction behavior to work, which is sometimes good and sometimes bad. And we also use event propagation (event paths) for catching errors in central locations while still allowing those errors to be handled and the rest of database operations run. The current API also enables, and tries to encourage, developers to fire off multiple operations at once and then wait for all of them to have finished. This offers performance advantages over starting operations one at a time and waiting for each to finish before starting the next. All of this is probably possible to maintain even with promises. Possibly promises can even solve it better. And obviously the syntax would be better with promises. But it's not immediately clear to me how to do it. But if someone does the work of putting together a working proposal then I'd be all for it. / Jonas
Re: Starting work on Indexed DB v2 spec - feedback wanted
On Fri, Apr 18, 2014 at 4:08 AM, Dale Harvey wrote: > Our current performance suite is @ > https://github.com/pouchdb/pouchdb/tree/master/tests/performance > > Its at a fairly abstract level above idb, and right now its not particularly > clean, but it should be easy enough to get running, instructions @ > https://github.com/pouchdb/pouchdb/blob/master/CONTRIBUTING.md > > We have only just started on and the tests may not be great representations, > but early signs are chrome and firefox are quite comparable with chrome > being noticeably faster for keyrange queries, and safari being orders of > magnitude faster Thanks. I assume you're using IndexedDB on Firefox and Chrome and WebSQL on Safari? I've opened https://bugzilla.mozilla.org/show_bug.cgi?id=998324 for investigating this on our end. - Kyle
Re: Starting work on Indexed DB v2 spec - feedback wanted
I don't see it on the wiki, so resuggesting storing blobs with partial blobs, as discussed here http://lists.w3.org/Archives/Public/public-webapps/2013OctDec/0657.html Regards Aymeric Le 16/04/2014 20:49, Joshua Bell a écrit : At the April 2014 WebApps WG F2F [1] there was general agreement that moving forward with an Indexed Database "v2" spec was a good idea. Ali Alabbas (Microsoft) has volunteered to co-edit the spec with me. Maintaining compatibility is the highest priority; this will not break the existing API. We've been tracking additional features for quite some time now, both on the wiki [2] and bug tracker [3]. Several are very straightforward (continuePrimaryKey, batch gets, binary keys, ...) and have already been implemented in some user agents, and it will be helpful to document these. Others proposals (URLs, Promises, full text search, ...) are much more complex and will require additional implementation feedback; we plan to add features to the v2 spec based on implementer acceptance. This is an informal call for feedback to implementers on what is missing from v1: * What features and functionality do you see as important to include? * How would you prioritize the features? If there's anything you think is missing from the wiki [2], or want to comment on the importance of a particular feature, please call it out - replying here is great. This will help implementers decide what work to prioritize, which will drive the spec work. We'd also like to keep the v2 cycle shorter than the v1 cycle was, so timely feedback is appreciated - there's always room for a "v3". [1] http://www.w3.org/2014/04/10-webapps-minutes.html [2] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures [3] https://www.w3.org/Bugs/Public/buglist.cgi?bug_status=RESOLVED&component=Indexed%20Database%20API&list_id=34841&product=WebAppsWG&query_format=advanced&resolution=LATER PS: Big thanks to Zhiqiang Zhang for his Indexed DB implementation report, also presented at the F2F. -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: Starting work on Indexed DB v2 spec - feedback wanted
Our current performance suite is @ https://github.com/pouchdb/pouchdb/tree/master/tests/performance Its at a fairly abstract level above idb, and right now its not particularly clean, but it should be easy enough to get running, instructions @ https://github.com/pouchdb/pouchdb/blob/master/CONTRIBUTING.md We have only just started on and the tests may not be great representations, but early signs are chrome and firefox are quite comparable with chrome being noticeably faster for keyrange queries, and safari being orders of magnitude faster On 18 April 2014 07:48, Kyle Huey wrote: > On Thu, Apr 17, 2014 at 5:16 PM, Ben Kelly wrote: > > On 4/17/2014 5:41 PM, Kyle Huey wrote: > >> > >> On Thu, Apr 17, 2014 at 2:10 PM, Dale Harvey > wrote: > >>> > >>> No features that slow it down, as with Tim I also implemented the same > >>> thing > >>> in node.js and see much better perfomance against straight leveldb, > with > >>> websql still being ~5x faster than idb > >> > >> > >> Do you have benchmarks for this? When we've profiled IndexedDB > >> performance for Gaia apps in the past the issue is invariably that the > >> main thread event loop is busy and IndexedDB's responses have to go to > >> the end of a long line. > > > > > > I would hazard a guess that some of SQL's more feature rich constructs > allow > > you to do more in a single API call. This could mean you need to hit the > > event loop less often to accomplish the same amount of work in many > cases. > > > > Just a theory. > > > > Ben > > Yes, that's entirely possible. Which is why I would like to see a > testcase ;-) > > - Kyle >
Re: Starting work on Indexed DB v2 spec - feedback wanted
https://gist.github.com/maxogden/11031041 Replying on behalf of maxogden who can't seem to get access to the mailing list. On Thu, Apr 17, 2014 at 1:56 PM, Joshua Bell wrote: > On Thu, Apr 17, 2014 at 1:22 PM, Tim Caswell wrote: > >> Personally, the main thing I want to see is expose simpler and lower >> level APIs. For my uses (backend to git server), the leveldb API is plenty >> powerful. Most of the time I'm using IndexedDB, I'm getting frustrated >> because it's way more complex than I need and gets in the way and slows >> things down. >> >> Would it make sense to standardize on a simpler set of APIs similar to >> what levelDB offers and expose that in addition to what indexedDB currently >> exposes? Or would this make sense as a new API apart from IDB? >> > > That sounds like a separate storage system to me, although you could > imagine it shares some primitives with Indexed DB (e.g. keys/ordering). > > How much of leveldb's API you consider part of the minimum set - write > batches? iterators? snapshots? custom comparators? multiple instances per > application? And are IDB-style keys / serialized script values appropriate, > or is that extra overhead over e.g. just strings? > > You may want to try prototyping this on top of Indexed DB as a library, > and see what others think. It'd basically just be hiding most of the IDB > API (versions, transactions, stores, indexes) behind functions that return > Promises. > > >> As a JS developer, I'd much rather see fast, simple, yet powerful >> primitives over application-level databases with indexes and transactions >> baked in. Chrome implements IDB on top of LevelDB, so it has just enough >> primitives to make more complex systems. >> >> But for applications like mine that use immutable storage and hashes for >> all lookups don't need or want the advanced features added on top. IDB is >> a serious performance bottleneck in my apps and when using LevelDB in >> node.js, my same logic runs a *lot* faster and using a lot less code. >> >> -Tim Caswell >> >> >> On Wed, Apr 16, 2014 at 1:49 PM, Joshua Bell wrote: >> >>> At the April 2014 WebApps WG F2F [1] there was general agreement that >>> moving forward with an Indexed Database "v2" spec was a good idea. Ali >>> Alabbas (Microsoft) has volunteered to co-edit the spec with me. >>> Maintaining compatibility is the highest priority; this will not break the >>> existing API. >>> >>> We've been tracking additional features for quite some time now, both on >>> the wiki [2] and bug tracker [3]. Several are very straightforward >>> (continuePrimaryKey, batch gets, binary keys, ...) and have already been >>> implemented in some user agents, and it will be helpful to document these. >>> Others proposals (URLs, Promises, full text search, ...) are much more >>> complex and will require additional implementation feedback; we plan to add >>> features to the v2 spec based on implementer acceptance. >>> >>> This is an informal call for feedback to implementers on what is missing >>> from v1: >>> >>> * What features and functionality do you see as important to include? >>> * How would you prioritize the features? >>> >>> If there's anything you think is missing from the wiki [2], or want to >>> comment on the importance of a particular feature, please call it out - >>> replying here is great. This will help implementers decide what work to >>> prioritize, which will drive the spec work. We'd also like to keep the v2 >>> cycle shorter than the v1 cycle was, so timely feedback is appreciated - >>> there's always room for a "v3". >>> >>> [1] http://www.w3.org/2014/04/10-webapps-minutes.html >>> [2] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures >>> [3] >>> https://www.w3.org/Bugs/Public/buglist.cgi?bug_status=RESOLVED&component=Indexed%20Database%20API&list_id=34841&product=WebAppsWG&query_format=advanced&resolution=LATER >>> >>> PS: Big thanks to Zhiqiang Zhang for his Indexed DB implementation >>> report, also presented at the F2F. >>> >> >> >
Re: Starting work on Indexed DB v2 spec - feedback wanted
On Thu, Apr 17, 2014 at 5:16 PM, Ben Kelly wrote: > On 4/17/2014 5:41 PM, Kyle Huey wrote: >> >> On Thu, Apr 17, 2014 at 2:10 PM, Dale Harvey wrote: >>> >>> No features that slow it down, as with Tim I also implemented the same >>> thing >>> in node.js and see much better perfomance against straight leveldb, with >>> websql still being ~5x faster than idb >> >> >> Do you have benchmarks for this? When we've profiled IndexedDB >> performance for Gaia apps in the past the issue is invariably that the >> main thread event loop is busy and IndexedDB's responses have to go to >> the end of a long line. > > > I would hazard a guess that some of SQL's more feature rich constructs allow > you to do more in a single API call. This could mean you need to hit the > event loop less often to accomplish the same amount of work in many cases. > > Just a theory. > > Ben Yes, that's entirely possible. Which is why I would like to see a testcase ;-) - Kyle
Re: Starting work on Indexed DB v2 spec - feedback wanted
Things I think are important:
* asynchronous get/put
* iterators on sorted keys (so we can implement efficient indexes and
nice streaming abstractions)
* batch put/del - not sure if it already has this, but these are super
necessary for implementing custom indexing schemes
* binary keys - combined with sorted key iterators, with binary keys
you can do some really interesting things in userspace with sorting
like typewise: https://www.npmjs.org/package/typewise
* binary values - store bulk data without the overhead of string conversion
* mutation observers - a simple event-based API that emits objects
with { type: 'put|del|batch', key: '...', value: '...' } would do fine
here, like leveldb. This might be a good place for modules to provide
hooks for indexing similar to `ev.preventDefault()` for dom elements.
After preventing the default mutation, the indexing modules could do
batch put/del into the database on their own and then have another
function to indicate back to the caller that triggered the mutation in
the first place that the operation failed or succeeded. Here is a
related module to look at for API inspiration:
https://github.com/dominictarr/level-hooks
Things I don't particularly care about:
* promises
* batched gets
* existence checks
* transactions - these seem like complications
These things should NOT be implemented:
* full text search - This is a very specific kind of index. Just give
us the primitives to implement general-purpose indexes instead.
* IDBKeyRange boolean operators - give userspace enough primitives to
implement indexing ourselves instead of baking in a DSL
A good starting point as Raynos has already mentioned is the leveldown
api, which has proved sufficient to sustain a huge ecosystem of
packages that implement features of traditional databases in a highly
modular way: https://github.com/rvagg/node-leveldown
Re: Starting work on Indexed DB v2 spec - feedback wanted
On 4/17/2014 5:41 PM, Kyle Huey wrote: On Thu, Apr 17, 2014 at 2:10 PM, Dale Harvey wrote: No features that slow it down, as with Tim I also implemented the same thing in node.js and see much better perfomance against straight leveldb, with websql still being ~5x faster than idb Do you have benchmarks for this? When we've profiled IndexedDB performance for Gaia apps in the past the issue is invariably that the main thread event loop is busy and IndexedDB's responses have to go to the end of a long line. I would hazard a guess that some of SQL's more feature rich constructs allow you to do more in a single API call. This could mean you need to hit the event loop less often to accomplish the same amount of work in many cases. Just a theory. Ben
Re: Starting work on Indexed DB v2 spec - feedback wanted
The level / node.js community already build a prototype on top of IDB (
https://github.com/maxogden/level.js )
It supports:
open(name)
get(key) => Uint8Array
del(key)
put(key, Uint8Array)
batch(Array<{ type: 'put' or 'del', key: String, value: Uint8Array>})
iterator({ start: String, key: String, order: 'asc' or 'desc' }) => {
next() => Uint8Array or null,
end()
}
close()
destroy() // cleans up the databases
So it supports async get / del / put + write btaches + iterators.
I dont think it supports snapshots or custom comparators.
It does support multiple instances, i.e. you can open multiple named
databases backed by a file location (which gets mapped to a unique idb db
name).
The values can just be String or Uint8Array and the same for the keys.
The level.js prototype is callback flavored but converting it to promises
would be trivial.
There is value in supporting batch & iterators as those are the primitives
that allow people to build their own transactions and indexes.
On Thu, Apr 17, 2014 at 1:56 PM, Joshua Bell wrote:
> On Thu, Apr 17, 2014 at 1:22 PM, Tim Caswell wrote:
>
>> Personally, the main thing I want to see is expose simpler and lower
>> level APIs. For my uses (backend to git server), the leveldb API is plenty
>> powerful. Most of the time I'm using IndexedDB, I'm getting frustrated
>> because it's way more complex than I need and gets in the way and slows
>> things down.
>>
>> Would it make sense to standardize on a simpler set of APIs similar to
>> what levelDB offers and expose that in addition to what indexedDB currently
>> exposes? Or would this make sense as a new API apart from IDB?
>>
>
> That sounds like a separate storage system to me, although you could
> imagine it shares some primitives with Indexed DB (e.g. keys/ordering).
>
> How much of leveldb's API you consider part of the minimum set - write
> batches? iterators? snapshots? custom comparators? multiple instances per
> application? And are IDB-style keys / serialized script values appropriate,
> or is that extra overhead over e.g. just strings?
>
> You may want to try prototyping this on top of Indexed DB as a library,
> and see what others think. It'd basically just be hiding most of the IDB
> API (versions, transactions, stores, indexes) behind functions that return
> Promises.
>
>
>> As a JS developer, I'd much rather see fast, simple, yet powerful
>> primitives over application-level databases with indexes and transactions
>> baked in. Chrome implements IDB on top of LevelDB, so it has just enough
>> primitives to make more complex systems.
>>
>> But for applications like mine that use immutable storage and hashes for
>> all lookups don't need or want the advanced features added on top. IDB is
>> a serious performance bottleneck in my apps and when using LevelDB in
>> node.js, my same logic runs a *lot* faster and using a lot less code.
>>
>> -Tim Caswell
>>
>>
>> On Wed, Apr 16, 2014 at 1:49 PM, Joshua Bell wrote:
>>
>>> At the April 2014 WebApps WG F2F [1] there was general agreement that
>>> moving forward with an Indexed Database "v2" spec was a good idea. Ali
>>> Alabbas (Microsoft) has volunteered to co-edit the spec with me.
>>> Maintaining compatibility is the highest priority; this will not break the
>>> existing API.
>>>
>>> We've been tracking additional features for quite some time now, both on
>>> the wiki [2] and bug tracker [3]. Several are very straightforward
>>> (continuePrimaryKey, batch gets, binary keys, ...) and have already been
>>> implemented in some user agents, and it will be helpful to document these.
>>> Others proposals (URLs, Promises, full text search, ...) are much more
>>> complex and will require additional implementation feedback; we plan to add
>>> features to the v2 spec based on implementer acceptance.
>>>
>>> This is an informal call for feedback to implementers on what is missing
>>> from v1:
>>>
>>> * What features and functionality do you see as important to include?
>>> * How would you prioritize the features?
>>>
>>> If there's anything you think is missing from the wiki [2], or want to
>>> comment on the importance of a particular feature, please call it out -
>>> replying here is great. This will help implementers decide what work to
>>> prioritize, which will drive the spec work. We'd also like to keep the v2
>>> cycle shorter than the v1 cycle was, so timely feedback is appreciated -
>>> there's always room for a "v3".
>>>
>>> [1] http://www.w3.org/2014/04/10-webapps-minutes.html
>>> [2] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures
>>> [3]
>>> https://www.w3.org/Bugs/Public/buglist.cgi?bug_status=RESOLVED&component=Indexed%20Database%20API&list_id=34841&product=WebAppsWG&query_format=advanced&resolution=LATER
>>>
>>> PS: Big thanks to Zhiqiang Zhang for his Indexed DB implementation
>>> report, also presented at the F2F.
>>>
>>
>>
>
Re: Starting work on Indexed DB v2 spec - feedback wanted
In my mind, promisifying the IndexedDB API would be the highest value thing to end users. After using a very early version of localForage ( https://github.com/mozilla/localForage), I started toying with a lower level, promise-based, API wrapper with my EasyIndexedDB library. Here's some example usage: https://github.com/ebryn/EasyIndexedDB/blob/master/test/basics_test.js#L42 I'd be happy to help provide continual feedback from a framework developer (Ember.js) & end users perspective. - Erik On Wed, Apr 16, 2014 at 11:49 AM, Joshua Bell wrote: > At the April 2014 WebApps WG F2F [1] there was general agreement that > moving forward with an Indexed Database "v2" spec was a good idea. Ali > Alabbas (Microsoft) has volunteered to co-edit the spec with me. > Maintaining compatibility is the highest priority; this will not break the > existing API. > > We've been tracking additional features for quite some time now, both on > the wiki [2] and bug tracker [3]. Several are very straightforward > (continuePrimaryKey, batch gets, binary keys, ...) and have already been > implemented in some user agents, and it will be helpful to document these. > Others proposals (URLs, Promises, full text search, ...) are much more > complex and will require additional implementation feedback; we plan to add > features to the v2 spec based on implementer acceptance. > > This is an informal call for feedback to implementers on what is missing > from v1: > > * What features and functionality do you see as important to include? > * How would you prioritize the features? > > If there's anything you think is missing from the wiki [2], or want to > comment on the importance of a particular feature, please call it out - > replying here is great. This will help implementers decide what work to > prioritize, which will drive the spec work. We'd also like to keep the v2 > cycle shorter than the v1 cycle was, so timely feedback is appreciated - > there's always room for a "v3". > > [1] http://www.w3.org/2014/04/10-webapps-minutes.html > [2] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures > [3] > https://www.w3.org/Bugs/Public/buglist.cgi?bug_status=RESOLVED&component=Indexed%20Database%20API&list_id=34841&product=WebAppsWG&query_format=advanced&resolution=LATER > > PS: Big thanks to Zhiqiang Zhang for his Indexed DB implementation report, > also presented at the F2F. >
Re: Starting work on Indexed DB v2 spec - feedback wanted
For my personal use case, I don't need much at all. I need the ability to store binary data by key and retrieve binary data by key. It would be nice if there was a quick existence check to see if a hash is already in the db since I'm using content-addressable structures. But again, only for performance reasons. If that wasn't there, I would just do a read and ignore the body. For the mutable half of a git db, I need basic string key with string value and some iterating on subsets. For example, I want to find all refs that start with "refs/heads/" to get the list of branches on a repo. But this metadata is tiny in comparison to everything else. A large git db may have hundreds of megabytes of git objects and less than 1000 refs for all the branches, tags, pull-requests, etc. The value of these refs is a string usually 41 bytes long. If the db didn't provide sorted and iteratable keys, I would just store all the metadata in a single object and replace it whenever I needed to change any part of it. Updates to this data is infrequent in my case. Ideally I'd have two databases. One for immutable binary values with fixed-length 20-byte binary (or 40 byte string) keys. This wouldn't need to be atomic at all. It would have: get(hash) => binary set(hash, binary) has(hash) => boolean del(hash) And one for the mutable metadata where the keys are strings and the values are strings. get(key) => string set(key, string) getRange(prefix) => object of key/value pairs that match the prefix del(key) This would all be async using ES6 promises or node-style callbacks or whatever. The reason I suggested LevelDB is because it's a nice medium between my minimal needs and what's needed to implement a more complex database in pure JS. On Thu, Apr 17, 2014 at 4:04 PM, Domenic Denicola < dome...@domenicdenicola.com> wrote: > *From:* Joshua Bell > >> How much of leveldb's API you consider part of the minimum set - > write batches? iterators? snapshots? custom comparators? multiple instances > per application? And are IDB-style keys / serialized script values > appropriate, or is that extra overhead over e.g. just strings? > > This is my question for Tim as well. My personal hope has always been for > something along the lines of async local storage [1], but that omits a lot > of LevelDB's API and complexity, so presumably it goes too far for LevelDB > proponents. > > [1]: https://github.com/slightlyoff/async-local-storage >
Re: Starting work on Indexed DB v2 spec - feedback wanted
On Thu, Apr 17, 2014 at 2:10 PM, Dale Harvey wrote: > No features that slow it down, as with Tim I also implemented the same thing > in node.js and see much better perfomance against straight leveldb, with > websql still being ~5x faster than idb Do you have benchmarks for this? When we've profiled IndexedDB performance for Gaia apps in the past the issue is invariably that the main thread event loop is busy and IndexedDB's responses have to go to the end of a long line. - Kyle
Re: Starting work on Indexed DB v2 spec - feedback wanted
My IndexedDB wishlist:
Ability to enumerate databases, I dont particularly want or care about the
transactional integrity of the api, if someone deletes a database while I
am in a callback in which I think it exists, meh
Change events / Observers, right now I have to fake it via localstorage
A transactional model that isnt tied to the event loop, sometimes I want to
do async things inside the transaction like converting to an ArrayBuffer
etc, I would like to open with an option to have the transaction stay open
till its explicitly closed
No features that slow it down, as with Tim I also implemented the same
thing in node.js and see much better perfomance against straight leveldb,
with websql still being ~5x faster than idb
I dont have any concrete suggestions, but making the transactional states
more visible would help, I have seen pretty much everyone errantly use
.put().onsuccess(function() { yay do stuff, to either lose data by assuming
the write is completed or get in a confusing state trying to open new
transactions. (see step 3 -
http://www.html5rocks.com/en/tutorials/indexeddb/todo/)
I think its also worth (and fairly trivial) to implement sugar syntax for a
key value storage like https://github.com/mozilla/localForage, as a large
amount of usage I have seen just wants to store some simple data and not
deal with transactions, object stores and schema migrations
Cheers
Dale
On 17 April 2014 21:22, Tim Caswell wrote:
> Personally, the main thing I want to see is expose simpler and lower level
> APIs. For my uses (backend to git server), the leveldb API is plenty
> powerful. Most of the time I'm using IndexedDB, I'm getting frustrated
> because it's way more complex than I need and gets in the way and slows
> things down.
>
> Would it make sense to standardize on a simpler set of APIs similar to
> what levelDB offers and expose that in addition to what indexedDB currently
> exposes? Or would this make sense as a new API apart from IDB?
>
> As a JS developer, I'd much rather see fast, simple, yet powerful
> primitives over application-level databases with indexes and transactions
> baked in. Chrome implements IDB on top of LevelDB, so it has just enough
> primitives to make more complex systems.
>
> But for applications like mine that use immutable storage and hashes for
> all lookups don't need or want the advanced features added on top. IDB is
> a serious performance bottleneck in my apps and when using LevelDB in
> node.js, my same logic runs a *lot* faster and using a lot less code.
>
> -Tim Caswell
>
>
> On Wed, Apr 16, 2014 at 1:49 PM, Joshua Bell wrote:
>
>> At the April 2014 WebApps WG F2F [1] there was general agreement that
>> moving forward with an Indexed Database "v2" spec was a good idea. Ali
>> Alabbas (Microsoft) has volunteered to co-edit the spec with me.
>> Maintaining compatibility is the highest priority; this will not break the
>> existing API.
>>
>> We've been tracking additional features for quite some time now, both on
>> the wiki [2] and bug tracker [3]. Several are very straightforward
>> (continuePrimaryKey, batch gets, binary keys, ...) and have already been
>> implemented in some user agents, and it will be helpful to document these.
>> Others proposals (URLs, Promises, full text search, ...) are much more
>> complex and will require additional implementation feedback; we plan to add
>> features to the v2 spec based on implementer acceptance.
>>
>> This is an informal call for feedback to implementers on what is missing
>> from v1:
>>
>> * What features and functionality do you see as important to include?
>> * How would you prioritize the features?
>>
>> If there's anything you think is missing from the wiki [2], or want to
>> comment on the importance of a particular feature, please call it out -
>> replying here is great. This will help implementers decide what work to
>> prioritize, which will drive the spec work. We'd also like to keep the v2
>> cycle shorter than the v1 cycle was, so timely feedback is appreciated -
>> there's always room for a "v3".
>>
>> [1] http://www.w3.org/2014/04/10-webapps-minutes.html
>> [2] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures
>> [3]
>> https://www.w3.org/Bugs/Public/buglist.cgi?bug_status=RESOLVED&component=Indexed%20Database%20API&list_id=34841&product=WebAppsWG&query_format=advanced&resolution=LATER
>>
>> PS: Big thanks to Zhiqiang Zhang for his Indexed DB implementation
>> report, also presented at the F2F.
>>
>
>
RE: Starting work on Indexed DB v2 spec - feedback wanted
From: Joshua Bell > How much of leveldb's API you consider part of the minimum set - write > batches? iterators? snapshots? custom comparators? multiple instances per > application? And are IDB-style keys / serialized script values appropriate, > or is that extra overhead over e.g. just strings? This is my question for Tim as well. My personal hope has always been for something along the lines of async local storage [1], but that omits a lot of LevelDB's API and complexity, so presumably it goes too far for LevelDB proponents. [1]: https://github.com/slightlyoff/async-local-storage
Re: Starting work on Indexed DB v2 spec - feedback wanted
On Thu, Apr 17, 2014 at 1:22 PM, Tim Caswell wrote: > Personally, the main thing I want to see is expose simpler and lower level > APIs. For my uses (backend to git server), the leveldb API is plenty > powerful. Most of the time I'm using IndexedDB, I'm getting frustrated > because it's way more complex than I need and gets in the way and slows > things down. > > Would it make sense to standardize on a simpler set of APIs similar to > what levelDB offers and expose that in addition to what indexedDB currently > exposes? Or would this make sense as a new API apart from IDB? > That sounds like a separate storage system to me, although you could imagine it shares some primitives with Indexed DB (e.g. keys/ordering). How much of leveldb's API you consider part of the minimum set - write batches? iterators? snapshots? custom comparators? multiple instances per application? And are IDB-style keys / serialized script values appropriate, or is that extra overhead over e.g. just strings? You may want to try prototyping this on top of Indexed DB as a library, and see what others think. It'd basically just be hiding most of the IDB API (versions, transactions, stores, indexes) behind functions that return Promises. > As a JS developer, I'd much rather see fast, simple, yet powerful > primitives over application-level databases with indexes and transactions > baked in. Chrome implements IDB on top of LevelDB, so it has just enough > primitives to make more complex systems. > > But for applications like mine that use immutable storage and hashes for > all lookups don't need or want the advanced features added on top. IDB is > a serious performance bottleneck in my apps and when using LevelDB in > node.js, my same logic runs a *lot* faster and using a lot less code. > > -Tim Caswell > > > On Wed, Apr 16, 2014 at 1:49 PM, Joshua Bell wrote: > >> At the April 2014 WebApps WG F2F [1] there was general agreement that >> moving forward with an Indexed Database "v2" spec was a good idea. Ali >> Alabbas (Microsoft) has volunteered to co-edit the spec with me. >> Maintaining compatibility is the highest priority; this will not break the >> existing API. >> >> We've been tracking additional features for quite some time now, both on >> the wiki [2] and bug tracker [3]. Several are very straightforward >> (continuePrimaryKey, batch gets, binary keys, ...) and have already been >> implemented in some user agents, and it will be helpful to document these. >> Others proposals (URLs, Promises, full text search, ...) are much more >> complex and will require additional implementation feedback; we plan to add >> features to the v2 spec based on implementer acceptance. >> >> This is an informal call for feedback to implementers on what is missing >> from v1: >> >> * What features and functionality do you see as important to include? >> * How would you prioritize the features? >> >> If there's anything you think is missing from the wiki [2], or want to >> comment on the importance of a particular feature, please call it out - >> replying here is great. This will help implementers decide what work to >> prioritize, which will drive the spec work. We'd also like to keep the v2 >> cycle shorter than the v1 cycle was, so timely feedback is appreciated - >> there's always room for a "v3". >> >> [1] http://www.w3.org/2014/04/10-webapps-minutes.html >> [2] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures >> [3] >> https://www.w3.org/Bugs/Public/buglist.cgi?bug_status=RESOLVED&component=Indexed%20Database%20API&list_id=34841&product=WebAppsWG&query_format=advanced&resolution=LATER >> >> PS: Big thanks to Zhiqiang Zhang for his Indexed DB implementation >> report, also presented at the F2F. >> > >
Re: Starting work on Indexed DB v2 spec - feedback wanted
Personally, the main thing I want to see is expose simpler and lower level APIs. For my uses (backend to git server), the leveldb API is plenty powerful. Most of the time I'm using IndexedDB, I'm getting frustrated because it's way more complex than I need and gets in the way and slows things down. Would it make sense to standardize on a simpler set of APIs similar to what levelDB offers and expose that in addition to what indexedDB currently exposes? Or would this make sense as a new API apart from IDB? As a JS developer, I'd much rather see fast, simple, yet powerful primitives over application-level databases with indexes and transactions baked in. Chrome implements IDB on top of LevelDB, so it has just enough primitives to make more complex systems. But for applications like mine that use immutable storage and hashes for all lookups don't need or want the advanced features added on top. IDB is a serious performance bottleneck in my apps and when using LevelDB in node.js, my same logic runs a *lot* faster and using a lot less code. -Tim Caswell On Wed, Apr 16, 2014 at 1:49 PM, Joshua Bell wrote: > At the April 2014 WebApps WG F2F [1] there was general agreement that > moving forward with an Indexed Database "v2" spec was a good idea. Ali > Alabbas (Microsoft) has volunteered to co-edit the spec with me. > Maintaining compatibility is the highest priority; this will not break the > existing API. > > We've been tracking additional features for quite some time now, both on > the wiki [2] and bug tracker [3]. Several are very straightforward > (continuePrimaryKey, batch gets, binary keys, ...) and have already been > implemented in some user agents, and it will be helpful to document these. > Others proposals (URLs, Promises, full text search, ...) are much more > complex and will require additional implementation feedback; we plan to add > features to the v2 spec based on implementer acceptance. > > This is an informal call for feedback to implementers on what is missing > from v1: > > * What features and functionality do you see as important to include? > * How would you prioritize the features? > > If there's anything you think is missing from the wiki [2], or want to > comment on the importance of a particular feature, please call it out - > replying here is great. This will help implementers decide what work to > prioritize, which will drive the spec work. We'd also like to keep the v2 > cycle shorter than the v1 cycle was, so timely feedback is appreciated - > there's always room for a "v3". > > [1] http://www.w3.org/2014/04/10-webapps-minutes.html > [2] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures > [3] > https://www.w3.org/Bugs/Public/buglist.cgi?bug_status=RESOLVED&component=Indexed%20Database%20API&list_id=34841&product=WebAppsWG&query_format=advanced&resolution=LATER > > PS: Big thanks to Zhiqiang Zhang for his Indexed DB implementation report, > also presented at the F2F. >
Starting work on Indexed DB v2 spec - feedback wanted
At the April 2014 WebApps WG F2F [1] there was general agreement that moving forward with an Indexed Database "v2" spec was a good idea. Ali Alabbas (Microsoft) has volunteered to co-edit the spec with me. Maintaining compatibility is the highest priority; this will not break the existing API. We've been tracking additional features for quite some time now, both on the wiki [2] and bug tracker [3]. Several are very straightforward (continuePrimaryKey, batch gets, binary keys, ...) and have already been implemented in some user agents, and it will be helpful to document these. Others proposals (URLs, Promises, full text search, ...) are much more complex and will require additional implementation feedback; we plan to add features to the v2 spec based on implementer acceptance. This is an informal call for feedback to implementers on what is missing from v1: * What features and functionality do you see as important to include? * How would you prioritize the features? If there's anything you think is missing from the wiki [2], or want to comment on the importance of a particular feature, please call it out - replying here is great. This will help implementers decide what work to prioritize, which will drive the spec work. We'd also like to keep the v2 cycle shorter than the v1 cycle was, so timely feedback is appreciated - there's always room for a "v3". [1] http://www.w3.org/2014/04/10-webapps-minutes.html [2] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures [3] https://www.w3.org/Bugs/Public/buglist.cgi?bug_status=RESOLVED&component=Indexed%20Database%20API&list_id=34841&product=WebAppsWG&query_format=advanced&resolution=LATER PS: Big thanks to Zhiqiang Zhang for his Indexed DB implementation report, also presented at the F2F.
