Re: [Bug 12321] New: Add compound keys to IndexedDB
On Fri, Mar 18, 2011 at 1:45 AM, Keean Schupke wrote: > I like BDB's solution. You have one primary key you cannot mess with (say > an integer for fast comparisons) you can then add any number of secondary > indexes. With a secondary index there is a callback to generate a binary > blob that is used for indexing. The callback has access to all the fields of > the object plus any info in the closure and can use that to generate the > index data any way it likes. We discussed this a while ago. IIRC, we decided to look at something like it for v2. It sounds like a good, general way to solve the problem though. And given the other discussion in this thread, it sounds like maybe this isn't a super important use case to fix in the mean time. J > This has the advantage of supporting any indexing scheme's the user may > wish to implement (by writing a custom callback), whist allowing a few > common options to be provided for the user (say a hash of all fields, or a > field name, international char set, and direction captured in a closure). > The user gets the power, the core implementation is simple, and common cases > can be implemented in an easy to use way. > > var lex_order = function(field, charset, direction) {return > function(object) {/* map indexed 'field' to blob in required order */ return > key;};}; > > Then create a new index: > > object_store.validate_index(1, lex_order('name', 'us', > 'ascending')).on_done(function(status) {/* status ok or error */}) > > validate index checks if the requested secondary index (1) exists, if it > does not it creates the index and calls the done callback (with a status > code indicating successful creation), if it does and it passes some > validation checks it also calls the done callback (with a status code > indicating successful validation). If anything goes wrong with either the > creation or validation of the secondary index if would call the done > callback with an error status code. > > > Cheers, > Keean. > > > On 18 March 2011 02:03, Jeremy Orlow wrote: > >> Here's one ugliness with A: There's no way to specify ascending >> or descending for the individual components of the key. So there's no way >> for me to open a cursor that looks at one field ascending and the other >> field descending. In addition, I can't think of any easy/good ways to hack >> around this. >> >> Any thoughts on how we could address this use case? >> >> J >> >> On Wed, Mar 16, 2011 at 4:50 PM, wrote: >> >>> http://www.w3.org/Bugs/Public/show_bug.cgi?id=12321 >>> >>> Summary: Add compound keys to IndexedDB >>> Product: WebAppsWG >>> Version: unspecified >>> Platform: PC >>>OS/Version: All >>>Status: NEW >>> Severity: normal >>> Priority: P2 >>> Component: Indexed Database API >>>AssignedTo: dave.n...@w3.org >>>ReportedBy: jor...@chromium.org >>> QAContact: member-webapi-...@w3.org >>>CC: m...@w3.org, public-webapps@w3.org >>> >>> >>> >From the thread "[IndexedDB] Compound and multiple keys" by Jonas >>> Sicking, >>> we're going to go with both options A and B. >>> >>> = >>> >>> Hi IndexedDB fans (yay!!), >>> >>> Problem description: >>> >>> One of the current shortcomings of IndexedDB is that it doesn't >>> support compound indexes. I.e. indexing on more than one value. For >>> example it's impossible to index on, and therefor efficiently search >>> for, firstname and lastname in an objectStore which stores people. Or >>> index on to-address and date sent in an objectStore holding emails. >>> >>> The way this is traditionally done is that multiple values are used as >>> key for each individual entry in an index or objectStore. For example >>> the CREATE INDEX statement in SQL can list multiple columns, and >>> CREATE TABLE statment can list several columns as PRIMARY KEY. >>> >>> There have been a couple of suggestions how to do this in IndexedDB >>> >>> Option A) >>> When specifying a key path in createObjectStore and createIndex, allow >>> an array of key-paths to be specified. Such as >>> >>> store = db.createObjectStore("mystore", ["firstName", "lastName"]); >>> store.add({firstName: "Benny", lastName: "Zysk", age: 28}); >>> store.add({firstName: "Benny", lastName: "Andersson", age: 63}); >>> store.add({firstName: "Charlie", lastName: "Brown", age: 8}); >>> >>> The records are stored in the following order >>> "Benny", "Andersson" >>> "Benny", "Zysk" >>> "Charlie", "Brown" >>> >>> Similarly, createIndex accepts the same syntax: >>> store.createIndex("myindex", ["lastName", "age"]); >>> >>> Option B) >>> Allowing arrays as an additional data type for keys. >>> store = db.createObjectStore("mystore", "fullName"); >>> store.add({fullName: ["Benny", "Zysk"], age: 28}); >>> store.add({fullName: ["Benny", "Andersson"], age: 63}); >>> store.add({fullName: ["Charlie", "Brown"], age: 8}); >>> >>> Also allows out-of-line keys using: >>>
Re: [Bug 12321] New: Add compound keys to IndexedDB
On Fri, Mar 18, 2011 at 12:51 PM, Glenn Maynard wrote: > Most SQL engines (Postgresql, SQLite) support "CREATE INDEX idx ON tbl (date > DESC, name ASC)" . This allows "ORDER BY date DESC, name ASC" (eg. "newest > events first, events per date sorted by name") and its reverse, "name DESC, > date ASC". MySQL is an outlier in not supporting this. In this as in many things. Nevertheless, in my experience using MySQL, it's rarely a problem, so solving it in a first pass is perhaps not essential. If you're going to solve it, I suggest allowing array keys to be a dictionary, like store = db.createObjectStore("mystore", [{col: "firstName", dir: "asc"}, {col: "lastName", dir: "desc"}]); where "foo" is equivalent to {col: "foo"} or whatever. This will be useful not just for sorting, but for any per-column option in the index, such as what collation you want to use for text (which is essential for international sorting). If this syntax is to be used, though, the current syntax would remain compatible, so column options can be pushed off to a later version of the standard.
Re: [Bug 12321] New: Add compound keys to IndexedDB
On Fri, Mar 18, 2011 at 12:27 PM, Aryeh Gregor wrote: > On Thu, Mar 17, 2011 at 10:03 PM, Jeremy Orlow > wrote: > > Here's one ugliness with A: There's no way to specify ascending > > or descending for the individual components of the key. So there's no > way > > for me to open a cursor that looks at one field ascending and the other > > field descending. In addition, I can't think of any easy/good ways to > hack > > around this. > > Any thoughts on how we could address this use case? > > For what it's worth, the way MySQL does it is it doesn't. If you have > an index on (a, b), then it can be used for ORDER BY a, b or ORDER BY > a DESC, b DESC, but not ORDER BY a DESC, b or ORDER BY a, b DESC. In > practice this usually works fine -- it's pretty rare that you really > want to sort different columns in a different order. > Most SQL engines (Postgresql, SQLite) support "CREATE INDEX idx ON tbl (date DESC, name ASC)" . This allows "ORDER BY date DESC, name ASC" (eg. "newest events first, events per date sorted by name") and its reverse, "name DESC, date ASC". MySQL is an outlier in not supporting this. -- Glenn Maynard
Re: [Bug 12321] New: Add compound keys to IndexedDB
On Thu, Mar 17, 2011 at 10:03 PM, Jeremy Orlow wrote: > Here's one ugliness with A: There's no way to specify ascending > or descending for the individual components of the key. So there's no way > for me to open a cursor that looks at one field ascending and the other > field descending. In addition, I can't think of any easy/good ways to hack > around this. > Any thoughts on how we could address this use case? For what it's worth, the way MySQL does it is it doesn't. If you have an index on (a, b), then it can be used for ORDER BY a, b or ORDER BY a DESC, b DESC, but not ORDER BY a DESC, b or ORDER BY a, b DESC. In practice this usually works fine -- it's pretty rare that you really want to sort different columns in a different order.
Re: [Bug 12321] New: Add compound keys to IndexedDB
I like BDB's solution. You have one primary key you cannot mess with (say an integer for fast comparisons) you can then add any number of secondary indexes. With a secondary index there is a callback to generate a binary blob that is used for indexing. The callback has access to all the fields of the object plus any info in the closure and can use that to generate the index data any way it likes. This has the advantage of supporting any indexing scheme's the user may wish to implement (by writing a custom callback), whist allowing a few common options to be provided for the user (say a hash of all fields, or a field name, international char set, and direction captured in a closure). The user gets the power, the core implementation is simple, and common cases can be implemented in an easy to use way. var lex_order = function(field, charset, direction) {return function(object) {/* map indexed 'field' to blob in required order */ return key;};}; Then create a new index: object_store.validate_index(1, lex_order('name', 'us', 'ascending')).on_done(function(status) {/* status ok or error */}) validate index checks if the requested secondary index (1) exists, if it does not it creates the index and calls the done callback (with a status code indicating successful creation), if it does and it passes some validation checks it also calls the done callback (with a status code indicating successful validation). If anything goes wrong with either the creation or validation of the secondary index if would call the done callback with an error status code. Cheers, Keean. On 18 March 2011 02:03, Jeremy Orlow wrote: > Here's one ugliness with A: There's no way to specify ascending > or descending for the individual components of the key. So there's no way > for me to open a cursor that looks at one field ascending and the other > field descending. In addition, I can't think of any easy/good ways to hack > around this. > > Any thoughts on how we could address this use case? > > J > > On Wed, Mar 16, 2011 at 4:50 PM, wrote: > >> http://www.w3.org/Bugs/Public/show_bug.cgi?id=12321 >> >> Summary: Add compound keys to IndexedDB >> Product: WebAppsWG >> Version: unspecified >> Platform: PC >>OS/Version: All >>Status: NEW >> Severity: normal >> Priority: P2 >> Component: Indexed Database API >>AssignedTo: dave.n...@w3.org >>ReportedBy: jor...@chromium.org >> QAContact: member-webapi-...@w3.org >>CC: m...@w3.org, public-webapps@w3.org >> >> >> >From the thread "[IndexedDB] Compound and multiple keys" by Jonas >> Sicking, >> we're going to go with both options A and B. >> >> = >> >> Hi IndexedDB fans (yay!!), >> >> Problem description: >> >> One of the current shortcomings of IndexedDB is that it doesn't >> support compound indexes. I.e. indexing on more than one value. For >> example it's impossible to index on, and therefor efficiently search >> for, firstname and lastname in an objectStore which stores people. Or >> index on to-address and date sent in an objectStore holding emails. >> >> The way this is traditionally done is that multiple values are used as >> key for each individual entry in an index or objectStore. For example >> the CREATE INDEX statement in SQL can list multiple columns, and >> CREATE TABLE statment can list several columns as PRIMARY KEY. >> >> There have been a couple of suggestions how to do this in IndexedDB >> >> Option A) >> When specifying a key path in createObjectStore and createIndex, allow >> an array of key-paths to be specified. Such as >> >> store = db.createObjectStore("mystore", ["firstName", "lastName"]); >> store.add({firstName: "Benny", lastName: "Zysk", age: 28}); >> store.add({firstName: "Benny", lastName: "Andersson", age: 63}); >> store.add({firstName: "Charlie", lastName: "Brown", age: 8}); >> >> The records are stored in the following order >> "Benny", "Andersson" >> "Benny", "Zysk" >> "Charlie", "Brown" >> >> Similarly, createIndex accepts the same syntax: >> store.createIndex("myindex", ["lastName", "age"]); >> >> Option B) >> Allowing arrays as an additional data type for keys. >> store = db.createObjectStore("mystore", "fullName"); >> store.add({fullName: ["Benny", "Zysk"], age: 28}); >> store.add({fullName: ["Benny", "Andersson"], age: 63}); >> store.add({fullName: ["Charlie", "Brown"], age: 8}); >> >> Also allows out-of-line keys using: >> store = db.createObjectStore("mystore"); >> store.add({age: 28}, ["Benny", "Zysk"]); >> store.add({age: 63}, ["Benny", "Andersson"]); >> store.add({age: 8}, ["Charlie", "Brown"]); >> >> (the sort order here is the same as in option A). >> >> Similarly, if an index pointed used a keyPath which points to an >> array, this would create an entry in the index which used a compound >> key consisting of the values in the array. >> >> There are of course advantages and disadvantages with bo
Re: [Bug 12321] New: Add compound keys to IndexedDB
- jessica On Thu, Mar 17, 2011 at 7:03 PM, Jeremy Orlow wrote: > Here's one ugliness with A: There's no way to specify ascending > or descending for the individual components of the key. So there's no way > for me to open a cursor that looks at one field ascending and the other > field descending. In addition, I can't think of any easy/good ways to hack > around this. > > Any thoughts on how we could address this use case? > > J > > On Wed, Mar 16, 2011 at 4:50 PM, wrote: > >> http://www.w3.org/Bugs/Public/show_bug.cgi?id=12321 >> >> Summary: Add compound keys to IndexedDB >> Product: WebAppsWG >> Version: unspecified >> Platform: PC >>OS/Version: All >>Status: NEW >> Severity: normal >> Priority: P2 >> Component: Indexed Database API >>AssignedTo: dave.n...@w3.org >>ReportedBy: jor...@chromium.org >> QAContact: member-webapi-...@w3.org >>CC: m...@w3.org, public-webapps@w3.org >> >> >> >From the thread "[IndexedDB] Compound and multiple keys" by Jonas >> Sicking, >> we're going to go with both options A and B. >> >> = >> >> Hi IndexedDB fans (yay!!), >> >> Problem description: >> >> One of the current shortcomings of IndexedDB is that it doesn't >> support compound indexes. I.e. indexing on more than one value. For >> example it's impossible to index on, and therefor efficiently search >> for, firstname and lastname in an objectStore which stores people. Or >> index on to-address and date sent in an objectStore holding emails. >> >> The way this is traditionally done is that multiple values are used as >> key for each individual entry in an index or objectStore. For example >> the CREATE INDEX statement in SQL can list multiple columns, and >> CREATE TABLE statment can list several columns as PRIMARY KEY. >> >> There have been a couple of suggestions how to do this in IndexedDB >> >> Option A) >> When specifying a key path in createObjectStore and createIndex, allow >> an array of key-paths to be specified. Such as >> >> store = db.createObjectStore("mystore", ["firstName", "lastName"]); >> store.add({firstName: "Benny", lastName: "Zysk", age: 28}); >> store.add({firstName: "Benny", lastName: "Andersson", age: 63}); >> store.add({firstName: "Charlie", lastName: "Brown", age: 8}); >> >> The records are stored in the following order >> "Benny", "Andersson" >> "Benny", "Zysk" >> "Charlie", "Brown" >> >> Similarly, createIndex accepts the same syntax: >> store.createIndex("myindex", ["lastName", "age"]); >> >> Option B) >> Allowing arrays as an additional data type for keys. >> store = db.createObjectStore("mystore", "fullName"); >> store.add({fullName: ["Benny", "Zysk"], age: 28}); >> store.add({fullName: ["Benny", "Andersson"], age: 63}); >> store.add({fullName: ["Charlie", "Brown"], age: 8}); >> >> Also allows out-of-line keys using: >> store = db.createObjectStore("mystore"); >> store.add({age: 28}, ["Benny", "Zysk"]); >> store.add({age: 63}, ["Benny", "Andersson"]); >> store.add({age: 8}, ["Charlie", "Brown"]); >> >> (the sort order here is the same as in option A). >> >> Similarly, if an index pointed used a keyPath which points to an >> array, this would create an entry in the index which used a compound >> key consisting of the values in the array. >> >> There are of course advantages and disadvantages with both options. >> >> Option A advantages: >> * Ensures that at objectStore/index creation time the number of keys >> are known. This allows the implementation to create and optimize the >> index using this information. This is especially useful in situations >> when the indexedDB implementation is backed by a SQL database which >> uses columns as a way to represent multiple keys. >> * Easy to use when key values appear as separate properties on the >> stored object. >> * Obvious how to sort entries. >> >> Option A disadvantages: >> * Doesn't allow compound out-of-line keys. >> * Requires multiple properties to be added to stored objects if the >> components of the key isn't available there (for example if it's >> out-of-line or stored in an array). >> >> Option B advantages: >> * Allows compound out-of-line keys. >> * Easy to use when the key values are handled as an array by other >> code. Both when using in-line and out-of-line keys. >> * Maximum flexibility since you can combine single-value keys and >> compound keys in one objectStore, as well as arrays of different >> length (we couldn't come up with use cases for this though). >> >> Option B disadvantages: >> * Requires defining sorting between single values and arrays, as well >> as between arrays of different length. >> * Requires a single property to be added to stored objects if the key >> isn't available there (for example if it's stored as separate >> properties). >> >> There is of course a third alternative: Do both Option A and Option B. >> This brings most of the advantages of both o
Re: [Bug 12321] New: Add compound keys to IndexedDB
Here's one ugliness with A: There's no way to specify ascending or descending for the individual components of the key. So there's no way for me to open a cursor that looks at one field ascending and the other field descending. In addition, I can't think of any easy/good ways to hack around this. Any thoughts on how we could address this use case? J On Wed, Mar 16, 2011 at 4:50 PM, wrote: > http://www.w3.org/Bugs/Public/show_bug.cgi?id=12321 > > Summary: Add compound keys to IndexedDB > Product: WebAppsWG > Version: unspecified > Platform: PC >OS/Version: All >Status: NEW > Severity: normal > Priority: P2 > Component: Indexed Database API >AssignedTo: dave.n...@w3.org >ReportedBy: jor...@chromium.org > QAContact: member-webapi-...@w3.org >CC: m...@w3.org, public-webapps@w3.org > > > >From the thread "[IndexedDB] Compound and multiple keys" by Jonas Sicking, > we're going to go with both options A and B. > > = > > Hi IndexedDB fans (yay!!), > > Problem description: > > One of the current shortcomings of IndexedDB is that it doesn't > support compound indexes. I.e. indexing on more than one value. For > example it's impossible to index on, and therefor efficiently search > for, firstname and lastname in an objectStore which stores people. Or > index on to-address and date sent in an objectStore holding emails. > > The way this is traditionally done is that multiple values are used as > key for each individual entry in an index or objectStore. For example > the CREATE INDEX statement in SQL can list multiple columns, and > CREATE TABLE statment can list several columns as PRIMARY KEY. > > There have been a couple of suggestions how to do this in IndexedDB > > Option A) > When specifying a key path in createObjectStore and createIndex, allow > an array of key-paths to be specified. Such as > > store = db.createObjectStore("mystore", ["firstName", "lastName"]); > store.add({firstName: "Benny", lastName: "Zysk", age: 28}); > store.add({firstName: "Benny", lastName: "Andersson", age: 63}); > store.add({firstName: "Charlie", lastName: "Brown", age: 8}); > > The records are stored in the following order > "Benny", "Andersson" > "Benny", "Zysk" > "Charlie", "Brown" > > Similarly, createIndex accepts the same syntax: > store.createIndex("myindex", ["lastName", "age"]); > > Option B) > Allowing arrays as an additional data type for keys. > store = db.createObjectStore("mystore", "fullName"); > store.add({fullName: ["Benny", "Zysk"], age: 28}); > store.add({fullName: ["Benny", "Andersson"], age: 63}); > store.add({fullName: ["Charlie", "Brown"], age: 8}); > > Also allows out-of-line keys using: > store = db.createObjectStore("mystore"); > store.add({age: 28}, ["Benny", "Zysk"]); > store.add({age: 63}, ["Benny", "Andersson"]); > store.add({age: 8}, ["Charlie", "Brown"]); > > (the sort order here is the same as in option A). > > Similarly, if an index pointed used a keyPath which points to an > array, this would create an entry in the index which used a compound > key consisting of the values in the array. > > There are of course advantages and disadvantages with both options. > > Option A advantages: > * Ensures that at objectStore/index creation time the number of keys > are known. This allows the implementation to create and optimize the > index using this information. This is especially useful in situations > when the indexedDB implementation is backed by a SQL database which > uses columns as a way to represent multiple keys. > * Easy to use when key values appear as separate properties on the > stored object. > * Obvious how to sort entries. > > Option A disadvantages: > * Doesn't allow compound out-of-line keys. > * Requires multiple properties to be added to stored objects if the > components of the key isn't available there (for example if it's > out-of-line or stored in an array). > > Option B advantages: > * Allows compound out-of-line keys. > * Easy to use when the key values are handled as an array by other > code. Both when using in-line and out-of-line keys. > * Maximum flexibility since you can combine single-value keys and > compound keys in one objectStore, as well as arrays of different > length (we couldn't come up with use cases for this though). > > Option B disadvantages: > * Requires defining sorting between single values and arrays, as well > as between arrays of different length. > * Requires a single property to be added to stored objects if the key > isn't available there (for example if it's stored as separate > properties). > > There is of course a third alternative: Do both Option A and Option B. > This brings most of the advantages of both options, but also many of > the disadvantages of both. It also adds a lot of API surface which > could conflict with future features, so it's something I'd really like > to avoid. > > > Questions: > >
[Bug 12321] New: Add compound keys to IndexedDB
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12321 Summary: Add compound keys to IndexedDB Product: WebAppsWG Version: unspecified Platform: PC OS/Version: All Status: NEW Severity: normal Priority: P2 Component: Indexed Database API AssignedTo: dave.n...@w3.org ReportedBy: jor...@chromium.org QAContact: member-webapi-...@w3.org CC: m...@w3.org, public-webapps@w3.org >From the thread "[IndexedDB] Compound and multiple keys" by Jonas Sicking, we're going to go with both options A and B. = Hi IndexedDB fans (yay!!), Problem description: One of the current shortcomings of IndexedDB is that it doesn't support compound indexes. I.e. indexing on more than one value. For example it's impossible to index on, and therefor efficiently search for, firstname and lastname in an objectStore which stores people. Or index on to-address and date sent in an objectStore holding emails. The way this is traditionally done is that multiple values are used as key for each individual entry in an index or objectStore. For example the CREATE INDEX statement in SQL can list multiple columns, and CREATE TABLE statment can list several columns as PRIMARY KEY. There have been a couple of suggestions how to do this in IndexedDB Option A) When specifying a key path in createObjectStore and createIndex, allow an array of key-paths to be specified. Such as store = db.createObjectStore("mystore", ["firstName", "lastName"]); store.add({firstName: "Benny", lastName: "Zysk", age: 28}); store.add({firstName: "Benny", lastName: "Andersson", age: 63}); store.add({firstName: "Charlie", lastName: "Brown", age: 8}); The records are stored in the following order "Benny", "Andersson" "Benny", "Zysk" "Charlie", "Brown" Similarly, createIndex accepts the same syntax: store.createIndex("myindex", ["lastName", "age"]); Option B) Allowing arrays as an additional data type for keys. store = db.createObjectStore("mystore", "fullName"); store.add({fullName: ["Benny", "Zysk"], age: 28}); store.add({fullName: ["Benny", "Andersson"], age: 63}); store.add({fullName: ["Charlie", "Brown"], age: 8}); Also allows out-of-line keys using: store = db.createObjectStore("mystore"); store.add({age: 28}, ["Benny", "Zysk"]); store.add({age: 63}, ["Benny", "Andersson"]); store.add({age: 8}, ["Charlie", "Brown"]); (the sort order here is the same as in option A). Similarly, if an index pointed used a keyPath which points to an array, this would create an entry in the index which used a compound key consisting of the values in the array. There are of course advantages and disadvantages with both options. Option A advantages: * Ensures that at objectStore/index creation time the number of keys are known. This allows the implementation to create and optimize the index using this information. This is especially useful in situations when the indexedDB implementation is backed by a SQL database which uses columns as a way to represent multiple keys. * Easy to use when key values appear as separate properties on the stored object. * Obvious how to sort entries. Option A disadvantages: * Doesn't allow compound out-of-line keys. * Requires multiple properties to be added to stored objects if the components of the key isn't available there (for example if it's out-of-line or stored in an array). Option B advantages: * Allows compound out-of-line keys. * Easy to use when the key values are handled as an array by other code. Both when using in-line and out-of-line keys. * Maximum flexibility since you can combine single-value keys and compound keys in one objectStore, as well as arrays of different length (we couldn't come up with use cases for this though). Option B disadvantages: * Requires defining sorting between single values and arrays, as well as between arrays of different length. * Requires a single property to be added to stored objects if the key isn't available there (for example if it's stored as separate properties). There is of course a third alternative: Do both Option A and Option B. This brings most of the advantages of both options, but also many of the disadvantages of both. It also adds a lot of API surface which could conflict with future features, so it's something I'd really like to avoid. Questions: The main question we had if there is a use case for having different number of compound-key-values for the entries in a index or objectStore? I.e. is there a case when you, in one objectStore, want to have one record with a compound key consisting of 2 different values, and another record consisting of 3, 4 or 5? In all the cases where I've used compound keys, each key-part has been vital. For example a table storing sale totals by quarter, country and price class. In this case it's obviously always going to be 3 parts to the compound key. Does anyone have counter examples? Similarly, a