Re: Fixing appcache: a proposal to get us started

[Nathan Kitchen]

One feature I'd like to see is respect for compression headers. I've got an
app which results in a 30Mb app cache, but it's only 8Mb over the wire due
to GZIP compression. I'd much prefer the appcache to see that the content
was served compressed, cache it compressed, and "serve" it to the browser
offline compressed too.

Whilst not strictly part of a JS API, it would probably be a worthwhile
"note to implementors" that the preferred behaviour is to store and "cache"
with the same compression headers with which the content was originally
served.

Another approach would be to modify the appcache manifest with a "compress"
property per item to cache, or perhaps transparently compress the entire
appcache for a given manifest within the browser (totally transparent to
the user/developer/API).

Either way, it'd be nice to prevent the scenario of downloading 8Mb content
and resulting in a 30Mb cache.

N




On 26 March 2013 08:48, James Graham  wrote:

> On 03/26/2013 08:02 AM, Jonas Sicking wrote:
>
>  Another "feature" that we are proposing is to drop the current
>> manifest format and instead use a JSON based one. The most simple
>> reason for this is that we noticed that the information we need to
>> express quickly became complex enough that using a format with simple
>> parsing rules was beneficial.
>>
>> A format based on extending the current appcache format would be no
>> problem for a UA to parse. However the complexity that we need to
>> express resulted in something that's too hard for a human to manually
>> write, or for a human to understand when looking at somebody else's
>> manifest in order to learn.
>>
>> The simple parsing rules for JSON seemed like a better fit. It also
>> provides more of an opportunity to extend the format in the future.
>> JSON also has advantages when it comes to creating APIs exposed to
>> webpages for interacting with appcaches. More about this below.
>>
>
> Some slightly trivial feedback: I am worried about using a format with no
> support for comments. I agree that some hypothetical JSON+comments format
> would be a good fit, but without the ability to document complex rulesets,
> it seems like we are going to create a maintenance nightmare.
>
>

UUID's

[Nathan Kitchen]

Hi.

I'm not sure if this is the place to raise this, it's basically a request
for consideration of an API which I don't believe exists and would probably
be most useful within web apps: a UUID generator.

Earlier this week I hit a real-world issue where I needed to generate a UUID
from JavaScript. It's possible to do this using Math.random, but these
aren't guaranteed unique in the same way as version 1 UUID's (generated via
a time-based algorithm). Decent write-up here (not mine):

http://www.broofa.com/2008/09/javascript-uuid-function/

And the spec:

http://www.ietf.org/rfc/rfc4122.txt

I'd expect a UUID generator might be useful in a number of use cases from
messaging to file IO to IndexedDb. Is this something Web Apps could
investigate, add to a meeting agenda, etc?

Thanks.

Re: Model-driven Views

[Nathan Kitchen]

Have you heard of knockout.js? It's an MVVM pattern based on JQuery, if
you're not aware of it you may be interested to see their approach.

Official site:
http://knockoutjs.com/

Recent MIX event:
http://channel9.msdn.com/Events/MIX/MIX11/FRM08

Just FYI as it was related...

On 23 April 2011 01:35, Rafael Weinstein  wrote:

> Myself and a few other chromium folks have been working on a design
> for a formalized separation between View and Model in the browser,
> with needs of web applications being the primary motivator.
>
> Our ideas are implemented as an experimental Javascript library:
> https://code.google.com/p/mdv/ and the basic design is described here:
> http://mdv.googlecode.com/svn/trunk/docs/design_intro.html. It's not
> complete and there are things we're not happy with, but it's
> self-consistent enough that you can start to imagine what a full
> design might look like.
>
> We hope to get others interested in collecting requirements/use cases
> and fleshing out a good solution.
>
> We're starting the discussion here because a few people in this group
> from whom we got early feedback felt that it would be most appropriate
> place and, further, that this work bears some relation to XBL.
>
> What do you think?
>
>
>

Re: [WebSQL] Any future plans, or has IndexedDB replaced WebSQL?

[Nathan Kitchen]

I agree that it'd be best to have a spec independent of database platform,
which is why I was asking about an idea along the lines of RelationalDB
or the example I gave in the email
which initiated this discussion, both of which are entirely abstracted from
the underlying database (SQLite or otherwise - doesn't even have to be SQL).

Was anything like this considered before WebSQL was canned?

Also, +1 on looking at a FireFox plugin to drum up some interest on this.

On 2 April 2011 05:32, Ian Hickson  wrote:

> On Fri, 1 Apr 2011, Arthur Barstow wrote:
> > On Apr/1/2011 3:39 PM, ext Glenn Maynard wrote:
> > > If SQLite was to be used as a web standard, I'd hope that it wouldn't
> show
> > > up in a spec as simply "do what SQLite does", but as a complete spec of
> > > SQLite's behavior.
> >
> > FYI, the Web SQL Database NOTE says:
> >
> > [[
> > http://www.w3.org/TR/2010/NOTE-webdatabase-20101118/#web-sql
> >
> > 5. Web SQL
> >
> > User agents must implement the SQL dialect supported by Sqlite 3.6.19.
> > ]]
> >
> > and I don't recall anyone ever committing to creating the later.
>
> Actually I did, multiple times. But nobody was interested in
> reimplementing that dialect independently, so all the points Jonas raised
> still apply.
>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>
>

Re: [WebSQL] Any future plans, or has IndexedDB replaced WebSQL?

[Nathan Kitchen]

Are there any browser vendor representatives on the mailing list who would
care to comment on the criteria for implementing something akin to Keean's
RelationalDB <https://github.com/keean/RelationalDB> idea? What would need
to be in place to start work on such an implementation?

I can think of a number of advantages to implementing this:

   1. Opportunity to explore more solutions to "offline data" than *just *
   IndexedDB.

   2. Many web developers have a working knowledge of SQL, so the concepts
   of a relational database may be more familiar. If adoption could be
   considered a proxy for the "success" of a standard, I'd suggest that aiming
   for something the web development community understands would be a large
   factor in adoption.

   3. It's probably (!) easier to implement RelationalDB than IndexedDB, as
   it maps fairly cleanly to existing relational database technologies. This
   would allow vendors to implement it using Sqlite, Access, etc independent of
   the spec.

   4. IndexedDB wouldn't necessarily be in competition with RelationalDB.
   Some use cases would be better served by IndexedDB, while others may be
   better served by RelationalDB. The two would compliment each other. Even so,
   a bit of healthy competition would probably be good for the technologies in
   general.

   5. Having two implementations allows third-party frameworks to pick the
   "best" underlying specification for their needs. Rather than having to write
   RelationalDB on top of IndexedDB (which I think is very likely to happen),
   there may as well be a native API for it which is properly standardized and
   more importantly *optimized*.

Have I convinced anyone to start work on this in a vNext browser?!

Pretty please...? There will be cake : )


On 31 March 2011 17:36, Keean Schupke  wrote:

> No real reason - just trying to implement a minimal framework. Date objects
> would be a definite must have going forward.
>
> I was interested in trying to get something like this standardised, as I
> believe it has none of the issues that stopped WebSQL, as it defines a
> complete relational API independent of the implementation of SQL behind it.
>
> The key thing is to get the browser implementors interested in implementing
> it. If even one of the main browser implementors is not interested in
> implementing it, then it will suffer the same fate as WebSQL.
>
> Independent of standardisation (which I would like) I intend to try and
> implement the same API on top of WebSQL and IndexedDB as a library. So
> people are free to use the backend with the best performance without
> changing their code. It aims to be as stateless as possible, and to
> implement relational algebra on relations.
>
>
> Cheers,
> Keean.
>
>
> On 31 March 2011 15:54, Nathan Kitchen  wrote:
>
>> That's nice, pretty much what I was thinking but somewhat more complete :
>> ) Is there not a w3 group progressing something like this? And if not, who
>> would need to be lobbied to get one started?!
>>
>> As an aside, I note you didn't implement "date" as a supported data type.
>> Was that a conscious decision, and if so what was the reasoning behind it?
>>
>> N
>>
>>
>> On 31 March 2011 16:33, Keean Schupke  wrote:
>>
>>> Have a look at my RelationalDB API
>>>
>>> https://github.com/keean/RelationalDB
>>>
>>> In particular examples/candy.html
>>>
>>> A lot of work went into the underlying concepts - Its work originally
>>> published by myself and others at the 2004 Haskell Workshop, and follows on
>>> from HaskellDB which was the original inspiration behind C#s Linq
>>> functionality).
>>>
>>> It implements the relational-algebra operators as methods that operate on
>>> relation objects.
>>>
>>> Let me know what you think.
>>>
>>>
>>> Cheers,
>>> Keean.
>>>
>>>
>>> On 31 March 2011 15:19, Nathan Kitchen  wrote:
>>>
>>>> Hi.
>>>>
>>>> I've been watching discussions on IndexedDB for a while now, and
>>>> wondered if anyone would mind spending a few moments to explain how
>>>> IndexedDB is related (or not) to WebSQL. Is IndexedDB seen as replacing the
>>>> functionality originally offered by WebSQL? If not, are there any plans to
>>>> make a cross-platform variant of Web SQL?
>>>>
>>>> If (?) most web developers know SQL, is there a case to be made for
>>>> abstracting SQL into JSON/JavaScript rather than moving to IndexedDB
>>>> document storage? Reasons for asking this:
>>>>
&g

Re: [WebSQL] Any future plans, or has IndexedDB replaced WebSQL?

[Nathan Kitchen]

That's nice, pretty much what I was thinking but somewhat more complete : )
Is there not a w3 group progressing something like this? And if not, who
would need to be lobbied to get one started?!

As an aside, I note you didn't implement "date" as a supported data type.
Was that a conscious decision, and if so what was the reasoning behind it?

N

On 31 March 2011 16:33, Keean Schupke  wrote:

> Have a look at my RelationalDB API
>
> https://github.com/keean/RelationalDB
>
> In particular examples/candy.html
>
> A lot of work went into the underlying concepts - Its work originally
> published by myself and others at the 2004 Haskell Workshop, and follows on
> from HaskellDB which was the original inspiration behind C#s Linq
> functionality).
>
> It implements the relational-algebra operators as methods that operate on
> relation objects.
>
> Let me know what you think.
>
>
> Cheers,
> Keean.
>
>
> On 31 March 2011 15:19, Nathan Kitchen  wrote:
>
>> Hi.
>>
>> I've been watching discussions on IndexedDB for a while now, and wondered
>> if anyone would mind spending a few moments to explain how IndexedDB is
>> related (or not) to WebSQL. Is IndexedDB seen as replacing the functionality
>> originally offered by WebSQL? If not, are there any plans to make a
>> cross-platform variant of Web SQL?
>>
>> If (?) most web developers know SQL, is there a case to be made for
>> abstracting SQL into JSON/JavaScript rather than moving to IndexedDB
>> document storage? Reasons for asking this:
>>
>>- Many of the posts appearing to come from "the dev community" rather
>>than W3C seem to expect more SQL-esque functionality from IndexedDB. If 
>> the
>>enthusiasts who get involved enough to post to the board are expecting
>>SQL/query type experience, maybe there is a driver for a native database 
>> API
>>supporting this.
>>- Several people have noted that third-party frameworks could
>>implement this functionality. This might be a daft question, but isn't it
>>easier to implement an "IndexedDB"-like framework on top of "WebSQL", 
>> than a
>>"WebSQL"-like framework on top of "IndexedDB" (overuse of quotes to 
>> indicate
>>the general concept).
>>
>> I had a ponder on how I'd like to see such a framework implemented (in
>> both Access & SQLite :p ), and came up with a stack of pseudo-code below in
>> my lunch break. Might make an interesting discussion point. It's not really
>> IndexedDB, it's WebSQL v2. Or maybe WebJSQL or something. I'd be really
>> interested to understand what advantages IndexedDB has over an
>> implementation like the one below though.
>>
>> // DATABASE
>> // First, open a database with the specified name. The number at the
>> // end denotes the version of the specification that the application
>> // plans to use. This allows forward-compatibility with vNext.
>> var db = window.openDatabase("shoppinglist", 1.0);
>>
>> // MIGRATIONS
>> // Next, create some migrations. These are predefined structures which
>> // are validated by the browser database engine. A migration consists
>> // of two actions: one "up", one "down". Each action specifies some
>> // operations and parameters. It's up to the browser database to read
>> // these and perform the appropriate action, as defined in the spec.
>>
>> // Other actions may include a "batch add" for "static" data. It could
>> // also be valid to have key and index creation and removal as separate
>> // actions.
>>
>> // SHOPPING TRIP
>> var createTripTableAction = {
>> action: "create-table",
>> params: {
>> name: "trip",
>> columns: [
>>   { name: "id", type: "whole-number", primaryKey: true },
>>   { name: "name", type: "string", length: 32, regex: "[A-Z]{1,32}"
>> } // regex: wouldn't that be nice...
>> ],
>> indexes: [
>>   {
>> columns: [
>>   { name: "name",   type: "full-text" }
>> ]
>>   } // More indexes here if required
>> ]
>>   }
>>   };
>>
>> var removeTripTableAction = {
>> action: "remove-table",
>> params: {
>> name: "shopping",
>> cleardata: true
>>   }
>&g

[WebSQL] Any future plans, or has IndexedDB replaced WebSQL?

[Nathan Kitchen]

Hi.

I've been watching discussions on IndexedDB for a while now, and wondered if
anyone would mind spending a few moments to explain how IndexedDB is related
(or not) to WebSQL. Is IndexedDB seen as replacing the functionality
originally offered by WebSQL? If not, are there any plans to make a
cross-platform variant of Web SQL?

If (?) most web developers know SQL, is there a case to be made for
abstracting SQL into JSON/JavaScript rather than moving to IndexedDB
document storage? Reasons for asking this:

   - Many of the posts appearing to come from "the dev community" rather
   than W3C seem to expect more SQL-esque functionality from IndexedDB. If the
   enthusiasts who get involved enough to post to the board are expecting
   SQL/query type experience, maybe there is a driver for a native database API
   supporting this.
   - Several people have noted that third-party frameworks could implement
   this functionality. This might be a daft question, but isn't it easier to
   implement an "IndexedDB"-like framework on top of "WebSQL", than a
   "WebSQL"-like framework on top of "IndexedDB" (overuse of quotes to indicate
   the general concept).

I had a ponder on how I'd like to see such a framework implemented (in both
Access & SQLite :p ), and came up with a stack of pseudo-code below in my
lunch break. Might make an interesting discussion point. It's not really
IndexedDB, it's WebSQL v2. Or maybe WebJSQL or something. I'd be really
interested to understand what advantages IndexedDB has over an
implementation like the one below though.

// DATABASE
// First, open a database with the specified name. The number at the
// end denotes the version of the specification that the application
// plans to use. This allows forward-compatibility with vNext.
var db = window.openDatabase("shoppinglist", 1.0);

// MIGRATIONS
// Next, create some migrations. These are predefined structures which
// are validated by the browser database engine. A migration consists
// of two actions: one "up", one "down". Each action specifies some
// operations and parameters. It's up to the browser database to read
// these and perform the appropriate action, as defined in the spec.

// Other actions may include a "batch add" for "static" data. It could
// also be valid to have key and index creation and removal as separate
// actions.

// SHOPPING TRIP
var createTripTableAction = {
action: "create-table",
params: {
name: "trip",
columns: [
  { name: "id", type: "whole-number", primaryKey: true },
  { name: "name", type: "string", length: 32, regex: "[A-Z]{1,32}" }
// regex: wouldn't that be nice...
],
indexes: [
  {
columns: [
  { name: "name",   type: "full-text" }
]
  } // More indexes here if required
]
  }
  };

var removeTripTableAction = {
action: "remove-table",
params: {
name: "shopping",
cleardata: true
  }
  };


// SHOPPING
var createShoppingTableAction = {
action: "create-table",
params: {
name: "shopping",
columns: [
  { name: "id", type: "whole-number", primaryKey: true },
  { name: "tripid", type: "whole-number" },
  { name: "name", type: "string", length: 128, nillable: false },
  { name: "cost", type: "decimal-number" }
],
keys: [
  { local: "tripid", foreign: { table: "trips", column: "id" },
cascade-delete: true } }
],
indexes: [
  {
columns: [
  { name: "tripid", type: "asc" },
  { name: "name",   type: "full-text" }
]
  }
]
  }
  };

var removeShoppingTableAction = {
action: "remove-table",
params: {
name: "shopping",
cleardata: true
  }
  };

var migTrip = { up: createTripTableAction, down: removeTripTableAction };
var migShopping = { up: createShoppingTableAction, down:
removeShoppingTableAction };

// Example calls which might be appropriate here, where the word
// "callback" is placeholder for async operations:

db.getVersion();// Returns 0
db.migrations.add(migTrip); // Persists the up/down actions as a
list somewhere
db.migrations.add(migShopping); // Persists the up/down actions as a
list somewhere
db.migrateUp(2, callback);  // Runs the two migrations
db.getVersion();// Returns 2
db.migrateDown(0, callback);// Runs all "down" migrations to
schema version 0.
db.getVersion();// Returns 0
db.destroy(true);   // Get rid of the database,
optionally running "down" migrations

// * Note that during migration operations the database should be locked: no
other threads should
// * be able to interact/access the database while this is taking place.

// QUERIES
// Invent a JSON structure for creating queries. There could be simple
// versio

Re: Offline Web Applications status

[Nathan Kitchen]

A couple of other app cache observations from a hobbyist who's played around
with Google's Gears...

I built an offline web application based on Gears, with the intention to
migrate to something a bit more standardized as it became available. That
was a good two years ago now, but the existing and proposed implementations
still don't offer the capability that I can get from Gears.

If you're in Chrome, point your browser at: http://my.offlinebible.com/. My
observations are going to be related to this app. I apologise if they're
quite specific, but they present a snapshot of some real-world issues
encountered while developing an offline app.

   - *App Cache*
   The first task that the app undertakes is downloading a large amount of
   offline data. This data includes the usual web page resources, plus a
   massive amount of data destined for the Gears/WebSQL database.

   Ideally, I want to keep this in a single easy installation process. With
   Gears I can do this entirely from JavaScript. First, the database is set up.
   Then, AJAX is used to request data in chunks (JSON, about 60Mb, gzipped for
   bandwidth gets it down to 10Mb iirc). Once it's all requested, it gets
   dropped into Gears/WebSQL. Finally all the usual web assets (html, css, js,
   img) are cached. Throw up a progress bar, run everything in a web worker to
   keep the UI from freezing, done.

   With the existing specs, I'd have to do something slightly different.
   First, I'd have to hit a page with a manifest, and hook the relevant events
   for when the cache was ready. If I want to offer the user different offline
   capabilities (i.e. customize what's cached), afaik I can't do that from
   JavaScript. It requires some server-side code/processing to output a
   different manifest.

   Once the cache was ready, I could carry on with the existing
   installation. However, I don't (think I) have the script-side control over
   adding and removing items from the cache to customize the user's offline
   experience.

   - *Search
   *This is the use-case that's most important to me. I want to be able to
   search all the data which I took offline. My current implementation is built
   using manually indexed items and joins. In theory I could use the full-text
   capabilities of the underlying SqlLite Gears implementation, but this was a
   step too proprietary for me. I built all the data indexes to see what
   performance was like. Throw some words into the search capability, you'll
   see how long it takes to search. It's *fairly* quick but there's a slight
   lag (which locks the UI, it's synchronous ATM).

   I know full-text indexing is on the cards for IndexedDB. I'd love to see
   some sample implementations of full-text to compare speed against the manual
   index. For single words there might not be too much difference, but for more
   complex multi-word or pattern-matching, the manual index is too
   slow/won't work.

I don't think that my scenario is particularly unusual. Taking a large
amount of data offline and making it available to search seems like a pretty
common use-case. To support this, there are three capabilities which I'd
like to see:

   - Script access to add or remove items from the application cache -
   document.manifest.add("");
   - Batch operations (or support for adding a lot of similar data as
   quickly as possible - this takes ages if you add each record as a single
   transaction)
   - Full-text search on data

I'm looking forward to this coming together eventually, might be worth an
IndexedDB implementation soon : )

On 24 March 2011 05:53, David John Burrowes
wrote:

> 2011/3/24 louis-rémi BABE 
>
>> ## Maybe Web devs don't use App Cache because they don't understand
>> what it is... ##
>>
>
> I think most webdevs are expecting more than what is offered. It seems like
> a half baked solution to a potentially useful requirement.
>
>
> I thought I'd add half a cent here, from the perspective of one who isn't a
> professional web developer... just a hobbyist.
>
> When I heard about the app cache, it seemed like a really great thing.
> Offline web apps! Cool! A way for the web to become even more ubiquitous!
>
> But, as the comment above hints, it really doesn't seem to be the full
> delivery of the solution (even when you get past the browser differences,
> setting up of mime types, debugging all this, etc).  An offline web app is
> certainly more than just caching the code and ui files, no?  It is also some
> kind of stand-in for the absent server... data storage, and cross-page state
> of some sort (e.g. I'd expected something like web workers that can live for
> a session, not a page).  These aren't all coming together at the same time,
> and aren't really being presented as a unified "feature" (indeed, I'm not
> sure that they are being thought of as that)
>
> I'm sure that as html continues it's forward evolution, these will all come
> into play and we'll eventually see more use of the feature.
>

Re: An HTML5 logo

[Nathan Kitchen]

I'm also a little confused. There was a recent announcement [1] from the
WHATWG that the version number was being dropped from HTML 5. This has been
reported elsewhere, usually directly referring back to the WHATWG
announcement [2].

Obviously this doesn't seem to fit with the smacking great "5" in the new
logo. For the record I think it looks great, there just seems to be a bit of
confusion around it at the moment.

[1] http://blog.whatwg.org/html-is-the-new-html5
[2]
http://arstechnica.com/web/news/2011/01/html-to-lose-the-version-number.ars

On 23 January 2011 06:54, Axel Rauschmayer  wrote:

> Yes, the logo looks awesome!
>
> But I think we should clarify who the logo is for: Techies or end users.
> All the logo mission statements that I have read point to the former group,
> but I think it makes more sense to target the latter group. Then HTML5 might
> not be a good brand [1].
>
> [1]
> http://www.2ality.com/2011/01/branding-web-technologies-and-new-html5.html
>
> On Jan 23, 2011, at 4:49 , Marcos Caceres wrote:
>
> > Hi Philippe,
> >
> > Logos look nice! however, I'm baffled as to why HTML5 logo site mix in
> > things like WebGL and CSS 3, which are clearly not part of the HTML
> > Standard (as it is now known;))? Who made the choice of what
> > technologes were to be included or excluded from the set of
> > technologies that make up the logo sets? I ask because it's poignant
> > and a little demoralizing to us who have been working for many years
> > to see W3C Widgets excluded from the list. I'm sure I'm not the only
> > one feeling a bit confused by the W3C's lack of communication about
> > this project or why the W3C chose to exclude widgets and other
> > technologies? Did I miss the memo?
> >
> > Kind regards,
> > Marcos
> >
> > On Tue, Jan 18, 2011 at 2:03 PM, Philippe Le Hegaret  wrote:
> >> Dear Web Application Working Group,
> >>
> >> Today W3C introduced an HTML5 logo for public consideration:
> >> http://www.w3.org/News/2011#entry-8992
> >>
> >> The W3C Communications Team is excited about the HTML5 logo, developed
> >> with community support, and hopes it will help you promote your work.
> >> The logo is intended to be a general purpose visual identity for HTML5
> >> and other web application technologies. It doesn't imply conformance;
> >> just "this is about open web application technologies."
> >>
> >> This is not yet the official W3C Communications Team logo for HTML5. We
> >> look forward to broad community adoption in order to make it so.
> >>
> >> For more information about the logo, see the logo home page [1] and faq
> >> [2].
> >>
> >> Thank you,
> >>
> >> Philippe Le Hégaret, Interaction Domain Lead
> >>
> >> [1] http://www.w3.org/html/logo/
> >> [2] http://www.w3.org/html/logo/faq
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > Marcos Caceres
> > Opera Software ASA, http://www.opera.com/
> > http://datadriven.com.au
> >
> >
>
> --
> Dr. Axel Rauschmayer
> a...@rauschma.de
> Home: http://rauschma.de
> Blog: http://2ality.com
>
>
>
>
>

Re: IndexedDB TPAC agenda

[Nathan Kitchen]

Not sure if this is covered by "index keys", but you may consider adding:

 - Full-text indexing

To the agenda.

N


On Tue, Nov 2, 2010 at 11:15 AM, Jonas Sicking  wrote:

> I suspect internationalization is another thing where we can quickly
> make progress so lets try to get to that one too.
>
> / Jonas
>
> On Tue, Nov 2, 2010 at 11:58 AM, Jeremy Orlow  wrote:
> > Great list!
> > I propose we start with the various keys issues (I think we can make a
> lot
> > of progress quickly and it's somewhat fresh on our minds), go to dynamic
> > transactions (mainly are we going to support them), and then go from
> there.
> > J
> >
> > On Tue, Nov 2, 2010 at 10:48 AM, Pablo Castro <
> pablo.cas...@microsoft.com>
> > wrote:
> >>
> >> To hit the ground running on this, here is a consolidated list of issues
> >> coming both from the thread below and various pending bugs/discussions
> we've
> >> had. I picked an arbitrary order and grouping, feel free to tweak in any
> >> way.
> >>
> >> - keys (arrays as keys, compound keys, general keypath restrictions)
> >> - index keys (arrays as keys, empty values, general keypath
> restrictions)
> >> - internationalization (collation specification, collation algorithm)
> >> - quotas (how do apps request more storage, is there a temp/permanent
> >> distinction?)
> >> - error handling (propagation, relationship to window.error, db scoped
> >> event handlers, errors vs return empty values)
> >> - blobs (be explicit about behavior of blobs in indexeddb objects)
> >> - transactions error modes (abort-on-unwind in error conditions; what
> >> happens when user leaves the page with pending transactions?)
> >> - transactions isolation/concurrent aspects
> >> - transactions scopes (dynamic support)
> >> - synchronous api
> >>
> >> Thanks
> >> -pablo
> >>
> >> -Original Message-
> >> From: public-webapps-requ...@w3.org [mailto:
> public-webapps-requ...@w3.org]
> >> On Behalf Of Pablo Castro
> >> Sent: Monday, November 01, 2010 10:39 PM
> >> To: Jeremy Orlow; Jonas Sicking
> >> Cc: public-webapps@w3.org
> >> Subject: RE: IndexedDB TPAC agenda
> >>
> >> A few other items to add to the list to discuss tomorrow:
> >>
> >> - Blobs support: have we discussed explicitly how things work when an
> >> object has a blob (file, array, etc.) as one of its properties?
> >> - Close on collation and international support
> >> - How do applications request that they need more storage? And related
> to
> >> this, at some point we discussed temporary vs permanent stores. Close on
> the
> >> whole story of how space is managed.
> >> - Database-wide exception handlers
> >>
> >> Looking forward to the discussion tomorrow.
> >>
> >> -pablo
> >>
> >>
> >> From: public-webapps-requ...@w3.org [mailto:
> public-webapps-requ...@w3.org]
> >> On Behalf Of Jeremy Orlow
> >> Sent: Monday, November 01, 2010 1:34 PM
> >> To: Jonas Sicking
> >> Cc: public-webapps@w3.org
> >> Subject: Re: IndexedDB TPAC agenda
> >>
> >> On Mon, Nov 1, 2010 at 12:23 PM, Jonas Sicking 
> wrote:
> >> On Mon, Nov 1, 2010 at 5:13 AM, Jeremy Orlow 
> wrote:
> >> > On Mon, Nov 1, 2010 at 11:53 AM, Jonas Sicking 
> wrote:
> >> >>
> >> >> On Mon, Nov 1, 2010 at 4:40 AM, Jeremy Orlow 
> >> >> wrote:
> >> >> > What items should we try to cover during the f2f?
> >> >> > On Mon, Nov 1, 2010 at 11:08 AM, Jonas Sicking 
> >> >> > wrote:
> >> >> >>
> >> >> >> > P.S. I'm happy to discuss all of this f2f tomorrow rather than
> >> >> >> > over
> >> >> >> > email
> >> >> >> > now.
> >> >> >>
> >> >> >> Speaking of which, would be great to have an agenda. Some of the
> >> >> >> bigger items are:
> >> >> >>
> >> >> >> * Dynamic transactions
> >> >> >> * Arrays-as-keys
> >> >> >> * Arrays and indexes (what to do if the keyPath for an index
> >> >> >> evaluates
> >> >> >> to an array)
> >> >> >> * Synchronous API
> >> >> >
> >> >> > * Compound keys.
> >> >> > * What should be allowed in a keyPath.
> >> >>
> >> >> Aren't "compound keys" same as "arrays-as-keys"?
> >> >
> >> > Sorry, I meant to say compound indexes.
> >> > We've talked about using indexes in many different ways--including
> >> > compound
> >> > indexes and allowing keys to include indexes.  I assumed you meant the
> >> > latter?
> >> I'm lost as to what you're saying here. Could you elaborate? Are you
> >> saying "index" when you mean "array" anywhere?
> >>
> >> oops.  Yes, I meant to say: "We've talked about using arrays in many
> >> different ways--including compound indexes and allowing keys to include
> >> arrays.  I assumed you meant the latter?"
> >>
> >> >> * What should happen if an index's keyPath points to a property which
> >> >> doesn't exist or which isn't a valid key-value? (same general topic
> as
> >> >> "arrays and indexes" above)
> >> >
> >> > We've talked about this several times.  It'd be great to settle on
> >> > something
> >> > once and for all.
> >> Agreed.
> >>
> >> >> * What happens if the user leaves a page in the middle of a
> >> >> transaction? (this might

Re: Replacing WebSQL with a Relational Data Model.

[Nathan Kitchen]

Sorry Keean, the main point of my post was to introduce the [featurecreep
/], not critique your suggestions. I don't honestly care about the
implementation of persistent browser storage, but I do care that it's
fully-featured. nat...@webr3.org noted that we just need something to
persist data from javascript. Although I agree with this, I think we
additionally need native full-text search *as well as* CRUD. The Gears
implementation of FTS (or rather, SQLite) exposed useful functionality but
that needs
*
*
In hindsight it was a little off-topic, but I saw it fly past and thought
that while we were discussing offline storage features it'd be a good point
to raise FTS.

I'm also not sure about persisted JSON structures vs relational objects, but
happy to see how the current spec pans out. It certainly involves thinking
about an application's data architecture in a different way though.

On Wed, Oct 27, 2010 at 9:10 AM, Keean Schupke  wrote:

> Hi Nathan,
>
> On 27 October 2010 08:58, Nathan Kitchen  wrote:
>
>> The most obvious problem was that it was tied so tightly to SQLite (which
>> I think everyone would be amazed if MS started shipping with IE10). They'd
>> want to use Access/SQL Compact, and suddenly we'd all have different SQL
>> dialects to code our offline applications to.
>
>
>
>
>
>> Which is why I agree 100% with this statement:
>>
>>
>> *The critical point here is that we need only one standardized interface,
>>> not a perfectly optimized for data-model-x one, not a uses
>>> query-language-foo one, just something that we can all use to persist data
>>> from javascript, and wrap in other APIs, that way any optimizations made
>>> will benefit everybody - regardless of their preferred interface, data model
>>> & query style.*
>>
>>
> And I totally agree with this statement, which is why I think it is
> critical a _relationally_complete_ API is standardised (either in this, or a
> later IndexedDB spec, or another spec entirely).
>
>
> Cheers,
> Keean.
>
>

Re: Replacing WebSQL with a Relational Data Model.

[Nathan Kitchen]

I've been keeping up with the work you guys have been doing for a while and
I like the direction that the current spec is headed in. Although it ended
up being a bit of a dud, Google's Gears project got some cool functionality
out for developers to feed back on. The most obvious problem was that it was
tied so tightly to SQLite (which I think everyone would be amazed if MS
started shipping with IE10). They'd want to use Access/SQL Compact, and
suddenly we'd all have different SQL dialects to code our offline
applications to.

Which is why I agree 100% with this statement:

*The critical point here is that we need only one standardized interface,
> not a perfectly optimized for data-model-x one, not a uses
> query-language-foo one, just something that we can all use to persist data
> from javascript, and wrap in other APIs, that way any optimizations made
> will benefit everybody - regardless of their preferred interface, data model
> & query style.*


[featurecreep]
There is one more thing that I would like to see in the spec though:
full-text indexing/inverted index support. It's all well and good having a
standardized interface to build CRUD operations on top of, but I'd be
surprised if anyone is going to be able to build performant text search on
top of an API without *some level* of native support for it.
[/featurecreep]

I for one would love to see some consideration given to this for v1, it'd be
a shame to have to wait x-years until the spec is further developed before
this functionality could be added.

Thanks.

Nathan
On Tue, Oct 26, 2010 at 7:03 PM, Nathan  wrote:

> Jonas Sicking wrote:
>
>> On Mon, Oct 25, 2010 at 10:24 AM, Keean Schupke  wrote:
>>
>>> We (www.fry-it.com) produce websites and mobile apps. We have been
>>> looking
>>> at HTML5 for developing mobile apps, and for the kind of apps that we
>>> want
>>> to produce the client side WebSQL API provided what we needed. The
>>> IndexedDB
>>> is not suitable (no joins, no compound indexes etc...).
>>>
>> [snip]
>> One thing that I'm wary of is going the full relational route of
>> requiring that values be split up into columns. It's really nice that
>> javascript objects can be stored as-is, and then use key-paths to
>> access specific values within an object. But I think that should be
>> doable while still keeping the benefits of your proposal.
>>
>
> Likewise, personally all data I use is either in graph or k/v format, and
> have found (like many others), that primary speed gains come from changing
> the underlying data model rather than trying to map forwards and backwards
> between objects, graphs and tabular data in a relational data model, via
> SQL.
>
> However, over and above any issues of speed or this-vs-that, I strongly
> feel that universality should be priority number one, what all of us *need*
> is a /single/ standardized storage interface implemented everywhere
> javascript is, and certainly in all the major browsers.
>
> Keean, I think what you are proposing is great, but should be an API
> layered on top of IndexedDB, and which uses IndexedDB - the fact is we're
> dealing with javascript here, focus has to be on handling the structures
> provided (object/array) and getting them in and out of a persistence layer
> as optimally as possible.
>
> Thus, once we have this single universal persistence interface we can wrap
> it up in other model or context specific APIs (such as the one you
> describe), lessons can be learned, bugs filed, and over time speed increases
> and optimizations will be introduced.
>
> The critical point here is that we need only one standardized interface,
> not a perfectly optimized for data-model-x one, not a uses
> query-language-foo one, just something that we can all use to persist data
> from javascript, and wrap in other APIs, that way any optimizations made
> will benefit everybody - regardless of their preferred interface, data model
> & query style.
>
> Best,
>
> Nathan
>
>
>

Re: [WebSQL/IndexedDB] Privacy issues in the wild

[Nathan Kitchen]

Thanks, I posted a comment on the original article to shed some light on
this. Looks like the iOS implementation either doesn't follow the
recommendations or doesn't give the user the means (information or
functionality) to prevent user tracking.

Hopefully this will stop people complaining about the spec and encourage
them take up their complaint with the vendors instead.

Cheers.

On Wed, Sep 8, 2010 at 10:51 AM, Jeremy Orlow  wrote:

> On Tue, Sep 7, 2010 at 7:51 PM, Nathan Kitchen wrote:
>
>> Hi all.
>>
>> Stumbled across this article on Ars Technica regarding the abuse of the
>> WebSQL spec. I thought I'd share it here for a couple of reasons:
>>
>>1. Someone might want to point out that it's part of the Offline
>>Storage Spec, not strictly HTML5.
>>
>> HTML5 is a buzz word.  Like AJAX or LAMP.  Very few people in this world
> (should) care about precisely what spec something came from.
>
>>
>>1. Security implications may inform some aspects of the spec.
>>
>> http://dev.w3.org/html5/webstorage/#user-tracking and
> http://dev.w3.org/html5/webdatabase/#user-tracking already addresses
> EXACTLY this.  I don't think there's anything to do from a spec standpoint.
>
>>
>>
>> Article: *Advertisers get hands stuck inside HTML5 database cookie jar* (
>> http://arstechnica.com/apple/news/2010/09/rldguid-tracking-cookies-in-safari-database-form.ars
>> )
>>
>> Thanks.
>>
>> Nathan
>>
>
>

[WebSQL/IndexedDB] Privacy issues in the wild

[Nathan Kitchen]

Hi all.

Stumbled across this article on Ars Technica regarding the abuse of the
WebSQL spec. I thought I'd share it here for a couple of reasons:

   1. Someone might want to point out that it's part of the Offline Storage
   Spec, not strictly HTML5.
   2. Security implications may inform some aspects of the spec.

Article: *Advertisers get hands stuck inside HTML5 database cookie jar* (
http://arstechnica.com/apple/news/2010/09/rldguid-tracking-cookies-in-safari-database-form.ars
)

Thanks.

Nathan

[IndexedDB] Full text indexing

[Nathan Kitchen]

Hi.

Just wondering if anyone could take a few moments to provide an update on
the state of full-text indexing in IndexedDB?

A quick google indicated that the question had been raised before, Jeremy
Orlow suggesting support for inverted indexes (back in February) and there
was a reference to ongoing discussions on the subject in June.

The current working draft doesn't appear to mention inverted indexes or full
text, and it was updated fairly recently.

Cheers.

[IndexedDB] Full text indexing

[Nathan Kitchen]

Hi all.

Disclaimer: last time I posted to this mailing list someone correctly
pointed out that I'd not read the spec properly. Apologies if I've
done the same again. I'm very enthusiastic about the whole offline web
app thing, and as this is a public forum I thought I may as well fire
off a couple of questions. They might be daft, so I apologise in
advance if so.

Onwards.

At present I can't see any reference to full text indexing via the
IndexedDB API. Is this something that is specifically out-of-scope, or
is it included in the way indexes work but not explicitly stated?

I can think of a couple of scenarios in which this would be useful,
all of which fall under the heading of "offline search". Few examples
below:

-- Reference material:
If users can take parts of a website offline, at some point someone
will want to search that data. If I build an offline application which
takes a stack of reference material offline, I'd also like to build a
database containing text from those pages. I can then use full-text
search to retrieve URL's of the offline pages and direct the user to
them.

-- Emails
Let's say a user takes their mailbox offline. Now they want to search
it for a particular phrase or subject. What feature of IndexedDB would
we expect developers to leverage to implement this?

While full text search would be possible with a regular index of
single keywords, this approach isn't as elegant as full-text indexing:

  * Searching for multiple keywords would probably be a second/third
query + join, which would be slow
  * Initially populating the database with individual keywords would
require the user to download a lot of data, whereas populating a
full-text index with a sentence would be more efficient (in some
(most?) scenarios).
  * A full-text index could expose more advanced functionality such as
searching for quoted terms, and other conditional operators (see Gears
implementation of full text search
[http://code.google.com/apis/gears/api_database.html - bottom of the
page]).

Unless this has been considered already, might I suggest either
extending KeyRange to include a "Match" property? Or perhaps introduce
a level of abstraction to KeyRanges along the lines of:

IRange (internal)
  - bool IncludeInResult( itemInIndex );

KeyRange : inherits IRange
  // properties as per spec

TextRange : inherits IRange
  - DOMString Match

Sure you can think of something more appropriate, but that explains
what I'd like to accomplish.

Is this something that can already be achieved via the IndexedDB spec?
If not, could it be included without too much effort?

Appreciate all your hard work.

Nathan

Re: [IndexedDB] Existence checking

[Nathan Kitchen]

That would do the trick, yes. Thanks for pointing it out, all my w3
spec reading skills are belong to you.

On Mon, Jul 5, 2010 at 4:44 PM, Shawn Wilsher  wrote:
>  On 7/5/2010 3:19 AM, Nathan Kitchen wrote:
>>
>> There are a couple of ways to do this:
>
> I think you missed one.  IDBDatabase has a property objectStores [1] that is
> an array of object stores.  Additionally, IDBObjectStore has a property
> called indexNames [2] that is an array of indexes for the object store.  We
> should probably look into having the same naming convention for both of
> those (objectStores and indexes or objectStoreNames and indexNames
> probably?).
>
> This accomplishes what you want, right?
>
> Cheers,
>
> Shawn
>
> [1]
> http://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#widl-IDBDatabase-objectStores
> [2]
> http://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#widl-IDBObjectStore-indexNames
>
>

[IndexedDB] Existence checking

[Nathan Kitchen]

Hi all.

I have a query/suggestion regarding the current spec for IndexedDB.
Currently there doesn't appear to be any specific way to check
existence of things like indexes and object stores. One scenario this
may be important could be a site which allows optional offline
functionality, for example allowing a user to have an offline store
for "cars", "planes" and "boats". The offline JavaScript code would
like to query which stores were available offline.

There are a couple of ways to do this:

  * Rely on developers to do something clever with the database version
Example: bitmask what facilities are available: 1 = car, 2 =
   plane, 4 = boat. Database version of "3" would imply that
   "car" and "plane" stores were available.
Comment: This may be an approach which would be recommended if
the intention of the spec is for the database version to be
the principle identifier of the state/version of
an IndexedDB.
IMHO, encouraging this method would lead to confusing
database version numbers and in turn, a maintenance
nightmare.

  * Try and catch errors
Example: Attempt to open a store, catch the appropriate exception
  and then attempt to create the store.
Comment: Code to do this is not particularly readable, and relies
on invoking an error condition to find out whether
a particular
store exists.

I'd much prefer to see a method to check for the existence of object
stores/indexes that doesn't rely on try/catching exceptions. An
"exists" check would lead to leaner, more readable code which doesn't
depend on exception handling to imply the state of the database.

I've not contributed here before, so please be nice if this suggestion
is stupid for reasons I don't know : )

Thanks.

Nathan