Re: Web Storage SQL
On Apr 17, 2009, at 2:39 PM, Jonas Sicking wrote: On Tue, Apr 14, 2009 at 9:08 AM, Nikunj Mehta nikunj.me...@oracle.com wrote: On Apr 11, 2009, at 12:39 AM, Jonas Sicking wrote: On Fri, Apr 10, 2009 at 10:55 PM, Nikunj Mehta nikunj.me...@oracle.com wrote: On Apr 10, 2009, at 3:13 PM, Ian Hickson wrote: On Fri, 10 Apr 2009, Nikunj Mehta wrote: Can someone state the various requirements for Web Storage? I did not find them enunciated anywhere. There's only one requirement that I know of: * Allow Web sites to store structured data on the client. There are many use cases, e.g. Google is interested in this to enable its applications to be taken offline. We recently released offline GMail using this SQL backend; one could easily imagine other applications like Calendar, Reader, DocsSpreadsheets, etc, supporting offline mode. A while back we released a demo of Reader using Gears' SQL database. Last time I tried this trick I was asked to come back with more precise use cases [1]. Then I put together more detailed use cases [2], and even those were not considered to be written precisely enough. So it looks like the bar for what constitutes a use case or requirement seems to be quite high. [1] http://lists.w3.org/Archives/Public/public-webapps/2008AprJun/0079.html [2] http://lists.w3.org/Archives/Public/public-webapps/2008OctDec/0104.html As far as I am concerned the use cases you enumerate in [2] were fine. However note that even the current WebStorage API makes it possible to address those use cases. Just in a way that is vastly different than the solution that you propose in [2]. Do you not agree? WebStorage does not, or for that matter any other speced API, make it possible to intercept PUT/POST/DELETE requests to perform offline behavior that can be later synchronized to the server. Indeed. But it does make it technically possible to address the use cases that you listed. Not it doesn't and that is why I have offered the BITSY proposal. http://www.oracle.com/technology/tech/feeds/spec/bitsy.html I think the main road block to accepting something like that is simply needing more experience in the WG. Since your requirement, or at least your proposed solution, require that the standard design how the synchronization should work, I personally would like to know more about other synchronization technologies before accepting your proposal. I have been working to simplify the requirements to allow application-specified synchronization provided: 1. The browser stores/caches certain URLs à la Gears LocalServer and the browser responds to GET/HEAD requests for those URLs 2. The browser allows JS interception of requests for non-GET/HEAD requests to certain URLs 3. The browser enforces cookie requirements for accessing those URLs 4. The browser provides some structured storage JS API for storing synchronization state (not the contents of the data itself) 5. The browser provides JS to contribute content to the browser store/cache as text (or blob) So it's entirely the responsibility of JS to synchronize the data? Using whatever means already exist, such as XHR etc? Nothing tied to AtomPub at all? This is correct. You can see this from the proposal. We have a JS library to synchronize AtomPub data, but this is completely optional. So it has nothing to do with lack of use cases, much more to do with that we're designing a different very API, and so we need different expertise and background data. At this point, the API that is required for BITSY is far simpler than it used to be - you can just think of it as a couple of extra methods to the Gears LocalServer API. That means we have a fair amount of expertise within this WG - both Google and Oracle have toyed with slightly different parts of this problem. Oracle has implemented the browser mechanisms above as a plug-in for both Safari and Firefox. Oracle can provide this specification as a member submission if that helps the WG. Of course in order to be able to evaluate a proposal we have to see it :) Hope to see a constructive discussion now that you can see it. [snip]
Re: Web Storage SQL
On Tue, Apr 14, 2009 at 9:08 AM, Nikunj Mehta nikunj.me...@oracle.com wrote: On Apr 11, 2009, at 12:39 AM, Jonas Sicking wrote: On Fri, Apr 10, 2009 at 10:55 PM, Nikunj Mehta nikunj.me...@oracle.com wrote: On Apr 10, 2009, at 3:13 PM, Ian Hickson wrote: On Fri, 10 Apr 2009, Nikunj Mehta wrote: Can someone state the various requirements for Web Storage? I did not find them enunciated anywhere. There's only one requirement that I know of: * Allow Web sites to store structured data on the client. There are many use cases, e.g. Google is interested in this to enable its applications to be taken offline. We recently released offline GMail using this SQL backend; one could easily imagine other applications like Calendar, Reader, DocsSpreadsheets, etc, supporting offline mode. A while back we released a demo of Reader using Gears' SQL database. Last time I tried this trick I was asked to come back with more precise use cases [1]. Then I put together more detailed use cases [2], and even those were not considered to be written precisely enough. So it looks like the bar for what constitutes a use case or requirement seems to be quite high. [1] http://lists.w3.org/Archives/Public/public-webapps/2008AprJun/0079.html [2] http://lists.w3.org/Archives/Public/public-webapps/2008OctDec/0104.html As far as I am concerned the use cases you enumerate in [2] were fine. However note that even the current WebStorage API makes it possible to address those use cases. Just in a way that is vastly different than the solution that you propose in [2]. Do you not agree? WebStorage does not, or for that matter any other speced API, make it possible to intercept PUT/POST/DELETE requests to perform offline behavior that can be later synchronized to the server. Indeed. But it does make it technically possible to address the use cases that you listed. I think the main road block to accepting something like that is simply needing more experience in the WG. Since your requirement, or at least your proposed solution, require that the standard design how the synchronization should work, I personally would like to know more about other synchronization technologies before accepting your proposal. I have been working to simplify the requirements to allow application-specified synchronization provided: 1. The browser stores/caches certain URLs à la Gears LocalServer and the browser responds to GET/HEAD requests for those URLs 2. The browser allows JS interception of requests for non-GET/HEAD requests to certain URLs 3. The browser enforces cookie requirements for accessing those URLs 4. The browser provides some structured storage JS API for storing synchronization state (not the contents of the data itself) 5. The browser provides JS to contribute content to the browser store/cache as text (or blob) So it's entirely the responsibility of JS to synchronize the data? Using whatever means already exist, such as XHR etc? Nothing tied to AtomPub at all? So it has nothing to do with lack of use cases, much more to do with that we're designing a different very API, and so we need different expertise and background data. At this point, the API that is required for BITSY is far simpler than it used to be - you can just think of it as a couple of extra methods to the Gears LocalServer API. That means we have a fair amount of expertise within this WG - both Google and Oracle have toyed with slightly different parts of this problem. Oracle has implemented the browser mechanisms above as a plug-in for both Safari and Firefox. Oracle can provide this specification as a member submission if that helps the WG. Of course in order to be able to evaluate a proposal we have to see it :) But we would rather use a standard API than rely on Gears. I think if we are serious about building a good foundation for local persistence, then we should have more precise requirements for Web Storage. Otherwise, we risk prematurely standardizing some dialect of SQL supported by SQLite as Web Storage. Not sure if it makes a difference, but I would be very surprised if we ended up with the same SQL dialect as what SQLite uses. I haven't worked with SQLite personally, but from what I understand it uses some extensions that doesn't exist in many other database engines. It's important to me that we don't lock ourselves into any particular database and so we should restrict ourselves to a dialect that is widely supported. So for example if you couldn't use an Oracle DB as a backend I would be very disappointed. Here's a compilation of requirements from what I have read in this thread and other work I have done in this area: queries may involve full-text search, parameterized or query by example search, or a combination What do you mean by query by example? From [1] and [2], QBE is a mechanism to create a query based on examples provided by a user searching data. Microsoft
Re: Web Storage SQL
On Apr 11, 2009, at 12:39 AM, Jonas Sicking wrote: On Fri, Apr 10, 2009 at 10:55 PM, Nikunj Mehta nikunj.me...@oracle.com wrote: On Apr 10, 2009, at 3:13 PM, Ian Hickson wrote: On Fri, 10 Apr 2009, Nikunj Mehta wrote: Can someone state the various requirements for Web Storage? I did not find them enunciated anywhere. There's only one requirement that I know of: * Allow Web sites to store structured data on the client. There are many use cases, e.g. Google is interested in this to enable its applications to be taken offline. We recently released offline GMail using this SQL backend; one could easily imagine other applications like Calendar, Reader, DocsSpreadsheets, etc, supporting offline mode. A while back we released a demo of Reader using Gears' SQL database. Last time I tried this trick I was asked to come back with more precise use cases [1]. Then I put together more detailed use cases [2], and even those were not considered to be written precisely enough. So it looks like the bar for what constitutes a use case or requirement seems to be quite high. [1] http://lists.w3.org/Archives/Public/public-webapps/2008AprJun/0079.html [2] http://lists.w3.org/Archives/Public/public-webapps/2008OctDec/0104.html As far as I am concerned the use cases you enumerate in [2] were fine. However note that even the current WebStorage API makes it possible to address those use cases. Just in a way that is vastly different than the solution that you propose in [2]. Do you not agree? WebStorage does not, or for that matter any other speced API, make it possible to intercept PUT/POST/DELETE requests to perform offline behavior that can be later synchronized to the server. However there are some requirements that I think you have which were not enumerated in [2] and that are not fulfilled by the current API. Specifically the ability to use the same code to implement a strictly online application, as one that supports seamless online/offline transitions. That is correct. I.e. the WebStorage APIs require that you monitor all submissions and loads to and from the server and redirect the save/load into queries into the WebStorage API. It would also be responsible for detecting when a user goes online again after having stored data and synchronize that to the server as needed. Your requirements include that a lot of that happens seamlessly, is that correct? Yes. I think the main road block to accepting something like that is simply needing more experience in the WG. Since your requirement, or at least your proposed solution, require that the standard design how the synchronization should work, I personally would like to know more about other synchronization technologies before accepting your proposal. I have been working to simplify the requirements to allow application- specified synchronization provided: 1. The browser stores/caches certain URLs à la Gears LocalServer and the browser responds to GET/HEAD requests for those URLs 2. The browser allows JS interception of requests for non-GET/HEAD requests to certain URLs 3. The browser enforces cookie requirements for accessing those URLs 4. The browser provides some structured storage JS API for storing synchronization state (not the contents of the data itself) 5. The browser provides JS to contribute content to the browser store/ cache as text (or blob) So it has nothing to do with lack of use cases, much more to do with that we're designing a different very API, and so we need different expertise and background data. At this point, the API that is required for BITSY is far simpler than it used to be - you can just think of it as a couple of extra methods to the Gears LocalServer API. That means we have a fair amount of expertise within this WG - both Google and Oracle have toyed with slightly different parts of this problem. Oracle has implemented the browser mechanisms above as a plug-in for both Safari and Firefox. Oracle can provide this specification as a member submission if that helps the WG. But we would rather use a standard API than rely on Gears. I think if we are serious about building a good foundation for local persistence, then we should have more precise requirements for Web Storage. Otherwise, we risk prematurely standardizing some dialect of SQL supported by SQLite as Web Storage. Not sure if it makes a difference, but I would be very surprised if we ended up with the same SQL dialect as what SQLite uses. I haven't worked with SQLite personally, but from what I understand it uses some extensions that doesn't exist in many other database engines. It's important to me that we don't lock ourselves into any particular database and so we should restrict ourselves to a dialect that is widely supported. So for example if you couldn't use an Oracle DB as a backend I would be very disappointed. Here's a compilation of requirements from what I have read in this
Re: Web Storage SQL
On 4/10/09 1:53 PM, Maciej Stachowiak wrote: I don't think this one point should be decisive by itself. But I don't think it should be given zero weight either. I do think the existence of an implementation of the current spec and Web content using it somewhat raises the burden of proof on anyone proposing a redesign. I'm not so sure of the usefulness of that argument. Just because company X implements and proposes feature Y, and convinces company Z to implement it on their site, it doesn't somehow get extra credibility (strictly a hypothetical situation). If there are serious issues with the proposal, they should be looked at and not glossed over. For what I've seen of this discussion, serious issues that are being raised are being glossed over or being dismissed as essentially not our problem. Note that one of the clients in question is the offline-enabled mobile version of GMail. I think this demonstrates that the SQL-based Database Storage can serve the needs of an advanced and polished Web application. In addition, we have a rough demonstration that it is practically implementable in a modern Web engine. Just because an API serves the needs of an advanced and polished web application doesn't mean it's the right API. We could let web authors write some form of assembly to do task X, and they could make it serve their needs, but it most certainly would not be the right API. An exaggeration, sure, but this is an API we are going to have to live with for a long time. We should take care to ensure that it's the right API for the web. My biggest problem with this draft is that I still don't see an answer for why SQL was chosen, especially given the drawbacks that have already been highlighted. Cheers, Shawn begin:vcard fn:Shawn Wilsher n:Wilsher;Shawn email;internet:sdwi...@mozilla.com title:Platform Engineer version:2.1 end:vcard
Re: Web Storage SQL
On Fri, Apr 10, 2009 at 6:13 PM, Ian Hickson i...@hixie.ch wrote: There are many use cases, e.g. Google is interested in this to enable its applications to be taken offline. We recently released offline GMail using this SQL backend; one could easily imagine other applications like Calendar, Reader, DocsSpreadsheets, etc, supporting offline mode. These are so obfuscated they are basically closed source applications. I have no problem with that, but their existence doesn't tell us that much, since it's hard to see how they're actually using the proposed API. -- Robert Sayre I would have written a shorter letter, but I did not have the time.
Re: Web Storage SQL
Hi :-), My opinion about the web storage SQL is splitted in two parts. On the one hand, making a subset of SQL instructions is very difficult. Each of us knows that and no-one would say the contrary. On the other hand, I have made an ORM in the past, build on two layers. The first one is a database abstract layer (DAL) that enables user to choose the RDBMS which he would like to use (what the /type/ argument of the openDatabase() function expects). The second one is an object-relational mapping layer (ORM) that builds SQL query through an object interface. The SQL instructions builder is delagated in many classes (with encapsulation): one class per supported database (typically, one class per RDBMS). A relevant and obvious compromise would be to make an ORM, with an API interface as jLinq proposes. It abstracts the database, proposes a standard API, enables multiple RDBMS in use etc. Thoughts ? Best regards. -- Ivan Enderlin Developper of Hoa Framework http://hoa-project.net/
Re: Web Storage SQL
On Apr 9, 2009, at 5:38 PM, Boris Zbarsky wrote: Maciej Stachowiak wrote: I agree that no such thing as standard SQL (or rather the fact that implementations all have extensions and divergences from the spec) is a problem. But I am not sure inventing a brand new query language and database model as proposed by Vlad is a good solution to this problem. That's fine; I'm not sure of that either. I have no particular opinion on that question, in fact. 1) Applications are starting to be deployed which use the SQL-based storage API, such as the mobile version of GMail. So it may be too late for us to remove SQL storage from WebKit entirely. This is a price of early adoption, sure. If we want this content to interoperate with non-WebKit-based user agents, then we will ultimately need a clear spec for the SQL dialect to use, even if we also added an OODB or a relational database using some other query language. That's true, but it's not a given that we want this content to interoperate as-is. Early adopters of known in-flux technologies typically realize that they might have to make changes; if a different data storage API is decided on, or if the subset of SQL that's decided on doesn't match what these apps are using, then they'll need to change. So while I agree that it might be difficult for Webkit to remove the SQL support it shipped as soon as some other approach is decided on (if that even happens), it doesn't follow that other UAs would need to ship SQL support at that point. There are strong arguments for not breaking existing content, of course, but there are also strong arguments for not having experimental implementations of early drafts completely dictate the standardization process. I don't think this one point should be decisive by itself. But I don't think it should be given zero weight either. I do think the existence of an implementation of the current spec and Web content using it somewhat raises the burden of proof on anyone proposing a redesign. Note that one of the clients in question is the offline-enabled mobile version of GMail. I think this demonstrates that the SQL-based Database Storage can serve the needs of an advanced and polished Web application. In addition, we have a rough demonstration that it is practically implementable in a modern Web engine. One clear problem identified despite these examples is that we do not have a precise enough spec for the query language to make truly independent interoperable implementations possible. It seems to me that significantly redesigning database storage is not necessary to address this. X is underspecified so let's do Y or Z instead is not a very strong argument in my opinion. Another issue raised is that a different database model (OODB for instance) may work better for content authors. I would say we do not have very compelling evidence yet that such a design would be better, or that it could meet the various requirements, and we do not even have a concrete strawman proposal that we could start evaluating. Regards, Maciej
Re: Web Storage SQL
On Apr 10, 2009, at 1:53 PM, Maciej Stachowiak wrote: One clear problem identified despite these examples is that we do not have a precise enough spec for the query language to make truly independent interoperable implementations possible. There are several different query languages that can be interoperably implemented - Lucene provides one example, and Couch DB is another. What makes you say that a truly interoperable implementation is not possible? Why does the query language have to be SQL? It seems to me that significantly redesigning database storage is not necessary to address this. X is underspecified so let's do Y or Z instead is not a very strong argument in my opinion. Another issue raised is that a different database model (OODB for instance) may work better for content authors. I would say we do not have very compelling evidence yet that such a design would be better, or that it could meet the various requirements, and we do not even have a concrete strawman proposal that we could start evaluating. Can someone state the various requirements for Web Storage? I did not find them enunciated anywhere.
Re: Web Storage SQL
Maciej Stachowiak wrote: I don't think this one point should be decisive by itself. But I don't think it should be given zero weight either. Agreed (in case that wasn't clear). Note that one of the clients in question is the offline-enabled mobile version of GMail. I think this demonstrates that the SQL-based Database Storage can serve the needs of an advanced and polished Web application. In addition, we have a rough demonstration that it is practically implementable in a modern Web engine. Both quite true. One clear problem identified despite these examples is that we do not have a precise enough spec for the query language to make truly independent interoperable implementations possible. It seems to me that significantly redesigning database storage is not necessary to address this. Yes; the two issues are somewhat orthogonal, in my opinion. -Boris
Re: Web Storage SQL
Nikunj Mehta wrote: One clear problem identified despite these examples is that we do not have a precise enough spec for the query language to make truly independent interoperable implementations possible. There are several different query languages that can be interoperably implemented - Lucene provides one example, and Couch DB is another. What makes you say that a truly interoperable implementation is not possible? What Maciej said is that truly interoperable implementations of the spec as currently written are not possible. This is so because it doesn't define the query language at all other than saying it's SQL. This says nothing about implementing a spec that actually specifies a query language. (I don't consider SQL to be specification of a well-defined query language, for reasons raised earlier in this thread.) Can someone state the various requirements for Web Storage? I did not find them enunciated anywhere. This would be very welcome, yes. -Boris
Re: Web Storage SQL
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Giovanni Campagna wrote: As far as I understand from this discussion and from the linked posts, there are currently three database types and their respective query languages: - relational databases and SQL - Ecmascript objects and JSONQuery - XML databases and XQuery Each one has its own merits: for example XML allows to use XML serialization and DOM, relational databases allow great masses of data with fast indexing, ES object allow for both typed and untyped (object) data. In addition, each one has its own community of followers. So why not adding a parameter on openDatabase() to specify what kind of database we want (and what kind of query language we will use)? I mean something like openDatabase(name, version, type, displayName, estimatedSize) where type can be any string so, for example, type = sql uses the standard SQL, type=sqlite uses SQLite extensions, type=-vendor-xyz is a vendor specific extension, etc. I think you would have to take the lite out the db name ;). I would think supporting three completely different data paradigms and three different query languages would be a large system. Also, just a clarification, our JSON/JS-oriented object style storage system in Persevere that uses JSONQuery has fully indexed tables, so it achieves the some level of scalability for querying massive tables (using JSONQuery) as its relational counterparts (that use SQL). I don't know of any scalability advantage to SQL. The same may be true of XQuery, I haven't dealt with XML dbs. - -- Kris Zyp SitePen (503) 806-1841 http://sitepen.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAknd8hcACgkQ9VpNnHc4zAy3owCdEpThZk9wpKBZZSJIlEtWg6T8 8owAn3s1dLUOHX+AHF3OqpIp3vZ9lCMm =BlmS -END PGP SIGNATURE-
Re: Web Storage SQL
Giovanni Campagna wrote: So why not adding a parameter on openDatabase() to specify what kind of database we want (and what kind of query language we will use)? I mean something like openDatabase(name, version, type, displayName, estimatedSize) where type can be any string so, for example, type = sql uses the standard SQL, type=sqlite uses SQLite extensions, type=-vendor-xyz is a vendor specific extension, etc. How does this solve the original no such thing as standard SQL, really issue? -Boris
Re: Web Storage SQL
2009/4/9 Boris Zbarsky bzbar...@mit.edu: Giovanni Campagna wrote: So why not adding a parameter on openDatabase() to specify what kind of database we want (and what kind of query language we will use)? I mean something like openDatabase(name, version, type, displayName, estimatedSize) where type can be any string so, for example, type = sql uses the standard SQL, type=sqlite uses SQLite extensions, type=-vendor-xyz is a vendor specific extension, etc. How does this solve the original no such thing as standard SQL, really issue? We have a standard SQL, and we have DBMS-specific extensions (for SQLite, for MySQL, for SQLServer, etc.). The latest version is ISO/IEC 9075:2008 Information technology -- Database languages -- SQL, released in 2008, but actively being revised, according to the ISO page. As usual, if you want interoperability, you use the existing, implemented, standard (or you ask the ISO to produce an updated standard with new features), else you use extensions. -Boris Giovanni
Re: Web Storage SQL
Giovanni Campagna wrote: We have a standard SQL, and we have DBMS-specific extensions (for SQLite, for MySQL, for SQLServer, etc.). The latest version is ISO/IEC 9075:2008 Information technology -- Database languages -- SQL, released in 2008, but actively being revised, according to the ISO page. Sort of. Extensions implies that as long as you write a query to the ISO standard in question the behavior will be the same in all databases and that you have to go out of your way to write non-interoperable queries. This is not the case. A query written based on that ISO standard has different behavior in different databases, in many cases. So just telling web authors write queries based on this standard and they will work is not sufficient. Let's ignore the fact that most web authors wouldn't be able to even get their hands on this standard to see whether their query complies with it without paying a fairly hefty sum [1]. You could also require UAs to implement queries exactly per this standard. Vladimir mentioned as a possible option in his original post. This is definitely the way forward (with either this SQL definition or some subset thereof) As usual, if you want interoperability, you use the existing, implemented, standard (or you ask the ISO to produce an updated standard with new features), else you use extensions. Use of the existing standard by authors doesn't give interoperability. -Boris [1] http://www.iso.org/iso/search.htm?qt=9075searchSubmit=Searchsort=reltype=simplepublished=on tells me that to get just the parts of ISO 9075:2008 that are obviously necessary to be able to do anything whatsoever will cost me either about $290 or $770 depending on what's in part 2 of the spec. I'm assuming none of parts 3,4,9,10,11,13,14 are relevant to our use case; this is probably a good assumption for most of them. That's for getting a PDF, of course; the PDF on CD prices are somewhat higher, and there seem to be no other provisions for looking at this specification.
Re: Web Storage SQL
Hi :-), Le 9/04/09 17:29, Giovanni Campagna a écrit : 2009/4/9 Boris Zbarskybzbar...@mit.edu: Giovanni Campagna wrote: So why not adding a parameter on openDatabase() to specify what kind of database we want (and what kind of query language we will use)? I mean something like openDatabase(name, version, type, displayName, estimatedSize) where type can be any string so, for example, type = sql uses the standard SQL, type=sqlite uses SQLite extensions, type=-vendor-xyz is a vendor specific extension, etc. How does this solve the original no such thing as standard SQL, really issue? We have a standard SQL, and we have DBMS-specific extensions (for SQLite, for MySQL, for SQLServer, etc.). The latest version is ISO/IEC 9075:2008 Information technology -- Database languages -- SQL, released in 2008, but actively being revised, according to the ISO page. As usual, if you want interoperability, you use the existing, implemented, standard (or you ask the ISO to produce an updated standard with new features), else you use extensions. A Database Abstract Layer (DAL, i.e. a system that enables user to select the Relational Database Management System, RDMS, to use) is a good idea but it does not sound standard anymore. I mean: we are postponing the problem, because who, why and how will decide what RDMS must be implemented? What about XML based database? Best regards. -- Ivan Enderlin Developper of Hoa Framework http://hoa-project.net/
[Fwd: Re: Web Storage SQL]
Message original Sujet: Re: Web Storage SQL Date : Thu, 9 Apr 2009 18:28:10 +0200 De :Giovanni Campagna scampa.giova...@gmail.com Pour : Ivan Enderlin @ Hoa w...@hoa-project.net 2009/4/9 Ivan Enderlin @ Hoaw...@hoa-project.net: Hi :-), Le 9/04/09 17:29, Giovanni Campagna a écrit : 2009/4/9 Boris Zbarskybzbar...@mit.edu: Giovanni Campagna wrote: So why not adding a parameter on openDatabase() to specify what kind of database we want (and what kind of query language we will use)? I mean something like openDatabase(name, version, type, displayName, estimatedSize) where type can be any string so, for example, type = sql uses the standard SQL, type=sqlite uses SQLite extensions, type=-vendor-xyz is a vendor specific extension, etc. How does this solve the original no such thing as standard SQL, really issue? We have a standard SQL, and we have DBMS-specific extensions (for SQLite, for MySQL, for SQLServer, etc.). The latest version is ISO/IEC 9075:2008 Information technology -- Database languages -- SQL, released in 2008, but actively being revised, according to the ISO page. As usual, if you want interoperability, you use the existing, implemented, standard (or you ask the ISO to produce an updated standard with new features), else you use extensions. A Database Abstract Layer (DAL, i.e. a system that enables user to select the Relational Database Management System, RDMS, to use) is a good idea but it does not sound standard anymore. I mean: we are postponing the problem, because who, why and how will decide what RDMS must be implemented? What about XML based database? Best regards. First, we have sql. This is the start of the Database proposals, and all conforming implementation must expose an SQL interface that is conforming to the ISO standard I cited before. Then we could add xml and json (or other formats) as standard to be implemented by everyone. This is only a choice of query language and data model (trees, objects and tables), not of the DBMS (SQLite vs SQLServer), and I hope this choice will be made by the WebApps Working Group, as this is the WG chartered for the WebStorage proposal. Thirdly, we have extension, in the form of -vendor-dbms. For example, I may expect that IE will have a -ms-sqlserver. These formats are by definition out of standard, but they allow the use of extensions beyond the ISO SQL standard. -- Ivan Enderlin Developper of Hoa Framework http://hoa-project.net/ Giovanni
Re: Web Storage SQL
2009/4/9 Boris Zbarsky bzbar...@mit.edu: Giovanni Campagna wrote: We have a standard SQL, and we have DBMS-specific extensions (for SQLite, for MySQL, for SQLServer, etc.). The latest version is ISO/IEC 9075:2008 Information technology -- Database languages -- SQL, released in 2008, but actively being revised, according to the ISO page. Sort of. Extensions implies that as long as you write a query to the ISO standard in question the behavior will be the same in all databases and that you have to go out of your way to write non-interoperable queries. Yes. With extensions I mean writing start transaction instead of begin, using the backquotes instead of double quotes for identifires, using INET and spatial data types, using table options... This is not the case. A query written based on that ISO standard has different behavior in different databases, in many cases. This is obviously an implementation bug, and should be corrected in the DBMS, not in the browser or in the web page. So just telling web authors write queries based on this standard and they will work is not sufficient. Let's ignore the fact that most web authors wouldn't be able to even get their hands on this standard to see whether their query complies with it without paying a fairly hefty sum [1]. We have tutorials and guides for that You could also require UAs to implement queries exactly per this standard. Vladimir mentioned as a possible option in his original post. This is definitely the way forward (with either this SQL definition or some subset thereof) But some extensions are useful (like spatial data types or non-transactional database engines): I don't see why we should not allow them, in a vendor (browser or dbms) specific form. As usual, if you want interoperability, you use the existing, implemented, standard (or you ask the ISO to produce an updated standard with new features), else you use extensions. Use of the existing standard by authors doesn't give interoperability. Again, this is an implementation bug, not something that should affect authors -Boris [1] http://www.iso.org/iso/search.htm?qt=9075searchSubmit=Searchsort=reltype=simplepublished=on tells me that to get just the parts of ISO 9075:2008 that are obviously necessary to be able to do anything whatsoever will cost me either about $290 or $770 depending on what's in part 2 of the spec. I'm assuming none of parts 3,4,9,10,11,13,14 are relevant to our use case; this is probably a good assumption for most of them. That's for getting a PDF, of course; the PDF on CD prices are somewhat higher, and there seem to be no other provisions for looking at this specification. Unfortunately, these are the problems of ISO standards. We cannot change that. Giovanni
Re: Web Storage SQL
On Apr 8, 2009, at 2:51 PM, Vladimir Vukicevic wrote: There's been a lot of interest around the Web Storage spec (formerly part of whatwg HTML5), which exposes a SQL database to web applications to use for data storage, both for online and offline use. It presents a simple API designed for executing SQL statements and reading result rows. But there's an interesting problem with this; unlike the rest of the HtML5, this section defines a core piece of functionality in terms of an undefined chunk referenced as SQL. Treating SQL as an undefined chunk is not unprecedented. Most database APIs platforms do not require a restricted syntax of SQL to be supported in the underlying database. For example, X/OPEN SQL CLI [1] was based on SQL 92 but its successors (JDBC, ODBC) go beyond this and support any additional SQL syntax supported by the underlying data source. The initial implementations of Web Storage are both based on SQLite, and expose the dialect of SQL understood by SQLite to web content. I'm actually a big fan of SQLite, and was one of the advocates for pulling it into the Gecko platform. However, SQLite implements a variant of SQL, with a number of deviations from other SQL engines, especially in terms of the types of data that can be placed in columns. Data types are certainly relevant here because with JavaScript you never know what arguments will translate to which values and types. For example, what does NULL translate to and what about undefined? My observation is that undefined is translated to the text value undefined and NULL translates to the SQL NULL. But there is no specification for this behavior. Web content that is created to use database storage with SQLite as the backing is unlikely to work with any other backend database. Similarly, if another database was chosen as a browser's backing implementation, web content that works with it is unlikely to work with anything else. This is a serious interop problem, the root of which is that there really isn't a useful core SQL standard. SQL92 is generally taken as a base, but is often extended or altered by implementations. Even beyond the parser issues (which could be resolved by defining a strict syntax to be used by Web Storage), the underlying implementation details will affect results. There is inherent challenge in embedding as potent a capability as data access inside a platform since there is a lot of variation in its own design and use. Still, the question in my mind is not whether an unanchored reference to SQL is fine as much as whether SQL is the right way (and for years to come, the only structured way) to think of Web application's (locally persistent) data. So, the only option is for the Web Storage portion of the spec to state do what SQLite does. This isn't specified in sufficient detail anywhere to be able to reimplement it from the documents, so it would be even worse — do what this exact version of SQLite does, because there are no guarantees that SQLite won't make any incompatible changes. For example, a future SQLite 4 may introduce some changes or some new syntax which wouldn't be supported by earlier versions. Thus, it requires every single browser developer to accept SQLite as part of their platform. This may not be possible for any number of reasons, not the least of which is it essentially means that every web browser is on the hook for potential security issues within SQLite. Instead of all of this, I think it's worth stepping back and consider exactly what functionality web developers actually want. Oracle certainly supports this endeavor to understand exactly what kind of local storage capabilities are required. It's certainly much easier to say well, server developers are used to working with SQL, so let's just put SQL into the client, but it's certainly not ideal — most people working with SQL tend to end up writing wrappers to map their database into a saner object API. There is no end to how much Oracle and various other companies shield developers using their platforms from using raw SQL. There are more reasons for that than I can list here, but suffice it to say that the Web Storage spec should consider techniques that are better matched to the Web as a data access platform - i.e., in terms of URLs and HTTP methods. So, I would propose stepping back from Web Storage as written and looking at the core pieces that we need to bring to web developers. I believe that the solution needs to have a few characteristics. First, it should be able to handle large data sets efficiently; in particular, it should not require that the entire data set fit into memory at one time. Second, it should be able to execute queries over the entire dataset. Finally, it should integrate well with the web, and in particular with JavaScript. With these needs in
Re: Web Storage SQL
On Apr 9, 2009, at 8:19 AM, Boris Zbarsky wrote: Giovanni Campagna wrote: So why not adding a parameter on openDatabase() to specify what kind of database we want (and what kind of query language we will use)? I mean something like openDatabase(name, version, type, displayName, estimatedSize) where type can be any string so, for example, type = sql uses the standard SQL, type=sqlite uses SQLite extensions, type=-vendor-xyz is a vendor specific extension, etc. How does this solve the original no such thing as standard SQL, really issue? I agree that no such thing as standard SQL (or rather the fact that implementations all have extensions and divergences from the spec) is a problem. But I am not sure inventing a brand new query language and database model as proposed by Vlad is a good solution to this problem. A few thoughts off the cuff in no particular order: 1) Applications are starting to be deployed which use the SQL-based storage API, such as the mobile version of GMail. So it may be too late for us to remove SQL storage from WebKit entirely. If we want this content to interoperate with non-WebKit-based user agents, then we will ultimately need a clear spec for the SQL dialect to use, even if we also added an OODB or a relational database using some other query language. 2) It's true that the server side code for many Web sites uses an object-relational mapping layer. However, so far as I know, very few use an actual OODB. Relational databases are dominant in the market and OODBs are a rarely used niche product. Thus, I question Vlad's suggestion than a client-side OODB would sufficiently meet the needs of authors. Rather, we should make sure that the platform supports adding an object-relational mapping on top of SQL storage. 3) It's not obvious to me that designing and clearly specifying a brand new query language would be easier than specifying a dialect of SQL. Note that this may require implementations to actually parse queries themselves and possibly change them, to ensure that the accepted syntax and semantics conform to the dialect. We are ok with this. 4) It's not obvious to me that writing a spec for a query language with (afaik) a single implementation, such as jLINQ, is easier than writing a clear and correct spec for what SQLite does or some subset thereof. Thus, I think the best path forward is to spec a particular SQL dialect, even though that task may be boring and unpleasant and not as fun as inventing a new kind of database. Regards, Maciej
Re: Web Storage SQL
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Maciej Stachowiak wrote: On Apr 9, 2009, at 8:19 AM, Boris Zbarsky wrote: Giovanni Campagna wrote: So why not adding a parameter on openDatabase() to specify what kind of database we want (and what kind of query language we will use)? I mean something like openDatabase(name, version, type, displayName, estimatedSize) where type can be any string so, for example, type = sql uses the standard SQL, type=sqlite uses SQLite extensions, type=-vendor-xyz is a vendor specific extension, etc. How does this solve the original no such thing as standard SQL, really issue? I agree that no such thing as standard SQL (or rather the fact that implementations all have extensions and divergences from the spec) is a problem. But I am not sure inventing a brand new query language and database model as proposed by Vlad is a good solution to this problem. A few thoughts off the cuff in no particular order: 1) Applications are starting to be deployed which use the SQL-based storage API, such as the mobile version of GMail. So it may be too late for us to remove SQL storage from WebKit entirely. If we want this content to interoperate with non-WebKit-based user agents, then we will ultimately need a clear spec for the SQL dialect to use, even if we also added an OODB or a relational database using some other query language. 2) It's true that the server side code for many Web sites uses an object-relational mapping layer. However, so far as I know, very few use an actual OODB. Relational databases are dominant in the market and OODBs are a rarely used niche product. Thus, I question Vlad's suggestion than a client-side OODB would sufficiently meet the needs of authors. Rather, we should make sure that the platform supports adding an object-relational mapping on top of SQL storage. First, OODB's seem to be on the rise, albiet under different titles lately (AppEngine, SimpleDB, CouchDB, Persevere). Second, when using relational DBs, most devs use ORMs to interact with the DB, so they are primarily working in the object-realm, even on the server. For situations where data is transferred to the client (in data form), devs stay in the object realm for the data transfer (JSON) and on the browser in JavaScript. I don't see why we would want to force data translation back to the relational realm in the last leg on the browser when we have worked so hard to stay within object paradigm. 3) It's not obvious to me that designing and clearly specifying a brand new query language would be easier than specifying a dialect of SQL. Note that this may require implementations to actually parse queries themselves and possibly change them, to ensure that the accepted syntax and semantics conform to the dialect. We are ok with this. I agree that we shouldn't be specifying a brand new query language. I thought the idea was looking at existing query languages that would be better fits for the web/JS environment. Nothing new would need to be invented here. 4) It's not obvious to me that writing a spec for a query language with (afaik) a single implementation, such as jLINQ, is easier than writing a clear and correct spec for what SQLite does or some subset thereof. JSONPath/JSONQuery seems like a far more mature path if an alternative to SQL is to be considered, and has a pretty good set of implementations (probably at least 5 different impls). Thus, I think the best path forward is to spec a particular SQL dialect, even though that task may be boring and unpleasant and not as fun as inventing a new kind of database. In view of point #1, this may be the best course, I don't know, but I mainly wanted to correct some of the statements above. - -- Kris Zyp SitePen (503) 806-1841 http://sitepen.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkneW+oACgkQ9VpNnHc4zAzs7wCeNh6qFrbomEi/wsx2FXo5GoQG kKgAn35tW9U3iUzBoeYhfmgq9eBphoU7 =G5bP -END PGP SIGNATURE-
Re: Web Storage SQL
SQLite wasn't the first browser-accessible DBMS, nor is it the most ubiquitous choice of target. IE's Jet database engine, which is the underlying engine for Access, would seem to be the most useful target specification. After all, the overwhelming majority of what I've heard over the past decade when discussing database support in a web browser can be paraphrased as I want to port my Access application to the browser. Can I do that? Changing the storage model or semantics away from those supported by Access would seem to be a great way to throw another roadblock in front of customers whose prior desktop applications have typically relied on Access, not Oracle, and certainly not Versant et. al. ss
Re: Web Storage SQL
Maciej Stachowiak wrote: I agree that no such thing as standard SQL (or rather the fact that implementations all have extensions and divergences from the spec) is a problem. But I am not sure inventing a brand new query language and database model as proposed by Vlad is a good solution to this problem. That's fine; I'm not sure of that either. I have no particular opinion on that question, in fact. 1) Applications are starting to be deployed which use the SQL-based storage API, such as the mobile version of GMail. So it may be too late for us to remove SQL storage from WebKit entirely. This is a price of early adoption, sure. If we want this content to interoperate with non-WebKit-based user agents, then we will ultimately need a clear spec for the SQL dialect to use, even if we also added an OODB or a relational database using some other query language. That's true, but it's not a given that we want this content to interoperate as-is. Early adopters of known in-flux technologies typically realize that they might have to make changes; if a different data storage API is decided on, or if the subset of SQL that's decided on doesn't match what these apps are using, then they'll need to change. So while I agree that it might be difficult for Webkit to remove the SQL support it shipped as soon as some other approach is decided on (if that even happens), it doesn't follow that other UAs would need to ship SQL support at that point. There are strong arguments for not breaking existing content, of course, but there are also strong arguments for not having experimental implementations of early drafts completely dictate the standardization process. No opinion on your other points; this is far from my area of expertise. -Boris
Web Storage SQL
(I originally blogged this at http://blog.vlad1.com/2009/04/06/html5-web-storage-and-sql/, but Hixie rightfully pointed out that I should post it here for discussion -- doing so! Blog post is copied pretty much verbatim below, so apologies if it sounds more blog-y than post-y.) There's been a lot of interest around the Web Storage spec (formerly part of whatwg HTML5), which exposes a SQL database to web applications to use for data storage, both for online and offline use. It presents a simple API designed for executing SQL statements and reading result rows. But there's an interesting problem with this; unlike the rest of the HtML5, this section defines a core piece of functionality in terms of an undefined chunk referenced as SQL. The initial implementations of Web Storage are both based on SQLite, and expose the dialect of SQL understood by SQLite to web content. I'm actually a big fan of SQLite, and was one of the advocates for pulling it into the Gecko platform. However, SQLite implements a variant of SQL, with a number of deviations from other SQL engines, especially in terms of the types of data that can be placed in columns. Web content that is created to use database storage with SQLite as the backing is unlikely to work with any other backend database. Similarly, if another database was chosen as a browser's backing implementation, web content that works with it is unlikely to work with anything else. This is a serious interop problem, the root of which is that there really isn't a useful core SQL standard. SQL92 is generally taken as a base, but is often extended or altered by implementations. Even beyond the parser issues (which could be resolved by defining a strict syntax to be used by Web Storage), the underlying implementation details will affect results. So, the only option is for the Web Storage portion of the spec to state do what SQLite does. This isn't specified in sufficient detail anywhere to be able to reimplement it from the documents, so it would be even worse — do what this exact version of SQLite does, because there are no guarantees that SQLite won't make any incompatible changes. For example, a future SQLite 4 may introduce some changes or some new syntax which wouldn't be supported by earlier versions. Thus, it requires every single browser developer to accept SQLite as part of their platform. This may not be possible for any number of reasons, not the least of which is it essentially means that every web browser is on the hook for potential security issues within SQLite. Instead of all of this, I think it's worth stepping back and consider exactly what functionality web developers actually want. It's certainly much easier to say well, server developers are used to working with SQL, so let's just put SQL into the client, but it's certainly not ideal — most people working with SQL tend to end up writing wrappers to map their database into a saner object API. So, I would propose stepping back from Web Storage as written and looking at the core pieces that we need to bring to web developers. I believe that the solution needs to have a few characteristics. First, it should be able to handle large data sets efficiently; in particular, it should not require that the entire data set fit into memory at one time. Second, it should be able to execute queries over the entire dataset. Finally, it should integrate well with the web, and in particular with JavaScript. With these needs in mind, I think there are other options that should be considered, even beyond a subset of SQL; for example, an object-oriented database approach might serve those needs better. A good prototype example of what such a system could look like is jLINQ, which implements client-side querying on JavaScript objects and arrays. As such, a basic implementation is simple; more complex ones can have browser support for efficient indexing, triggers, rapid serialization to and deserialization from disk, etc. An implementation could even map all of this on top of an underlying SQL engine. Another option is something like CouchDB. I was also just pointed at Persevere, which looks quite cool; much in the same way as CouchDB, the same API could be implemented both client-side and server-side, for efficient online/offline switching. An approach such as one of these could well serve the web better than just throwing a SQL dialect over the web content fence. This is a conversation that I think is worth having, both to figure out what could be done about the issues with directly exposing SQL/SQLite, and also to step back and explore alternate approaches to getting the same functionality in web developers' hands. - Vlad
