Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
Thanks for all the feedbacks so far. Now that the thread became quieter, I hope we're getting agreed, at least on the basic concept. To make the points clearer, let me summarize what is being proposed (and what is not) with the quota API: - To have unified, shared quotas between different storage APIs, except for localStorage. (I'd include FileSystem API to the same group as other storage APIs.) - To have a common distinction between temporary and persistent storages across different storage APIs. - To have some sort of unified quota management system which can handle quota/usage query and request up front. It is not mandatory to guarantee the requested quota though, and UA can return a quota error whenever it sees the actual or logical storage limit is reached. As for the UI prompting, the proposing API specifies nothing about the timing or frequency with which the prompting should be made, except for a general guideline like: - For persistent storage UA should properly communicate with the user about how much disk space should be used, and must not silently discard the user data without the user's permission. I'm happy to hear more feedbacks if there are any. Thanks, Kinuko
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
Some other topics I have not (explicitly) responded yet:
On Thu, Feb 3, 2011 at 9:32 PM, João Eiras joao.ei...@gmail.com wrote:
interface StorageInfo {
Should probably be QuotaInfo or even QuotaManager ? Storage can and
will be confused with Web Storage.
I have no strong opinion here, while I mildly preferred 'StrongInfo'
as the API is meant to provide a common interface to various storage APIs.
Maybe QuotaInfo is better?
On Fri, Feb 4, 2011 at 5:37 AM, Eric Uhrhane er...@google.com wrote:
Hum, right. But then my comment does apply to requestQuota no ? No
need to ping data files to store a new value for the quota, which is
usually just a pref.
That's an implementation detail. Plus, storing a preference to a
persistent database involves file IO, which can take an arbitrarily
long time and/or fail.
An async API leaves a lot of freedom for developers, so in general we
try to use them for everything new outside the worker context.
As for sync vs async interfaces, I really would like to keep it
asynchronous. As Eric suggested UA may need some disk IO to return
the info or make a decision.
On Fri, Feb 4, 2011 at 5:52 AM, Jonas Sicking jo...@sicking.cc wrote:
One thing that you are not discussing is UA behavior when the quota
limit is reached. In Firefox 4, for IndexedDB, once the page reaches
50MB, we prompt the user if he/she wants to allow the page to store
more information. Only if the user chooses no do we start failing
writes. If the user chooses yes the fact that we reached the quota
limit is transparent to the page (apart from that a write takes a bit
longer while waiting for the prompt).
Technically we don't need to change anything about the spec to handle
this, we can just allow the UA to increase the quota limit at any
point in time if it desires. But I figured I should bring it up.
Thanks for bringing it up. Currently I do not intend to include
the UI prompting policies/issues or how UA should behave when the
quota limit is reached in the spec. And also (as I noted in my several
previous emails) I generally think that prompting to the user should
be avoided as much as possible, as far as the request from the
page does not impose major/visible pressure to the user disk.
On Fri, Feb 4, 2011 at 6:56 AM, Glenn Maynard gl...@zewt.org wrote:
On Thu, Feb 3, 2011 at 7:32 AM, João Eiras joao.ei...@gmail.com wrote:
General note: using bytes as measure is probably overkill. While I
doubt localStorage would even exceed a couple megabytes, making an
entire e-mail account available offline with indexedDb or web sql db
could easily climb to a couple GBs. Perhaps kilobytes is better ?
Saying 4*1024*1024*1024 isn't a lot of work, and using different units means
I have to remember which units each function uses.
Using bytes might be overkill, but here I have a sympathy to Glenn's
opinion as I myself often get confused by different units in different APIs,
and I'd like to keep it to use bytes, the simplest units.
Thanks,
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
2011/2/7 Kinuko Yasuda kin...@chromium.org: On Sat, Feb 5, 2011 at 7:29 AM, Glenn Maynard gl...@zewt.org wrote: On Fri, Feb 4, 2011 at 12:07 AM, Kinuko Yasuda kin...@chromium.org wrote: If we want to make the quota API treat each API differently this would make a lot sense, but I'm not fully convinced by the idea. Putting aside the localStorage for now, do you still see significant issues in having a sshared single quota? Also let me note that this API does not and should not guarantee that the app can actually *write* that amount of data into the storage, even after the quota is granted, and the UA could still stop an API to write further even if it's within the quota. I suppose that even the 2-3x difference--requesting 256 MB and actually getting 512 MB over different APIs--is acceptable, since to users, requesting storage is an order-of-magnitude question more than a precise number. As long as implementations are still allowed to implement separate quotas if they want, it's probably acceptable for this API to not reflect them precisely and to be biased towards a shared quota. If we think that most of users/developers wouldn't be interested in specifying 'giving X bytes to storage A' and 'giving Y bytes to storage B' or such that while both storage A and B eats the user's disk, then probably UA should evolve in that direction. That's my assumption and why the proposed API look like that (i.e. biased towards a shared quota). Agreed. Users won't care about the distinction, so let's make it so developers don't have to care. 2011/2/4 Ian Fette (イアンフェッティ) ife...@google.com For instance, if a user has been using a site for months, uses it frequently, and the site hits its 5GB limit but there's still 300GB free on the drive, perhaps we just give the site another 5GB and give the user a passive indication that we've done so, and let them do something if they actually care. That's interesting; reducing the amount users are nagged about things that they probably don't care about is important. It would also need to suppress prompting from calls to requestQuota if the quota increase would have been allowed automatically. The proposing API itself doesn't specify the frequency of notifications or the behavior of out-of-quota scenarios, but probably it might be worth adding some note about calling 'requestQuota()' does not (and should not) always need to result in the UA prompting, and it must be generally prohibited prompting the user too frequently. Possibly in non-normative text, but I think it's implied if you just leave it out that implementers can do whatever they want with the UI. The bottom line of whether we should prompt or not is, I suppose, if UA ask for the user's permission to store some data in the storage, the UA shouldn't delete the data without the user's permission. Yup.
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On Sat, Feb 5, 2011 at 7:29 AM, Glenn Maynard gl...@zewt.org wrote: On Fri, Feb 4, 2011 at 12:07 AM, Kinuko Yasuda kin...@chromium.org wrote: If we want to make the quota API treat each API differently this would make a lot sense, but I'm not fully convinced by the idea. Putting aside the localStorage for now, do you still see significant issues in having a sshared single quota? Also let me note that this API does not and should not guarantee that the app can actually *write* that amount of data into the storage, even after the quota is granted, and the UA could still stop an API to write further even if it's within the quota. I suppose that even the 2-3x difference--requesting 256 MB and actually getting 512 MB over different APIs--is acceptable, since to users, requesting storage is an order-of-magnitude question more than a precise number. As long as implementations are still allowed to implement separate quotas if they want, it's probably acceptable for this API to not reflect them precisely and to be biased towards a shared quota. If we think that most of users/developers wouldn't be interested in specifying 'giving X bytes to storage A' and 'giving Y bytes to storage B' or such that while both storage A and B eats the user's disk, then probably UA should evolve in that direction. That's my assumption and why the proposed API look like that (i.e. biased towards a shared quota). 2011/2/4 Ian Fette (イアンフェッティ) ife...@google.com For instance, if a user has been using a site for months, uses it frequently, and the site hits its 5GB limit but there's still 300GB free on the drive, perhaps we just give the site another 5GB and give the user a passive indication that we've done so, and let them do something if they actually care. That's interesting; reducing the amount users are nagged about things that they probably don't care about is important. It would also need to suppress prompting from calls to requestQuota if the quota increase would have been allowed automatically. The proposing API itself doesn't specify the frequency of notifications or the behavior of out-of-quota scenarios, but probably it might be worth adding some note about calling 'requestQuota()' does not (and should not) always need to result in the UA prompting, and it must be generally prohibited prompting the user too frequently. The bottom line of whether we should prompt or not is, I suppose, if UA ask for the user's permission to store some data in the storage, the UA shouldn't delete the data without the user's permission. -- Glenn Maynard
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On 2/4/2011 2:29 PM, Glenn Maynard wrote: 2011/2/4 Ian Fette (イアンフェッティ) ife...@google.com mailto:ife...@google.com For instance, if a user has been using a site for months, uses it frequently, and the site hits its 5GB limit but there's still 300GB free on the drive, perhaps we just give the site another 5GB and give the user a passive indication that we've done so, and let them do something if they actually care. That's interesting; reducing the amount users are nagged about things that they probably don't care about is important. It would also need to suppress prompting from calls to requestQuota if the quota increase would have been allowed automatically. https://developer.mozilla.org/En/Offline_resources_in_Firefox [html manifest] causes Firefox to display the notification bar the first time the user loads your application In Firefox, the appCache manifest requires a passive indicator: all new users receive one on their first page load, as a kind of installation greeting. It seems unnecessary for apps within the 5MB localStorage quota.
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On Thu, Feb 3, 2011 at 5:40 PM, Charles Pritchard ch...@visc.us wrote: I rather like the prompt in the new FF builds; it's similar to the prompt on Mobile Safari; when you get into the site, it asks you if you're ok storing data, and it allows you to specify a quota stepping. FF does a great job on applicationCache + quota in that area. The FileSystem API is a tricky thing. indexedDB is more straightforward. Let's take localStorage off the table for quota control: it's set at 5 megs in practice, and it's a very different storage mechanism. In practice, websql, appCache and indexeddb are backed by the public domain sqlite library and they share a quota on implementations. I'd like to discuss FileSystem permissions as a distinct item, as localStorage is distinct. I'd be fine with exempting localStorage from API and just lock it at 5MB. It's tricky anyway since it's a synchronous API. Further, the current limits are not defined in terms of size-on-disk but rather size-of-data. Additionally, at least firefox grants 5MB of storage to localStorage without prompting. The same is not true for any of the other storage APIs that we support, and likely won't be for any future ones we'll support. But I don't see why filesystem should be treated separately? None of the above caveats applies to it. Use case for the quota API: When a user wants to download a large set of data for offline use, Scripting needs permission unlimited space. unlimited is often set at 100 megs. The user has to stick around until their quota is full, if they've set a low quota amount. Scripting should not have to require 100 megs explicitly when a user first lands on a page. The user should only see an increase quota request following an input action (such as clicking a link or button). You're setting a lot of, to me seemingly arbitrary, restrictions here. Why does unlimited equal 100 MB? Why shouldn't the page request quota up front? Why shouldn't the page be allowed to request quota any time it chooses? I also don't see why the scenario is specific to the FileSystem API. The same thing can happen with IndexedDB. Consider for example a webmail application which wants to download a large amount of emails and store locally for offline access. The solution here is for the page to estimate how much space it needs and then request that up front. Be that when the user clicks a sync now button, or as soon as the user arrives at the page. Whatever the page thinks produces the best UI. The user can immediately make a choice to grant the quota and then the page can take its time to download the data. This seems to work as well for filesystem as for indexedDB and app-cache. / Jonas
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On Fri, Feb 4, 2011 at 12:07 AM, Kinuko Yasuda kin...@chromium.org wrote: If we want to make the quota API treat each API differently this would make a lot sense, but I'm not fully convinced by the idea. Putting aside the localStorage for now, do you still see significant issues in having a sshared single quota? Also let me note that this API does not and should not guarantee that the app can actually *write* that amount of data into the storage, even after the quota is granted, and the UA could still stop an API to write further even if it's within the quota. I suppose that even the 2-3x difference--requesting 256 MB and actually getting 512 MB over different APIs--is acceptable, since to users, requesting storage is an order-of-magnitude question more than a precise number. As long as implementations are still allowed to implement separate quotas if they want, it's probably acceptable for this API to not reflect them precisely and to be biased towards a shared quota. 2011/2/4 Ian Fette (イアンフェッティ) ife...@google.com For instance, if a user has been using a site for months, uses it frequently, and the site hits its 5GB limit but there's still 300GB free on the drive, perhaps we just give the site another 5GB and give the user a passive indication that we've done so, and let them do something if they actually care. That's interesting; reducing the amount users are nagged about things that they probably don't care about is important. It would also need to suppress prompting from calls to requestQuota if the quota increase would have been allowed automatically. -- Glenn Maynard
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On 2/4/2011 1:30 AM, Jonas Sicking wrote: On Thu, Feb 3, 2011 at 5:40 PM, Charles Pritchardch...@visc.us wrote: The FileSystem API is a tricky thing. indexedDB is more straightforward. I'd be fine with exempting localStorage from API and just lock it at 5MB. It's tricky anyway since it's a synchronous API. Further, the ... But I don't see why filesystem should be treated separately? None of the above caveats applies to it. My concern with filesystem is that it's more of an OS-level service than a UA-level service. Other applications, outside of the UA, may interact with the files contained on the FS. At some point, I think this would be a good thing. I know that mount points have been brought up before. I'm just concerned that FS is not yet well defined, whereas idb is self-contained. Use case for the quota API: When a user wants to download a large set of data for offline use, Scripting needs permission unlimited space. unlimited is often set at 100 megs. The user has to stick around until their quota is full, if they've set a low quota amount. Scripting should not have to require 100 megs explicitly when a user first lands on a page. The user should only see an increase quota request following an input action (such as clicking a link or button). You're setting a lot of, to me seemingly arbitrary, restrictions here. Why does unlimited equal 100 MB? Why shouldn't the page request quota up front? Why shouldn't the page be allowed to request quota any time it chooses? From what I recall, Chrome had a 100 meg limit, and Safari did as well. I was just noting that unlimited is not unlimited. It's not relevant to the use case. I also don't see why the scenario is specific to the FileSystem API. The same thing can happen with IndexedDB. Consider for example a webmail application which wants to download a large amount of emails and store locally for offline access. I intended it to address the Quota API we're discussing, not the storage mechanism. The solution here is for the page to estimate how much space it needs and then request that up front. Be that when the user clicks a sync now button, or as soon as the user arrives at the page. Whatever the page thinks produces the best UI. The user can immediately make a choice to grant the quota and then the page can take its time to download the data. This seems to work as well for filesystem as for indexedDB and app-cache. That's what I think, too: when the user clicks a button. I don't like seeing a pop-up when I first load the page. There's about a 5Megs taken-for-granted with localStorage; I wouldn't mind if another 5 megs without prompt were granted for other services, bringing the total untrusted space to 10 megs.
Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
Hi,
Back then there has been a long thread [1] about how/whether we want to
allow web apps to request quotas for IndexedDB, or for any of the offline
storages (i.e. IndexedDB, FileSystem, appCache, localStorage and SQL DB).
In short there were two topics discussed:
1) introducing (at least) two different storage categories,
temporary/evictable/transient and persistent/non-evictable.
2) introducing a new app-facing API to request or query quota/usage.
There were basically two API proposals, one is to request more
(persistent) storage quota and the other is to query the current
quota and used amount.
As for 1) we seem to have roughly agreed on this. FileSystem API and
IndexedDB [2] are in that direction, and it was also discussed that
probably we could add Persistent storage option to other APIs. [3]
As for 2) we haven't had a concensus yet, but It sounds beneficial
for both webapps and UA to allow an app to request quotas up front,
as with the API neither webapps nor UA need to worry about quota
error/notification handling in the middle of storage operations.
Also we hear a lot of developers wanting to know how much storage
our app can use? information up front to make their apps more
usable with some storage constraints.
To reopen the quota API discussion, I'd like to propose following
asynchronous API that works across all the offline storage APIs:
interface StorageInfo {
// storage type
const unsigned short TEMPORARY = 0;
const unsigned short PERSISTENT = 1;
// To query how much storage is available and currently in use.
void queryUsage(unsigned short storageType,
UsageCallback usageCallback);
// To request a new quota size.
void requestQuota(unsigned short storageType,
unsigned long long newOriginQuotaInBytes,
QuotaCallback quotaCallback);
};
interface UsageCallback {
void handleEvent(unsigned long long currentOriginUsageInBytes,
unsigned long long currentOriginQuotaInBytes);
};
interface QuotaCallback {
void handleEvent(unsigned long long grantedOriginQuotaInBytes);
};
With this API I assume multiple offline storage APIs share the same
single (persistent) quota and usage info per origin, but that detail
could be probably left up to UA.
Thanks in advance for any feedbacks/comments.
[1]
http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/thread.html#msg102
[2] http://www.w3.org/Bugs/Public/show_bug.cgi?id=11350
[3] http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/0253.html
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
Howdy.
interface StorageInfo {
Should probably be QuotaInfo or even QuotaManager ? Storage can and
will be confused with Web Storage.
// storage type
const unsigned short TEMPORARY = 0;
const unsigned short PERSISTENT = 1;
Only two values seem not enough for me and I disagre with the
nomenclature. Would be betteer to have
LOCAL_STORAGE, SESSION_STORAGE, WEB_SQL_DBS, INDEXED_DB and so on. But
that would create artificial dependencies with the other specs, so it
would be better if type passed to the query and request functions to
just be an opaque string which would allow any offline storage spec to
refer to this one instead and specify a type of their own like
localStorage, sessionStorage, indexedDb.
// To query how much storage is available and currently in use.
void queryUsage(unsigned short storageType,
UsageCallback usageCallback);
Given that quota values are usually just preferences, hence
lightweight enough, this function should be sync for the sake of
simplicity.
General note: using bytes as measure is probably overkill. While I
doubt localStorage would even exceed a couple megabytes, making an
entire e-mail account available offline with indexedDb or web sql db
could easily climb to a couple GBs. Perhaps kilobytes is better ?
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On Thu, Feb 3, 2011 at 4:32 AM, João Eiras joao.ei...@gmail.com wrote:
Howdy.
interface StorageInfo {
Should probably be QuotaInfo or even QuotaManager ? Storage can and
will be confused with Web Storage.
// storage type
const unsigned short TEMPORARY = 0;
const unsigned short PERSISTENT = 1;
Only two values seem not enough for me and I disagre with the
nomenclature. Would be betteer to have
LOCAL_STORAGE, SESSION_STORAGE, WEB_SQL_DBS, INDEXED_DB and so on. But
that would create artificial dependencies with the other specs, so it
would be better if type passed to the query and request functions to
just be an opaque string which would allow any offline storage spec to
refer to this one instead and specify a type of their own like
localStorage, sessionStorage, indexedDb.
The point is that apps and users should be able to communicate about
how much disk space should be used. Why would users care which API a
web app is using for its storage? If google.com needs 10MB to store
my email, why should I care whether it's in indexedDB, localStorage,
or the FileSystem?
// To query how much storage is available and currently in use.
void queryUsage(unsigned short storageType,
UsageCallback usageCallback);
Given that quota values are usually just preferences, hence
lightweight enough, this function should be sync for the sake of
simplicity.
This doesn't request just the preference; it returns how much space is
actually used. That might be an expensive operation, might require
disk IO to fetch a value from an internal database, etc. By default,
we try to make all new APIs async so as not to constrain
implementation.
General note: using bytes as measure is probably overkill. While I
doubt localStorage would even exceed a couple megabytes, making an
entire e-mail account available offline with indexedDb or web sql db
could easily climb to a couple GBs. Perhaps kilobytes is better ?
Possibly, although there's no harm in allowing extra precision, and
nothing forcing UAs not to round values anyway.
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
Should probably be QuotaInfo or even QuotaManager ? Storage can and will be confused with Web Storage. // storage type const unsigned short TEMPORARY = 0; const unsigned short PERSISTENT = 1; Only two values seem not enough for me and I disagre with the nomenclature. Would be betteer to have LOCAL_STORAGE, SESSION_STORAGE, WEB_SQL_DBS, INDEXED_DB and so on. But that would create artificial dependencies with the other specs, so it would be better if type passed to the query and request functions to just be an opaque string which would allow any offline storage spec to refer to this one instead and specify a type of their own like localStorage, sessionStorage, indexedDb. The point is that apps and users should be able to communicate about how much disk space should be used. Why would users care which API a web app is using for its storage? If google.com needs 10MB to store my email, why should I care whether it's in indexedDB, localStorage, or the FileSystem? Before storing X amount of data with an API, the code requests enough space for the storage area that API covers. That's how I understand the use case. Then it's up to the user agent to communicate that appropriately to the user, if applicable. APIs are web dev scope, not user scope, so storage areas need to be differentiated. I doubt that any browser ATM shares a single quota between all different storage types. Opera doesn't for instance. // To query how much storage is available and currently in use. void queryUsage(unsigned short storageType, UsageCallback usageCallback); Given that quota values are usually just preferences, hence lightweight enough, this function should be sync for the sake of simplicity. This doesn't request just the preference; it returns how much space is actually used. That might be an expensive operation, might require disk IO to fetch a value from an internal database, etc. Hum, right. But then my comment does apply to requestQuota no ? No need to ping data files to store a new value for the quota, which is usually just a pref.
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On Thu, Feb 3, 2011 at 9:23 AM, João Eiras joao.ei...@gmail.com wrote: Should probably be QuotaInfo or even QuotaManager ? Storage can and will be confused with Web Storage. // storage type const unsigned short TEMPORARY = 0; const unsigned short PERSISTENT = 1; Only two values seem not enough for me and I disagre with the nomenclature. Would be betteer to have LOCAL_STORAGE, SESSION_STORAGE, WEB_SQL_DBS, INDEXED_DB and so on. But that would create artificial dependencies with the other specs, so it would be better if type passed to the query and request functions to just be an opaque string which would allow any offline storage spec to refer to this one instead and specify a type of their own like localStorage, sessionStorage, indexedDb. The point is that apps and users should be able to communicate about how much disk space should be used. Why would users care which API a web app is using for its storage? If google.com needs 10MB to store my email, why should I care whether it's in indexedDB, localStorage, or the FileSystem? Before storing X amount of data with an API, the code requests enough space for the storage area that API covers. That's how I understand the use case. Then it's up to the user agent to communicate that appropriately to the user, if applicable. Why would we need to have the app specify I want to store 5MB in indexedDB and 3MB in the FileSystem? The user won't care; at most, the user needs to know This app wants to use up 8MB of space. If the user doesn't care, why make the app make two requests? With two separate requests, you might prompt the user for the first 5MB, then immediately have to prompt them again for the next 3MB. Better just to do it all in one go: the app says I want to store up to 8MB, then uses it in whichever APIs it likes. APIs are web dev scope, not user scope, so storage areas need to be differentiated. I doubt that any browser ATM shares a single quota between all different storage types. Opera doesn't for instance. I believe that's what this proposal is trying to address. // To query how much storage is available and currently in use. void queryUsage(unsigned short storageType, UsageCallback usageCallback); Given that quota values are usually just preferences, hence lightweight enough, this function should be sync for the sake of simplicity. This doesn't request just the preference; it returns how much space is actually used. That might be an expensive operation, might require disk IO to fetch a value from an internal database, etc. Hum, right. But then my comment does apply to requestQuota no ? No need to ping data files to store a new value for the quota, which is usually just a pref. That's an implementation detail. Plus, storing a preference to a persistent database involves file IO, which can take an arbitrarily long time and/or fail. An async API leaves a lot of freedom for developers, so in general we try to use them for everything new outside the worker context.
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On Thu, Feb 3, 2011 at 2:25 AM, Kinuko Yasuda kin...@chromium.org wrote:
Hi,
Back then there has been a long thread [1] about how/whether we want to
allow web apps to request quotas for IndexedDB, or for any of the offline
storages (i.e. IndexedDB, FileSystem, appCache, localStorage and SQL DB).
In short there were two topics discussed:
1) introducing (at least) two different storage categories,
temporary/evictable/transient and persistent/non-evictable.
2) introducing a new app-facing API to request or query quota/usage.
There were basically two API proposals, one is to request more
(persistent) storage quota and the other is to query the current
quota and used amount.
As for 1) we seem to have roughly agreed on this. FileSystem API and
IndexedDB [2] are in that direction, and it was also discussed that
probably we could add Persistent storage option to other APIs. [3]
As for 2) we haven't had a concensus yet, but It sounds beneficial
for both webapps and UA to allow an app to request quotas up front,
as with the API neither webapps nor UA need to worry about quota
error/notification handling in the middle of storage operations.
Also we hear a lot of developers wanting to know how much storage
our app can use? information up front to make their apps more
usable with some storage constraints.
To reopen the quota API discussion, I'd like to propose following
asynchronous API that works across all the offline storage APIs:
interface StorageInfo {
// storage type
const unsigned short TEMPORARY = 0;
const unsigned short PERSISTENT = 1;
// To query how much storage is available and currently in use.
void queryUsage(unsigned short storageType,
UsageCallback usageCallback);
// To request a new quota size.
void requestQuota(unsigned short storageType,
unsigned long long newOriginQuotaInBytes,
QuotaCallback quotaCallback);
};
interface UsageCallback {
void handleEvent(unsigned long long currentOriginUsageInBytes,
unsigned long long currentOriginQuotaInBytes);
};
interface QuotaCallback {
void handleEvent(unsigned long long grantedOriginQuotaInBytes);
};
With this API I assume multiple offline storage APIs share the same
single (persistent) quota and usage info per origin, but that detail
could be probably left up to UA.
Thanks in advance for any feedbacks/comments.
[1]
http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/thread.html#msg102
[2] http://www.w3.org/Bugs/Public/show_bug.cgi?id=11350
[3] http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/0253.html
This seems to straddle the line between simplicity and usefulness
pretty well. Definitely agree that we want to treat this as a single
quota rather than trying to separate quota for the various storage
mechanisms.
One thing that you are not discussing is UA behavior when the quota
limit is reached. In Firefox 4, for IndexedDB, once the page reaches
50MB, we prompt the user if he/she wants to allow the page to store
more information. Only if the user chooses no do we start failing
writes. If the user chooses yes the fact that we reached the quota
limit is transparent to the page (apart from that a write takes a bit
longer while waiting for the prompt).
Technically we don't need to change anything about the spec to handle
this, we can just allow the UA to increase the quota limit at any
point in time if it desires. But I figured I should bring it up.
/ Jonas
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On Thu, Feb 3, 2011 at 5:25 AM, Kinuko Yasuda kin...@chromium.org wrote:
interface StorageInfo {
// storage type
const unsigned short TEMPORARY = 0;
const unsigned short PERSISTENT = 1;
// To query how much storage is available and currently in use.
void queryUsage(unsigned short storageType,
UsageCallback usageCallback);
// To request a new quota size.
void requestQuota(unsigned short storageType,
unsigned long long newOriginQuotaInBytes,
QuotaCallback quotaCallback);
};
With this API I assume multiple offline storage APIs share the same
single (persistent) quota and usage info per origin, but that detail
could be probably left up to UA.
It'd be good to be able to request quota with a consistent interface, rather
than each API having its own interface, which seems to be what's starting to
happen for example with LocalFileSystem.
What happens if the UA doesn't share quotas between different APIs?
Requesting 250 MB of storage would actually be granting much more; for
example, 250 MB for IndexedDB and 250 MB for localStorage. (Tracking quotas
separately seems very likely, especially with filesystem APIs.)
What happens if the practical limits for an API differ? For example,
filesystems can hold as much storage there is disk space, but localStorage
may have much lower reasonable limits. UAs may want to impose a quota cap
that differs between storage APIs. If code requests 4 GB of storage in
order to store assets with the filesystem API, there's no way for this API
to differentiate between the two--allowing the 4 GB for filesystem or
IndexedDB but not for localStorage.
One possible approach: rather than having a single implementation of
StorageInfo, each storage API implements the interface;
LocalFileSystem, LocalFileSystemSync,
IDBDatabase, Storage. This way, the API doesn't need to know how to specify
a particular category of each type (storageType goes away), and the question
of separated quotas is improved.
Note, though, that requesting quota for multiple APIs in this way should
*not* result in the UA prompting the user multiple times. If requests are
made to multiple APIs in quick succession (eg. before the script returns to
the browser), it should coalesce them and only ask the user once for the
total. Prompting the user two or more for quota permissions (Allow 100 MB
for IndexedDB? Allow 500 MB for filesystem access?) would be a major UI
failure.
On Thu, Feb 3, 2011 at 7:32 AM, João Eiras joao.ei...@gmail.com wrote:
General note: using bytes as measure is probably overkill. While I
doubt localStorage would even exceed a couple megabytes, making an
entire e-mail account available offline with indexedDb or web sql db
could easily climb to a couple GBs. Perhaps kilobytes is better ?
Saying 4*1024*1024*1024 isn't a lot of work, and using different units means
I have to remember which units each function uses.
--
Glenn Maynard
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
Before storing X amount of data with an API, the code requests enough space for the storage area that API covers. That's how I understand the use case. Then it's up to the user agent to communicate that appropriately to the user, if applicable. Why would we need to have the app specify I want to store 5MB in indexedDB and 3MB in the FileSystem? The user won't care; at most, the user needs to know This app wants to use up 8MB of space. If the user doesn't care, why make the app make two requests? Because the user agent needs to differentiate which api will use each quota.
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On 2/3/2011 3:59 PM, João Eiras wrote: Because the user agent needs to differentiate which api will use each quota. But why does a user agent need to do that? It seems like that is adding unnecessary complication to the API. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On Thu, Feb 3, 2011 at 3:59 PM, João Eiras joao.ei...@gmail.com wrote: Before storing X amount of data with an API, the code requests enough space for the storage area that API covers. That's how I understand the use case. Then it's up to the user agent to communicate that appropriately to the user, if applicable. Why would we need to have the app specify I want to store 5MB in indexedDB and 3MB in the FileSystem? The user won't care; at most, the user needs to know This app wants to use up 8MB of space. If the user doesn't care, why make the app make two requests? Because the user agent needs to differentiate which api will use each quota. I think the idea with this spec is that it *requires* that implementations share quota between its various storage mechanisms. As an implementor I think this is a good idea. / Jonas
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On , Shawn Wilsher sdwi...@mozilla.com wrote: On 2/3/2011 3:59 PM, João Eiras wrote: Because the user agent needs to differentiate which api will use each quota. But why does a user agent need to do that? It seems like that is adding unnecessary complication to the API. Or adding unnecessary complication to the implementation.
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On 2/3/2011 4:35 PM, João Eiras wrote: Or adding unnecessary complication to the implementation. I'm not looking to make my job easier (as an implementer); I'm looking to make it easy to use. At least with IndexedDB, we generally choose the option that is easier for the consumer as long as it isn't extremely difficult to implement (which I do not see this being the case for quotas). Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
I rather like the prompt in the new FF builds; it's similar to the prompt on Mobile Safari; when you get into the site, it asks you if you're ok storing data, and it allows you to specify a quota stepping. FF does a great job on applicationCache + quota in that area. The FileSystem API is a tricky thing. indexedDB is more straightforward. Let's take localStorage off the table for quota control: it's set at 5 megs in practice, and it's a very different storage mechanism. In practice, websql, appCache and indexeddb are backed by the public domain sqlite library and they share a quota on implementations. I'd like to discuss FileSystem permissions as a distinct item, as localStorage is distinct. Use case for the quota API: When a user wants to download a large set of data for offline use, Scripting needs permission unlimited space. unlimited is often set at 100 megs. The user has to stick around until their quota is full, if they've set a low quota amount. Scripting should not have to require 100 megs explicitly when a user first lands on a page. The user should only see an increase quota request following an input action (such as clicking a link or button). -Charles On 2/3/2011 5:08 PM, Shawn Wilsher wrote: On 2/3/2011 4:35 PM, João Eiras wrote: Or adding unnecessary complication to the implementation. I'm not looking to make my job easier (as an implementer); I'm looking to make it easy to use. At least with IndexedDB, we generally choose the option that is easier for the consumer as long as it isn't extremely difficult to implement (which I do not see this being the case for quotas). Cheers, Shawn
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
Thanks for all of your feedbacks. So it seems like one of the main discussion topic is whether we want to have shared quota between multiple storage mechanisms (per origin, per persistent/temporary storage class) or not. On Fri, Feb 4, 2011 at 9:33 AM, Jonas Sicking jo...@sicking.cc wrote: I think the idea with this spec is that it *requires* that implementations share quota between its various storage mechanisms. As an implementor I think this is a good idea. Yes, the primary idea of the proposal is to have a shared quota. (I should have noted it more explicitly.) I think and believe that in most cases users or webapps would see few point in having differentiated quota for different storages, as long as they all use the same persistent storage (i.e. disk) and belong to the same origin (=~ same webapp). As for localStorage, as Charles and Glenn expressed their concerns it might be better to take it off the table for now. Or I would simply say we should make it a temporary-only offline storage in the quota terminology, and requesting persistent quota should have no impact on localStorage quotas. On Fri, Feb 4, 2011 at 6:56 AM, Glenn Maynard gl...@zewt.org wrote: One possible approach: rather than having a single implementation of StorageInfo, each storage API implements the interface; LocalFileSystem, LocalFileSystemSync, IDBDatabase, Storage. This way, the API doesn't need to know how to specify a particular category of each type (storageType goes away), and the question of separated quotas is improved. If we want to make the quota API treat each API differently this would make a lot sense, but I'm not fully convinced by the idea. Putting aside the localStorage for now, do you still see significant issues in having a sshared single quota? Also let me note that this API does not and should not guarantee that the app can actually *write* that amount of data into the storage, even after the quota is granted, and the UA could still stop an API to write further even if it's within the quota. Thanks, Kinuko
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
I'm not sure FileSystem is necessarily any trickier from a user's perspective -- it's all storage that is taking up space on my HD (at least, for now the filesystem is just a directory under the user's profile in Chrome). I think it fits fine in the unified quota model. (And FWIW we are looking at replacing the SQLite backend for Indexed DB in Chrome, so it's not generally safe to make such assumptions about how implementations currently work remaining the same as you have below ;-) Still though, as an end user or developer using the API, I really shouldn't have to care about such details.) As for our plans in Chrome, currently we are planning to give 5GB quota when the site asks for persistent storage space, provided the site has asked for between 0 and 5GB (there will be a user prompt that simply states X wants to store data on your computer permanently. If the site asks for more than 5GB we will show a prompt X wants to store a large amount of data on your computer permanently and grant that amount, both cases after user consent. Still something of an open question as to what happens when a site hits the limit, whether we go the FF model or try something else. It's not clear to me that we actually have to prompt the user, or whether we can make inferences from user actions. For instance, if a user has been using a site for months, uses it frequently, and the site hits its 5GB limit but there's still 300GB free on the drive, perhaps we just give the site another 5GB and give the user a passive indication that we've done so, and let them do something if they actually care. I suspect most user's probably won't care, except perhaps on constrained (mobile) devices. I think there is a lot of room for innovation beyond just ask the user :) -Ian On Thu, Feb 3, 2011 at 5:40 PM, Charles Pritchard ch...@visc.us wrote: I rather like the prompt in the new FF builds; it's similar to the prompt on Mobile Safari; when you get into the site, it asks you if you're ok storing data, and it allows you to specify a quota stepping. FF does a great job on applicationCache + quota in that area. The FileSystem API is a tricky thing. indexedDB is more straightforward. Let's take localStorage off the table for quota control: it's set at 5 megs in practice, and it's a very different storage mechanism. In practice, websql, appCache and indexeddb are backed by the public domain sqlite library and they share a quota on implementations. I'd like to discuss FileSystem permissions as a distinct item, as localStorage is distinct. Use case for the quota API: When a user wants to download a large set of data for offline use, Scripting needs permission unlimited space. unlimited is often set at 100 megs. The user has to stick around until their quota is full, if they've set a low quota amount. Scripting should not have to require 100 megs explicitly when a user first lands on a page. The user should only see an increase quota request following an input action (such as clicking a link or button). -Charles On 2/3/2011 5:08 PM, Shawn Wilsher wrote: On 2/3/2011 4:35 PM, João Eiras wrote: Or adding unnecessary complication to the implementation. I'm not looking to make my job easier (as an implementer); I'm looking to make it easy to use. At least with IndexedDB, we generally choose the option that is easier for the consumer as long as it isn't extremely difficult to implement (which I do not see this being the case for quotas). Cheers, Shawn
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On 2/3/2011 9:39 PM, Ian Fette (イアンフェッティ) wrote: I'm not sure FileSystem is necessarily any trickier from a user's perspective -- it's all storage that is taking up space on my HD (at least, for now the filesystem is just a directory under the user's profile in Chrome). I think it fits fine in the unified quota model. (And FWIW we are looking at replacing the SQLite backend for Indexed DB in Chrome, so it's not generally safe to make such assumptions about how implementations currently work remaining the same as you have below ;-) Still though, as an end user or developer using the API, I really shouldn't have to care about such details.) Thanks for sharing. I'm sure you'll see good results replacing the backend. indexedDB can be more efficient than FileSystem. FileSystem is an API to the OS service, idb is not. There are no issues with idb keynames, innodes are not an issue: idb can be optimized for target platforms, which is good when the underlying file system is a bad match for the file sizes / dir lengths / file names. FileSystem is useful when you want OS-recognized files. It's great for moving files in and out of the browser sandbox; OS indexing services, and so on. That's been my experience: We're using file system, as we do want individual files to be show up to the OS, but for generated content, especially thumbnails of images, that data would be better stored in idb. Lots of small files which would be better stashed in a data store. -Charles
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On Thu, Feb 3, 2011 at 10:07 PM, Charles Pritchard ch...@visc.us wrote: On 2/3/2011 9:39 PM, Ian Fette (イアンフェッティ) wrote: I'm not sure FileSystem is necessarily any trickier from a user's perspective -- it's all storage that is taking up space on my HD (at least, for now the filesystem is just a directory under the user's profile in Chrome). I think it fits fine in the unified quota model. (And FWIW we are looking at replacing the SQLite backend for Indexed DB in Chrome, so it's not generally safe to make such assumptions about how implementations currently work remaining the same as you have below ;-) Still though, as an end user or developer using the API, I really shouldn't have to care about such details.) Thanks for sharing. I'm sure you'll see good results replacing the backend. indexedDB can be more efficient than FileSystem. FileSystem is an API to the OS service, idb is not. There are no issues with idb keynames, innodes are not an issue: idb can be optimized for target platforms, which is good when the underlying file system is a bad match for the file sizes / dir lengths / file names. FileSystem is useful when you want OS-recognized files. It's great for moving files in and out of the browser sandbox; OS indexing services, and so on. That's been my experience: We're using file system, as we do want individual files to be show up to the OS, but for generated content, especially thumbnails of images, that data would be better stored in idb. Lots of small files which would be better stashed in a data store. I'd rather not de-rail the thread, that said I am surprised you see this behaviour. It should be the exact opposite - we should be able to deliver much better performance from the filesystem api for binary data. Test cases where that's not the case are always welcome. -Charles
Re: Quota API to query/request quota for offline storages (e.g. IndexedDB, FileSystem)
On 2/3/2011 10:36 PM, Ian Fette (イアンフェッティ) wrote: On Thu, Feb 3, 2011 at 10:07 PM, Charles Pritchard ch...@visc.us mailto:ch...@visc.us wrote: On 2/3/2011 9:39 PM, Ian Fette (イアンフェッティ) wrote: I'm not sure FileSystem is necessarily any trickier from a user's perspective -- it's all storage that is taking up space on my HD (at least, for now the filesystem is just a directory under the user's profile in Chrome). I think it fits fine in the unified quota model. (And FWIW we are looking at replacing the SQLite backend for Indexed DB in Chrome, so it's not generally safe to make such assumptions about how implementations currently work remaining the same as you have below ;-) Still though, as an end user or developer using the API, I really shouldn't have to care about such details.) Thanks for sharing. I'm sure you'll see good results replacing the backend. indexedDB can be more efficient than FileSystem. FileSystem is an API to the OS service, idb is not. There are no issues with idb keynames, innodes are not an issue: idb can be optimized for target platforms, which is good when the underlying file system is a bad match for the file sizes / dir lengths / file names. FileSystem is useful when you want OS-recognized files. It's great for moving files in and out of the browser sandbox; OS indexing services, and so on. That's been my experience: We're using file system, as we do want individual files to be show up to the OS, but for generated content, especially thumbnails of images, that data would be better stored in idb. Lots of small files which would be better stashed in a data store. I'd rather not de-rail the thread, that said I am surprised you see this behaviour. It should be the exact opposite - we should be able to deliver much better performance from the filesystem api for binary data. Test cases where that's not the case are always welcome. I was referring to directory listing and disk usage. small files can waste a lot of space on older file systems. I mean to talk about the FileEntry -based APIs. I'm not seeing behavior where idb is beating the filesystem api; I'd prefer to use idb, but the sqlite implementation does not have an optimized large object store.
