Re: Updating Quota API: Promise, Events and some more
On Tue, Aug 27, 2013 at 4:26 AM, Jonas Sicking wrote: > Thanks, that explains the concern. > > However the proposed solution doesn't seem to solve the problem > particularly well, for two reasons: > > First off, an API which lets the website track how much data it is > storing, measured in bytes, doesn't seem like it solves the problem. > You also need to track quota as it is shrinking. Otherwise you won't > know when you are starting to butt up against the quota limit which > might cause you to get evicted more quickly. This also indicates that > setting limits in bytes, and getting notified when you cross those > limits, is the wrong API, since the limit would have to be adjusted > any time the available quota shrinks. > > The second problem is harder to address. The problem is that the used > quota could go down even when the website isn't open. We'd need > something like event pages, and an event fired at those, to tackle > this problem. > I believe the first problem can be addressed by polishing API details, but I have to agree that the second problem would be much better addressed in conjunction with event pages, and it might be better to wait until it matures. Unfortunately event pages, and how to fire events at them, is > something that's still being developed. Which makes it hard to know > what this API should look like. Until we have that figured out, maybe > something like this would be ok to put in the draft: > > [Constructor(StorageType type, unsigned long rate)] > interface StorageWatcher { > readonly attribute StorageType type; > readonly attribute unsigned long rate; > > attribute EventHandler onstoragechange; > attribute EventHandler onquotachange; > }; > I'm sold! Also I think in this way having only onstoragechange would be enough to start with. Let me summarize what came out so far: enum StorageType { "temporary", "persistent" }; partial interface Navigator { readonly attribute StorageQuota storageQuota; }; [NoInterfaceObject] interface StorageInfo { unsigned long long usage; unsigned long long quota; }; [NoInterfaceObject] interface StorageQuota { readonly attribute StorageType[] supportedTypes; // Queries the current storage info. Promise getInfo(StorageType type); // Requests a new quota for persistent storage. Promise requestPersistentQuota(unsigned long long newQuota); }; [Constructor(StorageType type, unsigned long rate)] interface StorageWatcher : EventTarget { readonly attribute StorageType type; readonly attribute unsigned long rate; attribute EventHandler onstoragechange; }; interface StorageEvent : Event { readonly attribute unsigned long long usage; readonly attribute unsigned long long quota; }; Thanks for all the discussion so far. I'll repost on public-webapps when I update the draft. (Would it be better to send the draft to public-script-coord too?) (And as always more feedback / comments are always welcome) Kinuko / Jonas >
Re: Updating Quota API: Promise, Events and some more
On Mon, Aug 26, 2013 at 1:39 AM, Kinuko Yasuda wrote: > > Going back to the use case 2, let me try to explain a sample scenario: > > Let's say website A has stored some data in temporary storage. The amount of > data was less than the per-site quota at the time (otherwise the write must > have failed), since there was lot of available space. But then the > remaining disk space has gotten smaller, and Chrome realizes that it needs > to make some more space. If website A has done nothing until then Chrome > will eventually evict site A (before or after evicting other sites), but if > it could have told A that its remaining storage was less than before or was > getting over-budget, A might have been able to react the events and to > reduce some data so that Chrome may not need to evict any data. This > scenario is over-simplified and when there're multiple websites just one > site reducing its data may not do any favor to the site (except that the > LRU-policy will implicitly help the site's data stay longer than others), > but in general if the site is constantly being used/opened and keeps being a > good citizen in terms of storage usage, the site would have much less > possibility to be evicted. Does it make sense to you? Thanks, that explains the concern. However the proposed solution doesn't seem to solve the problem particularly well, for two reasons: First off, an API which lets the website track how much data it is storing, measured in bytes, doesn't seem like it solves the problem. You also need to track quota as it is shrinking. Otherwise you won't know when you are starting to butt up against the quota limit which might cause you to get evicted more quickly. This also indicates that setting limits in bytes, and getting notified when you cross those limits, is the wrong API, since the limit would have to be adjusted any time the available quota shrinks. The second problem is harder to address. The problem is that the used quota could go down even when the website isn't open. We'd need something like event pages, and an event fired at those, to tackle this problem. Unfortunately event pages, and how to fire events at them, is something that's still being developed. Which makes it hard to know what this API should look like. Until we have that figured out, maybe something like this would be ok to put in the draft: [Constructor(StorageType type, unsigned long rate)] interface StorageWatcher { readonly attribute StorageType type; readonly attribute unsigned long rate; attribute EventHandler onstoragechange; attribute EventHandler onquotachange; }; / Jonas
Re: Updating Quota API: Promise, Events and some more
On Sat, Aug 24, 2013 at 3:45 PM, Jonas Sicking wrote: > On Aug 23, 2013 12:30 AM, "Kinuko Yasuda" wrote: > > On Tue, Aug 20, 2013 at 1:49 AM, Jonas Sicking wrote: > >> > >> On Mon, Aug 19, 2013 at 9:32 AM, Kinuko Yasuda > wrote: > >> >> Actually, *only* having onstoragechange would solve both the use case > >> >> of detecting when you are running low on storage, as well as when you > >> >> are "out of the woods". Especially if we provide both the quota and > >> >> the amount of used space in the event. > >> > > >> > Right. I think probably we should add either "a combination of > >> > storagelow + storageok" or "storagechange", and the latter would be > >> > more complete (and possibly be simpler). > >> > > >> > One concern for adding storagechange event is how frequently > >> > the event is going to be fired. In the former email I tentatively > wrote > >> > this to be fired every 1 second at most frequent (or whenever the > >> > storage info is changed), but this may be still too frequent if many > >> > apps are contending to a storage space. Do you have any opinion > >> > or preference on this? > >> > >> I think firing every second while writes are happening would actually > >> be ok. Seems like you are spending a lot of battery power anyway at > >> that point and so an extra callback a second doesn't seem like a huge > >> deal. > >> > >> That said, doing it more rarely, like every 5 seconds, sounds ok too. > >> > >> One solution is to punt the choice to the page. I.e. something like: > >> > >> [Constructor(StorageType type, unsigned long rate)] > >> interface StorageWatcher { > >> readonly attribute StorageType type; > >> readonly attribute unsigned long rate; > >> > >> attribute EventHandler onstoragechange; > >> }; > > > > > > > > Sorry for my slow response, while your suggestion looks pretty reasonable > > I've been thinking about the battery consumption and possible usecases > > when events could be useful. Major situations that came up to me are: > > > > 1. To watch the remaining storage capacity while the app is actively > writing > >or deleting data (<-- this is basically the case you mentioned) > > 2. To be notified when the (shared) storage capacity is getting tight and > >their data may be evicted if they don't adjust their data size > > > > 1. could be useful if an app wants to keep their data within a certain > > size (in temporary storage where they cannot specify an explicit quota), > > while 2. is the situation I often hear requests for, and would be nice > not > > to waste too much battery. > > > > In either case I guessed frequency wouldn't be interesting but the actual > > storage sizes may be. What if we allow apps to specify specific size > > thresholds? The resulting API looks a bit noisy, but say, > > > > [Constructor(StorageType type, long long remainingStorageLow, optional > long long usageHigh)] > > interface StorageWatcher { > > readonly attribute StorageType type; > > > > // Thresholds > > readonly long long remainingStorageLow; > > readonly long long usageHigh; > > > > attribute EventHandler onstoragelow; > > attribute EventHandler onstorageok; > > }; > > > > Too rich? > I don't quite understand usecase 2. Are you saying that a website want to > be notified if it is at risk of it itself getting evicted? Or if it is at > risk of evicting other websites? > Basically the former case, but maybe in a more implicit way. In general keeping bigger data while the remaining space is tight increases the chance of eviction, either on its own website or on other sites. If a website stores data that have different priorities in temporary storage, it may want to cleanup their less-important data to decrease the chance that their more-important data gets evicted (though it may just help other sites store more data). In the Gecko implementation neither really makes sense: > > As for a website getting itself evicted, we're planning on evicting > websites on a strictly LRU basis. That means that storing more data doesn't > increase the risk of getting evicted. There is no point at which a website > could write so much data that we'd immediately, or more aggressively would > evict it. The only limit is the quota limit that each website gets for > temporary storage, but that limit doesn't really affect eviction, but > rather just how much data can be written. > > As for a website causing other websites to get evicted, this is something > that can, and probably will, happen as soon as a websites starts using > temporary storage. Because the temporary storage has a global limit, and > can be used without prompting, we're counting on eventually reaching that > limit. Once that is done, as soon as a website starts writing *something* > will need to get evicted. So trying to avoid evicting others this likely > means not use temporary storage at all. > > Could you help me understand the policies that Chrome is using which > motivates use case 2? And exactly what
Re: Updating Quota API: Promise, Events and some more
On Aug 23, 2013 12:30 AM, "Kinuko Yasuda" wrote: > > On Tue, Aug 20, 2013 at 1:49 AM, Jonas Sicking wrote: >> >> On Mon, Aug 19, 2013 at 9:32 AM, Kinuko Yasuda wrote: >> >> Actually, *only* having onstoragechange would solve both the use case >> >> of detecting when you are running low on storage, as well as when you >> >> are "out of the woods". Especially if we provide both the quota and >> >> the amount of used space in the event. >> > >> > Right. I think probably we should add either "a combination of >> > storagelow + storageok" or "storagechange", and the latter would be >> > more complete (and possibly be simpler). >> > >> > One concern for adding storagechange event is how frequently >> > the event is going to be fired. In the former email I tentatively wrote >> > this to be fired every 1 second at most frequent (or whenever the >> > storage info is changed), but this may be still too frequent if many >> > apps are contending to a storage space. Do you have any opinion >> > or preference on this? >> >> I think firing every second while writes are happening would actually >> be ok. Seems like you are spending a lot of battery power anyway at >> that point and so an extra callback a second doesn't seem like a huge >> deal. >> >> That said, doing it more rarely, like every 5 seconds, sounds ok too. >> >> One solution is to punt the choice to the page. I.e. something like: >> >> [Constructor(StorageType type, unsigned long rate)] >> interface StorageWatcher { >> readonly attribute StorageType type; >> readonly attribute unsigned long rate; >> >> attribute EventHandler onstoragechange; >> }; > > > > Sorry for my slow response, while your suggestion looks pretty reasonable > I've been thinking about the battery consumption and possible usecases > when events could be useful. Major situations that came up to me are: > > 1. To watch the remaining storage capacity while the app is actively writing >or deleting data (<-- this is basically the case you mentioned) > 2. To be notified when the (shared) storage capacity is getting tight and >their data may be evicted if they don't adjust their data size > > 1. could be useful if an app wants to keep their data within a certain > size (in temporary storage where they cannot specify an explicit quota), > while 2. is the situation I often hear requests for, and would be nice not > to waste too much battery. > > In either case I guessed frequency wouldn't be interesting but the actual > storage sizes may be. What if we allow apps to specify specific size > thresholds? The resulting API looks a bit noisy, but say, > > [Constructor(StorageType type, long long remainingStorageLow, optional long long usageHigh)] > interface StorageWatcher { > readonly attribute StorageType type; > > // Thresholds > readonly long long remainingStorageLow; > readonly long long usageHigh; > > attribute EventHandler onstoragelow; > attribute EventHandler onstorageok; > }; > > Too rich? I don't quite understand usecase 2. Are you saying that a website want to be notified if it is at risk of it itself getting evicted? Or if it is at risk of evicting other websites? In the Gecko implementation neither really makes sense: As for a website getting itself evicted, we're planning on evicting websites on a strictly LRU basis. That means that storing more data doesn't increase the risk of getting evicted. There is no point at which a website could write so much data that we'd immediately, or more aggressively would evict it. The only limit is the quota limit that each website gets for temporary storage, but that limit doesn't really affect eviction, but rather just how much data can be written. As for a website causing other websites to get evicted, this is something that can, and probably will, happen as soon as a websites starts using temporary storage. Because the temporary storage has a global limit, and can be used without prompting, we're counting on eventually reaching that limit. Once that is done, as soon as a website starts writing *something* will need to get evicted. So trying to avoid evicting others this likely means not use temporary storage at all. Could you help me understand the policies that Chrome is using which motivates use case 2? And exactly what type of eviction that is the concern in that use case? / Jonas
Re: Updating Quota API: Promise, Events and some more
On Tue, Aug 20, 2013 at 1:49 AM, Jonas Sicking wrote: > On Mon, Aug 19, 2013 at 9:32 AM, Kinuko Yasuda > wrote: > >> Actually, *only* having onstoragechange would solve both the use case > >> of detecting when you are running low on storage, as well as when you > >> are "out of the woods". Especially if we provide both the quota and > >> the amount of used space in the event. > > > > Right. I think probably we should add either "a combination of > > storagelow + storageok" or "storagechange", and the latter would be > > more complete (and possibly be simpler). > > > > One concern for adding storagechange event is how frequently > > the event is going to be fired. In the former email I tentatively wrote > > this to be fired every 1 second at most frequent (or whenever the > > storage info is changed), but this may be still too frequent if many > > apps are contending to a storage space. Do you have any opinion > > or preference on this? > > I think firing every second while writes are happening would actually > be ok. Seems like you are spending a lot of battery power anyway at > that point and so an extra callback a second doesn't seem like a huge > deal. > > That said, doing it more rarely, like every 5 seconds, sounds ok too. > > One solution is to punt the choice to the page. I.e. something like: > > [Constructor(StorageType type, unsigned long rate)] > interface StorageWatcher { > readonly attribute StorageType type; > readonly attribute unsigned long rate; > > attribute EventHandler onstoragechange; > }; > Sorry for my slow response, while your suggestion looks pretty reasonable I've been thinking about the battery consumption and possible usecases when events could be useful. Major situations that came up to me are: 1. To watch the remaining storage capacity while the app is actively writing or deleting data (<-- this is basically the case you mentioned) 2. To be notified when the (shared) storage capacity is getting tight and their data may be evicted if they don't adjust their data size 1. could be useful if an app wants to keep their data within a certain size (in temporary storage where they cannot specify an explicit quota), while 2. is the situation I often hear requests for, and would be nice not to waste too much battery. In either case I guessed frequency wouldn't be interesting but the actual storage sizes may be. What if we allow apps to specify specific size thresholds? The resulting API looks a bit noisy, but say, [Constructor(StorageType type, long long remainingStorageLow, optional long long usageHigh)] interface StorageWatcher { readonly attribute StorageType type; // Thresholds readonly long long remainingStorageLow; readonly long long usageHigh; attribute EventHandler onstoragelow; attribute EventHandler onstorageok; }; Too rich? > / Jonas >
Re: Updating Quota API: Promise, Events and some more
On Mon, Aug 19, 2013 at 9:32 AM, Kinuko Yasuda wrote: >> Actually, *only* having onstoragechange would solve both the use case >> of detecting when you are running low on storage, as well as when you >> are "out of the woods". Especially if we provide both the quota and >> the amount of used space in the event. > > Right. I think probably we should add either "a combination of > storagelow + storageok" or "storagechange", and the latter would be > more complete (and possibly be simpler). > > One concern for adding storagechange event is how frequently > the event is going to be fired. In the former email I tentatively wrote > this to be fired every 1 second at most frequent (or whenever the > storage info is changed), but this may be still too frequent if many > apps are contending to a storage space. Do you have any opinion > or preference on this? I think firing every second while writes are happening would actually be ok. Seems like you are spending a lot of battery power anyway at that point and so an extra callback a second doesn't seem like a huge deal. That said, doing it more rarely, like every 5 seconds, sounds ok too. One solution is to punt the choice to the page. I.e. something like: [Constructor(StorageType type, unsigned long rate)] interface StorageWatcher { readonly attribute StorageType type; readonly attribute unsigned long rate; attribute EventHandler onstoragechange; }; / Jonas
Re: Updating Quota API: Promise, Events and some more
On Sat, Aug 17, 2013 at 3:01 AM, Jonas Sicking wrote: > On Fri, Aug 16, 2013 at 6:56 AM, Kinuko Yasuda > wrote: > > I had a few bike-sheddy comments as well. > > > Now the new draft looks like: > > > > enum StorageType { "temporary", "persistent" }; > > > > partial interface Navigator { > > readonly attribute StorageQuota storageQuota; > > }; > > > > [NoInterfaceObject] interface StorageInfo { > > unsigned long long usageInBytes; > > unsigned long long quotaInBytes; > > }; > > I think these can be simplified to simply 'usage' and 'quota'. The > fact that it's in bytes doesn't really need to be spelled out in the > property name I think. Feedback that we've gotten from authors is that > they prefer short names. Ok, I don't have strong opinion on the names. Some developers seem to prefer clearer names but I think just 'usage' and 'quota' would work in this case. > > [NoInterfaceObject] interface StorageQuota { > > readonly attribute StorageType[] supportedTypes; > > > > // Queries the current storage info. > > Promise queryStorageInfo(StorageType type); > > I think simply 'getInfo' might be enough here. It should already be > obvious that it's storage-quota related. Sounds good. Actually I wondered the same. > > // Requests a new quota for persistent storage. > > Promise requestPersistentQuota(unsigned long long > > newQuotaInBytes); > > > > // Creates a new StorageWatcher object to watch the storage changes. > > StorageWatcher createStorageWatcher(StorageType type) > > This could probably simply be done using a constructor. I.e. > > new StorageWatcher("persistent"); Yup... I once wondered we may not want to expose more constructors on global namespace, but constructor way actually makes it simpler. > [NoInterfaceObject] interface createStorageWatcher { > > // Fired when when the remaining storage space becomes lower > > // than 10% of all available space for the type, or before the quota > > backend > > // triggers eviction (for temporary case), whichever happens first. > > attribute EventHandler onstoragelow; > > }; > > Actually, the other two events made sense to me. Without onstorageok > the page has to poll queryStorageInfo to know when it's "out of the > woods". > > Actually, *only* having onstoragechange would solve both the use case > of detecting when you are running low on storage, as well as when you > are "out of the woods". Especially if we provide both the quota and > the amount of used space in the event. Right. I think probably we should add either "a combination of storagelow + storageok" or "storagechange", and the latter would be more complete (and possibly be simpler). One concern for adding storagechange event is how frequently the event is going to be fired. In the former email I tentatively wrote this to be fired every 1 second at most frequent (or whenever the storage info is changed), but this may be still too frequent if many apps are contending to a storage space. Do you have any opinion or preference on this? > / Jonas >
Re: Updating Quota API: Promise, Events and some more
On Fri, Aug 16, 2013 at 6:56 AM, Kinuko Yasuda wrote: I had a few bike-sheddy comments as well. > Now the new draft looks like: > > enum StorageType { "temporary", "persistent" }; > > partial interface Navigator { > readonly attribute StorageQuota storageQuota; > }; > > [NoInterfaceObject] interface StorageInfo { > unsigned long long usageInBytes; > unsigned long long quotaInBytes; > }; I think these can be simplified to simply 'usage' and 'quota'. The fact that it's in bytes doesn't really need to be spelled out in the property name I think. Feedback that we've gotten from authors is that they prefer short names. > [NoInterfaceObject] interface StorageQuota { > readonly attribute StorageType[] supportedTypes; > > // Queries the current storage info. > Promise queryStorageInfo(StorageType type); I think simply 'getInfo' might be enough here. It should already be obvious that it's storage-quota related. > // Requests a new quota for persistent storage. > Promise requestPersistentQuota(unsigned long long > newQuotaInBytes); > > // Creates a new StorageWatcher object to watch the storage changes. > StorageWatcher createStorageWatcher(StorageType type) This could probably simply be done using a constructor. I.e. new StorageWatcher("persistent"); > [NoInterfaceObject] interface createStorageWatcher { > // Fired when when the remaining storage space becomes lower > // than 10% of all available space for the type, or before the quota > backend > // triggers eviction (for temporary case), whichever happens first. > attribute EventHandler onstoragelow; > }; Actually, the other two events made sense to me. Without onstorageok the page has to poll queryStorageInfo to know when it's "out of the woods". Actually, *only* having onstoragechange would solve both the use case of detecting when you are running low on storage, as well as when you are "out of the woods". Especially if we provide both the quota and the amount of used space in the event. / Jonas
Re: Updating Quota API: Promise, Events and some more
On Fri, Aug 16, 2013 at 3:25 AM, Jonas Sicking wrote: > On Thu, Aug 15, 2013 at 9:44 AM, Kinuko Yasuda > wrote: > >> requestQuota for temporary storage: > >> I'd like to better understand the use case for requestQuota for > temporary > >> storage. Are implementations allowed to bring up a prompt when an > increased > >> temporary storage quota is requested? I thought one of the big use > cases for > >> temporary storage was that it would never trigger prompts, though > obviously > >> an exception could be made for the explicit requestQuota function. > >> > >> If the idea is for it not to bring up a prompt, why would we not simply > >> always allocate the largest value that could be requested? > >> > >> Also, I think that we for temporary storage in Gecko would not want to > >> guarantee that the allocated quota (as reported by queryQuota) for > temporary > >> storage will remain allocated for longer than the running page. I.e. > once a > >> page is closed, we might want to release the quota allocated and give > it to > >> another website. > > > > This is a very good feedback/question. In Chrome we actually don't > really > > support requestQuota for temporary storage, we just silently ignore the > > request but returns the current available quota to the requesting > webapp. I > > also agree that it won't be desirable to preserve the quota allocated for > > temporary storage. > > > > Seems like having this method take a storage type just confuses > > readers/implementors, we should probably just drop the storage type > > parameter and rename it to something like 'requestPersistentQuota', or > only > > add the method to 'navigator.persistentStorage' attribute? > > I think adding storageQuota.requestPersistentQuota would be a good > solution. It seems cleaner than having a separate > navigator.persistentStorage object. Agreed, sounds good to me too. Based on the feedbacks (thanks all) I've dropped {get,set}StorageType and storageok and storagechange events from the draft. (New events can be added later whenever we find them useful with more concrete use cases) Now the new draft looks like: enum StorageType { "temporary", "persistent" }; partial interface Navigator { readonly attribute StorageQuota storageQuota; }; [NoInterfaceObject] interface StorageInfo { unsigned long long usageInBytes; unsigned long long quotaInBytes; }; [NoInterfaceObject] interface StorageQuota { readonly attribute StorageType[] supportedTypes; // Queries the current storage info. Promise queryStorageInfo(StorageType type); // Requests a new quota for persistent storage. Promise requestPersistentQuota(unsigned long long newQuotaInBytes); // Creates a new StorageWatcher object to watch the storage changes. StorageWatcher createStorageWatcher(StorageType type) }; [NoInterfaceObject] interface createStorageWatcher { // Fired when when the remaining storage space becomes lower // than 10% of all available space for the type, or before the quota backend // triggers eviction (for temporary case), whichever happens first. attribute EventHandler onstoragelow; }; More feedback on this rough draft is always welcome. Also I'm going to start updating the editor's draft based on this one and discussions we had on this list. > / Jonas >
Re: Updating Quota API: Promise, Events and some more
On Thu, Aug 15, 2013 at 9:44 AM, Kinuko Yasuda wrote: >> requestQuota for temporary storage: >> I'd like to better understand the use case for requestQuota for temporary >> storage. Are implementations allowed to bring up a prompt when an increased >> temporary storage quota is requested? I thought one of the big use cases for >> temporary storage was that it would never trigger prompts, though obviously >> an exception could be made for the explicit requestQuota function. >> >> If the idea is for it not to bring up a prompt, why would we not simply >> always allocate the largest value that could be requested? >> >> Also, I think that we for temporary storage in Gecko would not want to >> guarantee that the allocated quota (as reported by queryQuota) for temporary >> storage will remain allocated for longer than the running page. I.e. once a >> page is closed, we might want to release the quota allocated and give it to >> another website. > > This is a very good feedback/question. In Chrome we actually don't really > support requestQuota for temporary storage, we just silently ignore the > request but returns the current available quota to the requesting webapp. I > also agree that it won't be desirable to preserve the quota allocated for > temporary storage. > > Seems like having this method take a storage type just confuses > readers/implementors, we should probably just drop the storage type > parameter and rename it to something like 'requestPersistentQuota', or only > add the method to 'navigator.persistentStorage' attribute? I think adding storageQuota.requestPersistentQuota would be a good solution. It seems cleaner than having a separate navigator.persistentStorage object. / Jonas
Re: Updating Quota API: Promise, Events and some more
Hi Jonas, On Thu, Aug 15, 2013 at 1:20 PM, Jonas Sicking wrote: > Hi Kinuko, > > Very excited to see movement on this API. We are very interested in > implementing this in Gecko once we reach an API that has the right feature > set and polish! The recent changes definitely brings us closer to that > which is great. Definitely looking forward to working with you on this. > It's great to know that Mozilla is interested in this API, and thanks very much for the valuable feedbacks. > I have a few pieces of high-level fedback, before getting into details. > > Persistence type granularity: > First off, I don't think that we in Gecko are interested in adding support > for per-objectStore or per-Entry persistence type. > > For a couple of reasons we want to use separate files for temporary and > persistent storage. This enables us to both measure the amount of > temporary/persistent storage, as well as clear temporary storage, without > doing slow operations involving running transactions or reading a lot of > files. > > Additionally, in our backends we wouldn't have the ability to measure how > much storage a particular objectStore is using. Or to measure how much the > metadata for a particular Entry uses. > > So I'd prefer to only support setting storage type on a per-database and > per-filesystem basis. > > getStorageType/setStorageType: > I'm not convinced that it makes sense to have a cross-storage-API API for > moving data between the different storage types. > > I definitely think that we need to support moving IDB databases or > Filesystems between temporary and permanent storage. In particular moving > data from temporary storage to persistent storage seems very useful. > > But I think this is something that we should do on a per-storage-API > basis. I.e. the API for moving an IDB database should probably be part of > IDB. And moving data between filesystems already exists in both the Google > Filesystem proposal as well as the Mozilla filesystem proposal. > I mostly agree with your points that 1. it's desirable to support moving data across temporary and persistent storage, and 2. it's more natural to support it on a per-storage-API basis. Also regarding the persistence type granularity we (Chrome) are actually in a very similar situation, i.e. it's not very easy to support per-objectStore or per-Entry persistence type. As I wrote in the response to Joshua's feedback the real 'desire' to have this kind of methods is to give more control to webapps on which data to be kept or evicted when the storage is getting tight, and I'm not willing to stick to the current proposal. If each storage API can support the desire of mobile/offline webapps that'd be good enough (if not great). After hearing feedbacks from you and Joshua, and knowing that IDB is going to support storage types in a near future, I feel we can drop these methods from this draft. (I'm also going to reach out the original requester to see if it works for them too) > requestQuota for temporary storage: > I'd like to better understand the use case for requestQuota for temporary > storage. Are implementations allowed to bring up a prompt when an increased > temporary storage quota is requested? I thought one of the big use cases > for temporary storage was that it would never trigger prompts, though > obviously an exception could be made for the explicit requestQuota function. > > If the idea is for it not to bring up a prompt, why would we not simply > always allocate the largest value that could be requested? > > Also, I think that we for temporary storage in Gecko would not want to > guarantee that the allocated quota (as reported by queryQuota) for > temporary storage will remain allocated for longer than the running page. > I.e. once a page is closed, we might want to release the quota allocated > and give it to another website. > This is a very good feedback/question. In Chrome we actually don't really support requestQuota for temporary storage, we just silently ignore the request but returns the current available quota to the requesting webapp. I also agree that it won't be desirable to preserve the quota allocated for temporary storage. Seems like having this method take a storage type just confuses readers/implementors, we should probably just drop the storage type parameter and rename it to something like 'requestPersistentQuota', or only add the method to 'navigator.persistentStorage' attribute? > / Jonas > > On Tue, Aug 13, 2013 at 10:57 PM, Kinuko Yasuda > wrote: > > Hi all, > > > > It's been a while since Quota API's FPWD ( > http://www.w3.org/TR/quota-api/) > > was published and we've gotten several requests/feedbacks so far. > > To address some of the requests and to gain more consensus, I'm thinking > > about making following changes to the Quota API: > > > > * Use Promises rather than callbacks > > * Add Events to notify webapps of important changes in the local storage > > space > > * Establish a way to get and
Re: Updating Quota API: Promise, Events and some more
Hi Joshua, On Thu, Aug 15, 2013 at 6:33 AM, Joshua Bell wrote: > > Promise getStorageType((IDBObjectStore or Database or >> Entry) object); >> Promise setStorageType((IDBObjectStore or Database or Entry) >> object, StorageType type); >> > > For IndexedDB, an object store is (probably) too low a level to specify a > storage type; ISTM that a database makes more sense as the level of > granularity for specifying storage, since that avoids the complexity of a > store disappearing out from within a database. Was the use of object store > here intentional? > > From an API perspective, passing an IDBObjectStore instance also doesn't > make much sense as that sort of object is really a transaction-specific > handle. Before delving deeply, my gut reaction is that to fit into this API > you would need to pass an IDBDatabase connection object, and it would > generate an error unless called during a versionchange transaction (which > guarantees there are no other connections). > Thanks for the very informative feedback. The use of object store wasn't really intentional (it was partly due to lack of my understanding), and if IDBDatabase connection object makes more sense I'm open to making the change (more discussion follows). That still feels like an odd mix of two APIs. An approach that we (Moz + > Google) have talked about would be to extend the IDBFactory.open() call > with an options dictionary, e.g. > > request = indexedDB.open({ name: ..., version: ..., storage: "temporary" > }); > Honestly I was debating whether I should add these methods to this proposal, as I had the same feeling as yours when I heard the original request, but if IDB is going to have the ability to set storage type at creation that sounds great, and would make the current situation much better. After all what is lacking in the current quota (+idb or other storage API) world is: (1) to know which storage type a storage object belongs to (or in other words how the data stored in idb is going to be managed by UA), and (2) to adjust the amount of data in each storage type to avoid undesirable data deletion/eviction by UA. On a tangent... > > An open question is if the storage type (1) can be assigned only when an > IDB database is created, or (2) can be changed, allowing an IDB database to > be moved while retaining data, or (3) defines a namespace between origin > and database, i.e. "example.com" / "permanent" / "db-1" and "example.com" > / "temporary" / "db-1" co-exist as separate databases. > > What are your thoughts on those 3 options with respect to other storage > systems? > If I were to develop a mobile offline app I could live with (1), though I may not fall into a typical webapp developer category. If the granularity to set storage type is IDB database (2) does not sound very useful. (3) sounds interesting and useful, though I'm not sure how much complexity it would add in the implementation. > StorageWatcher createStorageWatcher(StorageType type) >> }; >> >> This new draft uses string enums to specify storage types rather than >> separate attributes on navigator (e.g. navigator.temporaryStorage), >> mainly because some methods (like {get,set}StorageType do not fit well >> in split interface) and to preserve greater flexibility to add more >> storage >> types in a future. I'm open to discussions though. >> >> supportedTypes are list of all StorageType's supported by the UA. >> >> * queryStorageInfo and requestQuota are Promise version of >> queryUsageAndQuota and requestQuota, which is for querying the current >> storage info (usage and quota) and requesting a new quota, respectively. >> Both return the current (or updated) StorageInfo. >> >> * getStorageType and setStorageType are new methods which are intended to >> work horizontally across multiple storage APIs. getStorageType(object) >> returns the current storage type for the given storage object, and >> setStorageType(object, type) changes the object's storage type. >> They may fail if the storage backend of the object does not support >> Quota API or does not support getting or setting (changing) storage >> types. >> >> We're aware that this API may not work very well with FileSystem >> API(s), and >> also will need coordination with IndexedDB. Feedback is strongly >> encouraged >> here. >> >> > Since it doesn't seem like a good fit for FileSystem or (IMHO) IndexedDB, > can you give an example where this API does make sense? That might help > inform what the shape of such a horizontal API would be like that would > work with IDB, etc. > Putting aside the API details now (I admit these methods are not in a really good shape), the real desire behind this request (and another request to have more granularity in storage types, which is not addressed in this draft version) is, to allow webapps to specify a part of their data not to be evicted when the storage pressure is getting high, and the decision on which data should be kept or ev
Re: Updating Quota API: Promise, Events and some more
Hi Kinuko, Very excited to see movement on this API. We are very interested in implementing this in Gecko once we reach an API that has the right feature set and polish! The recent changes definitely brings us closer to that which is great. Definitely looking forward to working with you on this. I have a few pieces of high-level feedback, before getting into details. Persistence type granularity: First off, I don't think that we in Gecko are interested in adding support for per-objectStore or per-Entry persistence type. For a couple of reasons we want to use separate files for temporary and persistent storage. This enables us to both measure the amount of temporary/persistent storage, as well as clear temporary storage, without doing slow operations involving running transactions or reading a lot of files. Additionally, in our backends we wouldn't have the ability to measure how much storage a particular objectStore is using. Or to measure how much the metadata for a particular Entry uses. So I'd prefer to only support setting storage type on a per-database and per-filesystem basis. getStorageType/setStorageType: I'm not convinced that it makes sense to have a cross-storage-API API for moving data between the different storage types. I definitely think that we need to support moving IDB databases or Filesystems between temporary and permanent storage. In particular moving data from temporary storage to persistent storage seems very useful. But I think this is something that we should do on a per-storage-API basis. I.e. the API for moving an IDB database should probably be part of IDB. And moving data between filesystems already exists in both the Google Filesystem proposal as well as the Mozilla filesystem proposal. requestQuota for temporary storage: I'd like to better understand the use case for requestQuota for temporary storage. Are implementations allowed to bring up a prompt when an increased temporary storage quota is requested? I thought one of the big use cases for temporary storage was that it would never trigger prompts, though obviously an exception could be made for the explicit requestQuota function. If the idea is for it not to bring up a prompt, why would we not simply always allocate the largest value that could be requested? Also, I think that we for temporary storage in Gecko would not want to guarantee that the allocated quota (as reported by queryQuota) for temporary storage will remain allocated for longer than the running page. I.e. once a page is closed, we might want to release the quota allocated and give it to another website. / Jonas On Tue, Aug 13, 2013 at 10:57 PM, Kinuko Yasuda wrote: > Hi all, > > It's been a while since Quota API's FPWD (http://www.w3.org/TR/quota-api/) > was published and we've gotten several requests/feedbacks so far. > To address some of the requests and to gain more consensus, I'm thinking > about making following changes to the Quota API: > > * Use Promises rather than callbacks > * Add Events to notify webapps of important changes in the local storage > space > * Establish a way to get and set the storage types (temporary or persistent) > of each storage object > > This breaks compatibility in the existing implementation, but currently > it's implemented only in Chrome behind the flag, so I hope/assume it'll be > ok > to make incompatible changes. I'm also strongly hoping these changes > (and debate on them) help building more consensus. > > There're also some requests those are not (yet) addressed in this new draft: > > * More granularity in storage types or priorities, rather than sticking to > the > rigid two types, so that webapps can indicate which data should be evicted > first / when. > * Helper method to estimate 'actual' size of each storage object > * Helper method to trigger GC/compaction on the local storage > > While they look nice-to-have in some situations but may also add more > complexity in implementation, so I tentatively concluded that they can > be put off until the next iteration. > > New draft needs some more polish but I'd like to get early feedback > on the new draft. > > Detailed draft: > > enum StorageType { "temporary", "persistent" }; > > partial interface Navigator { > readonly attribute StorageQuota storageQuota; > }; > > [NoInterfaceObject] interface StorageInfo { > unsigned long long usageInBytes; > unsigned long long quotaInBytes; > }; > > [NoInterfaceObject] interface StorageQuota { > readonly attribute StorageType[] supportedTypes; > > Promise queryStorageInfo(StorageType type); > Promise requestQuota(StorageType type, unsigned long long > newQuotaInBytes); > > Promise getStorageType((IDBObjectStore or Database or > Entry) object); > Promise setStorageType((IDBObjectStore or Database or Entry) > object, StorageType type); > > StorageWatcher createStorageWatcher(StorageType type) > }; > > This new draft uses string enums to specify storage types rather than > separate attributes on navigator (e.g. navi
Re: Updating Quota API: Promise, Events and some more
> That still feels like an odd mix of two APIs. An approach that we (Moz + > Google) have talked about would be to extend the IDBFactory.open() call > with an options dictionary, e.g. > > request = indexedDB.open({ name: ..., version: ..., storage: "temporary" }); > > On a tangent... > > An open question is if the storage type (1) can be assigned only when an > IDB database is created, or (2) can be changed, allowing an IDB database to > be moved while retaining data, or (3) defines a namespace between origin > and database, i.e. "example.com" / "permanent" / "db-1" and "example.com" / > "temporary" / "db-1" co-exist as separate databases. Specifying StorageType during opening the database is good enough. Basically we can think a database as transaction group. I don't see it is necessary to change storage type. Most likely operation is when storage get low, we will delete not so important data. onstoragelow event cover pretty much. Another problem is we actually don't know the size of object we are storing. Option 3 is very interesting. Kyaw
Re: Updating Quota API: Promise, Events and some more
How an IndexedDB database use persistent storage?
Re: Updating Quota API: Promise, Events and some more
On Tue, Aug 13, 2013 at 10:57 PM, Kinuko Yasuda wrote: > Hi all, > > It's been a while since Quota API's FPWD (http://www.w3.org/TR/quota-api/) > was published and we've gotten several requests/feedbacks so far. > To address some of the requests and to gain more consensus, I'm thinking > about making following changes to the Quota API: > > * Use Promises rather than callbacks > * Add Events to notify webapps of important changes in the local storage > space > * Establish a way to get and set the storage types (temporary or > persistent) > of each storage object > > This breaks compatibility in the existing implementation, but currently > it's implemented only in Chrome behind the flag, so I hope/assume it'll be > ok > to make incompatible changes. I'm also strongly hoping these changes > (and debate on them) help building more consensus. > > There're also some requests those are not (yet) addressed in this new > draft: > > * More granularity in storage types or priorities, rather than sticking to > the > rigid two types, so that webapps can indicate which data should be > evicted > first / when. > * Helper method to estimate 'actual' size of each storage object > * Helper method to trigger GC/compaction on the local storage > > While they look nice-to-have in some situations but may also add more > complexity in implementation, so I tentatively concluded that they can > be put off until the next iteration. > > New draft needs some more polish but I'd like to get early feedback > on the new draft. > > Detailed draft: > > enum StorageType { "temporary", "persistent" }; > > partial interface Navigator { > readonly attribute StorageQuota storageQuota; > }; > > [NoInterfaceObject] interface StorageInfo { > unsigned long long usageInBytes; > unsigned long long quotaInBytes; > }; > > [NoInterfaceObject] interface StorageQuota { > readonly attribute StorageType[] supportedTypes; > > Promise queryStorageInfo(StorageType type); > Promise requestQuota(StorageType type, unsigned long long > newQuotaInBytes); > > Promise getStorageType((IDBObjectStore or Database or > Entry) object); > Promise setStorageType((IDBObjectStore or Database or Entry) > object, StorageType type); > > For IndexedDB, an object store is (probably) too low a level to specify a storage type; ISTM that a database makes more sense as the level of granularity for specifying storage, since that avoids the complexity of a store disappearing out from within a database. Was the use of object store here intentional? >From an API perspective, passing an IDBObjectStore instance also doesn't make much sense as that sort of object is really a transaction-specific handle. Before delving deeply, my gut reaction is that to fit into this API you would need to pass an IDBDatabase connection object, and it would generate an error unless called during a versionchange transaction (which guarantees there are no other connections). That still feels like an odd mix of two APIs. An approach that we (Moz + Google) have talked about would be to extend the IDBFactory.open() call with an options dictionary, e.g. request = indexedDB.open({ name: ..., version: ..., storage: "temporary" }); On a tangent... An open question is if the storage type (1) can be assigned only when an IDB database is created, or (2) can be changed, allowing an IDB database to be moved while retaining data, or (3) defines a namespace between origin and database, i.e. "example.com" / "permanent" / "db-1" and "example.com" / "temporary" / "db-1" co-exist as separate databases. What are your thoughts on those 3 options with respect to other storage systems? > StorageWatcher createStorageWatcher(StorageType type) > }; > > This new draft uses string enums to specify storage types rather than > separate attributes on navigator (e.g. navigator.temporaryStorage), > mainly because some methods (like {get,set}StorageType do not fit well > in split interface) and to preserve greater flexibility to add more storage > types in a future. I'm open to discussions though. > > supportedTypes are list of all StorageType's supported by the UA. > > * queryStorageInfo and requestQuota are Promise version of > queryUsageAndQuota and requestQuota, which is for querying the current > storage info (usage and quota) and requesting a new quota, respectively. > Both return the current (or updated) StorageInfo. > > * getStorageType and setStorageType are new methods which are intended to > work horizontally across multiple storage APIs. getStorageType(object) > returns the current storage type for the given storage object, and > setStorageType(object, type) changes the object's storage type. > They may fail if the storage backend of the object does not support > Quota API or does not support getting or setting (changing) storage > types. > > We're aware that this API may not work very well with FileSystem API(s), > and > also will need coordinat
Updating Quota API: Promise, Events and some more
Hi all, It's been a while since Quota API's FPWD (http://www.w3.org/TR/quota-api/) was published and we've gotten several requests/feedbacks so far. To address some of the requests and to gain more consensus, I'm thinking about making following changes to the Quota API: * Use Promises rather than callbacks * Add Events to notify webapps of important changes in the local storage space * Establish a way to get and set the storage types (temporary or persistent) of each storage object This breaks compatibility in the existing implementation, but currently it's implemented only in Chrome behind the flag, so I hope/assume it'll be ok to make incompatible changes. I'm also strongly hoping these changes (and debate on them) help building more consensus. There're also some requests those are not (yet) addressed in this new draft: * More granularity in storage types or priorities, rather than sticking to the rigid two types, so that webapps can indicate which data should be evicted first / when. * Helper method to estimate 'actual' size of each storage object * Helper method to trigger GC/compaction on the local storage While they look nice-to-have in some situations but may also add more complexity in implementation, so I tentatively concluded that they can be put off until the next iteration. New draft needs some more polish but I'd like to get early feedback on the new draft. Detailed draft: enum StorageType { "temporary", "persistent" }; partial interface Navigator { readonly attribute StorageQuota storageQuota; }; [NoInterfaceObject] interface StorageInfo { unsigned long long usageInBytes; unsigned long long quotaInBytes; }; [NoInterfaceObject] interface StorageQuota { readonly attribute StorageType[] supportedTypes; Promise queryStorageInfo(StorageType type); Promise requestQuota(StorageType type, unsigned long long newQuotaInBytes); Promise getStorageType((IDBObjectStore or Database or Entry) object); Promise setStorageType((IDBObjectStore or Database or Entry) object, StorageType type); StorageWatcher createStorageWatcher(StorageType type) }; This new draft uses string enums to specify storage types rather than separate attributes on navigator (e.g. navigator.temporaryStorage), mainly because some methods (like {get,set}StorageType do not fit well in split interface) and to preserve greater flexibility to add more storage types in a future. I'm open to discussions though. supportedTypes are list of all StorageType's supported by the UA. * queryStorageInfo and requestQuota are Promise version of queryUsageAndQuota and requestQuota, which is for querying the current storage info (usage and quota) and requesting a new quota, respectively. Both return the current (or updated) StorageInfo. * getStorageType and setStorageType are new methods which are intended to work horizontally across multiple storage APIs. getStorageType(object) returns the current storage type for the given storage object, and setStorageType(object, type) changes the object's storage type. They may fail if the storage backend of the object does not support Quota API or does not support getting or setting (changing) storage types. We're aware that this API may not work very well with FileSystem API(s), and also will need coordination with IndexedDB. Feedback is strongly encouraged here. * createStorageWatcher returns a new StorageWatcher object for the given storage type, with which webapps can listen to important changes in the local storage and can react appropriately (i.e. discard their lower-priority cache data before UA-initiated merciless eviction happens). [NoInterfaceObject] interface createStorageWatcher { attribute EventHandler onstoragelow; attribute EventHandler onstorageok; attribute EventHandler onstoragechange; }; storagelow and storageok events are modeled after DEVICE_STORAGE_LOW and DEVICE_STORAGE_OK intents on android, to give more chances to webapps to adjust their local data quickly. * storagelow event is fired when the remaining storage space becomes lower than 10% of all available space for the type, or before the quota backend triggers eviction (for temporary case), whichever happens first. This event must be also fired once when either one of these conditions has been already met when a StorageWatcher is created. * storageok event is fired when the remaining available storage goes out of 'storagelow' state. * storagechange event is fired about every 1 second (TBD) or every time the quota backend detects the amount of remaining storage is changed, whichever is least frequent. This event must be fired before storagelow or storageok when the remaining space crosses the threshold. Again, your review/feedback is really encouraged. Thanks! Kinuko