Re: Updating Quota API: Promise, Events and some more
On Tue, Aug 27, 2013 at 4:26 AM, Jonas Sicking wrote:
> Thanks, that explains the concern.
>
> However the proposed solution doesn't seem to solve the problem
> particularly well, for two reasons:
>
> First off, an API which lets the website track how much data it is
> storing, measured in bytes, doesn't seem like it solves the problem.
> You also need to track quota as it is shrinking. Otherwise you won't
> know when you are starting to butt up against the quota limit which
> might cause you to get evicted more quickly. This also indicates that
> setting limits in bytes, and getting notified when you cross those
> limits, is the wrong API, since the limit would have to be adjusted
> any time the available quota shrinks.
>
> The second problem is harder to address. The problem is that the used
> quota could go down even when the website isn't open. We'd need
> something like event pages, and an event fired at those, to tackle
> this problem.
>
I believe the first problem can be addressed by polishing API details,
but I have to agree that the second problem would be much better
addressed in conjunction with event pages, and it might be
better to wait until it matures.
Unfortunately event pages, and how to fire events at them, is
> something that's still being developed. Which makes it hard to know
> what this API should look like. Until we have that figured out, maybe
> something like this would be ok to put in the draft:
>
> [Constructor(StorageType type, unsigned long rate)]
> interface StorageWatcher {
> readonly attribute StorageType type;
> readonly attribute unsigned long rate;
>
> attribute EventHandler onstoragechange;
> attribute EventHandler onquotachange;
> };
>
I'm sold! Also I think in this way having only
onstoragechange would be enough to start with.
Let me summarize what came out so far:
enum StorageType { "temporary", "persistent" };
partial interface Navigator {
readonly attribute StorageQuota storageQuota;
};
[NoInterfaceObject] interface StorageInfo {
unsigned long long usage;
unsigned long long quota;
};
[NoInterfaceObject] interface StorageQuota {
readonly attribute StorageType[] supportedTypes;
// Queries the current storage info.
Promise getInfo(StorageType type);
// Requests a new quota for persistent storage.
Promise requestPersistentQuota(unsigned long long
newQuota);
};
[Constructor(StorageType type, unsigned long rate)]
interface StorageWatcher : EventTarget {
readonly attribute StorageType type;
readonly attribute unsigned long rate;
attribute EventHandler onstoragechange;
};
interface StorageEvent : Event {
readonly attribute unsigned long long usage;
readonly attribute unsigned long long quota;
};
Thanks for all the discussion so far.
I'll repost on public-webapps when I update the draft.
(Would it be better to send the draft to public-script-coord too?)
(And as always more feedback / comments are always welcome)
Kinuko
/ Jonas
>
Re: Updating Quota API: Promise, Events and some more
On Mon, Aug 26, 2013 at 1:39 AM, Kinuko Yasuda wrote:
>
> Going back to the use case 2, let me try to explain a sample scenario:
>
> Let's say website A has stored some data in temporary storage. The amount of
> data was less than the per-site quota at the time (otherwise the write must
> have failed), since there was lot of available space. But then the
> remaining disk space has gotten smaller, and Chrome realizes that it needs
> to make some more space. If website A has done nothing until then Chrome
> will eventually evict site A (before or after evicting other sites), but if
> it could have told A that its remaining storage was less than before or was
> getting over-budget, A might have been able to react the events and to
> reduce some data so that Chrome may not need to evict any data. This
> scenario is over-simplified and when there're multiple websites just one
> site reducing its data may not do any favor to the site (except that the
> LRU-policy will implicitly help the site's data stay longer than others),
> but in general if the site is constantly being used/opened and keeps being a
> good citizen in terms of storage usage, the site would have much less
> possibility to be evicted. Does it make sense to you?
Thanks, that explains the concern.
However the proposed solution doesn't seem to solve the problem
particularly well, for two reasons:
First off, an API which lets the website track how much data it is
storing, measured in bytes, doesn't seem like it solves the problem.
You also need to track quota as it is shrinking. Otherwise you won't
know when you are starting to butt up against the quota limit which
might cause you to get evicted more quickly. This also indicates that
setting limits in bytes, and getting notified when you cross those
limits, is the wrong API, since the limit would have to be adjusted
any time the available quota shrinks.
The second problem is harder to address. The problem is that the used
quota could go down even when the website isn't open. We'd need
something like event pages, and an event fired at those, to tackle
this problem.
Unfortunately event pages, and how to fire events at them, is
something that's still being developed. Which makes it hard to know
what this API should look like. Until we have that figured out, maybe
something like this would be ok to put in the draft:
[Constructor(StorageType type, unsigned long rate)]
interface StorageWatcher {
readonly attribute StorageType type;
readonly attribute unsigned long rate;
attribute EventHandler onstoragechange;
attribute EventHandler onquotachange;
};
/ Jonas
Re: Updating Quota API: Promise, Events and some more
On Sat, Aug 24, 2013 at 3:45 PM, Jonas Sicking wrote:
> On Aug 23, 2013 12:30 AM, "Kinuko Yasuda" wrote:
> > On Tue, Aug 20, 2013 at 1:49 AM, Jonas Sicking wrote:
> >>
> >> On Mon, Aug 19, 2013 at 9:32 AM, Kinuko Yasuda
> wrote:
> >> >> Actually, *only* having onstoragechange would solve both the use case
> >> >> of detecting when you are running low on storage, as well as when you
> >> >> are "out of the woods". Especially if we provide both the quota and
> >> >> the amount of used space in the event.
> >> >
> >> > Right. I think probably we should add either "a combination of
> >> > storagelow + storageok" or "storagechange", and the latter would be
> >> > more complete (and possibly be simpler).
> >> >
> >> > One concern for adding storagechange event is how frequently
> >> > the event is going to be fired. In the former email I tentatively
> wrote
> >> > this to be fired every 1 second at most frequent (or whenever the
> >> > storage info is changed), but this may be still too frequent if many
> >> > apps are contending to a storage space. Do you have any opinion
> >> > or preference on this?
> >>
> >> I think firing every second while writes are happening would actually
> >> be ok. Seems like you are spending a lot of battery power anyway at
> >> that point and so an extra callback a second doesn't seem like a huge
> >> deal.
> >>
> >> That said, doing it more rarely, like every 5 seconds, sounds ok too.
> >>
> >> One solution is to punt the choice to the page. I.e. something like:
> >>
> >> [Constructor(StorageType type, unsigned long rate)]
> >> interface StorageWatcher {
> >> readonly attribute StorageType type;
> >> readonly attribute unsigned long rate;
> >>
> >> attribute EventHandler onstoragechange;
> >> };
> >
> >
> >
> > Sorry for my slow response, while your suggestion looks pretty reasonable
> > I've been thinking about the battery consumption and possible usecases
> > when events could be useful. Major situations that came up to me are:
> >
> > 1. To watch the remaining storage capacity while the app is actively
> writing
> >or deleting data (<-- this is basically the case you mentioned)
> > 2. To be notified when the (shared) storage capacity is getting tight and
> >their data may be evicted if they don't adjust their data size
> >
> > 1. could be useful if an app wants to keep their data within a certain
> > size (in temporary storage where they cannot specify an explicit quota),
> > while 2. is the situation I often hear requests for, and would be nice
> not
> > to waste too much battery.
> >
> > In either case I guessed frequency wouldn't be interesting but the actual
> > storage sizes may be. What if we allow apps to specify specific size
> > thresholds? The resulting API looks a bit noisy, but say,
> >
> > [Constructor(StorageType type, long long remainingStorageLow, optional
> long long usageHigh)]
> > interface StorageWatcher {
> > readonly attribute StorageType type;
> >
> > // Thresholds
> > readonly long long remainingStorageLow;
> > readonly long long usageHigh;
> >
> > attribute EventHandler onstoragelow;
> > attribute EventHandler onstorageok;
> > };
> >
> > Too rich?
> I don't quite understand usecase 2. Are you saying that a website want to
> be notified if it is at risk of it itself getting evicted? Or if it is at
> risk of evicting other websites?
>
Basically the former case, but maybe in a more implicit way. In general
keeping bigger data while the remaining space is tight increases the chance
of eviction, either on its own website or on other sites. If a website
stores data that have different priorities in temporary storage, it may
want to cleanup their less-important data to decrease the chance that their
more-important data gets evicted (though it may just help other sites store
more data).
In the Gecko implementation neither really makes sense:
>
> As for a website getting itself evicted, we're planning on evicting
> websites on a strictly LRU basis. That means that storing more data doesn't
> increase the risk of getting evicted. There is no point at which a website
> could write so much data that we'd immediately, or more aggressively would
> evict it. The only limit is the quota limit that each website gets for
> temporary storage, but that limit doesn't really affect eviction, but
> rather just how much data can be written.
>
> As for a website causing other websites to get evicted, this is something
> that can, and probably will, happen as soon as a websites starts using
> temporary storage. Because the temporary storage has a global limit, and
> can be used without prompting, we're counting on eventually reaching that
> limit. Once that is done, as soon as a website starts writing *something*
> will need to get evicted. So trying to avoid evicting others this likely
> means not use temporary storage at all.
>
> Could you help me understand the policies that Chrome is using which
> motivates use case 2? And exactly what
Re: Updating Quota API: Promise, Events and some more
On Aug 23, 2013 12:30 AM, "Kinuko Yasuda" wrote:
>
> On Tue, Aug 20, 2013 at 1:49 AM, Jonas Sicking wrote:
>>
>> On Mon, Aug 19, 2013 at 9:32 AM, Kinuko Yasuda
wrote:
>> >> Actually, *only* having onstoragechange would solve both the use case
>> >> of detecting when you are running low on storage, as well as when you
>> >> are "out of the woods". Especially if we provide both the quota and
>> >> the amount of used space in the event.
>> >
>> > Right. I think probably we should add either "a combination of
>> > storagelow + storageok" or "storagechange", and the latter would be
>> > more complete (and possibly be simpler).
>> >
>> > One concern for adding storagechange event is how frequently
>> > the event is going to be fired. In the former email I tentatively
wrote
>> > this to be fired every 1 second at most frequent (or whenever the
>> > storage info is changed), but this may be still too frequent if many
>> > apps are contending to a storage space. Do you have any opinion
>> > or preference on this?
>>
>> I think firing every second while writes are happening would actually
>> be ok. Seems like you are spending a lot of battery power anyway at
>> that point and so an extra callback a second doesn't seem like a huge
>> deal.
>>
>> That said, doing it more rarely, like every 5 seconds, sounds ok too.
>>
>> One solution is to punt the choice to the page. I.e. something like:
>>
>> [Constructor(StorageType type, unsigned long rate)]
>> interface StorageWatcher {
>> readonly attribute StorageType type;
>> readonly attribute unsigned long rate;
>>
>> attribute EventHandler onstoragechange;
>> };
>
>
>
> Sorry for my slow response, while your suggestion looks pretty reasonable
> I've been thinking about the battery consumption and possible usecases
> when events could be useful. Major situations that came up to me are:
>
> 1. To watch the remaining storage capacity while the app is actively
writing
>or deleting data (<-- this is basically the case you mentioned)
> 2. To be notified when the (shared) storage capacity is getting tight and
>their data may be evicted if they don't adjust their data size
>
> 1. could be useful if an app wants to keep their data within a certain
> size (in temporary storage where they cannot specify an explicit quota),
> while 2. is the situation I often hear requests for, and would be nice not
> to waste too much battery.
>
> In either case I guessed frequency wouldn't be interesting but the actual
> storage sizes may be. What if we allow apps to specify specific size
> thresholds? The resulting API looks a bit noisy, but say,
>
> [Constructor(StorageType type, long long remainingStorageLow, optional
long long usageHigh)]
> interface StorageWatcher {
> readonly attribute StorageType type;
>
> // Thresholds
> readonly long long remainingStorageLow;
> readonly long long usageHigh;
>
> attribute EventHandler onstoragelow;
> attribute EventHandler onstorageok;
> };
>
> Too rich?
I don't quite understand usecase 2. Are you saying that a website want to
be notified if it is at risk of it itself getting evicted? Or if it is at
risk of evicting other websites?
In the Gecko implementation neither really makes sense:
As for a website getting itself evicted, we're planning on evicting
websites on a strictly LRU basis. That means that storing more data doesn't
increase the risk of getting evicted. There is no point at which a website
could write so much data that we'd immediately, or more aggressively would
evict it. The only limit is the quota limit that each website gets for
temporary storage, but that limit doesn't really affect eviction, but
rather just how much data can be written.
As for a website causing other websites to get evicted, this is something
that can, and probably will, happen as soon as a websites starts using
temporary storage. Because the temporary storage has a global limit, and
can be used without prompting, we're counting on eventually reaching that
limit. Once that is done, as soon as a website starts writing *something*
will need to get evicted. So trying to avoid evicting others this likely
means not use temporary storage at all.
Could you help me understand the policies that Chrome is using which
motivates use case 2? And exactly what type of eviction that is the concern
in that use case?
/ Jonas
Re: Updating Quota API: Promise, Events and some more
On Tue, Aug 20, 2013 at 1:49 AM, Jonas Sicking wrote:
> On Mon, Aug 19, 2013 at 9:32 AM, Kinuko Yasuda
> wrote:
> >> Actually, *only* having onstoragechange would solve both the use case
> >> of detecting when you are running low on storage, as well as when you
> >> are "out of the woods". Especially if we provide both the quota and
> >> the amount of used space in the event.
> >
> > Right. I think probably we should add either "a combination of
> > storagelow + storageok" or "storagechange", and the latter would be
> > more complete (and possibly be simpler).
> >
> > One concern for adding storagechange event is how frequently
> > the event is going to be fired. In the former email I tentatively wrote
> > this to be fired every 1 second at most frequent (or whenever the
> > storage info is changed), but this may be still too frequent if many
> > apps are contending to a storage space. Do you have any opinion
> > or preference on this?
>
> I think firing every second while writes are happening would actually
> be ok. Seems like you are spending a lot of battery power anyway at
> that point and so an extra callback a second doesn't seem like a huge
> deal.
>
> That said, doing it more rarely, like every 5 seconds, sounds ok too.
>
> One solution is to punt the choice to the page. I.e. something like:
>
> [Constructor(StorageType type, unsigned long rate)]
> interface StorageWatcher {
> readonly attribute StorageType type;
> readonly attribute unsigned long rate;
>
> attribute EventHandler onstoragechange;
> };
>
Sorry for my slow response, while your suggestion looks pretty reasonable
I've been thinking about the battery consumption and possible usecases
when events could be useful. Major situations that came up to me are:
1. To watch the remaining storage capacity while the app is actively writing
or deleting data (<-- this is basically the case you mentioned)
2. To be notified when the (shared) storage capacity is getting tight and
their data may be evicted if they don't adjust their data size
1. could be useful if an app wants to keep their data within a certain
size (in temporary storage where they cannot specify an explicit quota),
while 2. is the situation I often hear requests for, and would be nice not
to waste too much battery.
In either case I guessed frequency wouldn't be interesting but the actual
storage sizes may be. What if we allow apps to specify specific size
thresholds? The resulting API looks a bit noisy, but say,
[Constructor(StorageType type, long long remainingStorageLow, optional long
long usageHigh)]
interface StorageWatcher {
readonly attribute StorageType type;
// Thresholds
readonly long long remainingStorageLow;
readonly long long usageHigh;
attribute EventHandler onstoragelow;
attribute EventHandler onstorageok;
};
Too rich?
> / Jonas
>
Re: Updating Quota API: Promise, Events and some more
On Mon, Aug 19, 2013 at 9:32 AM, Kinuko Yasuda wrote:
>> Actually, *only* having onstoragechange would solve both the use case
>> of detecting when you are running low on storage, as well as when you
>> are "out of the woods". Especially if we provide both the quota and
>> the amount of used space in the event.
>
> Right. I think probably we should add either "a combination of
> storagelow + storageok" or "storagechange", and the latter would be
> more complete (and possibly be simpler).
>
> One concern for adding storagechange event is how frequently
> the event is going to be fired. In the former email I tentatively wrote
> this to be fired every 1 second at most frequent (or whenever the
> storage info is changed), but this may be still too frequent if many
> apps are contending to a storage space. Do you have any opinion
> or preference on this?
I think firing every second while writes are happening would actually
be ok. Seems like you are spending a lot of battery power anyway at
that point and so an extra callback a second doesn't seem like a huge
deal.
That said, doing it more rarely, like every 5 seconds, sounds ok too.
One solution is to punt the choice to the page. I.e. something like:
[Constructor(StorageType type, unsigned long rate)]
interface StorageWatcher {
readonly attribute StorageType type;
readonly attribute unsigned long rate;
attribute EventHandler onstoragechange;
};
/ Jonas
Re: Updating Quota API: Promise, Events and some more
On Sat, Aug 17, 2013 at 3:01 AM, Jonas Sicking wrote:
> On Fri, Aug 16, 2013 at 6:56 AM, Kinuko Yasuda
> wrote:
>
> I had a few bike-sheddy comments as well.
>
> > Now the new draft looks like:
> >
> > enum StorageType { "temporary", "persistent" };
> >
> > partial interface Navigator {
> > readonly attribute StorageQuota storageQuota;
> > };
> >
> > [NoInterfaceObject] interface StorageInfo {
> > unsigned long long usageInBytes;
> > unsigned long long quotaInBytes;
> > };
>
> I think these can be simplified to simply 'usage' and 'quota'. The
> fact that it's in bytes doesn't really need to be spelled out in the
> property name I think. Feedback that we've gotten from authors is that
> they prefer short names.
Ok, I don't have strong opinion on the names. Some developers
seem to prefer clearer names but I think just 'usage' and 'quota'
would work in this case.
> > [NoInterfaceObject] interface StorageQuota {
> > readonly attribute StorageType[] supportedTypes;
> >
> > // Queries the current storage info.
> > Promise queryStorageInfo(StorageType type);
>
> I think simply 'getInfo' might be enough here. It should already be
> obvious that it's storage-quota related.
Sounds good. Actually I wondered the same.
> > // Requests a new quota for persistent storage.
> > Promise requestPersistentQuota(unsigned long long
> > newQuotaInBytes);
> >
> > // Creates a new StorageWatcher object to watch the storage changes.
> > StorageWatcher createStorageWatcher(StorageType type)
>
> This could probably simply be done using a constructor. I.e.
>
> new StorageWatcher("persistent");
Yup... I once wondered we may not want to expose more
constructors on global namespace, but constructor way
actually makes it simpler.
> [NoInterfaceObject] interface createStorageWatcher {
> > // Fired when when the remaining storage space becomes lower
> > // than 10% of all available space for the type, or before the quota
> > backend
> > // triggers eviction (for temporary case), whichever happens first.
> > attribute EventHandler onstoragelow;
> > };
>
> Actually, the other two events made sense to me. Without onstorageok
> the page has to poll queryStorageInfo to know when it's "out of the
> woods".
>
> Actually, *only* having onstoragechange would solve both the use case
> of detecting when you are running low on storage, as well as when you
> are "out of the woods". Especially if we provide both the quota and
> the amount of used space in the event.
Right. I think probably we should add either "a combination of
storagelow + storageok" or "storagechange", and the latter would be
more complete (and possibly be simpler).
One concern for adding storagechange event is how frequently
the event is going to be fired. In the former email I tentatively wrote
this to be fired every 1 second at most frequent (or whenever the
storage info is changed), but this may be still too frequent if many
apps are contending to a storage space. Do you have any opinion
or preference on this?
> / Jonas
>
Re: Updating Quota API: Promise, Events and some more
On Fri, Aug 16, 2013 at 6:56 AM, Kinuko Yasuda wrote:
I had a few bike-sheddy comments as well.
> Now the new draft looks like:
>
> enum StorageType { "temporary", "persistent" };
>
> partial interface Navigator {
> readonly attribute StorageQuota storageQuota;
> };
>
> [NoInterfaceObject] interface StorageInfo {
> unsigned long long usageInBytes;
> unsigned long long quotaInBytes;
> };
I think these can be simplified to simply 'usage' and 'quota'. The
fact that it's in bytes doesn't really need to be spelled out in the
property name I think. Feedback that we've gotten from authors is that
they prefer short names.
> [NoInterfaceObject] interface StorageQuota {
> readonly attribute StorageType[] supportedTypes;
>
> // Queries the current storage info.
> Promise queryStorageInfo(StorageType type);
I think simply 'getInfo' might be enough here. It should already be
obvious that it's storage-quota related.
> // Requests a new quota for persistent storage.
> Promise requestPersistentQuota(unsigned long long
> newQuotaInBytes);
>
> // Creates a new StorageWatcher object to watch the storage changes.
> StorageWatcher createStorageWatcher(StorageType type)
This could probably simply be done using a constructor. I.e.
new StorageWatcher("persistent");
> [NoInterfaceObject] interface createStorageWatcher {
> // Fired when when the remaining storage space becomes lower
> // than 10% of all available space for the type, or before the quota
> backend
> // triggers eviction (for temporary case), whichever happens first.
> attribute EventHandler onstoragelow;
> };
Actually, the other two events made sense to me. Without onstorageok
the page has to poll queryStorageInfo to know when it's "out of the
woods".
Actually, *only* having onstoragechange would solve both the use case
of detecting when you are running low on storage, as well as when you
are "out of the woods". Especially if we provide both the quota and
the amount of used space in the event.
/ Jonas
Re: Updating Quota API: Promise, Events and some more
On Fri, Aug 16, 2013 at 3:25 AM, Jonas Sicking wrote:
> On Thu, Aug 15, 2013 at 9:44 AM, Kinuko Yasuda
> wrote:
> >> requestQuota for temporary storage:
> >> I'd like to better understand the use case for requestQuota for
> temporary
> >> storage. Are implementations allowed to bring up a prompt when an
> increased
> >> temporary storage quota is requested? I thought one of the big use
> cases for
> >> temporary storage was that it would never trigger prompts, though
> obviously
> >> an exception could be made for the explicit requestQuota function.
> >>
> >> If the idea is for it not to bring up a prompt, why would we not simply
> >> always allocate the largest value that could be requested?
> >>
> >> Also, I think that we for temporary storage in Gecko would not want to
> >> guarantee that the allocated quota (as reported by queryQuota) for
> temporary
> >> storage will remain allocated for longer than the running page. I.e.
> once a
> >> page is closed, we might want to release the quota allocated and give
> it to
> >> another website.
> >
> > This is a very good feedback/question. In Chrome we actually don't
> really
> > support requestQuota for temporary storage, we just silently ignore the
> > request but returns the current available quota to the requesting
> webapp. I
> > also agree that it won't be desirable to preserve the quota allocated for
> > temporary storage.
> >
> > Seems like having this method take a storage type just confuses
> > readers/implementors, we should probably just drop the storage type
> > parameter and rename it to something like 'requestPersistentQuota', or
> only
> > add the method to 'navigator.persistentStorage' attribute?
>
> I think adding storageQuota.requestPersistentQuota would be a good
> solution. It seems cleaner than having a separate
> navigator.persistentStorage object.
Agreed, sounds good to me too.
Based on the feedbacks (thanks all) I've dropped {get,set}StorageType and
storageok and storagechange events from the draft. (New events can be
added later whenever we find them useful with more concrete use cases)
Now the new draft looks like:
enum StorageType { "temporary", "persistent" };
partial interface Navigator {
readonly attribute StorageQuota storageQuota;
};
[NoInterfaceObject] interface StorageInfo {
unsigned long long usageInBytes;
unsigned long long quotaInBytes;
};
[NoInterfaceObject] interface StorageQuota {
readonly attribute StorageType[] supportedTypes;
// Queries the current storage info.
Promise queryStorageInfo(StorageType type);
// Requests a new quota for persistent storage.
Promise requestPersistentQuota(unsigned long long
newQuotaInBytes);
// Creates a new StorageWatcher object to watch the storage changes.
StorageWatcher createStorageWatcher(StorageType type)
};
[NoInterfaceObject] interface createStorageWatcher {
// Fired when when the remaining storage space becomes lower
// than 10% of all available space for the type, or before the quota
backend
// triggers eviction (for temporary case), whichever happens first.
attribute EventHandler onstoragelow;
};
More feedback on this rough draft is always welcome. Also I'm going to
start updating the editor's draft based on this one and discussions we had
on this list.
> / Jonas
>
Re: Updating Quota API: Promise, Events and some more
On Thu, Aug 15, 2013 at 9:44 AM, Kinuko Yasuda wrote: >> requestQuota for temporary storage: >> I'd like to better understand the use case for requestQuota for temporary >> storage. Are implementations allowed to bring up a prompt when an increased >> temporary storage quota is requested? I thought one of the big use cases for >> temporary storage was that it would never trigger prompts, though obviously >> an exception could be made for the explicit requestQuota function. >> >> If the idea is for it not to bring up a prompt, why would we not simply >> always allocate the largest value that could be requested? >> >> Also, I think that we for temporary storage in Gecko would not want to >> guarantee that the allocated quota (as reported by queryQuota) for temporary >> storage will remain allocated for longer than the running page. I.e. once a >> page is closed, we might want to release the quota allocated and give it to >> another website. > > This is a very good feedback/question. In Chrome we actually don't really > support requestQuota for temporary storage, we just silently ignore the > request but returns the current available quota to the requesting webapp. I > also agree that it won't be desirable to preserve the quota allocated for > temporary storage. > > Seems like having this method take a storage type just confuses > readers/implementors, we should probably just drop the storage type > parameter and rename it to something like 'requestPersistentQuota', or only > add the method to 'navigator.persistentStorage' attribute? I think adding storageQuota.requestPersistentQuota would be a good solution. It seems cleaner than having a separate navigator.persistentStorage object. / Jonas
Re: Updating Quota API: Promise, Events and some more
Hi Jonas, On Thu, Aug 15, 2013 at 1:20 PM, Jonas Sicking wrote: > Hi Kinuko, > > Very excited to see movement on this API. We are very interested in > implementing this in Gecko once we reach an API that has the right feature > set and polish! The recent changes definitely brings us closer to that > which is great. Definitely looking forward to working with you on this. > It's great to know that Mozilla is interested in this API, and thanks very much for the valuable feedbacks. > I have a few pieces of high-level fedback, before getting into details. > > Persistence type granularity: > First off, I don't think that we in Gecko are interested in adding support > for per-objectStore or per-Entry persistence type. > > For a couple of reasons we want to use separate files for temporary and > persistent storage. This enables us to both measure the amount of > temporary/persistent storage, as well as clear temporary storage, without > doing slow operations involving running transactions or reading a lot of > files. > > Additionally, in our backends we wouldn't have the ability to measure how > much storage a particular objectStore is using. Or to measure how much the > metadata for a particular Entry uses. > > So I'd prefer to only support setting storage type on a per-database and > per-filesystem basis. > > getStorageType/setStorageType: > I'm not convinced that it makes sense to have a cross-storage-API API for > moving data between the different storage types. > > I definitely think that we need to support moving IDB databases or > Filesystems between temporary and permanent storage. In particular moving > data from temporary storage to persistent storage seems very useful. > > But I think this is something that we should do on a per-storage-API > basis. I.e. the API for moving an IDB database should probably be part of > IDB. And moving data between filesystems already exists in both the Google > Filesystem proposal as well as the Mozilla filesystem proposal. > I mostly agree with your points that 1. it's desirable to support moving data across temporary and persistent storage, and 2. it's more natural to support it on a per-storage-API basis. Also regarding the persistence type granularity we (Chrome) are actually in a very similar situation, i.e. it's not very easy to support per-objectStore or per-Entry persistence type. As I wrote in the response to Joshua's feedback the real 'desire' to have this kind of methods is to give more control to webapps on which data to be kept or evicted when the storage is getting tight, and I'm not willing to stick to the current proposal. If each storage API can support the desire of mobile/offline webapps that'd be good enough (if not great). After hearing feedbacks from you and Joshua, and knowing that IDB is going to support storage types in a near future, I feel we can drop these methods from this draft. (I'm also going to reach out the original requester to see if it works for them too) > requestQuota for temporary storage: > I'd like to better understand the use case for requestQuota for temporary > storage. Are implementations allowed to bring up a prompt when an increased > temporary storage quota is requested? I thought one of the big use cases > for temporary storage was that it would never trigger prompts, though > obviously an exception could be made for the explicit requestQuota function. > > If the idea is for it not to bring up a prompt, why would we not simply > always allocate the largest value that could be requested? > > Also, I think that we for temporary storage in Gecko would not want to > guarantee that the allocated quota (as reported by queryQuota) for > temporary storage will remain allocated for longer than the running page. > I.e. once a page is closed, we might want to release the quota allocated > and give it to another website. > This is a very good feedback/question. In Chrome we actually don't really support requestQuota for temporary storage, we just silently ignore the request but returns the current available quota to the requesting webapp. I also agree that it won't be desirable to preserve the quota allocated for temporary storage. Seems like having this method take a storage type just confuses readers/implementors, we should probably just drop the storage type parameter and rename it to something like 'requestPersistentQuota', or only add the method to 'navigator.persistentStorage' attribute? > / Jonas > > On Tue, Aug 13, 2013 at 10:57 PM, Kinuko Yasuda > wrote: > > Hi all, > > > > It's been a while since Quota API's FPWD ( > http://www.w3.org/TR/quota-api/) > > was published and we've gotten several requests/feedbacks so far. > > To address some of the requests and to gain more consensus, I'm thinking > > about making following changes to the Quota API: > > > > * Use Promises rather than callbacks > > * Add Events to notify webapps of important changes in the local storage > > space > > * Establish a way to get and
Re: Updating Quota API: Promise, Events and some more
Hi Joshua,
On Thu, Aug 15, 2013 at 6:33 AM, Joshua Bell wrote:
>
> Promise getStorageType((IDBObjectStore or Database or
>> Entry) object);
>> Promise setStorageType((IDBObjectStore or Database or Entry)
>> object, StorageType type);
>>
>
> For IndexedDB, an object store is (probably) too low a level to specify a
> storage type; ISTM that a database makes more sense as the level of
> granularity for specifying storage, since that avoids the complexity of a
> store disappearing out from within a database. Was the use of object store
> here intentional?
>
> From an API perspective, passing an IDBObjectStore instance also doesn't
> make much sense as that sort of object is really a transaction-specific
> handle. Before delving deeply, my gut reaction is that to fit into this API
> you would need to pass an IDBDatabase connection object, and it would
> generate an error unless called during a versionchange transaction (which
> guarantees there are no other connections).
>
Thanks for the very informative feedback. The use of object store wasn't
really intentional (it was partly due to lack of my understanding), and if
IDBDatabase connection object makes more sense I'm open to making the
change (more discussion follows).
That still feels like an odd mix of two APIs. An approach that we (Moz +
> Google) have talked about would be to extend the IDBFactory.open() call
> with an options dictionary, e.g.
>
> request = indexedDB.open({ name: ..., version: ..., storage: "temporary"
> });
>
Honestly I was debating whether I should add these methods to this
proposal, as I had the same feeling as yours when I heard the original
request, but if IDB is going to have the ability to set storage type at
creation that sounds great, and would make the current situation much
better.
After all what is lacking in the current quota (+idb or other storage API)
world is:
(1) to know which storage type a storage object belongs to (or in other
words how the data stored in idb is going to be managed by UA), and
(2) to adjust the amount of data in each storage type to avoid undesirable
data deletion/eviction by UA.
On a tangent...
>
> An open question is if the storage type (1) can be assigned only when an
> IDB database is created, or (2) can be changed, allowing an IDB database to
> be moved while retaining data, or (3) defines a namespace between origin
> and database, i.e. "example.com" / "permanent" / "db-1" and "example.com"
> / "temporary" / "db-1" co-exist as separate databases.
>
> What are your thoughts on those 3 options with respect to other storage
> systems?
>
If I were to develop a mobile offline app I could live with (1), though I
may not fall into a typical webapp developer category. If the granularity
to set storage type is IDB database (2) does not sound very useful. (3)
sounds interesting and useful, though I'm not sure how much complexity it
would add in the implementation.
> StorageWatcher createStorageWatcher(StorageType type)
>> };
>>
>> This new draft uses string enums to specify storage types rather than
>> separate attributes on navigator (e.g. navigator.temporaryStorage),
>> mainly because some methods (like {get,set}StorageType do not fit well
>> in split interface) and to preserve greater flexibility to add more
>> storage
>> types in a future. I'm open to discussions though.
>>
>> supportedTypes are list of all StorageType's supported by the UA.
>>
>> * queryStorageInfo and requestQuota are Promise version of
>> queryUsageAndQuota and requestQuota, which is for querying the current
>> storage info (usage and quota) and requesting a new quota, respectively.
>> Both return the current (or updated) StorageInfo.
>>
>> * getStorageType and setStorageType are new methods which are intended to
>> work horizontally across multiple storage APIs. getStorageType(object)
>> returns the current storage type for the given storage object, and
>> setStorageType(object, type) changes the object's storage type.
>> They may fail if the storage backend of the object does not support
>> Quota API or does not support getting or setting (changing) storage
>> types.
>>
>> We're aware that this API may not work very well with FileSystem
>> API(s), and
>> also will need coordination with IndexedDB. Feedback is strongly
>> encouraged
>> here.
>>
>>
> Since it doesn't seem like a good fit for FileSystem or (IMHO) IndexedDB,
> can you give an example where this API does make sense? That might help
> inform what the shape of such a horizontal API would be like that would
> work with IDB, etc.
>
Putting aside the API details now (I admit these methods are not in a
really good shape), the real desire behind this request (and another
request to have more granularity in storage types, which is not addressed
in this draft version) is, to allow webapps to specify a part of their data
not to be evicted when the storage pressure is getting high, and the
decision on which data should be kept or ev
Re: Updating Quota API: Promise, Events and some more
Hi Kinuko,
Very excited to see movement on this API. We are very interested in
implementing this in Gecko once we reach an API that has the right feature
set and polish! The recent changes definitely brings us closer to that
which is great. Definitely looking forward to working with you on this.
I have a few pieces of high-level feedback, before getting into details.
Persistence type granularity:
First off, I don't think that we in Gecko are interested in adding support
for per-objectStore or per-Entry persistence type.
For a couple of reasons we want to use separate files for temporary and
persistent storage. This enables us to both measure the amount of
temporary/persistent storage, as well as clear temporary storage, without
doing slow operations involving running transactions or reading a lot of
files.
Additionally, in our backends we wouldn't have the ability to measure how
much storage a particular objectStore is using. Or to measure how much the
metadata for a particular Entry uses.
So I'd prefer to only support setting storage type on a per-database and
per-filesystem basis.
getStorageType/setStorageType:
I'm not convinced that it makes sense to have a cross-storage-API API for
moving data between the different storage types.
I definitely think that we need to support moving IDB databases or
Filesystems between temporary and permanent storage. In particular moving
data from temporary storage to persistent storage seems very useful.
But I think this is something that we should do on a per-storage-API basis.
I.e. the API for moving an IDB database should probably be part of IDB. And
moving data between filesystems already exists in both the Google
Filesystem proposal as well as the Mozilla filesystem proposal.
requestQuota for temporary storage:
I'd like to better understand the use case for requestQuota for temporary
storage. Are implementations allowed to bring up a prompt when an increased
temporary storage quota is requested? I thought one of the big use cases
for temporary storage was that it would never trigger prompts, though
obviously an exception could be made for the explicit requestQuota function.
If the idea is for it not to bring up a prompt, why would we not simply
always allocate the largest value that could be requested?
Also, I think that we for temporary storage in Gecko would not want to
guarantee that the allocated quota (as reported by queryQuota) for
temporary storage will remain allocated for longer than the running page.
I.e. once a page is closed, we might want to release the quota allocated
and give it to another website.
/ Jonas
On Tue, Aug 13, 2013 at 10:57 PM, Kinuko Yasuda wrote:
> Hi all,
>
> It's been a while since Quota API's FPWD (http://www.w3.org/TR/quota-api/)
> was published and we've gotten several requests/feedbacks so far.
> To address some of the requests and to gain more consensus, I'm thinking
> about making following changes to the Quota API:
>
> * Use Promises rather than callbacks
> * Add Events to notify webapps of important changes in the local storage
> space
> * Establish a way to get and set the storage types (temporary or
persistent)
> of each storage object
>
> This breaks compatibility in the existing implementation, but currently
> it's implemented only in Chrome behind the flag, so I hope/assume it'll be
> ok
> to make incompatible changes. I'm also strongly hoping these changes
> (and debate on them) help building more consensus.
>
> There're also some requests those are not (yet) addressed in this new
draft:
>
> * More granularity in storage types or priorities, rather than sticking to
> the
> rigid two types, so that webapps can indicate which data should be evicted
> first / when.
> * Helper method to estimate 'actual' size of each storage object
> * Helper method to trigger GC/compaction on the local storage
>
> While they look nice-to-have in some situations but may also add more
> complexity in implementation, so I tentatively concluded that they can
> be put off until the next iteration.
>
> New draft needs some more polish but I'd like to get early feedback
> on the new draft.
>
> Detailed draft:
>
> enum StorageType { "temporary", "persistent" };
>
> partial interface Navigator {
> readonly attribute StorageQuota storageQuota;
> };
>
> [NoInterfaceObject] interface StorageInfo {
> unsigned long long usageInBytes;
> unsigned long long quotaInBytes;
> };
>
> [NoInterfaceObject] interface StorageQuota {
> readonly attribute StorageType[] supportedTypes;
>
> Promise queryStorageInfo(StorageType type);
> Promise requestQuota(StorageType type, unsigned long long
> newQuotaInBytes);
>
> Promise getStorageType((IDBObjectStore or Database or
> Entry) object);
> Promise setStorageType((IDBObjectStore or Database or Entry)
> object, StorageType type);
>
> StorageWatcher createStorageWatcher(StorageType type)
> };
>
> This new draft uses string enums to specify storage types rather than
> separate attributes on navigator (e.g. navi
Re: Updating Quota API: Promise, Events and some more
> That still feels like an odd mix of two APIs. An approach that we (Moz +
> Google) have talked about would be to extend the IDBFactory.open() call
> with an options dictionary, e.g.
>
> request = indexedDB.open({ name: ..., version: ..., storage: "temporary" });
>
> On a tangent...
>
> An open question is if the storage type (1) can be assigned only when an
> IDB database is created, or (2) can be changed, allowing an IDB database to
> be moved while retaining data, or (3) defines a namespace between origin
> and database, i.e. "example.com" / "permanent" / "db-1" and "example.com" /
> "temporary" / "db-1" co-exist as separate databases.
Specifying StorageType during opening the database is good enough.
Basically we can think a database as transaction group.
I don't see it is necessary to change storage type. Most likely
operation is when storage get low, we will delete not so important
data. onstoragelow event cover pretty much. Another problem is we
actually don't know the size of object we are storing.
Option 3 is very interesting.
Kyaw
Re: Updating Quota API: Promise, Events and some more
How an IndexedDB database use persistent storage?
Re: Updating Quota API: Promise, Events and some more
On Tue, Aug 13, 2013 at 10:57 PM, Kinuko Yasuda wrote:
> Hi all,
>
> It's been a while since Quota API's FPWD (http://www.w3.org/TR/quota-api/)
> was published and we've gotten several requests/feedbacks so far.
> To address some of the requests and to gain more consensus, I'm thinking
> about making following changes to the Quota API:
>
> * Use Promises rather than callbacks
> * Add Events to notify webapps of important changes in the local storage
> space
> * Establish a way to get and set the storage types (temporary or
> persistent)
> of each storage object
>
> This breaks compatibility in the existing implementation, but currently
> it's implemented only in Chrome behind the flag, so I hope/assume it'll be
> ok
> to make incompatible changes. I'm also strongly hoping these changes
> (and debate on them) help building more consensus.
>
> There're also some requests those are not (yet) addressed in this new
> draft:
>
> * More granularity in storage types or priorities, rather than sticking to
> the
> rigid two types, so that webapps can indicate which data should be
> evicted
> first / when.
> * Helper method to estimate 'actual' size of each storage object
> * Helper method to trigger GC/compaction on the local storage
>
> While they look nice-to-have in some situations but may also add more
> complexity in implementation, so I tentatively concluded that they can
> be put off until the next iteration.
>
> New draft needs some more polish but I'd like to get early feedback
> on the new draft.
>
> Detailed draft:
>
> enum StorageType { "temporary", "persistent" };
>
> partial interface Navigator {
> readonly attribute StorageQuota storageQuota;
> };
>
> [NoInterfaceObject] interface StorageInfo {
> unsigned long long usageInBytes;
> unsigned long long quotaInBytes;
> };
>
> [NoInterfaceObject] interface StorageQuota {
> readonly attribute StorageType[] supportedTypes;
>
> Promise queryStorageInfo(StorageType type);
> Promise requestQuota(StorageType type, unsigned long long
> newQuotaInBytes);
>
> Promise getStorageType((IDBObjectStore or Database or
> Entry) object);
> Promise setStorageType((IDBObjectStore or Database or Entry)
> object, StorageType type);
>
>
For IndexedDB, an object store is (probably) too low a level to specify a
storage type; ISTM that a database makes more sense as the level of
granularity for specifying storage, since that avoids the complexity of a
store disappearing out from within a database. Was the use of object store
here intentional?
>From an API perspective, passing an IDBObjectStore instance also doesn't
make much sense as that sort of object is really a transaction-specific
handle. Before delving deeply, my gut reaction is that to fit into this API
you would need to pass an IDBDatabase connection object, and it would
generate an error unless called during a versionchange transaction (which
guarantees there are no other connections).
That still feels like an odd mix of two APIs. An approach that we (Moz +
Google) have talked about would be to extend the IDBFactory.open() call
with an options dictionary, e.g.
request = indexedDB.open({ name: ..., version: ..., storage: "temporary" });
On a tangent...
An open question is if the storage type (1) can be assigned only when an
IDB database is created, or (2) can be changed, allowing an IDB database to
be moved while retaining data, or (3) defines a namespace between origin
and database, i.e. "example.com" / "permanent" / "db-1" and "example.com" /
"temporary" / "db-1" co-exist as separate databases.
What are your thoughts on those 3 options with respect to other storage
systems?
> StorageWatcher createStorageWatcher(StorageType type)
> };
>
> This new draft uses string enums to specify storage types rather than
> separate attributes on navigator (e.g. navigator.temporaryStorage),
> mainly because some methods (like {get,set}StorageType do not fit well
> in split interface) and to preserve greater flexibility to add more storage
> types in a future. I'm open to discussions though.
>
> supportedTypes are list of all StorageType's supported by the UA.
>
> * queryStorageInfo and requestQuota are Promise version of
> queryUsageAndQuota and requestQuota, which is for querying the current
> storage info (usage and quota) and requesting a new quota, respectively.
> Both return the current (or updated) StorageInfo.
>
> * getStorageType and setStorageType are new methods which are intended to
> work horizontally across multiple storage APIs. getStorageType(object)
> returns the current storage type for the given storage object, and
> setStorageType(object, type) changes the object's storage type.
> They may fail if the storage backend of the object does not support
> Quota API or does not support getting or setting (changing) storage
> types.
>
> We're aware that this API may not work very well with FileSystem API(s),
> and
> also will need coordinat
Updating Quota API: Promise, Events and some more
Hi all,
It's been a while since Quota API's FPWD (http://www.w3.org/TR/quota-api/)
was published and we've gotten several requests/feedbacks so far.
To address some of the requests and to gain more consensus, I'm thinking
about making following changes to the Quota API:
* Use Promises rather than callbacks
* Add Events to notify webapps of important changes in the local storage
space
* Establish a way to get and set the storage types (temporary or persistent)
of each storage object
This breaks compatibility in the existing implementation, but currently
it's implemented only in Chrome behind the flag, so I hope/assume it'll be
ok
to make incompatible changes. I'm also strongly hoping these changes
(and debate on them) help building more consensus.
There're also some requests those are not (yet) addressed in this new draft:
* More granularity in storage types or priorities, rather than sticking to
the
rigid two types, so that webapps can indicate which data should be evicted
first / when.
* Helper method to estimate 'actual' size of each storage object
* Helper method to trigger GC/compaction on the local storage
While they look nice-to-have in some situations but may also add more
complexity in implementation, so I tentatively concluded that they can
be put off until the next iteration.
New draft needs some more polish but I'd like to get early feedback
on the new draft.
Detailed draft:
enum StorageType { "temporary", "persistent" };
partial interface Navigator {
readonly attribute StorageQuota storageQuota;
};
[NoInterfaceObject] interface StorageInfo {
unsigned long long usageInBytes;
unsigned long long quotaInBytes;
};
[NoInterfaceObject] interface StorageQuota {
readonly attribute StorageType[] supportedTypes;
Promise queryStorageInfo(StorageType type);
Promise requestQuota(StorageType type, unsigned long long
newQuotaInBytes);
Promise getStorageType((IDBObjectStore or Database or
Entry) object);
Promise setStorageType((IDBObjectStore or Database or Entry)
object, StorageType type);
StorageWatcher createStorageWatcher(StorageType type)
};
This new draft uses string enums to specify storage types rather than
separate attributes on navigator (e.g. navigator.temporaryStorage),
mainly because some methods (like {get,set}StorageType do not fit well
in split interface) and to preserve greater flexibility to add more storage
types in a future. I'm open to discussions though.
supportedTypes are list of all StorageType's supported by the UA.
* queryStorageInfo and requestQuota are Promise version of
queryUsageAndQuota and requestQuota, which is for querying the current
storage info (usage and quota) and requesting a new quota, respectively.
Both return the current (or updated) StorageInfo.
* getStorageType and setStorageType are new methods which are intended to
work horizontally across multiple storage APIs. getStorageType(object)
returns the current storage type for the given storage object, and
setStorageType(object, type) changes the object's storage type.
They may fail if the storage backend of the object does not support
Quota API or does not support getting or setting (changing) storage types.
We're aware that this API may not work very well with FileSystem API(s),
and
also will need coordination with IndexedDB. Feedback is strongly
encouraged
here.
* createStorageWatcher returns a new StorageWatcher object for the given
storage type, with which webapps can listen to important changes in the
local storage and can react appropriately (i.e. discard their
lower-priority
cache data before UA-initiated merciless eviction happens).
[NoInterfaceObject] interface createStorageWatcher {
attribute EventHandler onstoragelow;
attribute EventHandler onstorageok;
attribute EventHandler onstoragechange;
};
storagelow and storageok events are modeled after DEVICE_STORAGE_LOW
and DEVICE_STORAGE_OK intents on android, to give more chances to
webapps to adjust their local data quickly.
* storagelow event is fired when the remaining storage space becomes lower
than 10% of all available space for the type, or before the quota backend
triggers eviction (for temporary case), whichever happens first.
This event must be also fired once when either one of these conditions has
been already met when a StorageWatcher is created.
* storageok event is fired when the remaining available storage goes out of
'storagelow' state.
* storagechange event is fired about every 1 second (TBD) or every time
the quota backend detects the amount of remaining storage is changed,
whichever is least frequent. This event must be fired before storagelow
or storageok when the remaining space crosses the threshold.
Again, your review/feedback is really encouraged. Thanks!
Kinuko
