Re: [whatwg] Shared storage

[Brett Zamir]

(Apologies...resending due to my inadvertent use of generic "whatwg" subject...)

On 10/28/2014 12:06 PM,whatwg-requ...@lists.whatwg.org  wrote:


Date: Tue, 28 Oct 2014 17:33:02 + (UTC)
From: Ian Hickson
Subject: Re: [whatwg] Shared storage
On Sat, 15 Feb 2014, Brett Zamir wrote:

The desktop PC thankfully evolved into allowing third-party software
which could create and edit files shareable by other third-party
software which would have the same rights to do the same. The importance
of this can hardly be overestimated.

Yet today, on the web, there appears to be no standard way to create
content in such an agnostic manner whereby users have full, built-in,
locally-controlled portability of their data.

Why can't you just do the same as used to be done? Download the resource
locally ("save", using ), then upload it to the new site
("open", using )?


Yes, as mentioned by others, this can become a terrible user experience,
and this is not to speak of user agent reasons.

Besides the added challenges mentioned by Katelyn Gadd when live updates
occur on multiple instances of the file across different applications,
there is also the even more common use case of a particular app being
able to remember the files used previously and let the user access them
without needing to remember their exact location in a hierarchy (though
allowing a hierarchy is desirable to the user for flexibility in
organization).

Such recall of, and access to, previously used files without repeated
need for manual selection is commonly found in apps which use the likes
of a "recent files" drop-down or, perhaps even more commonly, by a set
of tabs which open with the last set of used files.

There is also the specific desirability for functionality to iterate
through file names (or other shared data) so that apps can provide their
own UI, perhaps filtered down by file type according to the types of
files consumable by the app, such as an IDE project viewer which still
allows the user to group files app-agnostically and where they wish
along with other file types.

The ability to store files of different types within the same
user-viewable and user-creatable folder is compelling because the user
has freedom to group like content together even if the file types
differ. A user might wish to store an email draft, a word processing
file, and a set of images all in the same folder to keep track of them,
even if a given web app might not utilize all of these types. While
there are apps which aggregate data from different sources and then let
the user tag them in such a manner as to mimic this functionality, this
is again application-specific and not necessarily portable or as
flexibly under user control as would be a shared and hierarchical file
storage area.

Best,
Brett

Re: [whatwg] Shared storage

[Nils Dagsson Moskopp]

Melvin Carvalho  writes:

> On 28 October 2014 21:32, Nils Dagsson Moskopp <
> n...@dieweltistgarnichtso.net> wrote:
>
>> Melvin Carvalho  writes:
>>
>> > On 15 February 2014 03:04, Brett Zamir  wrote:
>> >
>> >> *The opportunity and current obstacles*
>> >>
>> >> The desktop PC thankfully evolved into allowing third-party software
>> which
>> >> could create and edit files shareable by other third-party software
>> which
>> >> would have the same rights to do the same. The importance of this can
>> >> hardly be overestimated.
>> >>
>> >> Yet today, on the web, there appears to be no standard way to create
>> >> content in such an agnostic manner whereby users have full, built-in,
>> >> locally-controlled portability of their data.
>> >>
>> >> *Workarounds*
>> >>
>> >> Sure, there is postMessage or CORS requests which can be used to allow
>> one
>> >> site to be the arbiter of this data.
>> >>
>> >> And one could conceivably create a shared data store built upon even
>> >> postMessage alone, even one which can work fully offline through cache
>> >> manifests and localStorage or IndexedDB (I have begun some work on this
>> >> concept at https://gist.github.com/brettz9/8876920 ), but this can only
>> >> work if:
>> >>
>> >> 1. A site or set of domains is trusted to host the shared content.
>> >> 2. Instead of being built into the browser, it requires that the shared
>> >> storage site be visited at least one time.
>> >>
>> >> *Proposal*
>> >>
>> >> 1. Add support for sharedStorage (similar to globalStorage but requiring
>> >> approval), SharedIndexedDB, and SharedFileWriter/SharedFileSystem which,
>> >> when used, would cause the browser to prompt the user to require user
>> >> approval whenever storing or retrieving from such data stores (with an
>> >> option to remember the choice for a particular site/domain), informing
>> >> users of potential risks depending on how the data might be used, and
>> >> potentially allowing them to view, on the spot, the specific data that
>> was
>> >> being stored.
>> >>
>> >> Optional API methods could deter XSS by doing selective escaping, but
>> the
>> >> potential for abuse should not be used as an excuse for preventing
>> >> arbitrary shared storage, since again, it is worked well on the desktop,
>> >> despite risks there, and as works with postMessage despite it also
>> having
>> >> risks.
>> >>
>> >> 2. Add support for corresponding ReadonlyShared storage mechanisms,
>> >> namespaced by the origin site of the data. A site, http://example.com
>> >> might add such shared storage under "example.com" which
>> >> http://some-other-site.example could retrieve but not alter or delete
>> >> (unless perhaps a grave warning were given to users about the fact that
>> >> this was not the same domain). This would have the benefit above
>> >> postMessage in that if the origin site goes down, third party sites
>> would
>> >> still be able to have access to the data.
>> >>
>> >> 3. Encourage browsers to allow direct editing of this stored data in a
>> >> human-readable manner (with files at least being ideally directly
>> viewable
>> >> from the OS desktop).
>> >>
>> >> I proposed something similar earlier, and received a reply about doing
>> >> this through shared workers, but as I understood it, I did not like that
>> >> possibility because:
>> >>
>> >> a. it would limit the neutrality of the storage, creating one site
>> as
>> >> an unchallengeable arbiter of the data
>> >> b. it would increase complexity for developers
>> >> c. it would presumably depend on the setting of CORS directives to
>> >> distinguish it from same-domain shared workers.
>> >>
>> >> While https://wiki.mozilla.org/WebAPI/DeviceStorageAPI appears to meet
>> a
>> >> subset of these needs, it does not meet all.
>> >>
>> >
>> > +1
>>
>> Stop doing this.
>>
>
> Excuse me?
>

If I am not mistaken, you full-quoted an entire Email just to add „+1“.

If you would have done similar in the Google+ Ghetto or on Facebook, I
would have no problem with your behaviour. However, on a mailing list,
you are annoying people, probably the few hundred or thousand who are
subscribed in this case. You are wasting their time – and yours, if I
might add … since the WHATWG was not a democracy last time I checked.

-- 
Nils Dagsson Moskopp // erlehmann

Re: [whatwg] Shared storage

[Melvin Carvalho]

On 28 October 2014 21:32, Nils Dagsson Moskopp <
n...@dieweltistgarnichtso.net> wrote:

> Melvin Carvalho  writes:
>
> > On 15 February 2014 03:04, Brett Zamir  wrote:
> >
> >> *The opportunity and current obstacles*
> >>
> >> The desktop PC thankfully evolved into allowing third-party software
> which
> >> could create and edit files shareable by other third-party software
> which
> >> would have the same rights to do the same. The importance of this can
> >> hardly be overestimated.
> >>
> >> Yet today, on the web, there appears to be no standard way to create
> >> content in such an agnostic manner whereby users have full, built-in,
> >> locally-controlled portability of their data.
> >>
> >> *Workarounds*
> >>
> >> Sure, there is postMessage or CORS requests which can be used to allow
> one
> >> site to be the arbiter of this data.
> >>
> >> And one could conceivably create a shared data store built upon even
> >> postMessage alone, even one which can work fully offline through cache
> >> manifests and localStorage or IndexedDB (I have begun some work on this
> >> concept at https://gist.github.com/brettz9/8876920 ), but this can only
> >> work if:
> >>
> >> 1. A site or set of domains is trusted to host the shared content.
> >> 2. Instead of being built into the browser, it requires that the shared
> >> storage site be visited at least one time.
> >>
> >> *Proposal*
> >>
> >> 1. Add support for sharedStorage (similar to globalStorage but requiring
> >> approval), SharedIndexedDB, and SharedFileWriter/SharedFileSystem which,
> >> when used, would cause the browser to prompt the user to require user
> >> approval whenever storing or retrieving from such data stores (with an
> >> option to remember the choice for a particular site/domain), informing
> >> users of potential risks depending on how the data might be used, and
> >> potentially allowing them to view, on the spot, the specific data that
> was
> >> being stored.
> >>
> >> Optional API methods could deter XSS by doing selective escaping, but
> the
> >> potential for abuse should not be used as an excuse for preventing
> >> arbitrary shared storage, since again, it is worked well on the desktop,
> >> despite risks there, and as works with postMessage despite it also
> having
> >> risks.
> >>
> >> 2. Add support for corresponding ReadonlyShared storage mechanisms,
> >> namespaced by the origin site of the data. A site, http://example.com
> >> might add such shared storage under "example.com" which
> >> http://some-other-site.example could retrieve but not alter or delete
> >> (unless perhaps a grave warning were given to users about the fact that
> >> this was not the same domain). This would have the benefit above
> >> postMessage in that if the origin site goes down, third party sites
> would
> >> still be able to have access to the data.
> >>
> >> 3. Encourage browsers to allow direct editing of this stored data in a
> >> human-readable manner (with files at least being ideally directly
> viewable
> >> from the OS desktop).
> >>
> >> I proposed something similar earlier, and received a reply about doing
> >> this through shared workers, but as I understood it, I did not like that
> >> possibility because:
> >>
> >> a. it would limit the neutrality of the storage, creating one site
> as
> >> an unchallengeable arbiter of the data
> >> b. it would increase complexity for developers
> >> c. it would presumably depend on the setting of CORS directives to
> >> distinguish it from same-domain shared workers.
> >>
> >> While https://wiki.mozilla.org/WebAPI/DeviceStorageAPI appears to meet
> a
> >> subset of these needs, it does not meet all.
> >>
> >
> > +1
>
> Stop doing this.
>

Excuse me?


>
> --
> Nils Dagsson Moskopp // erlehmann
> 
>

Re: [whatwg] Shared storage

[Melvin Carvalho]

On 15 February 2014 03:04, Brett Zamir  wrote:

> *The opportunity and current obstacles*
>
> The desktop PC thankfully evolved into allowing third-party software which
> could create and edit files shareable by other third-party software which
> would have the same rights to do the same. The importance of this can
> hardly be overestimated.
>
> Yet today, on the web, there appears to be no standard way to create
> content in such an agnostic manner whereby users have full, built-in,
> locally-controlled portability of their data.
>
> *Workarounds*
>
> Sure, there is postMessage or CORS requests which can be used to allow one
> site to be the arbiter of this data.
>
> And one could conceivably create a shared data store built upon even
> postMessage alone, even one which can work fully offline through cache
> manifests and localStorage or IndexedDB (I have begun some work on this
> concept at https://gist.github.com/brettz9/8876920 ), but this can only
> work if:
>
> 1. A site or set of domains is trusted to host the shared content.
> 2. Instead of being built into the browser, it requires that the shared
> storage site be visited at least one time.
>
> *Proposal*
>
> 1. Add support for sharedStorage (similar to globalStorage but requiring
> approval), SharedIndexedDB, and SharedFileWriter/SharedFileSystem which,
> when used, would cause the browser to prompt the user to require user
> approval whenever storing or retrieving from such data stores (with an
> option to remember the choice for a particular site/domain), informing
> users of potential risks depending on how the data might be used, and
> potentially allowing them to view, on the spot, the specific data that was
> being stored.
>
> Optional API methods could deter XSS by doing selective escaping, but the
> potential for abuse should not be used as an excuse for preventing
> arbitrary shared storage, since again, it is worked well on the desktop,
> despite risks there, and as works with postMessage despite it also having
> risks.
>
> 2. Add support for corresponding ReadonlyShared storage mechanisms,
> namespaced by the origin site of the data. A site, http://example.com
> might add such shared storage under "example.com" which
> http://some-other-site.example could retrieve but not alter or delete
> (unless perhaps a grave warning were given to users about the fact that
> this was not the same domain). This would have the benefit above
> postMessage in that if the origin site goes down, third party sites would
> still be able to have access to the data.
>
> 3. Encourage browsers to allow direct editing of this stored data in a
> human-readable manner (with files at least being ideally directly viewable
> from the OS desktop).
>
> I proposed something similar earlier, and received a reply about doing
> this through shared workers, but as I understood it, I did not like that
> possibility because:
>
> a. it would limit the neutrality of the storage, creating one site as
> an unchallengeable arbiter of the data
> b. it would increase complexity for developers
> c. it would presumably depend on the setting of CORS directives to
> distinguish it from same-domain shared workers.
>
> While https://wiki.mozilla.org/WebAPI/DeviceStorageAPI appears to meet a
> subset of these needs, it does not meet all.
>

+1


>
> Thank you,
> Brett
>
>

Re: [whatwg] Shared storage

[Katelyn Gadd]

A smarter file-picker is not going to address the significant UX and
performance issues with save->open for moving files between
applications.

Many common workflows involve having files open in multiple editors at
once, saving in one app, reloading in others. Some applications have
built in affordances for this, like text editors and IDEs that
automatically reload when the content has changed on disk. One
real-world example: Unity's project management lets you pull various
resources into your game, then live-edit the resources in Maya,
Photoshop, etc with the IDE automatically pulling in the changes and
running builds to update the content.

If this kind of interchange and interaction requires literally doing
save->confirm save->open->pick saved file from wherever it went *every
time* nobody will use these applications. The applications that get
used will be ones that figure out a clever set of workarounds like the
postmessage/blessed domain one mentioned in the original post. That
sort of approach is something that will easily lead to a walled garden
library that 'solves the problem' and becomes widely adopted, when
HTML5 itself could probably tackle the issue without a bunch of new
scaffolding or significant new APIs.

Mind you, I'm not saying this is an easy problem to solve - just that
it's worth solving and that save->open is not a solution.

One idea: 'transferable' support for indexedDB such that a blessed
domain can postMessage an indexedDB database over to another domain,
to allow shared access to storage for a specific database. This
doesn't address all the problems in the original post, but simplifies
the task of sharing content between applications. This would probably
also pay dividends for web workers.
(Actually, is this already specced? I only know that worker access to
indexedDB wasn't available in Gecko or Blink the last time I checked
and didn't seem to be a planned addition.)

Another option: Some simple protocol for creating 'shared' databases
using a special token in the name you pass to indexedDB.open? I can
see how this might be a compat problem, but it would be a pretty
low-overhead way to expose basic support for this. Perhaps the unique
token you pass would be created by a feature-detectable API, sort of
like how createObjectURL works. That would give you a single point
where you can insert approval/feedback UI and do any necessary I/O to
set up sharing, while the rest of the database flow remains unchanged.

Synchronization between two processes mutually touching the DB is hard
but you could just punt on that like is currently done with typed
arrays - neutering or equivalent. Alternately, a single database-wide
lock would probably cover things - that's the approach SQLite used for
a long time.

-kg


On 28 October 2014 12:17, Nils Dagsson Moskopp
 wrote:
> Tobie Langel  writes:
>
>> On Tue, Oct 28, 2014 at 6:33 PM, Ian Hickson  wrote:
>>
>>> On Sat, 15 Feb 2014, Brett Zamir wrote:
>>> >
>>> > The desktop PC thankfully evolved into allowing third-party software
>>> > which could create and edit files shareable by other third-party
>>> > software which would have the same rights to do the same. The importance
>>> > of this can hardly be overestimated.
>>> >
>>> > Yet today, on the web, there appears to be no standard way to create
>>> > content in such an agnostic manner whereby users have full, built-in,
>>> > locally-controlled portability of their data.
>>>
>>> Why can't you just do the same as used to be done? Download the resource
>>> locally ("save", using ), then upload it to the new site
>>> ("open", using )?
>>>
>>
>> Because that's a terrible user experience?
>
> If that is indeed the case, the terrible user experience is likely a
> feature of your user agent. Many mobile UAs currently offer several
> alternatives to the standard file-picker, no change in HTML needed.
>
> --
> Nils Dagsson Moskopp // erlehmann
> 

Re: [whatwg] Shared storage

[Nils Dagsson Moskopp]

Tobie Langel  writes:

> On Tue, Oct 28, 2014 at 6:33 PM, Ian Hickson  wrote:
>
>> On Sat, 15 Feb 2014, Brett Zamir wrote:
>> >
>> > The desktop PC thankfully evolved into allowing third-party software
>> > which could create and edit files shareable by other third-party
>> > software which would have the same rights to do the same. The importance
>> > of this can hardly be overestimated.
>> >
>> > Yet today, on the web, there appears to be no standard way to create
>> > content in such an agnostic manner whereby users have full, built-in,
>> > locally-controlled portability of their data.
>>
>> Why can't you just do the same as used to be done? Download the resource
>> locally ("save", using ), then upload it to the new site
>> ("open", using )?
>>
>
> Because that's a terrible user experience?

If that is indeed the case, the terrible user experience is likely a
feature of your user agent. Many mobile UAs currently offer several
alternatives to the standard file-picker, no change in HTML needed.

-- 
Nils Dagsson Moskopp // erlehmann

Re: [whatwg] Shared storage

[Tobie Langel]

On Tue, Oct 28, 2014 at 6:33 PM, Ian Hickson  wrote:

> On Sat, 15 Feb 2014, Brett Zamir wrote:
> >
> > The desktop PC thankfully evolved into allowing third-party software
> > which could create and edit files shareable by other third-party
> > software which would have the same rights to do the same. The importance
> > of this can hardly be overestimated.
> >
> > Yet today, on the web, there appears to be no standard way to create
> > content in such an agnostic manner whereby users have full, built-in,
> > locally-controlled portability of their data.
>
> Why can't you just do the same as used to be done? Download the resource
> locally ("save", using ), then upload it to the new site
> ("open", using )?
>

Because that's a terrible user experience?

--tobie

Re: [whatwg] Shared storage

[Ian Hickson]

On Sat, 15 Feb 2014, Brett Zamir wrote:
> 
> The desktop PC thankfully evolved into allowing third-party software 
> which could create and edit files shareable by other third-party 
> software which would have the same rights to do the same. The importance 
> of this can hardly be overestimated.
> 
> Yet today, on the web, there appears to be no standard way to create 
> content in such an agnostic manner whereby users have full, built-in, 
> locally-controlled portability of their data.

Why can't you just do the same as used to be done? Download the resource 
locally ("save", using ), then upload it to the new site 
("open", using )?

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'