Re: [XHR] chunked

[Anne van Kesteren]

On Thu, Oct 11, 2012 at 1:25 PM, Anne van Kesteren  wrote:
> [...]

Thanks all for your patience.

http://xhr.spec.whatwg.org/ now includes the Streams API.

https://github.com/whatwg/xhr/commit/4b56e0f353362d50b87539ff906519892ee13652
documents the change.


-- 
http://annevankesteren.nl/

Re: [XHR] chunked

[Anne van Kesteren]

On Tue, Oct 2, 2012 at 2:56 PM, Adrian Bateman  wrote:
> On Thursday, September 27, 2012 10:43 AM, Travis Leithead wrote:
>> In my observation of the current IE behavior, the Stream is for download
>> only. XHR gets the data from the server and buffers it. The consumer of the
>> stream then pulls data as needed which is extracted from the buffer.
>
> In IE10, we only implemented the download part of our proposal. The idea is 
> that
> you should be able to upload a continuing stream using the stream builder API.
> However, not many services support chunked upload nor does our underlying 
> network
> stack on the client so it was a low priority and not something we've tackled 
> yet.

Just to be clear here, initially I thought
http://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm#send0 to
be too unclear for the streaming scenario, but I suppose it can work
if you assume the raw data by the stream object is not a constant
until the object is closed. I will take a stab at updating
http://xhr.spec.whatwg.org/ with the relevant text and references.


-- 
http://annevankesteren.nl/

RE: [XHR] chunked

[Adrian Bateman]

On Thursday, September 27, 2012 10:43 AM, Travis Leithead wrote:
> > From: annevankeste...@gmail.com [mailto:annevankeste...@gmail.com] On
> >
> > On Thu, Sep 27, 2012 at 7:00 PM, Travis Leithead
> >  wrote:
> > > It hasn't been updated in a while, but we're still keen on seeing it
> > move forward AFAIK:
> > > http://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm
> >
> > Cool. I can integrate the relevant bits into XMLHttpRequest. Do I
> > understand it correctly that it does not allow for streaming data
> > towards the server? It seems to just copy the data from the Stream
> > object that it currently represents and that's that. Is that really
> > what we want here?
> 
> In my observation of the current IE behavior, the Stream is for download
> only. XHR gets the data from the server and buffers it. The consumer of the
> stream then pulls data as needed which is extracted from the buffer.

In IE10, we only implemented the download part of our proposal. The idea is that
you should be able to upload a continuing stream using the stream builder API.
However, not many services support chunked upload nor does our underlying 
network
stack on the client so it was a low priority and not something we've tackled 
yet.

Adrian.

Re: [XHR] chunked

[Wenbo Zhu]

On Thu, Sep 27, 2012 at 12:21 PM, Travis Leithead <
travis.leith...@microsoft.com> wrote:

> > From: annevankeste...@gmail.com [mailto:annevankeste...@gmail.com]
> >
> > On Thu, Sep 27, 2012 at 7:42 PM, Travis Leithead
> >  wrote:
> > > In my observation of the current IE behavior, the Stream is for
> download
> > only. XHR gets the data from the server and buffers it. The consumer of
> > the stream then pulls data as needed which is extracted from the buffer.
> >
> > I see, so the bit where it says you can pass it to send() we should
> > maybe not add for now if it's not going to do something useful.
>
> I honestly haven't tested that part, but this seems safe (it can be added-
> in later if need be).
>
Could you confirm if it actually works ... even if the spec fails to talk
about it? :)

Re: [XHR] chunked

[Wenbo Zhu]

On Thu, Sep 27, 2012 at 12:21 PM, Travis Leithead <
travis.leith...@microsoft.com> wrote:

> > From: annevankeste...@gmail.com [mailto:annevankeste...@gmail.com]
> >
> > On Thu, Sep 27, 2012 at 7:42 PM, Travis Leithead
> >  wrote:
> > > In my observation of the current IE behavior, the Stream is for
> download
> > only. XHR gets the data from the server and buffers it. The consumer of
> > the stream then pulls data as needed which is extracted from the buffer.
> >
> > I see, so the bit where it says you can pass it to send() we should
> > maybe not add for now if it's not going to do something useful.
>
> I honestly haven't tested that part, but this seems safe (it can be added-
> in later if need be).
>
The send() version, i.e. chunked requests, will actually be more useful,
but I guess someone has to use it first, e.g. to cut the number of requests.

RE: [XHR] chunked

[Travis Leithead]

> From: annevankeste...@gmail.com [mailto:annevankeste...@gmail.com]
> 
> On Thu, Sep 27, 2012 at 7:42 PM, Travis Leithead
>  wrote:
> > In my observation of the current IE behavior, the Stream is for download
> only. XHR gets the data from the server and buffers it. The consumer of
> the stream then pulls data as needed which is extracted from the buffer.
> 
> I see, so the bit where it says you can pass it to send() we should
> maybe not add for now if it's not going to do something useful.

I honestly haven't tested that part, but this seems safe (it can be added-
in later if need be).

Re: [XHR] chunked

[Anne van Kesteren]

On Thu, Sep 27, 2012 at 7:42 PM, Travis Leithead
 wrote:
> In my observation of the current IE behavior, the Stream is for download 
> only. XHR gets the data from the server and buffers it. The consumer of the 
> stream then pulls data as needed which is extracted from the buffer.

I see, so the bit where it says you can pass it to send() we should
maybe not add for now if it's not going to do something useful.


-- 
http://annevankesteren.nl/

RE: [XHR] chunked

[Travis Leithead]

> From: annevankeste...@gmail.com [mailto:annevankeste...@gmail.com] On
> 
> On Thu, Sep 27, 2012 at 7:00 PM, Travis Leithead
>  wrote:
> > It hasn't been updated in a while, but we're still keen on seeing it
> move forward AFAIK:
> > http://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm
> 
> Cool. I can integrate the relevant bits into XMLHttpRequest. Do I
> understand it correctly that it does not allow for streaming data
> towards the server? It seems to just copy the data from the Stream
> object that it currently represents and that's that. Is that really
> what we want here?

In my observation of the current IE behavior, the Stream is for download only. 
XHR gets the data from the server and buffers it. The consumer of the stream 
then pulls data as needed which is extracted from the buffer.

Re: [XHR] chunked

[Anne van Kesteren]

On Thu, Sep 27, 2012 at 7:00 PM, Travis Leithead
 wrote:
> It hasn't been updated in a while, but we're still keen on seeing it move 
> forward AFAIK:
> http://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

Cool. I can integrate the relevant bits into XMLHttpRequest. Do I
understand it correctly that it does not allow for streaming data
towards the server? It seems to just copy the data from the Stream
object that it currently represents and that's that. Is that really
what we want here?


-- 
http://annevankesteren.nl/

RE: [XHR] chunked

[Travis Leithead]

> From: Jonas Sicking [mailto:jo...@sicking.cc]
> 
> On Thu, Sep 27, 2012 at 9:26 AM, Anne van Kesteren 
> wrote:
> > On Thu, Sep 27, 2012 at 6:23 PM, Jonas Sicking  wrote:
> >> I do somewhat agree that if we had a full "stream" solution in the
> >> form of a Stream primitive and .responseType="stream", then a better
> >> solution might be to use that in combination with having chunked
> >> delivery on the Stream class instead.
> >
> > Was Microsoft going to take a stab at that? I could write something
> > like that up in XMLHttpRequest I suppose. Although maybe that would
> > require a bunch of coordination with the MediaStream stuff?
> 
> I can't speak to anyone else's plans. But it does seem like a proposal
> was made quite a long time ago and as I recall it it received
> favorable feedback, but so far nothing else has happened.

It hasn't been updated in a while, but we're still keen on seeing it move 
forward AFAIK:
http://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

Re: [XHR] chunked

[Jonas Sicking]

On Thu, Sep 27, 2012 at 9:26 AM, Anne van Kesteren  wrote:
> On Thu, Sep 27, 2012 at 6:23 PM, Jonas Sicking  wrote:
>> I do somewhat agree that if we had a full "stream" solution in the
>> form of a Stream primitive and .responseType="stream", then a better
>> solution might be to use that in combination with having chunked
>> delivery on the Stream class instead.
>
> Was Microsoft going to take a stab at that? I could write something
> like that up in XMLHttpRequest I suppose. Although maybe that would
> require a bunch of coordination with the MediaStream stuff?

I can't speak to anyone else's plans. But it does seem like a proposal
was made quite a long time ago and as I recall it it received
favorable feedback, but so far nothing else has happened.

/ Jonas

Re: [XHR] chunked

[Anne van Kesteren]

On Thu, Sep 27, 2012 at 6:23 PM, Jonas Sicking  wrote:
> I do somewhat agree that if we had a full "stream" solution in the
> form of a Stream primitive and .responseType="stream", then a better
> solution might be to use that in combination with having chunked
> delivery on the Stream class instead.

Was Microsoft going to take a stab at that? I could write something
like that up in XMLHttpRequest I suppose. Although maybe that would
require a bunch of coordination with the MediaStream stuff?


-- 
http://annevankesteren.nl/

Re: [XHR] chunked

[Jonas Sicking]

On Thu, Sep 27, 2012 at 6:27 AM, Anne van Kesteren  wrote:
> On Thu, Jun 7, 2012 at 11:48 AM, Wenbo Zhu  wrote:
>> On Thu, Jun 7, 2012 at 2:34 AM, Anne van Kesteren  wrote:
>>> Can I take this as Chrome being interested in implementing the chunked
>>> proposal as well?
>>
>> Will have to get back to you on this, since I am mostly working on
>> servers and HTTP/application protocols.
>
> Still trying to figure out what to do here. Any other XMLHttpRequest
> implementors hoping to implement this feature? Has it been adopted by
> developers?

I know pdf.js uses this in order to implement incremental rendering of
pdf files.

I do somewhat agree that if we had a full "stream" solution in the
form of a Stream primitive and .responseType="stream", then a better
solution might be to use that in combination with having chunked
delivery on the Stream class instead.

/ Jonas

Re: [XHR] chunked

[Anne van Kesteren]

On Thu, Jun 7, 2012 at 11:48 AM, Wenbo Zhu  wrote:
> On Thu, Jun 7, 2012 at 2:34 AM, Anne van Kesteren  wrote:
>> Can I take this as Chrome being interested in implementing the chunked
>> proposal as well?
>
> Will have to get back to you on this, since I am mostly working on
> servers and HTTP/application protocols.

Still trying to figure out what to do here. Any other XMLHttpRequest
implementors hoping to implement this feature? Has it been adopted by
developers?


-- 
http://annevankesteren.nl/

Re: [XHR] chunked

[Wenbo Zhu]

On Thu, Jun 7, 2012 at 3:11 AM, Anne van Kesteren  wrote:

> On Thu, Jun 7, 2012 at 11:48 AM, Wenbo Zhu  wrote:
> > I suppose you still favor the idea of exposing chunked HTTP to
> > applications. Without it, native clients will be increasingly deployed
> > in place of HTML5 apps.
>
> Lets not play scare-tactics.

Merely by presenting facts? Certainly in the mobile world, you have options
to not rely on HTTP or browsers.


> And besides, there's server-sent events
> and the WebSocket protocol.
>
Can't argue about this.


>
>
> --
> Anne — Opera Software
> http://annevankesteren.nl/
> http://www.opera.com/
>

Re: [XHR] chunked

[Anne van Kesteren]

On Thu, Jun 7, 2012 at 11:48 AM, Wenbo Zhu  wrote:
> I suppose you still favor the idea of exposing chunked HTTP to
> applications. Without it, native clients will be increasingly deployed
> in place of HTML5 apps.

Lets not play scare-tactics. And besides, there's server-sent events
and the WebSocket protocol.


-- 
Anne — Opera Software
http://annevankesteren.nl/
http://www.opera.com/

Re: [XHR] chunked

[Wenbo Zhu]

On Thu, Jun 7, 2012 at 2:34 AM, Anne van Kesteren  wrote:
> On Thu, Jun 7, 2012 at 11:30 AM, Wenbo Zhu  wrote:
>> Yes, this will be the desired behavior.
>
> Can I take this as Chrome being interested in implementing the chunked
> proposal as well?
Will have to get back to you on this, since I am mostly working on
servers and HTTP/application protocols.

>
> Because if only Mozilla is interested in these I don't see much point
> adding them to the standard, as I indicated before.
I suppose you still favor the idea of exposing chunked HTTP to
applications. Without it, native clients will be increasingly deployed
in place of HTML5 apps.
>
>
> --
> Anne — Opera Software
> http://annevankesteren.nl/
> http://www.opera.com/

Re: [XHR] chunked

[Anne van Kesteren]

On Thu, Jun 7, 2012 at 11:30 AM, Wenbo Zhu  wrote:
> Yes, this will be the desired behavior.

Can I take this as Chrome being interested in implementing the chunked
proposal as well?

Because if only Mozilla is interested in these I don't see much point
adding them to the standard, as I indicated before.


-- 
Anne — Opera Software
http://annevankesteren.nl/
http://www.opera.com/

Re: [XHR] chunked

[Wenbo Zhu]

On Thu, Jun 7, 2012 at 2:14 AM, Henri Sivonen  wrote:

> On Thu, May 24, 2012 at 8:59 AM, Anne van Kesteren 
> wrote:
> > On Thu, May 24, 2012 at 2:54 AM, Jonas Sicking  wrote:
> >> Is there a reason not to add "chunked-text" and "chunked-arraybuffer"
> >> to the spec right now?
> >
> > 1. Why not just have
> > http://html5labs.interoperabilitybridges.com/streamsapi/ to address
> > this case?
>
> It appears that Microsoft's proposal involves potentially buffering
> the XHR response body until the Web author-supplied script chooses to
> initiate I read from the stream object. Or am I missing something?
>
> The "chunked-text" and the "chunked-arraybuffer" response types are
> both simpler than the stream proposal and have the advantage that they
> don't involve the browser engine buffering the XHR response body: if
> the Web author-provided script fails to handle the chunks as they
> arrive, the data is gone and doesn't fill up buffer space.
>
Yes, this will be the desired behavior.

>
> (Some of my remarks on IRC were confused, because I mistook the
> Worker-specific API for the API proposed for the main thread.)
>
> --
> Henri Sivonen
> hsivo...@iki.fi
> http://hsivonen.iki.fi/
>
>

Re: [XHR] chunked

[Henri Sivonen]

On Thu, May 24, 2012 at 8:59 AM, Anne van Kesteren  wrote:
> On Thu, May 24, 2012 at 2:54 AM, Jonas Sicking  wrote:
>> Is there a reason not to add "chunked-text" and "chunked-arraybuffer"
>> to the spec right now?
>
> 1. Why not just have
> http://html5labs.interoperabilitybridges.com/streamsapi/ to address
> this case?

It appears that Microsoft's proposal involves potentially buffering
the XHR response body until the Web author-supplied script chooses to
initiate I read from the stream object. Or am I missing something?

The "chunked-text" and the "chunked-arraybuffer" response types are
both simpler than the stream proposal and have the advantage that they
don't involve the browser engine buffering the XHR response body: if
the Web author-provided script fails to handle the chunks as they
arrive, the data is gone and doesn't fill up buffer space.

(Some of my remarks on IRC were confused, because I mistook the
Worker-specific API for the API proposed for the main thread.)

-- 
Henri Sivonen
hsivo...@iki.fi
http://hsivonen.iki.fi/

Re: [XHR] chunked

[Anne van Kesteren]

On Thu, May 24, 2012 at 2:54 AM, Jonas Sicking  wrote:
> No change came out of this thread. I think we established that there
> is a need for "chunked-arraybuffer". This thread contained some
> controversy about whether "chunked-text" is needed, but in the end the
> controversy didn't actually seem relevant to the question at hand.
>
> Is there a reason not to add "chunked-text" and "chunked-arraybuffer"
> to the spec right now?

1. Why not just have
http://html5labs.interoperabilitybridges.com/streamsapi/ to address
this case? 2. Is anyone else planning on implementing these?


-- 
Anne — Opera Software
http://annevankesteren.nl/
http://www.opera.com/

Re: [XHR] chunked

[Jonas Sicking]

On Wed, Nov 30, 2011 at 7:28 AM, Anne van Kesteren  wrote:
>
> A while ago sicking proposed adding chunked support to XMLHttpRequest:
>
> http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/0741.html
> https://bugzilla.mozilla.org/show_bug.cgi?id=687087
>
> A use case I remember was downloading a large file of some kind that
> presumably can be incrementally rendered as otherwise responseType "blob"
> should be sufficient. More use cases appreciated. Would help with the
> design.
>
> As for the feature, basically have responseType "chunked-text" and
> "chunked-arraybuffer" values and reset rather than update the response
> entity body with each progress event. And make sure that a progress event is
> dispatched when the last fetch event is queued. And make sure that this is
> only available for asynchronous usage.
>
> Charles asked whether "chunked-text" was really needed (and whether we
> should have "chunked" which implies ArrayBuffer instead). Nobody got back to
> him on that. If it is needed, how does it work when you just have some of
> the bytes of a multi-byte character in a single chunk? Fails to decode as
> per the normal algorithm?
>
> Also, this basically makes it possible to write EventSource on top of
> XMLHttpRequest. Is that acceptable? If it encourages more people to use a
> lower-level API, higher-level optimizations for mobile phones might become
> harder down the road.

No change came out of this thread. I think we established that there
is a need for "chunked-arraybuffer". This thread contained some
controversy about whether "chunked-text" is needed, but in the end the
controversy didn't actually seem relevant to the question at hand.

Is there a reason not to add "chunked-text" and "chunked-arraybuffer"
to the spec right now?

/ Jonas

Re: [XHR] chunked requests

[Robert O'Callahan]

On Sat, Dec 10, 2011 at 2:14 AM, Anne van Kesteren  wrote:

> On Thu, 08 Dec 2011 23:16:37 +0100, Jonas Sicking 
> wrote:
>
>> I think Microsoft's stream proposal would address this use case.
>>
>
> So that would be: http://html5labs.interoperabilitybridges.com/streamsapi/
>


> How does that relate to the various APIs for streaming media?
>

They're entirely unrelated (except for an unfortunate naming similarity).
Microsoft's Streams are a very different abstraction to MediaStreams.

Rob
-- 
"If we claim to be without sin, we deceive ourselves and the truth is not
in us. If we confess our sins, he is faithful and just and will forgive us
our sins and purify us from all unrighteousness. If we claim we have not
sinned, we make him out to be a liar and his word is not in us." [1 John
1:8-10]

Re: [XHR] chunked requests

[Eric Rescorla]

On Tue, Dec 20, 2011 at 2:47 PM, Anne van Kesteren  wrote:
> On Tue, 20 Dec 2011 22:55:40 +0100, Eric Rescorla  wrote:
>>
>> That isn't to say that the browser stacks aren't adding 1/n+1
>> splitting. NSS, for instance, has such a fix. However, I don't think
>> there's anything to do from a TLS standards perspective.
>
>
> What I would like is a standard that actually defines what of TLS user
> agents need to implement. Instead of every user agent having to figure that
> out for themselves. Those unknowns (TLS is not alone here) create a barrier
> to entry and make it harder to create user agents that interoperate with
> "legacy" content.

Again, the TLS spec does tell you what to do: it tells you to
implement TLS 1.1 and/or 1.2.

What you're asking for is for it *also* to tell you what to do in
cases where you choose
to use a downrev version of TLS. But unfortunately, that guidance is
necessarily application
specific. In particular, the use of 1/n+1 splitting is the result of
discovering that some
HTTP stacks don't properly handle empty TLS records and even then some
stacks choke
Moreover, many non-HTTPS stacks have no need to do any sort of countermeasure
because Rizzo-Duong-style attacks don't apply.

If/when IETF publishes a revision of TLS, I suspect we'll add a
description of 1/n+1 splitting
as a backward compatibility measure, but I don't really think it's the
role of standards
to document security hacks for downrev versions in perpetuity.

-Ekr

Re: [XHR] chunked requests

[Anne van Kesteren]

On Tue, 20 Dec 2011 22:55:40 +0100, Eric Rescorla  wrote:

That isn't to say that the browser stacks aren't adding 1/n+1
splitting. NSS, for instance, has such a fix. However, I don't think  
there's anything to do from a TLS standards perspective.


What I would like is a standard that actually defines what of TLS user  
agents need to implement. Instead of every user agent having to figure  
that out for themselves. Those unknowns (TLS is not alone here) create a  
barrier to entry and make it harder to create user agents that  
interoperate with "legacy" content.



--
Anne van Kesteren
http://annevankesteren.nl/

Re: [XHR] chunked requests

[Eric Rescorla]

On Tue, Dec 20, 2011 at 12:11 PM, Anne van Kesteren  wrote:
> On Tue, 20 Dec 2011 21:06:28 +0100, Eric Rescorla  wrote:
>>
>> On Tue, Dec 20, 2011 at 9:36 AM, Anne van Kesteren 
>> wrote:
>>>
>>> Surely this should be patched in the base
>>> specification rather than in every API that interacts with it. I do not
>>> want to make the life of the guy implementing XMLHttpRequest more difficult
>>> if the problem is supposed to be addressed at the TLS layer anyway.
>>
>>
>> The problem was addressed at the TLS layer 5 years ago when we issued
>> TLS 1.1.
>
>
> If support for the old version cannot be removed it does not seem to me that
> the problem is actually addressed as far as user agents that have to support
> legacy servers are concerned.

I see your point, but it's not clear what you would like done about
it. The TLS WG
specifies protocols, not APIs, and this is an API issue. Even if it
were within scope,
all we would do is add some sort of security considerations note to
the TLS spec,
but again, it would only apply to RFC 2246, since TLS 1.1 and 1.2 don't have
this problem.

That isn't to say that the browser stacks aren't adding 1/n+1
splitting. NSS, for
instance, has such a fix. However, I don't think there's anything to do from a
TLS standards perspective.

-Ekr

Re: [XHR] chunked requests

[Anne van Kesteren]

On Tue, 20 Dec 2011 21:06:28 +0100, Eric Rescorla  wrote:
On Tue, Dec 20, 2011 at 9:36 AM, Anne van Kesteren   
wrote:

Surely this should be patched in the base
specification rather than in every API that interacts with it. I do not  
want to make the life of the guy implementing XMLHttpRequest more  
difficult if the problem is supposed to be addressed at the TLS layer  
anyway.


The problem was addressed at the TLS layer 5 years ago when we issued
TLS 1.1.


If support for the old version cannot be removed it does not seem to me  
that the problem is actually addressed as far as user agents that have to  
support legacy servers are concerned.



--
Anne van Kesteren
http://annevankesteren.nl/

Re: [XHR] chunked requests

[Eric Rescorla]

On Tue, Dec 20, 2011 at 9:36 AM, Anne van Kesteren  wrote:
> On Sun, 18 Dec 2011 13:12:57 +0100, Eric Rescorla  wrote:
>>
>> Sorry, I forgot to mention the 1/n+1 splitting countermeasure in my
>> response.
>>
>> With that said, this isn't TLS 1.1, but rather a specific, more
>> backwards-compatible countermeasure. It's fine for the security
>> considerations section to say here that browsers must do either TLS 1.1 or
>> 1/n+1 splitting, but it should say something, since it's not like 1/n+1
>> splitting is required by TLS (any version).
>
>
> Who's in charge of updating TLS?

Me.


> Surely this should be patched in the base
> specification rather than in every API that interacts with it. I do not want
> to make the life of the guy implementing XMLHttpRequest more difficult if
> the problem is supposed to be addressed at the TLS layer anyway.

The problem was addressed at the TLS layer 5 years ago when we issued
TLS 1.1.

-Ekr

Re: [XHR] chunked requests

[Anne van Kesteren]

On Sun, 18 Dec 2011 13:12:57 +0100, Eric Rescorla  wrote:
Sorry, I forgot to mention the 1/n+1 splitting countermeasure in my  
response.


With that said, this isn't TLS 1.1, but rather a specific, more
backwards-compatible countermeasure. It's fine for the security  
considerations section to say here that browsers must do either TLS 1.1  
or 1/n+1 splitting, but it should say something, since it's not like  
1/n+1 splitting is required by TLS (any version).


Who's in charge of updating TLS? Surely this should be patched in the base  
specification rather than in every API that interacts with it. I do not  
want to make the life of the guy implementing XMLHttpRequest more  
difficult if the problem is supposed to be addressed at the TLS layer  
anyway.



--
Anne van Kesteren
http://annevankesteren.nl/

Re: [XHR] chunked requests

[Eric Rescorla]

On Sat, Dec 17, 2011 at 6:11 AM, Anne van Kesteren  wrote:
> On Fri, 09 Dec 2011 19:54:31 +0100, Eric Rescorla  wrote:
>>
>> Unfortunately, many servers do not support TLS 1.1, and to make matters
>> worse, they do so in a way that is not securely verifiable. By which I
>> mean that an active attacker can force a client/server pair both of which
>> support TLS 1.1 down to TLS 1.0. This may be detectable in some way, but not
>> by TLS's built-in mechanisms. And since the threat model here is an active
>> attacker, this is a problem.
>
>
> It seems user agents are addressing this issue in general by simply removing
> support for those servers so we might not have to define anything here and
> just leave it to the TLS standards:
>
> http://my.opera.com/securitygroup/blog/2011/12/11/opera-11-60-and-new-problems-with-some-secure-servers

Sorry, I forgot to mention the 1/n+1 splitting countermeasure in my response.

With that said, this isn't TLS 1.1, but rather a specific, more
backwards-compatible
countermeasure. It's fine for the security considerations section to say here
that browsers must do either TLS 1.1 or 1/n+1 splitting, but it should say
something, since it's not like 1/n+1 splitting is required by TLS (any version).

-Ekr

Re: [XHR] chunked requests

[Adam Barth]

On Sat, Dec 17, 2011 at 6:11 AM, Anne van Kesteren  wrote:
> On Fri, 09 Dec 2011 19:54:31 +0100, Eric Rescorla  wrote:
>>
>> Unfortunately, many servers do not support TLS 1.1, and to make matters
>> worse, they do so in a way that is not securely verifiable. By which I
>> mean that an active attacker can force a client/server pair both of which
>> support TLS 1.1 down to TLS 1.0. This may be detectable in some way, but not
>> by TLS's built-in mechanisms. And since the threat model here is an active
>> attacker, this is a problem.
>
> It seems user agents are addressing this issue in general by simply removing
> support for those servers so we might not have to define anything here and
> just leave it to the TLS standards:
>
> http://my.opera.com/securitygroup/blog/2011/12/11/opera-11-60-and-new-problems-with-some-secure-servers

I would still add a security consideration so folks who implement this
are aware that the two issues are related.

Adam

Re: [XHR] chunked requests

[Anne van Kesteren]

On Fri, 09 Dec 2011 19:54:31 +0100, Eric Rescorla  wrote:

Unfortunately, many servers do not support TLS 1.1, and to make matters
worse, they do so in a way that is not securely verifiable. By which I  
mean that an active attacker can force a client/server pair both of  
which support TLS 1.1 down to TLS 1.0. This may be detectable in some  
way, but not
by TLS's built-in mechanisms. And since the threat model here is an  
active attacker, this is a problem.


It seems user agents are addressing this issue in general by simply  
removing support for those servers so we might not have to define anything  
here and just leave it to the TLS standards:


http://my.opera.com/securitygroup/blog/2011/12/11/opera-11-60-and-new-problems-with-some-secure-servers


--
Anne van Kesteren
http://annevankesteren.nl/

Re: [XHR] chunked requests

[Anne van Kesteren]

On Wed, 14 Dec 2011 20:17:01 +0100, Feras Moussa   
wrote:
We've only recently uploaded the draft for the Streams API and shared it  
with the Working Group [1], which also has the new location.


There is a conversation currently taking place in the media capture WG  
[2] discussing how to better align Stream and MediaStream. I think it's  
an interesting topic worth investigating, and there are a few ideas that  
have been proposed. We'll have to see where the conversation lands.


Cool! Let me know when I should start adding the relevant text to  
XMLHttpRequest.



--
Anne van Kesteren
http://annevankesteren.nl/

RE: [XHR] chunked requests

[Feras Moussa]

We've only recently uploaded the draft for the Streams API and shared it with 
the Working Group [1], which also has the new location.

There is a conversation currently taking place in the media capture WG [2] 
discussing how to better align Stream and MediaStream. I think it's an 
interesting topic worth investigating, and there are a few ideas that have been 
proposed. We'll have to see where the conversation lands.

Thanks,
Feras

[1] http://lists.w3.org/Archives/Public/public-webapps/2011OctDec/1494.html
[2] http://lists.w3.org/Archives/Public/public-media-capture/2011Dec/0037.html


> -Original Message-
> From: Anne van Kesteren [mailto:ann...@opera.com]
> Sent: Friday, December 09, 2011 5:15 AM
> To: Wenbo Zhu; Jonas Sicking; Robert O'Callahan; Feras Moussa
> Cc: WebApps WG
> Subject: Re: [XHR] chunked requests
> 
> On Thu, 08 Dec 2011 23:16:37 +0100, Jonas Sicking  wrote:
> > I think Microsoft's stream proposal would address this use case.
> 
> So that would be: http://html5labs.interoperabilitybridges.com/streamsapi/
> 
> How does that relate to the various APIs for streaming media?
> 
> (I added roc and Feras Moussa.)
> 
> 
> --
> Anne van Kesteren
> http://annevankesteren.nl/

Re: [XHR] chunked requests

[Wenbo Zhu]

On Thu, Dec 8, 2011 at 2:36 PM, Charles Pritchard  wrote:

>
>
>
>
> On Dec 8, 2011, at 1:04 PM, Wenbo Zhu  wrote:
>
>
>
> On Wed, Dec 7, 2011 at 6:04 PM, Charles Pritchard < 
> ch...@jumis.com> wrote:
>
>> **
>> I think the Web Sockets spec is intended for client to server sessions
>> like this.
>>
> WebSocket protocol isn't yet well-supported by proxies. Besides, all we
> need here is a throwaway RPC-like request/response, and it's a bit heavy
> duty to use WebSocket for this use case.
>
>
> Isn't a chunked stream the anti-thesis of RPC?
>
Streaming request or response does not imply bi-directional messaging, i.e.
the (non-RPC) semantics of WebSocket.


> It looks like you're acknowledging WebSockets works while asking about
> what to do with current web infrastructure. It seems to me that current
> infrastructure is -mostly- Flash based and assumes a level of control over
> input: one can't send arbitrary information as a developer.. It has to be
> directly from a webcam or similar media stream.
>
> Do you have specific examples of existing infrastructure you're hoping to
> support?
>
See below.


>
>
>
>>
>> As for the peer-to-peer communication, I imagine the WebRTC group is
>> where you'll see more activity on this issue.
>>
> WebRTC is mostly peer-to-peer UDP (e.g. RTP), and the client-to-client
> communication has nothing to do with either HTTP or WebSocket.
>
>
>
> And MediaStream/Stream, which has to so with your request.
>
Not sure about the status of this spec. My original email merely tries to
show the need to process/post data locally ...
http://developers.whatwg.org/video-conferencing-and-peer-to-peer-communication.html#video-conferencing-and-peer-to-peer-communication


>
>
> I don't know that arbitrary binary data is on their agenda -- I hope it is
>> -- for p2p communication.
>>
>>
>>
>> On 12/7/11 5:59 PM, Wenbo Zhu wrote:
>>
>> One use case that we have which is not currently handled by
>> XMLHttpRequest is incrementally sending data that takes a long time to
>> generate _from the client to the server_. For example, if we were to record
>> data from a microphone, we couldn't upload it in real time to the server
>> with the current API.
>>
>>  The MediaStreaming spec also mentioned several use cases which would
>> require streaming request data via an API:
>>  - Sending the locally-produced streams to remote peers and receiving
>> streams from remote peers.
>> - Sending arbitrary data to remote peers.
>>
>>
>> http://www.whatwg.org/specs/web-apps/current-work/multipage/video-conferencing-and-peer-to-peer-communication.html
>>
>>  - Wenbo
>>
>>
>>
>

Re: [XHR] chunked requests

[Eric Rescorla]

On Fri, Dec 9, 2011 at 10:37 AM, Adam Barth  wrote:
> On Fri, Dec 9, 2011 at 7:59 AM, Anne van Kesteren  wrote:
>> On Fri, 09 Dec 2011 16:33:08 +0100, Eric Rescorla  wrote:
>>> Same-origin requests should be OK because the JS would have access
>>> to the relevant sensitive data in any case.
>>
>> Okay, I guess we can make that difference.
>
> Correct me if I'm wrong, but I believe these issues are fixed in TLS
> 1.1.  Most user agents implement TLS 1.1 anyway, so this seems mostly
> like a requirement to put in the security considerations section.

Would that it were this easy.

Unfortunately, many servers do not support TLS 1.1, and to make matters
worse, they do so in a way that is not securely verifiable. By which I mean
that an active attacker can force a client/server pair both of which support
TLS 1.1 down to TLS 1.0. This may be detectable in some way, but not
by TLS's built-in mechanisms. And since the threat model here is an active
attacker, this is a problem.

-Ekr

Re: [XHR] chunked requests

[Adam Barth]

On Fri, Dec 9, 2011 at 7:59 AM, Anne van Kesteren  wrote:
> On Fri, 09 Dec 2011 16:33:08 +0100, Eric Rescorla  wrote:
>> On Fri, Dec 9, 2011 at 4:59 AM, Anne van Kesteren 
>> wrote:
>>> Are you saying that if responseType is set to "stream" and the server
>>> only supports TLS 1.0 the connection should fail, but if it is greater than
>>> that it is okay?
>>
>> I'm not sure I understand this feature well enough. The issue is streaming
>> content from the client, not from the server, and in particular the ability
>> of the JS to provide additional content to be sent after the data has
>> started to be transmitted.
>
> My bad. I meant send(Stream) which would indeed allow for that.
>
>> As for what should happen in this setting if the negotiated TLS version
>> is 1.0, I could imagine a number of possibilities, including:
>>
>> 1. The client refuses to send
>> 2. There is a pre-flight and the server has to explicitly accept.
>> 3. There is a big nasty warning.
>> 4. We just warn people in the spec and hope they do something sensible
>
> Okay. I think I would very much prefer 1 here.
>
>>> Same-origin requests are always okay? (Though it seems we should just
>>> require TLS 1.1 there too then to not make matters too confusing.)
>>
>>
>> Same-origin requests should be OK because the JS would have access
>> to the relevant sensitive data in any case.
>
> Okay, I guess we can make that difference.

Correct me if I'm wrong, but I believe these issues are fixed in TLS
1.1.  Most user agents implement TLS 1.1 anyway, so this seems mostly
like a requirement to put in the security considerations section.

Adam

Re: [XHR] chunked requests

[Anne van Kesteren]

On Fri, 09 Dec 2011 16:33:08 +0100, Eric Rescorla  wrote:
On Fri, Dec 9, 2011 at 4:59 AM, Anne van Kesteren   
wrote:
Are you saying that if responseType is set to "stream" and the server  
only supports TLS 1.0 the connection should fail, but if it is greater  
than that it is okay?


I'm not sure I understand this feature well enough. The issue is  
streaming content from the client, not from the server, and in  
particular the ability of the JS to provide additional content to be  
sent after the data has

started to be transmitted.


My bad. I meant send(Stream) which would indeed allow for that.



As for what should happen in this setting if the negotiated TLS version
is 1.0, I could imagine a number of possibilities, including:

1. The client refuses to send
2. There is a pre-flight and the server has to explicitly accept.
3. There is a big nasty warning.
4. We just warn people in the spec and hope they do something  
sensible


Okay. I think I would very much prefer 1 here.



Same-origin requests are always okay? (Though it seems we should just
require TLS 1.1 there too then to not make matters too confusing.)


Same-origin requests should be OK because the JS would have access
to the relevant sensitive data in any case.


Okay, I guess we can make that difference.

Thanks,


--
Anne van Kesteren
http://annevankesteren.nl/

Re: [XHR] chunked requests

[Eric Rescorla]

On Fri, Dec 9, 2011 at 4:59 AM, Anne van Kesteren  wrote:
> On Fri, 09 Dec 2011 02:13:50 +0100, Eric Rescorla  wrote:
>>
>> On Thu, Dec 8, 2011 at 5:07 PM, Adam Barth  wrote:
>>>
>>> Whatever spec we end up going with should note in its security
>>> consideration that the user agent must implement TLS 1.2 or greater to
>>> avoid this attack.
>>
>>
>> I believe it's actually TLS 1.1, since the relevant feature is
>> explicit IVs. Or you could allow RC4, I guess.
>
>
> Are you saying that if responseType is set to "stream" and the server only
> supports TLS 1.0 the connection should fail, but if it is greater than that
> it is okay?

I'm not sure I understand this feature well enough. The issue is streaming
content from the client, not from the server, and in particular the ability
of the JS to provide additional content to be sent after the data has
started to be transmitted.

As for what should happen in this setting if the negotiated TLS version
is 1.0, I could imagine a number of possibilities, including:

1. The client refuses to send
2. There is a pre-flight and the server has to explicitly accept.
3. There is a big nasty warning.
4. We just warn people in the spec and hope they do something sensible



> Same-origin requests are always okay? (Though it seems we should just
> require TLS 1.1 there too then to not make matters too confusing.)

Same-origin requests should be OK because the JS would have access
to the relevant sensitive data in any case.

-Ekr

Re: [XHR] chunked requests

[Anne van Kesteren]

On Thu, 08 Dec 2011 23:16:37 +0100, Jonas Sicking  wrote:

I think Microsoft's stream proposal would address this use case.


So that would be: http://html5labs.interoperabilitybridges.com/streamsapi/

How does that relate to the various APIs for streaming media?

(I added roc and Feras Moussa.)


--
Anne van Kesteren
http://annevankesteren.nl/

Re: [XHR] chunked requests

[Anne van Kesteren]

On Fri, 09 Dec 2011 02:13:50 +0100, Eric Rescorla  wrote:

On Thu, Dec 8, 2011 at 5:07 PM, Adam Barth  wrote:

Whatever spec we end up going with should note in its security
consideration that the user agent must implement TLS 1.2 or greater to
avoid this attack.


I believe it's actually TLS 1.1, since the relevant feature is
explicit IVs. Or you could allow RC4, I guess.


Are you saying that if responseType is set to "stream" and the server only  
supports TLS 1.0 the connection should fail, but if it is greater than  
that it is okay?


Same-origin requests are always okay? (Though it seems we should just  
require TLS 1.1 there too then to not make matters too confusing.)



--
Anne van Kesteren
http://annevankesteren.nl/

Re: [XHR] chunked requests

[Eric Rescorla]

On Thu, Dec 8, 2011 at 5:07 PM, Adam Barth  wrote:
> Keep in mind that streamed or chunked uploads will expose the ability
> to exploit the BEAST vulnerability in SSL:
>
> http://www.educatedguesswork.org/2011/09/security_impact_of_the_rizzodu.html

Right. Specifically, it needs to be a cross-origin streamed request without
significant uncontrollable headers and/or masking.


> Whatever spec we end up going with should note in its security
> consideration that the user agent must implement TLS 1.2 or greater to
> avoid this attack.

I believe it's actually TLS 1.1, since the relevant feature is
explicit IVs. Or you
could allow RC4, I guess.

Best,
-Ekr

Re: [XHR] chunked requests

[Adam Barth]

Keep in mind that streamed or chunked uploads will expose the ability
to exploit the BEAST vulnerability in SSL:

http://www.educatedguesswork.org/2011/09/security_impact_of_the_rizzodu.html

Whatever spec we end up going with should note in its security
consideration that the user agent must implement TLS 1.2 or greater to
avoid this attack.

Adam


On Thu, Dec 8, 2011 at 2:16 PM, Jonas Sicking  wrote:
> I think Microsoft's stream proposal would address this use case.
>
> / Jonas
>
> On Wed, Dec 7, 2011 at 5:59 PM, Wenbo Zhu  wrote:
>> One use case that we have which is not currently handled by XMLHttpRequest
>> is incrementally sending data that takes a long time to generate _from the
>> client to the server_. For example, if we were to record data from a
>> microphone, we couldn't upload it in real time to the server with the
>> current API.
>>
>> The MediaStreaming spec also mentioned several use cases which would require
>> streaming request data via an API:
>> - Sending the locally-produced streams to remote peers and receiving streams
>> from remote peers.
>> - Sending arbitrary data to remote peers.
>>
>> http://www.whatwg.org/specs/web-apps/current-work/multipage/video-conferencing-and-peer-to-peer-communication.html
>>
>> - Wenbo
>

Re: [XHR] chunked requests

[Charles Pritchard]

On Dec 8, 2011, at 1:04 PM, Wenbo Zhu  wrote:

> 
> 
> On Wed, Dec 7, 2011 at 6:04 PM, Charles Pritchard  wrote:
> I think the Web Sockets spec is intended for client to server sessions like 
> this.
> WebSocket protocol isn't yet well-supported by proxies. Besides, all we need 
> here is a throwaway RPC-like request/response, and it's a bit heavy duty to 
> use WebSocket for this use case. 
> 

Isn't a chunked stream the anti-thesis of RPC? It looks like you're 
acknowledging WebSockets works while asking about what to do with current web 
infrastructure. It seems to me that current infrastructure is -mostly- Flash 
based and assumes a level of control over input: one can't send arbitrary 
information as a developer.. It has to be directly from a webcam or similar 
media stream.

Do you have specific examples of existing infrastructure you're hoping to 
support?


>  
> 
> As for the peer-to-peer communication, I imagine the WebRTC group is where 
> you'll see more activity on this issue.
> WebRTC is mostly peer-to-peer UDP (e.g. RTP), and the client-to-client 
> communication has nothing to do with either HTTP or WebSocket. 


And MediaStream/Stream, which has to so with your request.


> 
> I don't know that arbitrary binary data is on their agenda -- I hope it is -- 
> for p2p communication.
> 
> 
> 
> On 12/7/11 5:59 PM, Wenbo Zhu wrote:
>> 
>> One use case that we have which is not currently handled by XMLHttpRequest 
>> is incrementally sending data that takes a long time to generate _from the 
>> client to the server_. For example, if we were to record data from a 
>> microphone, we couldn't upload it in real time to the server with the 
>> current API.
>> 
>> The MediaStreaming spec also mentioned several use cases which would require 
>> streaming request data via an API:
>> - Sending the locally-produced streams to remote peers and receiving streams 
>> from remote peers.
>> - Sending arbitrary data to remote peers.
>> 
>> http://www.whatwg.org/specs/web-apps/current-work/multipage/video-conferencing-and-peer-to-peer-communication.html
>> 
>> - Wenbo
> 
> 

Re: [XHR] chunked requests

[Jonas Sicking]

I think Microsoft's stream proposal would address this use case.

/ Jonas

On Wed, Dec 7, 2011 at 5:59 PM, Wenbo Zhu  wrote:
> One use case that we have which is not currently handled by XMLHttpRequest
> is incrementally sending data that takes a long time to generate _from the
> client to the server_. For example, if we were to record data from a
> microphone, we couldn't upload it in real time to the server with the
> current API.
>
> The MediaStreaming spec also mentioned several use cases which would require
> streaming request data via an API:
> - Sending the locally-produced streams to remote peers and receiving streams
> from remote peers.
> - Sending arbitrary data to remote peers.
>
> http://www.whatwg.org/specs/web-apps/current-work/multipage/video-conferencing-and-peer-to-peer-communication.html
>
> - Wenbo

Re: [XHR] chunked requests

[Wenbo Zhu]

On Wed, Dec 7, 2011 at 6:04 PM, Charles Pritchard  wrote:

> **
> I think the Web Sockets spec is intended for client to server sessions
> like this.
>
WebSocket protocol isn't yet well-supported by proxies. Besides, all we
need here is a throwaway RPC-like request/response, and it's a bit heavy
duty to use WebSocket for this use case.



>
> As for the peer-to-peer communication, I imagine the WebRTC group is where
> you'll see more activity on this issue.
>
WebRTC is mostly peer-to-peer UDP (e.g. RTP), and the client-to-client
communication has nothing to do with either HTTP or WebSocket.

I don't know that arbitrary binary data is on their agenda -- I hope it is
> -- for p2p communication.
>
>
>
> On 12/7/11 5:59 PM, Wenbo Zhu wrote:
>
> One use case that we have which is not currently handled by XMLHttpRequest
> is incrementally sending data that takes a long time to generate _from the
> client to the server_. For example, if we were to record data from a
> microphone, we couldn't upload it in real time to the server with the
> current API.
>
>  The MediaStreaming spec also mentioned several use cases which would
> require streaming request data via an API:
>  - Sending the locally-produced streams to remote peers and receiving
> streams from remote peers.
> - Sending arbitrary data to remote peers.
>
>
> http://www.whatwg.org/specs/web-apps/current-work/multipage/video-conferencing-and-peer-to-peer-communication.html
>
>  - Wenbo
>
>
>

[XHR] chunked requests

[Wenbo Zhu]

One use case that we have which is not currently handled by XMLHttpRequest
is incrementally sending data that takes a long time to generate _from the
client to the server_. For example, if we were to record data from a
microphone, we couldn't upload it in real time to the server with the
current API.

The MediaStreaming spec also mentioned several use cases which would
require streaming request data via an API:
- Sending the locally-produced streams to remote peers and receiving
streams from remote peers.
- Sending arbitrary data to remote peers.

http://www.whatwg.org/specs/web-apps/current-work/multipage/video-conferencing-and-peer-to-peer-communication.html

- Wenbo

Re: [XHR] chunked

[Wenbo Zhu]

On Wed, Nov 30, 2011 at 7:28 AM, Anne van Kesteren  wrote:

> A while ago sicking proposed adding chunked support to XMLHttpRequest:
>
> http://lists.w3.org/Archives/**Public/public-webapps/**
> 2011JulSep/0741.html
> https://bugzilla.mozilla.org/**show_bug.cgi?id=687087
>
> A use case I remember was downloading a large file of some kind that
> presumably can be incrementally rendered as otherwise responseType "blob"
> should be sufficient. More use cases appreciated. Would help with the
> design.
>
E.g. voice/image search, translation ... IMHO any single resource that
involves non-trivial processing to produce would fit the use case.


>
> As for the feature, basically have responseType "chunked-text" and
> "chunked-arraybuffer" values and reset rather than update the response
> entity body with each progress event. And make sure that a progress event
> is dispatched when the last fetch event is queued. And make sure that this
> is only available for asynchronous usage.
>
> Charles asked whether "chunked-text" was really needed (and whether we
> should have "chunked" which implies ArrayBuffer instead). Nobody got back
> to him on that. If it is needed, how does it work when you just have some
> of the bytes of a multi-byte character in a single chunk? Fails to decode
> as per the normal algorithm?
>
When text is consumed as chunked streams, my take is that the application
code has to deal with partial frames, and partial chars are just one
sub-problem. So, I wouldn't consider multi-byte characters a particular
limitation.


>
> Also, this basically makes it possible to write EventSource on top of
> XMLHttpRequest. Is that acceptable? If it encourages more people to use a
> lower-level API, higher-level optimizations for mobile phones might become
> harder down the road.
>
At the same time, lower-level APIs that match the underlying wire-protocol
(i.e. HTTP) would be equally important for optimization purposes.

Thanks,
Wenbo

>
>
> --
> Anne van Kesteren
> http://annevankesteren.nl/
>
>

Re: [XHR] chunked

[Jonas Sicking]

On Wed, Nov 30, 2011 at 6:16 PM, Charles Pritchard  wrote:
> On 11/30/11 6:04 PM, Glenn Maynard wrote:
>
> On Wed, Nov 30, 2011 at 8:45 PM, Charles Pritchard  wrote:
>>
>> I don't worry about character sets at all. I treat the content as opaque.
>
>
> It doesn't sound like you're describing text processing at all, but binary
> processing of data that happens to contain text.
>
>
> Right, for chunked encoding, I start by treating it as binary data, and go
> from there.
> So, I'm not a good source for the usefulness of chunked-text.
>
> When I want text, I convert, but my default is more often to work with
> buffers.
>
> I think Jonas made a good point about multibyte boundaries in text. That's
> not something that can be solved by a simple method:
> str = UintToString(IntArrayBufferChunk);
>
> But it can be solved simply by delegating it to XHR and a chunked-type
> response.
>
> My only requirement is for chunked-arraybuffer, but I didn't mean to stand
> in the way of a chunked-text type.

The part of the original email that you responded to was specifically
talking about weather we should have chunked-text *as well as*
chunked-arraybuffer. So it seems like this sub-thread was off-topic to
that question.

/ Jonas

Re: [XHR] chunked

[Charles Pritchard]

On 11/30/11 6:04 PM, Glenn Maynard wrote:
On Wed, Nov 30, 2011 at 8:45 PM, Charles Pritchard > wrote:


I don't worry about character sets at all. I treat the content as
opaque. 



It doesn't sound like you're describing text processing at all, but 
binary processing of data that happens to contain text.


Right, for chunked encoding, I start by treating it as binary data, and 
go from there.

So, I'm not a good source for the usefulness of chunked-text.

When I want text, I convert, but my default is more often to work with 
buffers.


I think Jonas made a good point about multibyte boundaries in text. 
That's not something that can be solved by a simple method:

str = UintToString(IntArrayBufferChunk);

But it can be solved simply by delegating it to XHR and a chunked-type 
response.


My only requirement is for chunked-arraybuffer, but I didn't mean to 
stand in the way of a chunked-text type.


-Charles

Re: [XHR] chunked

[Glenn Maynard]

On Wed, Nov 30, 2011 at 8:45 PM, Charles Pritchard  wrote:

> I don't worry about character sets at all. I treat the content as opaque.


It doesn't sound like you're describing text processing at all, but binary
processing of data that happens to contain text.

-- 
Glenn Maynard

Re: [XHR] chunked

[Charles Pritchard]

On 11/30/11 3:44 PM, Jonas Sicking wrote:

On Wed, Nov 30, 2011 at 1:03 PM, Charles Pritchard  wrote:

On Nov 30, 2011, at 12:41 PM, Jonas Sicking  wrote:


On Wed, Nov 30, 2011 at 12:13 PM, Charles Pritchard  wrote:

On Nov 30, 2011, at 11:32 AM, Jonas Sicking  wrote:


Charles asked whether "chunked-text" was really needed (and whether we
should have "chunked" which implies ArrayBuffer instead). Nobody got back to
him on that.

Any text based format would benefit from chunked-text. While the
example above uses a binary format, it applies equally to text based
formats. And given how much we in this group seem to prefer text based
formats, (HTML, CSS, Javascript, EventSource, JSON) I think we should
assume that other people at least use them, if not prefer them.

My thinking was that ArrayBuffer can easily be converted to String by authors. 
Even with text-based formats, I prefer to fetch data as blob and buffer.

Why?

Because it helps with general methods to pass buffers around, and stream 
processing is buffer based. I don't need to worry about character sets or 
errant binary data. It's raw and easy to port, being a Transferable object.

Why couldn't you use strings instead of buffers when dealing with
textual data? And why don't you need need to worry less about


I can, it just requires extra calls to and from String.fromCharCode and 
charCodeAt. Using buffers is going to be faster, of course, as they are 
based on Typed Arrays, whenever I'm doing heavy computation. I do use 
strings. But for heavy processing and abstracted methods, I use buffers 
and typed arrays.



characters sets if you are doing the charset conversion rather than
the UA? I would imagine you'd have to worry more about character sets
if you take on that burden rather than let the UA do it.


I don't worry about character sets at all. I treat the content as 
opaque. If there's an error in the character set, that's up to some 
other part of the process to figure out. I treat the data as fairly 
opaque. I rarely am trying to sanitize or validate data. I'm just 
looking for particular values in it.



I can see the argument of being a Transferable object. But why
transfer the chunked data between threads? Just do the load on the
thread that is going to interpret the data.


That may often be the case, but when I transfer processed data back to 
the main thread, I still may want it as a buffer.
I've worked a lot on string processing, in old JS and JS with typed 
arrays. I've certainly ridden the ins and outs and use all methods 
available.



Many of my encoding and decoding methods expect byte arrays.

And these methods are dealing with textual data? Note that no-one is
proposing the ability to do chunked-arraybuffer. The question is if we
should have chunked-text as well.


It's a good question, I don't have an answer. I'd sure like an 
ArrayBuffer to string method. Converting to a Blob then using 
FileReader, is bit of extra work, as is running fromCharCode over the 
array to build a string.  A conversion method would make 
chunked-arraybuffer quite easy to convert into a string...


Your suggestion that chunked-text would always return complete 
multi-byte codes was pretty good, as is the notion that whatever 
chunked-text returns is DOMString-safe, so it'd always work with 
something like localStorage.setItem. And those are perhaps features that 
a conversion method would not handle as well,.


Sorry if I created confusion or chaos here. Just reporting on what I 
tend to do.



-Charles

Re: [XHR] chunked

[Jonas Sicking]

On Wed, Nov 30, 2011 at 1:03 PM, Charles Pritchard  wrote:
> On Nov 30, 2011, at 12:41 PM, Jonas Sicking  wrote:
>
>> On Wed, Nov 30, 2011 at 12:13 PM, Charles Pritchard  wrote:
>>> On Nov 30, 2011, at 11:32 AM, Jonas Sicking  wrote:
>>>
> Charles asked whether "chunked-text" was really needed (and whether we
> should have "chunked" which implies ArrayBuffer instead). Nobody got back 
> to
> him on that.

 Any text based format would benefit from chunked-text. While the
 example above uses a binary format, it applies equally to text based
 formats. And given how much we in this group seem to prefer text based
 formats, (HTML, CSS, Javascript, EventSource, JSON) I think we should
 assume that other people at least use them, if not prefer them.
>>>
>>> My thinking was that ArrayBuffer can easily be converted to String by 
>>> authors. Even with text-based formats, I prefer to fetch data as blob and 
>>> buffer.
>>
>> Why?
>
> Because it helps with general methods to pass buffers around, and stream 
> processing is buffer based. I don't need to worry about character sets or 
> errant binary data. It's raw and easy to port, being a Transferable object.

Why couldn't you use strings instead of buffers when dealing with
textual data? And why don't you need need to worry less about
characters sets if you are doing the charset conversion rather than
the UA? I would imagine you'd have to worry more about character sets
if you take on that burden rather than let the UA do it.

I can see the argument of being a Transferable object. But why
transfer the chunked data between threads? Just do the load on the
thread that is going to interpret the data.

> Many of my encoding and decoding methods expect byte arrays.

And these methods are dealing with textual data? Note that no-one is
proposing the ability to do chunked-arraybuffer. The question is if we
should have chunked-text as well.

/ Jonas

Re: [XHR] chunked

[Charles Pritchard]

On Nov 30, 2011, at 12:41 PM, Jonas Sicking  wrote:

> On Wed, Nov 30, 2011 at 12:13 PM, Charles Pritchard  wrote:
>> On Nov 30, 2011, at 11:32 AM, Jonas Sicking  wrote:
>> 
 Charles asked whether "chunked-text" was really needed (and whether we
 should have "chunked" which implies ArrayBuffer instead). Nobody got back 
 to
 him on that.
>>> 
>>> Any text based format would benefit from chunked-text. While the
>>> example above uses a binary format, it applies equally to text based
>>> formats. And given how much we in this group seem to prefer text based
>>> formats, (HTML, CSS, Javascript, EventSource, JSON) I think we should
>>> assume that other people at least use them, if not prefer them.
>> 
>> My thinking was that ArrayBuffer can easily be converted to String by 
>> authors. Even with text-based formats, I prefer to fetch data as blob and 
>> buffer.
> 
> Why?

Because it helps with general methods to pass buffers around, and stream 
processing is buffer based. I don't need to worry about character sets or 
errant binary data. It's raw and easy to port, being a Transferable object.

Many of my encoding and decoding methods expect byte arrays.

Re: [XHR] chunked

[Jonas Sicking]

On Wed, Nov 30, 2011 at 12:13 PM, Charles Pritchard  wrote:
> On Nov 30, 2011, at 11:32 AM, Jonas Sicking  wrote:
>
>>> Charles asked whether "chunked-text" was really needed (and whether we
>>> should have "chunked" which implies ArrayBuffer instead). Nobody got back to
>>> him on that.
>>
>> Any text based format would benefit from chunked-text. While the
>> example above uses a binary format, it applies equally to text based
>> formats. And given how much we in this group seem to prefer text based
>> formats, (HTML, CSS, Javascript, EventSource, JSON) I think we should
>> assume that other people at least use them, if not prefer them.
>
> My thinking was that ArrayBuffer can easily be converted to String by 
> authors. Even with text-based formats, I prefer to fetch data as blob and 
> buffer.

Why?

/ Jonas

Re: [XHR] chunked

[Charles Pritchard]

On Nov 30, 2011, at 11:32 AM, Jonas Sicking  wrote:

>> Charles asked whether "chunked-text" was really needed (and whether we
>> should have "chunked" which implies ArrayBuffer instead). Nobody got back to
>> him on that.
> 
> Any text based format would benefit from chunked-text. While the
> example above uses a binary format, it applies equally to text based
> formats. And given how much we in this group seem to prefer text based
> formats, (HTML, CSS, Javascript, EventSource, JSON) I think we should
> assume that other people at least use them, if not prefer them.

My thinking was that ArrayBuffer can easily be converted to String by authors. 
Even with text-based formats, I prefer to fetch data as blob and buffer.


> 
>> If it is needed, how does it work when you just have some of
>> the bytes of a multi-byte character in a single chunk? Fails to decode as
>> per the normal algorithm?
> 
> Any multi-byte characters which are only partially downloaded are not
> returned in that "chunk". I.e. you only expose fully decoded
> characters. The value in progressEvent.loaded doesn't match up with
> the accumulated xhr.response.length for multi-byte encodings anyway.


This is an interesting case and may be a good reason to have a chunked text 
type.

When I'm parsing streams, I'm going to be waiting on various delimiters, and 
appending unprocessed bits... So the multibyte issue won't be an issue for me. 
Consider in any XML format, the developer is going to be seeking closing tags, 
or such.

But, getting the behavior you're looking for, where the callback is ensured 
that the string has full multibyte characters, that's not particularly easy for 
developers. While I wouldn't use it, as I'd use chunked arraybuffer, I can see 
how it'd be useful to achieve those parameters. There are various APIs that 
expect valid DOMString.

Re: [XHR] chunked

[Jonas Sicking]

On Wed, Nov 30, 2011 at 7:28 AM, Anne van Kesteren  wrote:
> A while ago sicking proposed adding chunked support to XMLHttpRequest:
>
> http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/0741.html
> https://bugzilla.mozilla.org/show_bug.cgi?id=687087
>
> A use case I remember was downloading a large file of some kind that
> presumably can be incrementally rendered as otherwise responseType "blob"
> should be sufficient. More use cases appreciated. Would help with the
> design.

Incremental rendering as well as incremental parsing. Imagine for
example a webapp like google docs but with the ability to read MS-Word
documents directly client-side (currently I believe google docs does
their MS-Word file parsing server-side).

Even if the application wouldn't want to render the document right
away, parsing it incrementally rather than after the full download has
several advantages. First of all, external resources referenced from
the document can be loaded immediately and in parallel with the main
document. Second not having to keep the unparsed data of the full
document can save a bunch of memory. Third, it means that the document
is available for use sooner after the download is finished since most
if it is already parsed.

> Charles asked whether "chunked-text" was really needed (and whether we
> should have "chunked" which implies ArrayBuffer instead). Nobody got back to
> him on that.

Any text based format would benefit from chunked-text. While the
example above uses a binary format, it applies equally to text based
formats. And given how much we in this group seem to prefer text based
formats, (HTML, CSS, Javascript, EventSource, JSON) I think we should
assume that other people at least use them, if not prefer them.

> If it is needed, how does it work when you just have some of
> the bytes of a multi-byte character in a single chunk? Fails to decode as
> per the normal algorithm?

Any multi-byte characters which are only partially downloaded are not
returned in that "chunk". I.e. you only expose fully decoded
characters. The value in progressEvent.loaded doesn't match up with
the accumulated xhr.response.length for multi-byte encodings anyway.

> Also, this basically makes it possible to write EventSource on top of
> XMLHttpRequest. Is that acceptable? If it encourages more people to use a
> lower-level API, higher-level optimizations for mobile phones might become
> harder down the road.

It seems more than acceptable to me! It's a good sign if our low-level
primitives, like XHR, are extended to the point that they can be used
to implement our higher level APIs. That shows that we're building our
low-level primitives correctly.

And note that EventSource is already implementable on top of XHR. The
only thing that "chunked" is buying us is less memory usage.

But I also think that EventSource still has some significant
advantages in that it supplies things like automatic reconnection and
possibly in the future events delivered over SMS or other "push"
mechanisms.

/ Jonas

[XHR] chunked

[Anne van Kesteren]

A while ago sicking proposed adding chunked support to XMLHttpRequest:

http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/0741.html
https://bugzilla.mozilla.org/show_bug.cgi?id=687087

A use case I remember was downloading a large file of some kind that  
presumably can be incrementally rendered as otherwise responseType "blob"  
should be sufficient. More use cases appreciated. Would help with the  
design.


As for the feature, basically have responseType "chunked-text" and  
"chunked-arraybuffer" values and reset rather than update the response  
entity body with each progress event. And make sure that a progress event  
is dispatched when the last fetch event is queued. And make sure that this  
is only available for asynchronous usage.


Charles asked whether "chunked-text" was really needed (and whether we  
should have "chunked" which implies ArrayBuffer instead). Nobody got back  
to him on that. If it is needed, how does it work when you just have some  
of the bytes of a multi-byte character in a single chunk? Fails to decode  
as per the normal algorithm?


Also, this basically makes it possible to write EventSource on top of  
XMLHttpRequest. Is that acceptable? If it encourages more people to use a  
lower-level API, higher-level optimizations for mobile phones might become  
harder down the road.



--
Anne van Kesteren
http://annevankesteren.nl/