Re: [whatwg] canvas, img, file api and blobs

[Stefan Haustein]

On Wed, Feb 17, 2010 at 1:04 PM, Stefan Haustein wrote:

> On Wed, Feb 17, 2010 at 8:35 AM, Stef Epardaud  wrote:
>
>> On Tue, Feb 16, 2010 at 07:25:34PM +, Stefan Haustein wrote:
>> >  We've been getting pretty good traction on Vlad's ArrayBuffers
>> proposal,
>> >  which was taken from the WebGL spec. Our current plan is to change
>> the
>> >  names in the browsers (WebKit, Chrome and Mozilla) to the
>> "non-WebGL
>> >  specific" names Vlad proposes in his spec. We'd really like this to
>> be the
>> >  "one true binary data access" mechanism for HTML. We're talking to
>> the
>> >  File API guys about this and I think this API can be adapted in all
>> the
>> >  other places as well.
>> >  As far as performance goes, can you point me at some quantitative
>> data?
>> >  When you say it's an "orders-of-magnitude" bottleneck, what are you
>> >  comparing it to? The API is very new and we certainly want to
>> improve it
>> >  for the various purposes it can be put to. We've even talked about
>> >  optimizations inside the JS implementations to improve access
>> performance.
>>
>> If we can get something akin to Java's System.arraycopy (
>>
>> http://java.sun.com/j2se/1.4.2/docs/api/java/lang/System.html#arraycopy%28java.lang.Object,%20int,%20java.lang.Object,%20int,%20int%29
>> ) then the ArrayBuffer proposal would work for me :)
>>
>> If we cannot copy ArrayBuffer ranges by blocks in an effecient manner,
>> then it's going to be very limiting.
>>
>
> The array based set method would let you do this:
>
> function arrayCopy(src, spos, dst, dpos, len) {
>   dst.set(src.slice(spos, len), dpos);
> }
>
> If I understand Vladimir's response correctly, its omission from his
> ECAMScript proposal is unintentional (it is present in the WebGL spec) and
> will be fixed.
>
> Stefan
>

p.s. This would probably also address some of the reference vs. copy issues
raised in section 5 (
http://people.mozilla.com/~vladimir/jsvec/TypedArray-spec.html#5 )


>
>
>
>
>
>> --
>> Stéphane Epardaud
>>
>
>
>
> --
> Stefan Haustein
> Google UK Limited
>
> Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W
> 9TQ; Registered in England Number: 3977902
>
>


-- 
Stefan Haustein
Google UK Limited

Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W
9TQ; Registered in England Number: 3977902

Re: [whatwg] canvas, img, file api and blobs

[Stefan Haustein]

On Wed, Feb 17, 2010 at 8:35 AM, Stef Epardaud  wrote:

> On Tue, Feb 16, 2010 at 07:25:34PM +, Stefan Haustein wrote:
> >  We've been getting pretty good traction on Vlad's ArrayBuffers
> proposal,
> >  which was taken from the WebGL spec. Our current plan is to change
> the
> >  names in the browsers (WebKit, Chrome and Mozilla) to the "non-WebGL
> >  specific" names Vlad proposes in his spec. We'd really like this to
> be the
> >  "one true binary data access" mechanism for HTML. We're talking to
> the
> >  File API guys about this and I think this API can be adapted in all
> the
> >  other places as well.
> >  As far as performance goes, can you point me at some quantitative
> data?
> >  When you say it's an "orders-of-magnitude" bottleneck, what are you
> >  comparing it to? The API is very new and we certainly want to
> improve it
> >  for the various purposes it can be put to. We've even talked about
> >  optimizations inside the JS implementations to improve access
> performance.
>
> If we can get something akin to Java's System.arraycopy (
>
> http://java.sun.com/j2se/1.4.2/docs/api/java/lang/System.html#arraycopy%28java.lang.Object,%20int,%20java.lang.Object,%20int,%20int%29
> ) then the ArrayBuffer proposal would work for me :)
>
> If we cannot copy ArrayBuffer ranges by blocks in an effecient manner,
> then it's going to be very limiting.
>

The array based set method would let you do this:

function arrayCopy(src, spos, dst, dpos, len) {
  dst.set(src.slice(spos, len), dpos);
}

If I understand Vladimir's response correctly, its omission from his
ECAMScript proposal is unintentional (it is present in the WebGL spec) and
will be fixed.

Stefan





> --
> Stéphane Epardaud
>



-- 
Stefan Haustein
Google UK Limited

Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W
9TQ; Registered in England Number: 3977902

Re: [whatwg] canvas, img, file api and blobs

[Stef Epardaud]

On Tue, Feb 16, 2010 at 07:25:34PM +, Stefan Haustein wrote:
>  We've been getting pretty good traction on Vlad's ArrayBuffers proposal,
>  which was taken from the WebGL spec. Our current plan is to change the
>  names in the browsers (WebKit, Chrome and Mozilla) to the "non-WebGL
>  specific" names Vlad proposes in his spec. We'd really like this to be 
> the
>  "one true binary data access" mechanism for HTML. We're talking to the
>  File API guys about this and I think this API can be adapted in all the
>  other places as well.
>  As far as performance goes, can you point me at some quantitative data?
>  When you say it's an "orders-of-magnitude" bottleneck, what are you
>  comparing it to? The API is very new and we certainly want to improve it
>  for the various purposes it can be put to. We've even talked about
>  optimizations inside the JS implementations to improve access 
> performance.

If we can get something akin to Java's System.arraycopy (
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/System.html#arraycopy%28java.lang.Object,%20int,%20java.lang.Object,%20int,%20int%29
) then the ArrayBuffer proposal would work for me :)

If we cannot copy ArrayBuffer ranges by blocks in an effecient manner,
then it's going to be very limiting.
-- 
Stéphane Epardaud

Re: [whatwg] canvas, img, file api and blobs

[Lino Mastrodomenico]

2010/2/16 Joel Webber :
> On Tue, Feb 16, 2010 at 2:25 PM, Stefan Haustein 
> wrote:
>>
>> Constructing ints / longs from bytes in Javascript will probably be slow,
>> so in addition to the typed arrays, we'd love to have some kind of access
>> that would be similar to Java's DataInput (+DataOutput, see
>> http://java.sun.com/j2se/1.4.2/docs/api/java/io/DataInput.html ), but with
>> endianess support
>
> Agreed with Stefan's point, although technically you could implement
> getByte(), getShort(), et al with a bunch of TypedArray views onto a single
> ArrayBuffer. But this is really, really ugly, because in the general case
> you'd need 4 ByteArrays, 2 ShortArrays, etc. to deal with arbitrary offsets.

A mini language that describes arbitrary sequences of data types in
binary data is the one used by the Python module "struct"
().

E.g. "< 3b x 2i" represents 3 signed chars, one byte of padding and 2
signed 32-bit ints, little-endian.

It can be an inspiration for functions or methods that extract data
out of byte arrays and convert JS variables back to binary.

Another thing that people will probably ask is the possibility of
converting binary data to and from strings. Obviously for both
directions the user has to provide an encoding (e.g. UTF-8).

-- 
Lino Mastrodomenico

Re: [whatwg] canvas, img, file api and blobs

[Joel Webber]

On Tue, Feb 16, 2010 at 2:25 PM, Stefan Haustein wrote:

> On Tue, Feb 16, 2010 at 6:22 PM, Chris Marrin  wrote:
>
>> We've been getting pretty good traction on Vlad's ArrayBuffers proposal,
>> which was taken from the WebGL spec. Our current plan is to change the names
>> in the browsers (WebKit, Chrome and Mozilla) to the "non-WebGL specific"
>> names Vlad proposes in his spec. We'd really like this to be the "one true
>> binary data access" mechanism for HTML. We're talking to the File API guys
>> about this and I think this API can be adapted in all the other places as
>> well.
>>
>> As far as performance goes, can you point me at some quantitative data?
>> When you say it's an "orders-of-magnitude" bottleneck, what are you
>> comparing it to? The API is very new and we certainly want to improve it for
>> the various purposes it can be put to. We've even talked about optimizations
>> inside the JS implementations to improve access performance.
>>
>
> If we can get a webgl buffer from an XHR response (which would be a *huge*
> improvement), we'd still need to parse the binary data when decoding JPEG
> headers, protocol buffers etc.
>
> Constructing ints / longs from bytes in Javascript will probably be slow,
> so in addition to the typed arrays, we'd love to have some kind of access
> that would be similar to Java's DataInput (+DataOutput, see
> http://java.sun.com/j2se/1.4.2/docs/api/java/io/DataInput.html ), but with
> endianess support
>

Agreed with Stefan's point, although technically you could implement
getByte(), getShort(), et al with a bunch of TypedArray views onto a single
ArrayBuffer. But this is really, really ugly, because in the general case
you'd need 4 ByteArrays, 2 ShortArrays, etc. to deal with arbitrary offsets.
It would be much cleaner (and probably a bit more efficient) to implement
these getters directly in C++. This is pretty closely analogous to Java's
nio.ByteBuffer interface, though I think it would be better to hoist it out
into a separate interface.

As for quantitative data, I'm working on getting some together right now.
What I'd like to do (roughly) is to define a mesh format that looks very
roughly like this:

Mesh {
  int nVerts;
  float[] verts; // nVerts * 3
  int nFaces;
  int[] triIndices; // Maybe in strips
  // etc for materials and the like
}

And implement the fastest Javascript/JSON/whatever implementation for
getting one over XHR and loading it into WebGL.

I then plan on doing the equivalent with TypedArrays coming directly from
XHR into WebGL without Javascript having to touch every element. I've hacked
the WebGLArrays into TypedArrays, and implemented { ByteBuffer
XMLHttpRequest.resultBuffer } in my local WebKit, which I can then use to
compare performance. I strongly suspect that the latter will be enormously
faster, but could be proven wrong.

Note that to get reasonable performance in js, without TypedArray, you
pretty much *have* to switch to a text format like JSON. You can implement
(and we have) parsing of an arbitrary binary data structure in js, but it's
just freakishly slow and memory-inefficient.

Re: [whatwg] canvas, img, file api and blobs

[Stefan Haustein]

On Tue, Feb 16, 2010 at 6:22 PM, Chris Marrin  wrote:

>
> On Feb 16, 2010, at 9:00 AM, Eric Carlson wrote:
>
> > Chris -
> >
> >   Welcome to the HTML5 WG email torrent ;-)
> >
> >   Here is a message that you might actually care to read.
> >
> > eric
> >
> >
> >
> > Begin forwarded message:
> >
> >> From: Joel Webber 
> >> Date: February 16, 2010 8:39:31 AM PST
> >> To: Stefan Haustein 
> >> Cc: Maciej Stachowiak , wha...@whatwg.org, Jonas Sicking
> , Stef Epardaud 
> >> Subject: Re: [whatwg] canvas, img, file api and blobs
> >>
> >> On Tue, Feb 16, 2010 at 7:38 AM, Stefan Haustein 
> wrote:
> >> On Tue, Feb 16, 2010 at 10:08 AM, Maciej Stachowiak 
> wrote:
> >>
> >> On Feb 16, 2010, at 12:13 AM, Jonas Sicking wrote:
> >>
> >>
> >> Absolutely! I definitely agree that we need a type like this. The
> >> sooner the better. On that note, do you know what the latest status is
> >> within ECMA on this? I know you made a proposal on the webapps list
> >> (or was it here?), did that go anywhere?
> >>
> >> I made my proposal to ECMA TC-39 (the relevant committee). I will try to
> polish it and request it as an agenda item for the next face-to-face (in
> March). Independently, WebGL's typed arrays have been proposed.
> >>
> >> Hi Maciej,
> >>
> >> do you have a link to your proposal?
> >>
> >> And in particular, does it bear any resemblance to the WebGLArray
> interfaces, as proposed in (
> http://people.mozilla.com/~vladimir/jsvec/TypedArray-spec.html)? I'm
> particularly concerned with the interfaces among all these different
> subsystems (WebGL, Canvas, XHR, File, etc., as being discussed on this
> thread) that want to operate on binary data.
> >>
> >> We've found that getting data from XHR to WebGL via WebGLArrays to be a
> huge (read: probably orders-of-magnitude) bottleneck; but being able to
> slice mesh and texture data out of arrays directly from XHR responses would
> completely fix this.
> >
>
> We've been getting pretty good traction on Vlad's ArrayBuffers proposal,
> which was taken from the WebGL spec. Our current plan is to change the names
> in the browsers (WebKit, Chrome and Mozilla) to the "non-WebGL specific"
> names Vlad proposes in his spec. We'd really like this to be the "one true
> binary data access" mechanism for HTML. We're talking to the File API guys
> about this and I think this API can be adapted in all the other places as
> well.
>
> As far as performance goes, can you point me at some quantitative data?
> When you say it's an "orders-of-magnitude" bottleneck, what are you
> comparing it to? The API is very new and we certainly want to improve it for
> the various purposes it can be put to. We've even talked about optimizations
> inside the JS implementations to improve access performance.
>

If we can get a webgl buffer from an XHR response (which would be a *huge*
improvement), we'd still need to parse the binary data when decoding JPEG
headers, protocol buffers etc.

Constructing ints / longs from bytes in Javascript will probably be slow, so
in addition to the typed arrays, we'd love to have some kind of access that
would be similar to Java's DataInput (+DataOutput, see
http://java.sun.com/j2se/1.4.2/docs/api/java/io/DataInput.html ), but with
endianess support

Stefan


>
> -
> ~Chris
> cmar...@apple.com
>
>
>
>
>


-- 
Stefan Haustein
Google UK Limited

Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W
9TQ; Registered in England Number: 3977902

Re: [whatwg] canvas, img, file api and blobs

[Chris Marrin]

On Feb 16, 2010, at 9:00 AM, Eric Carlson wrote:

> Chris -
> 
>   Welcome to the HTML5 WG email torrent ;-)
> 
>   Here is a message that you might actually care to read.
> 
> eric
> 
> 
> 
> Begin forwarded message:
> 
>> From: Joel Webber 
>> Date: February 16, 2010 8:39:31 AM PST
>> To: Stefan Haustein 
>> Cc: Maciej Stachowiak , wha...@whatwg.org, Jonas Sicking 
>> , Stef Epardaud 
>> Subject: Re: [whatwg] canvas, img, file api and blobs
>> 
>> On Tue, Feb 16, 2010 at 7:38 AM, Stefan Haustein  wrote:
>> On Tue, Feb 16, 2010 at 10:08 AM, Maciej Stachowiak  wrote:
>> 
>> On Feb 16, 2010, at 12:13 AM, Jonas Sicking wrote:
>> 
>> 
>> Absolutely! I definitely agree that we need a type like this. The
>> sooner the better. On that note, do you know what the latest status is
>> within ECMA on this? I know you made a proposal on the webapps list
>> (or was it here?), did that go anywhere?
>> 
>> I made my proposal to ECMA TC-39 (the relevant committee). I will try to 
>> polish it and request it as an agenda item for the next face-to-face (in 
>> March). Independently, WebGL's typed arrays have been proposed.
>> 
>> Hi Maciej,
>> 
>> do you have a link to your proposal?
>> 
>> And in particular, does it bear any resemblance to the WebGLArray 
>> interfaces, as proposed in 
>> (http://people.mozilla.com/~vladimir/jsvec/TypedArray-spec.html)? I'm 
>> particularly concerned with the interfaces among all these different 
>> subsystems (WebGL, Canvas, XHR, File, etc., as being discussed on this 
>> thread) that want to operate on binary data.
>> 
>> We've found that getting data from XHR to WebGL via WebGLArrays to be a huge 
>> (read: probably orders-of-magnitude) bottleneck; but being able to slice 
>> mesh and texture data out of arrays directly from XHR responses would 
>> completely fix this.
> 

We've been getting pretty good traction on Vlad's ArrayBuffers proposal, which 
was taken from the WebGL spec. Our current plan is to change the names in the 
browsers (WebKit, Chrome and Mozilla) to the "non-WebGL specific" names Vlad 
proposes in his spec. We'd really like this to be the "one true binary data 
access" mechanism for HTML. We're talking to the File API guys about this and I 
think this API can be adapted in all the other places as well.

As far as performance goes, can you point me at some quantitative data? When 
you say it's an "orders-of-magnitude" bottleneck, what are you comparing it to? 
The API is very new and we certainly want to improve it for the various 
purposes it can be put to. We've even talked about optimizations inside the JS 
implementations to improve access performance.

-
~Chris
cmar...@apple.com

Re: [whatwg] canvas, img, file api and blobs

[Stefan Haustein]

On Tue, Feb 16, 2010 at 5:09 PM, Stef Epardaud  wrote:

> On Tue, Feb 16, 2010 at 11:39:31AM -0500, Joel Webber wrote:
> >And in particular, does it bear any resemblance to the WebGLArray
> >interfaces, as proposed in
> >([3]http://people.mozilla.com/~vladimir/jsvec/TypedArray-spec.html)?
> I'm
> >particularly concerned with the interfaces among all these different
> >subsystems (WebGL, Canvas, XHR, File, etc., as being discussed on this
> >thread) that want to operate on binary data.
> >We've found that getting data from XHR to WebGL via WebGLArrays to be
> a huge
> >(read: probably orders-of-magnitude) bottleneck; but being able to
> slice
> >mesh and texture data out of arrays directly from XHR responses would
> >completely fix this.
>
> The TypedArray proposal seems to lack a way to efficiently copy ranges
> of bytes from one array to another. This is essential if we want to
> resize arrays (which is going to be needed if we want to add EXIF data
> to an image for example). Copying byte by byte would be unacceptable.
>
> It is mentioned in section 5, but a proposed solution is a constructor,
> which is a bit restrictive (how would we concatenate two byte arrays?)
>

The WebGL spec has a corresponding set method (see
https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/WebGL-spec.html#5.14.3.3):

set(array, offset)

Set multiple values, reading input values from the array. The optional
offset value indicates the index in the current array where values are
written. If omitted, it is assumed to be 0.

If the offset plus the length of the given array is out of range for the
current WebGLArray, an exception is raised.
Not sure why this was omitted in Vladimir's version...



> --
> Stéphane Epardaud
>



-- 
Stefan Haustein
Google UK Limited

Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W
9TQ; Registered in England Number: 3977902

Re: [whatwg] canvas, img, file api and blobs

[Stef Epardaud]

On Tue, Feb 16, 2010 at 11:39:31AM -0500, Joel Webber wrote:
>And in particular, does it bear any resemblance to the WebGLArray
>interfaces, as proposed in
>([3]http://people.mozilla.com/~vladimir/jsvec/TypedArray-spec.html)? I'm
>particularly concerned with the interfaces among all these different
>subsystems (WebGL, Canvas, XHR, File, etc., as being discussed on this
>thread) that want to operate on binary data.
>We've found that getting data from XHR to WebGL via WebGLArrays to be a 
> huge
>(read: probably orders-of-magnitude) bottleneck; but being able to slice
>mesh and texture data out of arrays directly from XHR responses would
>completely fix this.

The TypedArray proposal seems to lack a way to efficiently copy ranges
of bytes from one array to another. This is essential if we want to
resize arrays (which is going to be needed if we want to add EXIF data
to an image for example). Copying byte by byte would be unacceptable.

It is mentioned in section 5, but a proposed solution is a constructor,
which is a bit restrictive (how would we concatenate two byte arrays?)
-- 
Stéphane Epardaud

Re: [whatwg] canvas, img, file api and blobs

[Joel Webber]

On Tue, Feb 16, 2010 at 7:38 AM, Stefan Haustein wrote:

> On Tue, Feb 16, 2010 at 10:08 AM, Maciej Stachowiak  wrote:
>
>>
>> On Feb 16, 2010, at 12:13 AM, Jonas Sicking wrote:
>>
>>
>>> Absolutely! I definitely agree that we need a type like this. The
>>> sooner the better. On that note, do you know what the latest status is
>>> within ECMA on this? I know you made a proposal on the webapps list
>>> (or was it here?), did that go anywhere?
>>>
>>
>> I made my proposal to ECMA TC-39 (the relevant committee). I will try to
>> polish it and request it as an agenda item for the next face-to-face (in
>> March). Independently, WebGL's typed arrays have been proposed.
>
>
> Hi Maciej,
>
> do you have a link to your proposal?
>

And in particular, does it bear any resemblance to the WebGLArray
interfaces, as proposed in (
http://people.mozilla.com/~vladimir/jsvec/TypedArray-spec.html)? I'm
particularly concerned with the interfaces among all these different
subsystems (WebGL, Canvas, XHR, File, etc., as being discussed on this
thread) that want to operate on binary data.

We've found that getting data from XHR to WebGL via WebGLArrays to be a huge
(read: probably orders-of-magnitude) bottleneck; but being able to slice
mesh and texture data out of arrays directly from XHR responses would
completely fix this.

Re: [whatwg] canvas, img, file api and blobs

[Stef Epardaud]

On Tue, Feb 16, 2010 at 01:24:14PM +0100, Anne van Kesteren wrote:
> You need to use scripts anyway and at this experimental stage it is not  
> really clear whether your use case hits the 80/20 or is actually rather  
> uncommon and can be dealt with using a jQuery wrapper or some such.

Right, if this use-case is not common enough and it can be dealt with
using some JS, then I agree it doesn't _have_ to be in the spec.
-- 
Stéphane Epardaud

Re: [whatwg] canvas, img, file api and blobs

[Stefan Haustein]

On Tue, Feb 16, 2010 at 10:08 AM, Maciej Stachowiak  wrote:

>
> On Feb 16, 2010, at 12:13 AM, Jonas Sicking wrote:
>
>
>> Absolutely! I definitely agree that we need a type like this. The
>> sooner the better. On that note, do you know what the latest status is
>> within ECMA on this? I know you made a proposal on the webapps list
>> (or was it here?), did that go anywhere?
>>
>
> I made my proposal to ECMA TC-39 (the relevant committee). I will try to
> polish it and request it as an agenda item for the next face-to-face (in
> March). Independently, WebGL's typed arrays have been proposed.


Hi Maciej,

do you have a link to your proposal?

Stefan


>
>
>
>> However I suspect it'd be useful to be able to extract as Blob object
>> as well, for storing in places like databases or to extract a url.
>>
>
> If we end up with APIs in some places that only take Blobs as input, then
> we should probably have easy ways to make a Blob out of a binary data object
> or a string (like StringStreams in other languages).
>
> Regards,
> Maciej
>
>


-- 
Stefan Haustein
Google UK Limited

Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W
9TQ; Registered in England Number: 3977902

Re: [whatwg] canvas, img, file api and blobs

[Anne van Kesteren]

On Tue, 16 Feb 2010 13:17:55 +0100, Stef Epardaud  wrote:

This is all fine when you want to do AJAX, but unless HTML5 drops
regular non-AJAX  submission, I don't understand why we would
mandate that processed File submission be done only using AJAX.


You need to use scripts anyway and at this experimental stage it is not  
really clear whether your use case hits the 80/20 or is actually rather  
uncommon and can be dealt with using a jQuery wrapper or some such.



--
Anne van Kesteren
http://annevankesteren.nl/

Re: [whatwg] canvas, img, file api and blobs

[Stef Epardaud]

On Tue, Feb 16, 2010 at 04:10:38AM -0800, Jonas Sicking wrote:
> Ah, sorry, it is indeed the idea that you can get a pre-filled
> FormData from a . Something like:
> 
> formData = myForm.getFormData();
> 
> after which you can do:
> 
> formData.append("thefile", myProcessedFile);
> xhr.send(formData);
> 
> This isn't in any spec yet, so the exact syntax might change, but it
> won't get more complex than this.

Right, this is simpler, but I still need to:

- Clear the  selection I've replaced with processed
  files, so they don't get submitted as well.
- Allocate the XMLHttpRequest
- Handle the response
- Process any eventual redirects.

This is all fine when you want to do AJAX, but unless HTML5 drops
regular non-AJAX  submission, I don't understand why we would
mandate that processed File submission be done only using AJAX.
-- 
Stéphane Epardaud

Re: [whatwg] canvas, img, file api and blobs

[Jonas Sicking]

On Tue, Feb 16, 2010 at 4:01 AM, Stef Epardaud  wrote:
> On Tue, Feb 16, 2010 at 03:48:04AM -0800, Jonas Sicking wrote:
>> If you have the FormData object [1], then why is it simpler to use
>>  submission?
>>
>> [1] 
>> http://dev.w3.org/2006/webapi/XMLHttpRequest-2/Overview.html#the-formdata-interface
>
> Unless I'm missing the point, I find it simpler to use a regular 
> submission, which would use the processed files as well as other 
> elements rather than build the XMLHttpRequest, build the FormData (via
> the constructor, unless we can obtain it from the filled-in  but I
> didn't see any mention of FormData in the HTML5 spec), then send the
> request and deal with the answer.
>
> If I can turn the contents of a  into a File (possibly via Blob
> and/or Storage), then replace the 's list of files,
> then all the submitting is done by the browser.
>
> I think it is simpler than doing it AJAX, which is not to say that the
> more complex AJAX way isn't better, I think both should be possible.
> Also it would be consistent with the fact that every other  type
> can be set/filtered by JS after user input, and still rely on regular
>  submission.

Ah, sorry, it is indeed the idea that you can get a pre-filled
FormData from a . Something like:

formData = myForm.getFormData();

after which you can do:

formData.append("thefile", myProcessedFile);
xhr.send(formData);

This isn't in any spec yet, so the exact syntax might change, but it
won't get more complex than this.

/ Jonas

Re: [whatwg] canvas, img, file api and blobs

[Stef Epardaud]

On Tue, Feb 16, 2010 at 03:48:04AM -0800, Jonas Sicking wrote:
> If you have the FormData object [1], then why is it simpler to use
>  submission?
> 
> [1] 
> http://dev.w3.org/2006/webapi/XMLHttpRequest-2/Overview.html#the-formdata-interface

Unless I'm missing the point, I find it simpler to use a regular 
submission, which would use the processed files as well as other 
elements rather than build the XMLHttpRequest, build the FormData (via
the constructor, unless we can obtain it from the filled-in  but I
didn't see any mention of FormData in the HTML5 spec), then send the
request and deal with the answer.

If I can turn the contents of a  into a File (possibly via Blob
and/or Storage), then replace the 's list of files,
then all the submitting is done by the browser.

I think it is simpler than doing it AJAX, which is not to say that the
more complex AJAX way isn't better, I think both should be possible.
Also it would be consistent with the fact that every other  type
can be set/filtered by JS after user input, and still rely on regular
 submission.
-- 
Stéphane Epardaud

Re: [whatwg] canvas, img, file api and blobs

[Jonas Sicking]

On Tue, Feb 16, 2010 at 1:58 AM, Stef Epardaud  wrote:
> On Tue, Feb 16, 2010 at 01:47:04AM -0800, Jonas Sicking wrote:
>> This is something I've been thinking about too.
>>
>> First of all, what is the use case? Once we have the FormData object,
>> you will be able to submit processed files using XMLHttpRequest
>> easily. So why do you want to use s?
>>
>> Another problem is that if this is a displayed , then
>> the user could overwrite the processed file with a user-selected file
>> at any time. This seems undesirable.
>>
>> Trying to get a feel for what you're trying to build to make sure that
>> we have all use cases covered.
>
> The rationale is simple: a JS action can filter what the user types in a
> , and still allow the  to be sent regularly
> (non-AJAX). Why not the same for files? In my use-case I have two
> different options, sending processed files using XMLHttpRequest, or with
> a normal  submit. XMLHttpRequest gives me better progress
> notification (how I wish we could get progress notifications for a
> regular  when POSTing), but if the set of processed files is part
> of a  with more input elements (in the case of photos, comments,
> date, whatever), it is simpler to use the regular  submission, and
> just filter the selected Files as the user adds them to the  type=file>, process them, then replace them in the . If
> the user then changes his mind about the selected file list, hey it
> still works.

If you have the FormData object [1], then why is it simpler to use
 submission?

[1] 
http://dev.w3.org/2006/webapi/XMLHttpRequest-2/Overview.html#the-formdata-interface

/ Jonas

Re: [whatwg] canvas, img, file api and blobs

[Maciej Stachowiak]

On Feb 16, 2010, at 12:13 AM, Jonas Sicking wrote:



Absolutely! I definitely agree that we need a type like this. The
sooner the better. On that note, do you know what the latest status is
within ECMA on this? I know you made a proposal on the webapps list
(or was it here?), did that go anywhere?


I made my proposal to ECMA TC-39 (the relevant committee). I will try  
to polish it and request it as an agenda item for the next face-to- 
face (in March). Independently, WebGL's typed arrays have been proposed.




However I suspect it'd be useful to be able to extract as Blob object
as well, for storing in places like databases or to extract a url.


If we end up with APIs in some places that only take Blobs as input,  
then we should probably have easy ways to make a Blob out of a binary  
data object or a string (like StringStreams in other languages).


Regards,
Maciej

Re: [whatwg] canvas, img, file api and blobs

[Stef Epardaud]

On Tue, Feb 16, 2010 at 01:47:04AM -0800, Jonas Sicking wrote:
> This is something I've been thinking about too.
> 
> First of all, what is the use case? Once we have the FormData object,
> you will be able to submit processed files using XMLHttpRequest
> easily. So why do you want to use s?
> 
> Another problem is that if this is a displayed , then
> the user could overwrite the processed file with a user-selected file
> at any time. This seems undesirable.
> 
> Trying to get a feel for what you're trying to build to make sure that
> we have all use cases covered.

The rationale is simple: a JS action can filter what the user types in a
, and still allow the  to be sent regularly
(non-AJAX). Why not the same for files? In my use-case I have two
different options, sending processed files using XMLHttpRequest, or with
a normal  submit. XMLHttpRequest gives me better progress
notification (how I wish we could get progress notifications for a
regular  when POSTing), but if the set of processed files is part
of a  with more input elements (in the case of photos, comments,
date, whatever), it is simpler to use the regular  submission, and
just filter the selected Files as the user adds them to the , process them, then replace them in the . If
the user then changes his mind about the selected file list, hey it
still works.
-- 
Stéphane Epardaud

Re: [whatwg] canvas, img, file api and blobs

[Jonas Sicking]

On Tue, Feb 16, 2010 at 1:15 AM, Stef Epardaud  wrote:
> Now that I think about it, is it possible to replace a regular form's
> file input list (the list of File objects selected from an input of type
> "file" with multiple files enabled) with "processed" files? Like data
> coming out of a canvas, possibly by storing them locally (via storage)
> to get a File instance. Or are we then limited to sending those
> processed files via XmlHttpRequest?

This is something I've been thinking about too.

First of all, what is the use case? Once we have the FormData object,
you will be able to submit processed files using XMLHttpRequest
easily. So why do you want to use s?

Another problem is that if this is a displayed , then
the user could overwrite the processed file with a user-selected file
at any time. This seems undesirable.

Trying to get a feel for what you're trying to build to make sure that
we have all use cases covered.

/ Jonas

Re: [whatwg] canvas, img, file api and blobs

[Stef Epardaud]

On Tue, Feb 16, 2010 at 10:23:09AM +0100, Anne van Kesteren wrote:
> It is XMLHttpRequest and being the editor and all I can assure you it will  
> be modified the moment ECMA (or someone) gives us binary data in  
> ECMAScript. I've been wanting to add that as both a sending and receiving  
> type since before I started working on XMLHttpRequest Level 2.

Great!

Sorry about getting the XMLHttpRequest name wrong, I always have problems with
acronyms in CamelCase, especially here with two acronyms :)
-- 
Stéphane Epardaud

Re: [whatwg] canvas, img, file api and blobs

[Anne van Kesteren]

On Tue, 16 Feb 2010 10:15:03 +0100, Stef Epardaud  wrote:

I am not sure I like this "as well" ;) In my case I need to extract
binary data from a canvas, modify this binary data, then send it using
XmlHttpRequest (which only accepts a File or Blob IIRC), so this binary
type has to be accepted by XmlHttpRequest as well, for sending.


It is XMLHttpRequest and being the editor and all I can assure you it will  
be modified the moment ECMA (or someone) gives us binary data in  
ECMAScript. I've been wanting to add that as both a sending and receiving  
type since before I started working on XMLHttpRequest Level 2.



--
Anne van Kesteren
http://annevankesteren.nl/

Re: [whatwg] canvas, img, file api and blobs

[Stef Epardaud]

On Tue, Feb 16, 2010 at 12:13:11AM -0800, Jonas Sicking wrote:
> On Mon, Feb 15, 2010 at 11:47 PM, Maciej Stachowiak  wrote:
> > I still think we need a type for binary data that can represent in-memory
> > resources. Blobs only allow asynchronous access, since they are meant to
> > represent something like a file, a slice of a file, or a chunk of data in an
> > on-disk database. It doesn't make sense to use such an inconvenient
> > interface to pull out the contents of the canvas, which pretty much have to
> > already be in memory.
> >
> > Even something like WebGL's typed arrays would be better, if the ECMAScript
> > committee doesn't come up with a good solution for basic binary data soon.
> 
> Absolutely! I definitely agree that we need a type like this. The
> sooner the better. On that note, do you know what the latest status is
> within ECMA on this? I know you made a proposal on the webapps list
> (or was it here?), did that go anywhere?

I agree we need more access than what the Blob has to offer, like
sequential and random access at least, and read/write, especially if we
are to copy EXIF data from one source image to a computed image.

> However I suspect it'd be useful to be able to extract as Blob object
> as well, for storing in places like databases or to extract a url.

I am not sure I like this "as well" ;) In my case I need to extract
binary data from a canvas, modify this binary data, then send it using
XmlHttpRequest (which only accepts a File or Blob IIRC), so this binary
type has to be accepted by XmlHttpRequest as well, for sending.

Now that I think about it, is it possible to replace a regular form's
file input list (the list of File objects selected from an input of type
"file" with multiple files enabled) with "processed" files? Like data
coming out of a canvas, possibly by storing them locally (via storage)
to get a File instance. Or are we then limited to sending those
processed files via XmlHttpRequest?

-- 
Stéphane Epardaud

Re: [whatwg] canvas, img, file api and blobs

[Jonas Sicking]

On Mon, Feb 15, 2010 at 11:47 PM, Maciej Stachowiak  wrote:
>
> On Feb 15, 2010, at 1:53 PM, Jonas Sicking wrote:
>
>> On Mon, Feb 15, 2010 at 6:43 AM, Stef Epardaud  wrote:
>>>
>>> Hello,
>>>
>>> I am trying to write a client-side application in HTML5 that resizes
>>> images before uploading them to the server. I saw several demos that did
>>> this resizing using canvas and img, but I have only seen how to get a
>>> data URL out of a canvas, and since Base64 is about 1.37% larger than
>>> the equivalent binary data (according to
>>> http://en.wikipedia.org/wiki/Base64),
>>> I wonder why it is not possible to get a Blob out of the canvas?
>>> Especially since the File API supports uploading Blobs.
>>
>> It has been suggested earlier on this list that we add a
>>
>> canvas.toFile
>> or
>> canvas.toBlob
>>
>> function, which would return a binary Blob instead of a data URL. For
>> exactly the reasons that you mention. I hope to write a test
>> implementation of this for firefox in the near future.
>
> I still think we need a type for binary data that can represent in-memory
> resources. Blobs only allow asynchronous access, since they are meant to
> represent something like a file, a slice of a file, or a chunk of data in an
> on-disk database. It doesn't make sense to use such an inconvenient
> interface to pull out the contents of the canvas, which pretty much have to
> already be in memory.
>
> Even something like WebGL's typed arrays would be better, if the ECMAScript
> committee doesn't come up with a good solution for basic binary data soon.

Absolutely! I definitely agree that we need a type like this. The
sooner the better. On that note, do you know what the latest status is
within ECMA on this? I know you made a proposal on the webapps list
(or was it here?), did that go anywhere?

However I suspect it'd be useful to be able to extract as Blob object
as well, for storing in places like databases or to extract a url.

/ Jonas

/ Jonas

Re: [whatwg] canvas, img, file api and blobs

[Maciej Stachowiak]

On Feb 15, 2010, at 1:53 PM, Jonas Sicking wrote:

On Mon, Feb 15, 2010 at 6:43 AM, Stef Epardaud   
wrote:

Hello,

I am trying to write a client-side application in HTML5 that resizes
images before uploading them to the server. I saw several demos  
that did

this resizing using canvas and img, but I have only seen how to get a
data URL out of a canvas, and since Base64 is about 1.37% larger than
the equivalent binary data (according to http://en.wikipedia.org/wiki/Base64 
),

I wonder why it is not possible to get a Blob out of the canvas?
Especially since the File API supports uploading Blobs.


It has been suggested earlier on this list that we add a

canvas.toFile
or
canvas.toBlob

function, which would return a binary Blob instead of a data URL. For
exactly the reasons that you mention. I hope to write a test
implementation of this for firefox in the near future.


I still think we need a type for binary data that can represent in- 
memory resources. Blobs only allow asynchronous access, since they are  
meant to represent something like a file, a slice of a file, or a  
chunk of data in an on-disk database. It doesn't make sense to use  
such an inconvenient interface to pull out the contents of the canvas,  
which pretty much have to already be in memory.


Even something like WebGL's typed arrays would be better, if the  
ECMAScript committee doesn't come up with a good solution for basic  
binary data soon.


Regards,
Macie

Re: [whatwg] canvas, img, file api and blobs

[Jonas Sicking]

On Mon, Feb 15, 2010 at 6:43 AM, Stef Epardaud  wrote:
> Hello,
>
> I am trying to write a client-side application in HTML5 that resizes
> images before uploading them to the server. I saw several demos that did
> this resizing using canvas and img, but I have only seen how to get a
> data URL out of a canvas, and since Base64 is about 1.37% larger than
> the equivalent binary data (according to http://en.wikipedia.org/wiki/Base64),
> I wonder why it is not possible to get a Blob out of the canvas?
> Especially since the File API supports uploading Blobs.

It has been suggested earlier on this list that we add a

canvas.toFile
or
canvas.toBlob

function, which would return a binary Blob instead of a data URL. For
exactly the reasons that you mention. I hope to write a test
implementation of this for firefox in the near future.

> In a related question, is there any guarantee that when we draw an image
> with EXIF information into a canvas for resizing, we get (or not) the
> EXIF back in the resized image (currently via toDataURL())?

Like Boris says, no, copying an image into a canvas doesn't copy the
EXIF data. It would make sense to add some API to canvas that allows
importing metadata from a given image.

/ Jonas

Re: [whatwg] canvas, img, file api and blobs

[Stef Epardaud]

On Mon, Feb 15, 2010 at 05:16:43PM +0100, Anne van Kesteren wrote:
> >You don't.  The canvas is just a bitmap; it has no attached metadata.
> >
> >This could be changed, but various things would need to be defined about  
> >what happens when multiple images are drawn in, when images are drawn on  
> >top of other things in the canvas, when stuff is drawn on top of images  
> >in the canvas, etc.

Very true. Doesn't look right to expect EXIF out of a canvas.

> Seems easier to take care of this by allowing extraction of metadata (some  
> W3C WG is working on this I think) and maybe a way to put it back in...

That would be very welcome, the Blob type doesn't even seem to have a
way to iterate or modify the contents, which would be required if we
were to manually scan the original metadata and move it to the resized
image.
-- 
Stéphane Epardaud

Re: [whatwg] canvas, img, file api and blobs

[Anne van Kesteren]

On Mon, 15 Feb 2010 17:10:30 +0100, Boris Zbarsky  wrote:

On 2/15/10 9:43 AM, Stef Epardaud wrote:

In a related question, is there any guarantee that when we draw an image
with EXIF information into a canvas for resizing, we get (or not) the
EXIF back in the resized image (currently via toDataURL())?


You don't.  The canvas is just a bitmap; it has no attached metadata.

This could be changed, but various things would need to be defined about  
what happens when multiple images are drawn in, when images are drawn on  
top of other things in the canvas, when stuff is drawn on top of images  
in the canvas, etc.


Seems easier to take care of this by allowing extraction of metadata (some  
W3C WG is working on this I think) and maybe a way to put it back in...



--
Anne van Kesteren
http://annevankesteren.nl/

Re: [whatwg] canvas, img, file api and blobs

[Boris Zbarsky]

On 2/15/10 9:43 AM, Stef Epardaud wrote:

In a related question, is there any guarantee that when we draw an image
with EXIF information into a canvas for resizing, we get (or not) the
EXIF back in the resized image (currently via toDataURL())?


You don't.  The canvas is just a bitmap; it has no attached metadata.

This could be changed, but various things would need to be defined about 
what happens when multiple images are drawn in, when images are drawn on 
top of other things in the canvas, when stuff is drawn on top of images 
in the canvas, etc.


-Boris