[webkit-dev] image downscaling during decoding
Hi, We are looking for ways to improve page loading speed and reduce memory usage for WebKit in general and for GTK port of WebKit in particular. One of the ideas is to implement downscaling of the images during decoding for image elements with rectangle less then original image size. At the moment such images are full decoded to a full size buffer and get downscaled during rendering. It can be quite beneficial in term of memory usage and should speed up rendering of the pages like image galleries for example. So what are your thoughts about it? Regards, Anton ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] image downscaling during decoding
Hi, At the moment such images are full decoded to a full size buffer and get downscaled during rendering. WebKit has already a hard coded ImageSource::s_maxPixelsPerDecodedImage value, in WebCode/platform/graphics/ImageSource.cpp which defaults to 1024 * 1024 when WebKit's option ENABLE(IMAGE_DECODER_DOWN_SAMPLING) gets turned on. s_maxPixelsPerDecodedImage is used to compute a scaling factor for both the horizontal and vertical directions and serves as a cap to the maximum picture size that a decoder would produce (so already a way to control the total memory allocated for pictures). Now if those new width and height are smaller than the original picture's dimensions, then the decoder if clever enough would then produce a downscaled picture (preferably while decoding and not in a second pass) that fit s_maxPixelsPerDecodedImage. For example, it's is already possible for libjpeg to downscale JPEGs (via a modified IDCT, so we get the resize for free), by specifying a resize ratio via cinfo's scale_num and scale_denom fields. Finally, it would make sense to allocate all those graphics buffers from video memory instead of system memory (malloc()) (and yes that's rather platform specific) and let the h/w to perform the final resize and layout/rendering. -Ilyes On Mon, Aug 13, 2012 at 2:39 PM, Anton Obzhirov a.obzhi...@samsung.com wrote: Hi, We are looking for ways to improve page loading speed and reduce memory usage for WebKit in general and for GTK port of WebKit in particular. One of the ideas is to implement downscaling of the images during decoding for image elements with rectangle less then original image size. At the moment such images are full decoded to a full size buffer and get downscaled during rendering. It can be quite beneficial in term of memory usage and should speed up rendering of the pages like image galleries for example. So what are your thoughts about it? Regards, Anton ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] image downscaling during decoding
On Monday 13 August 2012, Anton Obzhirov wrote: Hi, We are looking for ways to improve page loading speed and reduce memory usage for WebKit in general and for GTK port of WebKit in particular. One of the ideas is to implement downscaling of the images during decoding for image elements with rectangle less then original image size. At the moment such images are full decoded to a full size buffer and get downscaled during rendering. It can be quite beneficial in term of memory usage and should speed up rendering of the pages like image galleries for example. So what are your thoughts about it? There is already code for that. Check the IMAGE_DECODER_DOWN_SAMPLING flag, it currently only downscales image over a certain size and it is not adaptable (it does not decoder to a higher res if you zoom), but it works during decoding and throws away excess accuracy. Improvements to this would be great. If you look into the iOS fork of WebKit, you will find they has something less precise, but faster, where they can down-scale by powers of 2 (1/2, 1/4, 1/8..). Note btw, that some loosy image-formats notably JPEG has a methods of decoding, where you get increased pixel-accuracy when decoding to lower resolutions, but we currently do not take advantage of this, and it only supports down-scale by powers of 2. Just a few thoughts. `Allan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] image downscaling during decoding
Libjpeg has some internal support for downscaling already (2x, 4x). Not sure about Libpng and other libraries. In general probably it can be implemented by downscaling decoded row of the pixels on the fly using callback API provided. From: tomhud...@google.com [mailto:tomhud...@google.com] On Behalf Of Tom Hudson Sent: 13 August 2012 15:00 To: webkit-dev@lists.webkit.org; Anton Obzhirov Subject: Fwd: [webkit-dev] image downscaling during decoding On Mon, Aug 13, 2012 at 9:39 AM, Anton Obzhirov a.obzhi...@samsung.com wrote: We are looking for ways to improve page loading speed and reduce memory usage for WebKit in general and for GTK port of WebKit in particular. One of the ideas is to implement downscaling of the images during decoding for image elements with rectangle less then original image size. At the moment such images are full decoded to a full size buffer and get downscaled during rendering. It can be quite beneficial in term of memory usage and should speed up rendering of the pages like image galleries for example. So what are your thoughts about it? Interesting idea! I know Chrome sees memory pressure from the downsampling two-step on some pages, so I'd think this is could be useful for us, too. Isn't it going to require fairly large and intrusive changes to several different third-party libraries, though? Also, different ports use different image downscaling algorithms; I can think of ways to try to enable incremental downscaling callbacks so you don't have to implement N downscales in each of M decoders, but none that are both general and performant. I'd love to see a proposal on this. Tom (from the right account this time) ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] image downscaling during decoding
Yes, I noticed IMAGE_DECODER_DOWN_SAMPLING option. I planned to reuse existing ImageDecoder API as much as possible. May be will need to add generic image downscaler with platform API. I need to think it through and I will send my proposal. Should I create a bug for it? -Original Message- From: Ilyes Gouta [mailto:ilyes.go...@gmail.com] Sent: 13 August 2012 15:37 To: Anton Obzhirov Cc: webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] image downscaling during decoding Hi, At the moment such images are full decoded to a full size buffer and get downscaled during rendering. WebKit has already a hard coded ImageSource::s_maxPixelsPerDecodedImage value, in WebCode/platform/graphics/ImageSource.cpp which defaults to 1024 * 1024 when WebKit's option ENABLE(IMAGE_DECODER_DOWN_SAMPLING) gets turned on. s_maxPixelsPerDecodedImage is used to compute a scaling factor for both the horizontal and vertical directions and serves as a cap to the maximum picture size that a decoder would produce (so already a way to control the total memory allocated for pictures). Now if those new width and height are smaller than the original picture's dimensions, then the decoder if clever enough would then produce a downscaled picture (preferably while decoding and not in a second pass) that fit s_maxPixelsPerDecodedImage. For example, it's is already possible for libjpeg to downscale JPEGs (via a modified IDCT, so we get the resize for free), by specifying a resize ratio via cinfo's scale_num and scale_denom fields. Finally, it would make sense to allocate all those graphics buffers from video memory instead of system memory (malloc()) (and yes that's rather platform specific) and let the h/w to perform the final resize and layout/rendering. -Ilyes On Mon, Aug 13, 2012 at 2:39 PM, Anton Obzhirov a.obzhi...@samsung.com wrote: Hi, We are looking for ways to improve page loading speed and reduce memory usage for WebKit in general and for GTK port of WebKit in particular. One of the ideas is to implement downscaling of the images during decoding for image elements with rectangle less then original image size. At the moment such images are full decoded to a full size buffer and get downscaled during rendering. It can be quite beneficial in term of memory usage and should speed up rendering of the pages like image galleries for example. So what are your thoughts about it? Regards, Anton ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] image downscaling during decoding
Hi, On Mon, Aug 13, 2012 at 3:46 PM, Anton Obzhirov a.obzhi...@samsung.com wrote: Libjpeg has some internal support for downscaling already (2x, 4x). Not sure about Libpng and other libraries. In general probably it can be implemented by downscaling decoded row of the pixels on the fly using callback API provided. If such a callback API for row-based resizing comes to fruition, then let's (also) not rule out whole-picture h/w accelerated resizing - such as by making that new API a mandatory stage in the decoding process. -Ilyes From: tomhud...@google.com [mailto:tomhud...@google.com] On Behalf Of Tom Hudson Sent: 13 August 2012 15:00 To: webkit-dev@lists.webkit.org; Anton Obzhirov Subject: Fwd: [webkit-dev] image downscaling during decoding On Mon, Aug 13, 2012 at 9:39 AM, Anton Obzhirov a.obzhi...@samsung.com wrote: We are looking for ways to improve page loading speed and reduce memory usage for WebKit in general and for GTK port of WebKit in particular. One of the ideas is to implement downscaling of the images during decoding for image elements with rectangle less then original image size. At the moment such images are full decoded to a full size buffer and get downscaled during rendering. It can be quite beneficial in term of memory usage and should speed up rendering of the pages like image galleries for example. So what are your thoughts about it? Interesting idea! I know Chrome sees memory pressure from the downsampling two-step on some pages, so I'd think this is could be useful for us, too. Isn't it going to require fairly large and intrusive changes to several different third-party libraries, though? Also, different ports use different image downscaling algorithms; I can think of ways to try to enable incremental downscaling callbacks so you don't have to implement N downscales in each of M decoders, but none that are both general and performant. I'd love to see a proposal on this. Tom (from the right account this time) ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] image downscaling during decoding
I agree we should have a separate row-based downscaling phase. JPEGImageDecoder.cpp already supports downscaling with a given width and height, but it's using point sampling which has poor quality. It would be nice to have a callback based downscaling API that we can implement better algorithms. Alpha 2012/8/13 Alpha (Hin-Chung) Lam hc...@google.com I agree we should have a separate row-based downscaling phase. JPEGImageDecoder.cpp already supports downscaling with a given width and height, but it's using point sampling which has poor quality. It would be nice to have a callback based downscaling API that we can implement better algorithms. Alpha 2012/8/13 Ilyes Gouta ilyes.go...@gmail.com Hi, On Mon, Aug 13, 2012 at 3:46 PM, Anton Obzhirov a.obzhi...@samsung.com wrote: Libjpeg has some internal support for downscaling already (2x, 4x). Not sure about Libpng and other libraries. In general probably it can be implemented by downscaling decoded row of the pixels on the fly using callback API provided. If such a callback API for row-based resizing comes to fruition, then let's (also) not rule out whole-picture h/w accelerated resizing - such as by making that new API a mandatory stage in the decoding process. -Ilyes From: tomhud...@google.com [mailto:tomhud...@google.com] On Behalf Of Tom Hudson Sent: 13 August 2012 15:00 To: webkit-dev@lists.webkit.org; Anton Obzhirov Subject: Fwd: [webkit-dev] image downscaling during decoding On Mon, Aug 13, 2012 at 9:39 AM, Anton Obzhirov a.obzhi...@samsung.com wrote: We are looking for ways to improve page loading speed and reduce memory usage for WebKit in general and for GTK port of WebKit in particular. One of the ideas is to implement downscaling of the images during decoding for image elements with rectangle less then original image size. At the moment such images are full decoded to a full size buffer and get downscaled during rendering. It can be quite beneficial in term of memory usage and should speed up rendering of the pages like image galleries for example. So what are your thoughts about it? Interesting idea! I know Chrome sees memory pressure from the downsampling two-step on some pages, so I'd think this is could be useful for us, too. Isn't it going to require fairly large and intrusive changes to several different third-party libraries, though? Also, different ports use different image downscaling algorithms; I can think of ways to try to enable incremental downscaling callbacks so you don't have to implement N downscales in each of M decoders, but none that are both general and performant. I'd love to see a proposal on this. Tom (from the right account this time) ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] image downscaling during decoding
Hi, If such a callback API for row-based resizing comes to fruition, then let's (also) not rule out whole-picture h/w accelerated resizing - such as by making that new API a mandatory stage in the decoding that's actually: such as by *not* making that new API a mandatory stage in the decoding process. I agree we should have a separate row-based downscaling phase. Resizing and filtering on a single row isn't enough. For a good quality and depending on the scaling ratio, we should maintain N rows, with N depends on the maximum downscaling or upscaling factor. -Ilyes nice to have a callback based downscaling API that we can implement better algorithms. On Mon, Aug 13, 2012 at 6:52 PM, Alpha Lam hc...@chromium.org wrote: I agree we should have a separate row-based downscaling phase. JPEGImageDecoder.cpp already supports downscaling with a given width and height, but it's using point sampling which has poor quality. It would be nice to have a callback based downscaling API that we can implement better algorithms. Alpha 2012/8/13 Alpha (Hin-Chung) Lam hc...@google.com I agree we should have a separate row-based downscaling phase. JPEGImageDecoder.cpp already supports downscaling with a given width and height, but it's using point sampling which has poor quality. It would be nice to have a callback based downscaling API that we can implement better algorithms. Alpha 2012/8/13 Ilyes Gouta ilyes.go...@gmail.com Hi, On Mon, Aug 13, 2012 at 3:46 PM, Anton Obzhirov a.obzhi...@samsung.com wrote: Libjpeg has some internal support for downscaling already (2x, 4x). Not sure about Libpng and other libraries. In general probably it can be implemented by downscaling decoded row of the pixels on the fly using callback API provided. If such a callback API for row-based resizing comes to fruition, then let's (also) not rule out whole-picture h/w accelerated resizing - such as by making that new API a mandatory stage in the decoding process. -Ilyes From: tomhud...@google.com [mailto:tomhud...@google.com] On Behalf Of Tom Hudson Sent: 13 August 2012 15:00 To: webkit-dev@lists.webkit.org; Anton Obzhirov Subject: Fwd: [webkit-dev] image downscaling during decoding On Mon, Aug 13, 2012 at 9:39 AM, Anton Obzhirov a.obzhi...@samsung.com wrote: We are looking for ways to improve page loading speed and reduce memory usage for WebKit in general and for GTK port of WebKit in particular. One of the ideas is to implement downscaling of the images during decoding for image elements with rectangle less then original image size. At the moment such images are full decoded to a full size buffer and get downscaled during rendering. It can be quite beneficial in term of memory usage and should speed up rendering of the pages like image galleries for example. So what are your thoughts about it? Interesting idea! I know Chrome sees memory pressure from the downsampling two-step on some pages, so I'd think this is could be useful for us, too. Isn't it going to require fairly large and intrusive changes to several different third-party libraries, though? Also, different ports use different image downscaling algorithms; I can think of ways to try to enable incremental downscaling callbacks so you don't have to implement N downscales in each of M decoders, but none that are both general and performant. I'd love to see a proposal on this. Tom (from the right account this time) ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] image downscaling during decoding
2012/8/13 Ilyes Gouta ilyes.go...@gmail.com
Hi,
If such a callback API for row-based resizing comes to fruition, then
let's (also) not rule out whole-picture h/w accelerated resizing -
such as by making that new API a mandatory stage in the decoding
that's actually: such as by *not* making that new API a mandatory
stage in the decoding
process.
I agree we should have a separate row-based downscaling phase.
Resizing and filtering on a single row isn't enough. For a good
quality and depending on the scaling ratio, we should maintain N rows,
with N depends on the maximum downscaling or upscaling factor.
Maintaining N rows is an implementation detail of the scaler. I mean by
having a simple one-scanline-in, zero-or-more-scanlines-out interface, like
this:
scaler.setSourceSize(10, 10);
scaler.setDestSize(100, 100);
while (decoder.hasMoreScanines()) {
scaler.supplyScanLine(decoder.nextScanine());
while (scaler.hasMoreScanlines())
memcpy(dest.nextOutputScanline(), scaler.nextScanline(), width * 4);
}
Then what algorithm to use and quality is up to the scaler implementation
to decide.
Alpha
-Ilyes
nice to have a callback based downscaling API that we can implement
better
algorithms.
On Mon, Aug 13, 2012 at 6:52 PM, Alpha Lam hc...@chromium.org wrote:
I agree we should have a separate row-based downscaling phase.
JPEGImageDecoder.cpp already supports downscaling with a given width and
height, but it's using point sampling which has poor quality. It would be
nice to have a callback based downscaling API that we can implement
better
algorithms.
Alpha
2012/8/13 Alpha (Hin-Chung) Lam hc...@google.com
I agree we should have a separate row-based downscaling phase.
JPEGImageDecoder.cpp already supports downscaling with a given width and
height, but it's using point sampling which has poor quality. It would
be
nice to have a callback based downscaling API that we can implement
better
algorithms.
Alpha
2012/8/13 Ilyes Gouta ilyes.go...@gmail.com
Hi,
On Mon, Aug 13, 2012 at 3:46 PM, Anton Obzhirov
a.obzhi...@samsung.com
wrote:
Libjpeg has some internal support for downscaling already (2x, 4x).
Not
sure
about
Libpng and other libraries. In general probably it can be
implemented
by
downscaling decoded row of the pixels on the fly using callback API
provided.
If such a callback API for row-based resizing comes to fruition, then
let's (also) not rule out whole-picture h/w accelerated resizing -
such as by making that new API a mandatory stage in the decoding
process.
-Ilyes
From: tomhud...@google.com [mailto:tomhud...@google.com] On Behalf
Of
Tom
Hudson
Sent: 13 August 2012 15:00
To: webkit-dev@lists.webkit.org; Anton Obzhirov
Subject: Fwd: [webkit-dev] image downscaling during decoding
On Mon, Aug 13, 2012 at 9:39 AM, Anton Obzhirov
a.obzhi...@samsung.com
wrote:
We are looking for ways to improve page loading speed and reduce
memory
usage for WebKit in general and for GTK port of WebKit in particular.
One of the ideas is to implement downscaling of the images during
decoding
for image elements with rectangle less then original image size.
At the moment such images are full decoded to a full size buffer and
get
downscaled during rendering.
It can be quite beneficial in term of memory usage and should speed
up
rendering of the pages like image galleries for example.
So what are your thoughts about it?
Interesting idea! I know Chrome sees memory pressure from the
downsampling
two-step on some pages, so I'd think this is could be useful for us,
too.
Isn't it going to require fairly large and intrusive changes to
several
different third-party libraries, though?
Also, different ports use different image downscaling algorithms; I
can
think of ways to try to enable incremental downscaling callbacks so
you
don't have to implement N downscales in each of M decoders, but none
that
are both general and performant.
I'd love to see a proposal on this.
Tom
(from the right account this time)
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] image downscaling during decoding
Hi,
Maintaining N rows is an implementation detail of the scaler. I mean by
having a simple one-scanline-in, zero-or-more-scanlines-out interface, like
this:
scaler.setSourceSize(10, 10);
scaler.setDestSize(100, 100);
while (decoder.hasMoreScanines()) {
scaler.supplyScanLine(decoder.nextScanine());
while (scaler.hasMoreScanlines())
memcpy(dest.nextOutputScanline(), scaler.nextScanline(), width * 4);
}
Then what algorithm to use and quality is up to the scaler implementation to
decide.
I'm still thinking that any scaling (if any) performed by image
decoder should remain implicit, i.e not exposed to the upper rendering
layer.
The abstract ImageDecoder class has a setSize(unsigned width, unsigned
height) method, that's called by the specialized decoder (let's say
JPEG) to set the original width and height of the picture, and a
ImageFrame* frameBufferAtIndex(size_t) = 0 method that return a
decoded ImageFrame frame. Now ImageFrame (or a derived class!) on its
turn has a NativeImagePtr asNewNativeImage() function that returns the
native representation of that picture, so the native buffer. It could
a system memory allocated buffer, or a surface allocated by the
underlying toolkit.
So in order to downscale when decoding a picture, the base class
ImageDecoder would just need to be augmented (a new method) to also
take the desired decoding target dimensions that the upper
rendering/compositing layer wants/expects. With that information, the
decoder can choose to decimate, downscale, upscale, etc. its original
picture in order to meet that target dimension. libjpeg can take
advantage of such info, to downscale *while* decoding (so it comes for
free). frameBufferAtIndex() would then return a frame with the hinted
target dimensions (they could be close to the expected dimensions, not
necessarily equal to) rather than the original ones. The drawing
toolkit/libraries such as Cairo or Qt could then leverage their
resizing routines (which could be h/w accelerated) when blitting the
ImageFrame onscreen when compositing for display.
I'm not sure about the utility of a s/w scaler anymore.
-Ilyes
Alpha
-Ilyes
nice to have a callback based downscaling API that we can implement
better
algorithms.
On Mon, Aug 13, 2012 at 6:52 PM, Alpha Lam hc...@chromium.org wrote:
I agree we should have a separate row-based downscaling phase.
JPEGImageDecoder.cpp already supports downscaling with a given width and
height, but it's using point sampling which has poor quality. It would
be
nice to have a callback based downscaling API that we can implement
better
algorithms.
Alpha
2012/8/13 Alpha (Hin-Chung) Lam hc...@google.com
I agree we should have a separate row-based downscaling phase.
JPEGImageDecoder.cpp already supports downscaling with a given width
and
height, but it's using point sampling which has poor quality. It would
be
nice to have a callback based downscaling API that we can implement
better
algorithms.
Alpha
2012/8/13 Ilyes Gouta ilyes.go...@gmail.com
Hi,
On Mon, Aug 13, 2012 at 3:46 PM, Anton Obzhirov
a.obzhi...@samsung.com
wrote:
Libjpeg has some internal support for downscaling already (2x, 4x).
Not
sure
about
Libpng and other libraries. In general probably it can be
implemented
by
downscaling decoded row of the pixels on the fly using callback API
provided.
If such a callback API for row-based resizing comes to fruition, then
let's (also) not rule out whole-picture h/w accelerated resizing -
such as by making that new API a mandatory stage in the decoding
process.
-Ilyes
From: tomhud...@google.com [mailto:tomhud...@google.com] On Behalf
Of
Tom
Hudson
Sent: 13 August 2012 15:00
To: webkit-dev@lists.webkit.org; Anton Obzhirov
Subject: Fwd: [webkit-dev] image downscaling during decoding
On Mon, Aug 13, 2012 at 9:39 AM, Anton Obzhirov
a.obzhi...@samsung.com
wrote:
We are looking for ways to improve page loading speed and reduce
memory
usage for WebKit in general and for GTK port of WebKit in
particular.
One of the ideas is to implement downscaling of the images during
decoding
for image elements with rectangle less then original image size.
At the moment such images are full decoded to a full size buffer and
get
downscaled during rendering.
It can be quite beneficial in term of memory usage and should speed
up
rendering of the pages like image galleries for example.
So what are your thoughts about it?
Interesting idea! I know Chrome sees memory pressure from the
downsampling
two-step on some pages, so I'd think this is could be useful for us,
too.
Isn't it going to require fairly large and intrusive changes to
several
different third-party libraries, though?
Also, different ports use different image downscaling algorithms; I
