Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-11-18 Thread Mark
On Sat, Nov 14, 2009 at 5:18 PM, Mark mark...@gmail.com wrote: On Mon, Oct 5, 2009 at 3:05 PM, Mark mark...@gmail.com wrote: On Mon, Oct 5, 2009 at 3:26 PM, Dr. Michael J. Chudobiak m...@avtechpulse.com wrote: On 10/03/2009 02:08 PM, Mark wrote: So what's the conclusion? The existing

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-11-14 Thread Mark
On Mon, Oct 5, 2009 at 3:05 PM, Mark mark...@gmail.com wrote: On Mon, Oct 5, 2009 at 3:26 PM, Dr. Michael J. Chudobiak m...@avtechpulse.com wrote: On 10/03/2009 02:08 PM, Mark wrote: So what's the conclusion? The existing Nautilus code is OK, except that it should be threaded? That sounds

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-10-05 Thread Dr. Michael J. Chudobiak
On 10/03/2009 02:08 PM, Mark wrote: So what's the conclusion? The existing Nautilus code is OK, except that it should be threaded? That sounds like a good conclusion for now but are we going to do something with it? I'm not a Nautilus developer, but I'd guess that a benchmarked patch

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-10-05 Thread Mark
On Mon, Oct 5, 2009 at 3:26 PM, Dr. Michael J. Chudobiak m...@avtechpulse.com wrote: On 10/03/2009 02:08 PM, Mark wrote: So what's the conclusion? The existing Nautilus code is OK, except that it should be threaded? That sounds like a good conclusion for now but are we going to do

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-10-03 Thread Mark
On Fri, Oct 2, 2009 at 2:11 PM, Dr. Michael J. Chudobiak m...@avtechpulse.com wrote: The one with just 21 seconds is where all my images are in cache. That's also where all my cores are working at 100% The other thread benchmark (70 seconds) vauses all cores to work at ~40% So what's the

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-10-02 Thread Dr. Michael J. Chudobiak
The one with just 21 seconds is where all my images are in cache. That's also where all my cores are working at 100% The other thread benchmark (70 seconds) vauses all cores to work at ~40% So what's the conclusion? The existing Nautilus code is OK, except that it should be threaded? - Mike

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-10-01 Thread Petr Tomasek
On Wed, Sep 30, 2009 at 05:51:56PM +0100, Ross Burton wrote: On Wed, 2009-09-30 at 12:32 -0400, David Zeuthen wrote: (Btw, this infrastructure is not specific to the gphoto2:// GVfs backend; any GVfs backend can use it - say, a Flickr backend). Bad example, downloading the original file

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-10-01 Thread Ross Burton
On Thu, 2009-10-01 at 16:00 +0200, Petr Tomasek wrote: On Wed, Sep 30, 2009 at 05:51:56PM +0100, Ross Burton wrote: On Wed, 2009-09-30 at 12:32 -0400, David Zeuthen wrote: (Btw, this infrastructure is not specific to the gphoto2:// GVfs backend; any GVfs backend can use it - say, a Flickr

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Alexander Larsson
On Tue, 2009-09-29 at 22:59 +0200, Mark wrote: hehe that was the idea indeed ^_^ and i will continue with that. I will test the large factors tomorrow. For now i'm happy with 100% cpu usage on all my cores (4). with the code posted in my previous message i only had 70% cpu usage so there

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Mark
On Wed, Sep 30, 2009 at 11:46 AM, Alexander Larsson al...@redhat.com wrote: On Tue, 2009-09-29 at 22:59 +0200, Mark wrote: hehe that was the idea indeed ^_^ and i will continue with that. I will test the large factors tomorrow. For now i'm happy with 100% cpu usage on all my cores (4). with

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Alexander Larsson
On Wed, 2009-09-30 at 12:36 +0200, Mark wrote: On Wed, Sep 30, 2009 at 11:46 AM, Alexander Larsson al...@redhat.com wrote: On Tue, 2009-09-29 at 22:59 +0200, Mark wrote: hehe that was the idea indeed ^_^ and i will continue with that. I will test the large factors tomorrow. For now

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Alexander Larsson
On Wed, 2009-09-30 at 12:58 +0200, Mark wrote: Its faster because it uses a simpler algorithm, so its not generally useful, however it might be nice to allow it optionally, like adding a new filter type and let you pass in a filter to the scaling loader. the filter structure in glib

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Xavier Bestel
Hi, On Wed, 2009-09-30 at 11:46 +0200, Alexander Larsson wrote: On Tue, 2009-09-29 at 22:59 +0200, Mark wrote: hehe that was the idea indeed ^_^ and i will continue with that. I will test the large factors tomorrow. For now i'm happy with 100% cpu usage on all my cores (4). with the

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Alexander Larsson
On Wed, 2009-09-30 at 14:53 +0200, Xavier Bestel wrote: Hi, On Wed, 2009-09-30 at 11:46 +0200, Alexander Larsson wrote: On Tue, 2009-09-29 at 22:59 +0200, Mark wrote: hehe that was the idea indeed ^_^ and i will continue with that. I will test the large factors tomorrow. For

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Philip Van Hoof
On Fri, 2009-08-28 at 22:11 +0200, Mark wrote: Now for the results: Glib -- 1927 images thumbnailed in 2.29 minutes. That is roughly 0.07 seconds per thumbnail GraphicsMagick -- 1927 images thumbnailed in 3.08 minutes. That is roughly 0.09

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Dr. Michael J. Chudobiak
On 09/30/2009 09:27 AM, Philip Van Hoof wrote: You want to override GdkPixbuf's support for Jpeg and thumbnail all JPEG images using EPeg. It'll beat GdkPixbuf by 5 times or something. That's because afaik does EPeg use libjpeg in a way that it skips rows and columns, and that way performs

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Mark
On Wed, Sep 30, 2009 at 3:27 PM, Philip Van Hoof s...@pvanhoof.be wrote: On Fri, 2009-08-28 at 22:11 +0200, Mark wrote: Now for the results: Glib -- 1927 images thumbnailed in 2.29 minutes. That is roughly 0.07 seconds per thumbnail GraphicsMagick

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Alexander Larsson
On Wed, 2009-09-30 at 15:27 +0200, Philip Van Hoof wrote: On Fri, 2009-08-28 at 22:11 +0200, Mark wrote: Now for the results: Glib -- 1927 images thumbnailed in 2.29 minutes. That is roughly 0.07 seconds per thumbnail GraphicsMagick

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Alexander Larsson
On Wed, 2009-09-30 at 16:07 +0200, Alexander Larsson wrote: The first part (DCT stuff) we already do in the jpeg loader (its the jpeg loader trick talked about above). The second we could easily do ourself in the jpeg loader for scaled image loads by scaling in YUV/GRAY8 space before

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Philip Van Hoof
On Wed, 2009-09-30 at 16:05 +0200, Mark wrote: On Wed, Sep 30, 2009 at 3:27 PM, Philip Van Hoof s...@pvanhoof.be wrote: sounds interesting. A few questions for that lib: - Where can it be downloaded (i only found the documentation of it) The websites Alexander pointed to in his reply have

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Mark
On Wed, Sep 30, 2009 at 4:58 PM, Alexander Larsson al...@redhat.com wrote: On Wed, 2009-09-30 at 16:07 +0200, Alexander Larsson wrote: On Wed, 2009-09-30 at 15:27 +0200, Philip Van Hoof wrote: On Fri, 2009-08-28 at 22:11 +0200, Mark wrote: Now for the results: Glib

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Ross Burton
On Wed, 2009-09-30 at 16:57 +0200, Philip Van Hoof wrote: I don't think it has an upstream repository anymore, just source packages of distributions. I do find that using software which lacks a maintainer and canonical repository is far preferable than software with a repository but no

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread David Zeuthen
On Wed, 2009-09-30 at 16:07 +0200, Alexander Larsson wrote: Another thumbnailing performance trick is to use thumbnails in EXIF data, which many cameras add, if availible. FWIW, we do this in the gphoto2:// GVfs backend. Browsing through a USB connected camera in Nautilus is really fast these

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Ross Burton
On Wed, 2009-09-30 at 12:32 -0400, David Zeuthen wrote: (Btw, this infrastructure is not specific to the gphoto2:// GVfs backend; any GVfs backend can use it - say, a Flickr backend). Bad example, downloading the original file (no seeking in HTTP) just to extract the thumbnail would be pretty

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Bastien Nocera
On Wed, 2009-09-30 at 17:51 +0100, Ross Burton wrote: On Wed, 2009-09-30 at 12:32 -0400, David Zeuthen wrote: (Btw, this infrastructure is not specific to the gphoto2:// GVfs backend; any GVfs backend can use it - say, a Flickr backend). Bad example, downloading the original file (no

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread David Zeuthen
On Wed, 2009-09-30 at 17:51 +0100, Ross Burton wrote: On Wed, 2009-09-30 at 12:32 -0400, David Zeuthen wrote: (Btw, this infrastructure is not specific to the gphoto2:// GVfs backend; any GVfs backend can use it - say, a Flickr backend). Bad example, downloading the original file (no

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Ross Burton
On Wed, 2009-09-30 at 18:01 +0100, Bastien Nocera wrote: On Wed, 2009-09-30 at 17:51 +0100, Ross Burton wrote: On Wed, 2009-09-30 at 12:32 -0400, David Zeuthen wrote: (Btw, this infrastructure is not specific to the gphoto2:// GVfs backend; any GVfs backend can use it - say, a Flickr

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-30 Thread Ross Burton
On Wed, 2009-09-30 at 13:06 -0400, David Zeuthen wrote: No, it's actually a great example. The way it works is that a GVfs backend can set the preview::icon file attribute (which is a GIcon) [1] to whatever it wants. In GVfs we have a class, GVfsIcon, that implements GLoadableIcon and GIcon.

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-29 Thread Mark
On Mon, Aug 31, 2009 at 4:06 PM, Dr. Michael J. Chudobiak m...@avtechpulse.com wrote: On 08/31/2009 09:03 AM, jcup...@gmail.com wrote: a very quick load-at-1/8th-size read where it just decompresses enough to be able to get the DC component of each 8x8 block. If you use libjpeg like this you

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-29 Thread Dr. Michael J. Chudobiak
On 09/29/2009 12:00 PM, Mark wrote: Well, the benchmarks ran above are resizing 1927 wallpaper sized images to a max width or height of 200 and that function clearly loses. The solution to this is extremely simple. Grab this file:

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-09-29 Thread Mark
On Tue, Sep 29, 2009 at 9:46 PM, Dr. Michael J. Chudobiak m...@avtechpulse.com wrote: On 09/29/2009 12:00 PM, Mark wrote: Well, the benchmarks ran above are resizing 1927 wallpaper sized images to a max width or height of 200 and that function clearly loses. The solution to this is extremely

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-31 Thread Dr. Michael J. Chudobiak
On 08/30/2009 09:51 AM, jcup...@gmail.com wrote: able to supply pixels at a certain size. In particular, libjpeg can do a very quick load-at-1/8th-size read where it just decompresses enough to be able to get the DC component of each 8x8 block. If you use libjpeg like this you can expect around

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-31 Thread jcupitt
2009/8/31 Dr. Michael J. Chudobiak m...@avtechpulse.com: On 08/30/2009 09:51 AM, jcup...@gmail.com wrote: able to supply pixels at a certain size. In particular, libjpeg can do a very quick load-at-1/8th-size read where it just decompresses enough to be able to get the DC component of each 8x8

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-31 Thread Dr. Michael J. Chudobiak
On 08/31/2009 09:03 AM, jcup...@gmail.com wrote: a very quick load-at-1/8th-size read where it just decompresses enough to be able to get the DC component of each 8x8 block. If you use libjpeg like this you can expect around a 100x speedup of the decompress step. The gdk-pixbuf jpeg loader

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-30 Thread Matthias Clasen
On Sun, Aug 30, 2009 at 9:51 AM, jcup...@gmail.com wrote: 2009/8/28 Mark mark...@gmail.com: static GdkPixbuf * scale_pixbuf_preserve_aspect_ratio (GdkPixbuf *pixbuf,                                    gint size,                                    GdkInterpType interp) One more idea: this

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-30 Thread Mark
On Sun, Aug 30, 2009 at 3:51 PM, jcup...@gmail.com wrote: 2009/8/28 Mark mark...@gmail.com: static GdkPixbuf * scale_pixbuf_preserve_aspect_ratio (GdkPixbuf *pixbuf,                                    gint size,                                    GdkInterpType interp) One more idea: this

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-30 Thread muppet
On Aug 29, 2009, at 8:08 AM, Mark wrote: Oke, i did some ltracing now. Here is the ltrace output for 5 images that get thumbnailed. I hope anyone here could help me a bit in explaining this output since it's not all obvious to me. Before you read the output. i put a 2 second sleep after a

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-29 Thread Mark
On Sat, Aug 29, 2009 at 1:04 AM, Christian Hergertch...@dronelabs.com wrote: On Fri, Aug 28, 2009 at 11:49 PM, Christian Hergertch...@dronelabs.com  wrote: Hi, What you mentioned is good information to start hunting.  Was the CPU time related to IO wait at all?  Always get accurate

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-29 Thread Mark
Oke, i did some ltracing now. Here is the ltrace output for 5 images that get thumbnailed. I hope anyone here could help me a bit in explaining this output since it's not all obvious to me. Before you read the output. i put a 2 second sleep after a image is done and the pixbuffs are removed

Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-28 Thread Mark
Hi, Before you read on. I wasn't sure if this needs to go to the gtk-devel or the nautilus mailing lists. I did gtk for now. I hope that's the right list for things like this. It has been a few years ago since i made a topic about this (on the nautilus list) with the title (rough guess)

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-28 Thread Christian Hergert
Have you profiled to see if the bottleneck is CPU bound? If its IO bound, you will only cause more contention by adding threading. At minimum, using a thread (or async gio) to load files and another thread that just thumbnails might be a good idea. Cheers, -- Christian Mark wrote: is

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-28 Thread Mark
On Fri, Aug 28, 2009 at 10:45 PM, Christian Hergertch...@dronelabs.com wrote: Have you profiled to see if the bottleneck is CPU bound?  If its IO bound, you will only cause more contention by adding threading. At minimum, using a thread (or async gio) to load files and another thread that

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-28 Thread David Zeuthen
On Fri, 2009-08-28 at 23:15 +0200, Mark wrote: I haven't done io profiling but i did calculate the disc usage for those 1927 files. and every benchmark was WAY below what my hdd could handle (Spinpoint F1 1TB hdd and it can handle roughly 100MB/sec). Uhm, wait, that's only true for sequential

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-28 Thread Christian Hergert
Hi, What you mentioned is good information to start hunting. Was the CPU time related to IO wait at all? Always get accurate numbers before performance tuning. Measure, measure, measure or so the mantra goes. Unfortunately, the symptom you see regarding IO will very likely change under a

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-28 Thread Mark
On Fri, Aug 28, 2009 at 11:38 PM, David Zeuthenda...@fubar.dk wrote: On Fri, 2009-08-28 at 23:15 +0200, Mark wrote: I haven't done io profiling but i did calculate the disc usage for those 1927 files. and every benchmark was WAY below what my hdd could handle (Spinpoint F1 1TB hdd and it can

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-28 Thread Christian Hergert
On Fri, Aug 28, 2009 at 11:49 PM, Christian Hergertch...@dronelabs.com wrote: Hi, What you mentioned is good information to start hunting. Was the CPU time related to IO wait at all? Always get accurate numbers before performance tuning. Measure, measure, measure or so the mantra goes.

Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

2009-08-28 Thread Mark
On Sat, Aug 29, 2009 at 1:04 AM, Christian Hergertch...@dronelabs.com wrote: On Fri, Aug 28, 2009 at 11:49 PM, Christian Hergertch...@dronelabs.com  wrote: Hi, What you mentioned is good information to start hunting.  Was the CPU time related to IO wait at all?  Always get accurate