Re: [Qgis-developer] Slow raster loading

2012-02-15 Thread Marco Hugentobler

Hi Paolo, Tim


Marco, any idea about when this will happen?


I've pushed the raster_resampler branch into the qgis/Quantum-GIS 
repository. Like this, developers can directly make changes in it.
I think it is important to do enough testing before integrating into 
master branch. Tim already found and fixed an issue, I'm sure there are 
more issues around.


On the other hand, once people are used to the average resampling for 
zoomed-out rasters, I'm sure they don't like to go back to 'nearest 
neighbour'.


Regards,
Marco

On 15.02.2012 08:25, Paolo Cavallini wrote:

Il 09/02/2012 10:03, Tim Sutton ha scritto:


I would suggest that we wait until after Marco's resampling branch is
merged before looking at these things as many parts of raster
implementation are changing with it.

quite sensible - Marco, any idea about when this will happen?
Thanks.




--
Dr. Marco Hugentobler
Sourcepole -  Linux  Open Source Solutions
Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland
marco.hugentob...@sourcepole.ch http://www.sourcepole.ch
Technical Advisor QGIS Project Steering Committee

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] Slow raster loading

2012-02-14 Thread Paolo Cavallini
Il 09/02/2012 10:03, Tim Sutton ha scritto:

 I would suggest that we wait until after Marco's resampling branch is
 merged before looking at these things as many parts of raster
 implementation are changing with it.

quite sensible - Marco, any idea about when this will happen?
Thanks.

-- 
Paolo Cavallini - Faunalia
www.faunalia.eu
Full contact details at www.faunalia.eu/pc
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] Slow raster loading

2012-02-10 Thread Giovanni Manghi
Hi Tim,


  *) computing of stats takes a long time, it is easy to see that also
  with not that big rasters it takes a lot, especially if multi-band,
  hurting the user experience. The proposal is to make computing of stats
  optional.
 
 
 Computing stats should be a once off event since I adapted the gdal
 provider to use gdal native stats. Is your experience on first load or
 on subsequent loads too?


subsequent loads are fast, with the exception of GRASS rasters, probably
because stats are not written anywhere.

computing rasterlite stats is very slow if compared with all other
formats.

I agree with others that when computing stats should at least show a
progress bar, but (also) having them computed optionally would be even
better.

When stretching is active standard deviation is enabled is set to 2 by
default. In my experience this is causing to render rasters with
strange colormaps, while a smaller value (1.7) seems to give better
results.



 I am not sure it is so easy to avoid collecting stats. We used to have
 some logic that set stats to be only collected when they were needed,
 I'll need to check if that is still the case or not (but I suspect
 not).
 
  *) in the raster properties there are two properties (contrast
  enhancement and standard deviation) that have a button to save the
  option in a way to allow the setting be persistent among qgis sessions.
  This obviously should be moved to the general options, allowing to
  override at a layer level.
 
 That makes sense to me.


ok


cheers

-- Giovanni --

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] Slow raster loading

2012-02-09 Thread Tim Sutton
Hi

On Mon, Jan 30, 2012 at 5:07 PM, Giovanni Manghi
giovanni.man...@gmail.com wrote:
 Hi all,

 On Mon, 2012-01-30 at 13:16 +0100, Paolo Cavallini wrote:
 Hi all.
 Currently, loading coloured rasters with stretching to minmax on is 
 extremely slow
 (almost unusable):
 http://hub.qgis.org/issues/4421
 E.g. I could load a large RasterLite in 1 sec instead of minutes (!).
 The trick, from the user point of view, is to disable stretching by default, 
 and
 enable it only for rasters where the option makes sense (e.g. a monoband, 
 greyscale).
 Obviously we could do it better, disabling the stretch for the rasters where 
 it does
 not make sense.
 In addition, Giovanni noticed several other small bugs.


 my notes are in the ticket.

 There a few bugs (see ticket) and a couple of things need to be changed:

 *) computing of stats takes a long time, it is easy to see that also
 with not that big rasters it takes a lot, especially if multi-band,
 hurting the user experience. The proposal is to make computing of stats
 optional.


Computing stats should be a once off event since I adapted the gdal
provider to use gdal native stats. Is your experience on first load or
on subsequent loads too?

I am not sure it is so easy to avoid collecting stats. We used to have
some logic that set stats to be only collected when they were needed,
I'll need to check if that is still the case or not (but I suspect
not).

 *) in the raster properties there are two properties (contrast
 enhancement and standard deviation) that have a button to save the
 option in a way to allow the setting be persistent among qgis sessions.
 This obviously should be moved to the general options, allowing to
 override at a layer level.

That makes sense to me.

I would suggest that we wait until after Marco's resampling branch is
merged before looking at these things as many parts of raster
implementation are changing with it.

Regards

Tim


 cheers

 -- Giovanni --

 ___
 Qgis-developer mailing list
 Qgis-developer@lists.osgeo.org
 http://lists.osgeo.org/mailman/listinfo/qgis-developer



-- 
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==
Please do not email me off-list with technical
support questions. Using the lists will gain
more exposure for your issues and the knowledge
surrounding your issue will be shared with all.

Visit http://linfiniti.com to find out about:
 * QGIS programming and support services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux
Irc: timlinux on #qgis at freenode.net
==
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] Slow raster loading

2012-02-09 Thread Nathan Woodrow
On Thu, Feb 9, 2012 at 7:03 PM, Tim Sutton li...@linfiniti.com wrote:

 Computing stats should be a once off event since I adapted the gdal
 provider to use gdal native stats. Is your experience on first load or
 on subsequent loads too?

 I am not sure it is so easy to avoid collecting stats. We used to have
 some logic that set stats to be only collected when they were needed,
 I'll need to check if that is still the case or not (but I suspect
 not).


Tim,

IMO we would should really thread these operations, and report the progress
to the user in a friendly cancel-able way.  If the user doesn't care about
stats, which to me when loading something that is just for viewing is most
of the time, QGIS should just say stats not loaded and show the file
anyway. If you want to get the stats again we should have a recalculate
stats button, with threading and progress bar.

I had a person email me today (a council trying to me to QGIS) about trying
to load a 2gb ecw and it just locks up the UI with no idea of how much is
left to be done or what is going on. I have experienced the same thing but
managed to get around it by making a fake stats file, QGIS then just opens
thinking the stats are fine.

- Nathan
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] Slow raster loading

2012-02-09 Thread MORREALE Jean Roc

Le 09/02/2012 10:38, Nathan Woodrow a écrit :

I had a person email me today (a council trying to me to QGIS) about trying
to load a 2gb ecw and it just locks up the UI with no idea of how much is
left to be done or what is going on. I have experienced the same thing but
managed to get around it by making a fake stats file, QGIS then just opens
thinking the stats are fine.


Oh I'm often getting a similar kind of report : the user opens the 
properties, clicks on the wrong tab - too bad the file is a huge pct 
raster, kill your project.

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] Slow raster loading

2012-02-09 Thread Jim Hammack

  
  
I'm having
the same problem when loading large MrSID maps. I don't recall
QGIS reading/building the stats file in earlier versions. Is
this necessary?

On 02/09/2012 11:51 AM, MORREALE Jean Roc wrote:
Le
  09/02/2012 10:38, Nathan Woodrow a crit :
  
  I had a person email me today (a council
trying to me to QGIS) about trying

to load a 2gb ecw and it just locks up the UI with no idea of
how much is

left to be done or what is going on. I have experienced the same
thing but

managed to get around it by making a fake stats file, QGIS then
just opens

thinking the stats are fine.

  
  
  Oh I'm often getting a similar kind of report : the user opens the
  properties, clicks on the wrong tab - too bad the file is a
  huge pct raster, kill your project.
  
  ___
  
  Qgis-developer mailing list
  
  Qgis-developer@lists.osgeo.org
  
  http://lists.osgeo.org/mailman/listinfo/qgis-developer
  
  

  

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] Slow raster loading

2012-01-30 Thread Giovanni Manghi
Hi all,

On Mon, 2012-01-30 at 13:16 +0100, Paolo Cavallini wrote:
 Hi all.
 Currently, loading coloured rasters with stretching to minmax on is extremely 
 slow
 (almost unusable):
 http://hub.qgis.org/issues/4421
 E.g. I could load a large RasterLite in 1 sec instead of minutes (!).
 The trick, from the user point of view, is to disable stretching by default, 
 and
 enable it only for rasters where the option makes sense (e.g. a monoband, 
 greyscale).
 Obviously we could do it better, disabling the stretch for the rasters where 
 it does
 not make sense.
 In addition, Giovanni noticed several other small bugs.


my notes are in the ticket.

There a few bugs (see ticket) and a couple of things need to be changed:

*) computing of stats takes a long time, it is easy to see that also
with not that big rasters it takes a lot, especially if multi-band,
hurting the user experience. The proposal is to make computing of stats
optional.

*) in the raster properties there are two properties (contrast
enhancement and standard deviation) that have a button to save the
option in a way to allow the setting be persistent among qgis sessions.
This obviously should be moved to the general options, allowing to
override at a layer level.

cheers

-- Giovanni --

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-13 Thread Giovanni Manghi
Hi Tim,


 They should be.

 Try going to the histogram tab of your raster properties or do
 something else that requires stats. In many cases, stats are not
 needed so you may not be seeing thm gathered. Also for read only
 medai, the stats should land up in the proxy dir which is set to
 ~/.qgis/gdal_pam. Otherwise can you give me a specific procedure to
 follow to replicate?


sorry for the noise, yes it works ok.

Cheers

-- Giovanni --

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-11 Thread Giovanni Manghi
Hi Tim,

I'm wrong or after the last commits QGIS is not gathering raster stats
anymore?

At least in my box the aux.xml files are not created anywhere.


Cheers

-- Giovanni --


On Wed, 2011-08-10 at 11:50 +0200, Tim Sutton wrote:
 Hi
 
 On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler
 marco.hugentob...@sourcepole.ch wrote:
  Btw, is it possible to call computeMinMax() on demand, e.g. in
  QgsGDALProvider::minimumValue /maximumValue instead of the provider
  constructor?
 
  It probably does not matter much for the desktop experience. However, for
  server throughput, avoidance of opening and parsing the aux.xml file could 
  make
  a difference.
 
 
 I think the overhead for reading the aux.xml file is really minimal
 and the min/max population of that if its not there is quite quick and
 a once off event so I wouldnt think it is going to impact the server
 much. I'd suggest I first run some benchmarks to see how much overhead
 it adds.
 
 Regards
 
 Tim
 
  Regards,
  Marco
 
 
  Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton:
  Hi Paolo
 
  Does it do that every time you load that layer or only the first time?
  Does the filesystem permissions allow writing of gdal .aux.xml files?
 
  My raster stats branch merge changed some behaviours and it is
  probably a side effect of that. One of the changes is that gdal gets
  full stats when trying to determine the min max of a band. It will
  fetch those stats quickly from aux.xml if present of if not its going
  to do a full stats gather just for the basic min max stats which is
  where you probably experience the slowness. I will see if I can find a
  better way to do the basic min/max stats gathering.
 
  Regards
 
  Tim
 
  On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it
  wrote:
   Hi all.
   With current master, I get bad performances in loading rasters (a 12Mb
   takes 5 sec); when double-clicking it from the QGIS browser, it is far
   slower (20 sec). From the console:
   Warning: Object::connect: No such signal QgsGdalProvider::statusChanged(
   QString ) in
   /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any
   hint?
   All the best.
   --
   Paolo Cavallini: http://www.faunalia.it/pc
   ___
   Qgis-developer mailing list
   Qgis-developer@lists.osgeo.org
   http://lists.osgeo.org/mailman/listinfo/qgis-developer
 
 
  --
  Dr. Marco Hugentobler
  Sourcepole -  Linux  Open Source Solutions
  Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland
  marco.hugentob...@sourcepole.ch http://www.sourcepole.ch
  Technical Advisor QGIS Project Steering Committee
 
 
 
 


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-10 Thread Marco Hugentobler
Btw, is it possible to call computeMinMax() on demand, e.g. in 
QgsGDALProvider::minimumValue /maximumValue instead of the provider 
constructor?

It probably does not matter much for the desktop experience. However, for 
server throughput, avoidance of opening and parsing the aux.xml file could make 
a difference.

Regards,
Marco


Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton:
 Hi Paolo
 
 Does it do that every time you load that layer or only the first time?
 Does the filesystem permissions allow writing of gdal .aux.xml files?
 
 My raster stats branch merge changed some behaviours and it is
 probably a side effect of that. One of the changes is that gdal gets
 full stats when trying to determine the min max of a band. It will
 fetch those stats quickly from aux.xml if present of if not its going
 to do a full stats gather just for the basic min max stats which is
 where you probably experience the slowness. I will see if I can find a
 better way to do the basic min/max stats gathering.
 
 Regards
 
 Tim
 
 On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it 
wrote:
  Hi all.
  With current master, I get bad performances in loading rasters (a 12Mb
  takes 5 sec); when double-clicking it from the QGIS browser, it is far
  slower (20 sec). From the console:
  Warning: Object::connect: No such signal QgsGdalProvider::statusChanged(
  QString ) in
  /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any
  hint?
  All the best.
  --
  Paolo Cavallini: http://www.faunalia.it/pc
  ___
  Qgis-developer mailing list
  Qgis-developer@lists.osgeo.org
  http://lists.osgeo.org/mailman/listinfo/qgis-developer


-- 
Dr. Marco Hugentobler
Sourcepole -  Linux  Open Source Solutions
Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland
marco.hugentob...@sourcepole.ch http://www.sourcepole.ch
Technical Advisor QGIS Project Steering Committee
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-10 Thread Tim Sutton
Hi

On Wed, Aug 10, 2011 at 12:20 AM, Peter Borissow
peter.boris...@yahoo.com wrote:
 Also, might be a good idea to spawn a separate thread to generate the aux
 file in the background...


It could be a possibility but the way things are set up now, stats are
only gathered when really needed, and if you need then for the
activity you are doing you would have to wait for them anyway. The
only merrit of using a separate thread would be if we decided to start
gathering complete stats on first load of a raster in the background
always so that they are preemptively there when the user does
something that needs stats. However I don't know if that is a good
idea - if you have many rasters in a project that need stats it will
spawn many threads and the IO / CPU overhead may be unpleasant.

Regards

Tim


 
 From: Frank Warmerdam warmer...@pobox.com
 To: Tim Sutton li...@linfiniti.com
 Cc: qgis-developer qgis-developer@lists.osgeo.org
 Sent: Tuesday, August 9, 2011 5:51 PM
 Subject: Re: [Qgis-developer] slow raster loading

 On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote:

 Yes thats why I was asking if the file permissions exist to write the
 aux.xml. If they don't behaviour should be the same as prior to my
 native raster stats commits i.e. stats will be recalced each time you
 open a project / add a layer and trigger something that needs the
 stats.

 Tim, and others,

 By the way, there is a mechanism in GDAL to support storing .aux.xml
 information in an alternative directory when the file cannot be
 created on the target directory due to permissions, read-only media, etc.

 This is enabled by setting the GDAL_PAM_PROXY_DIR option to
 point to a directory where files can be written.  If there is a place
 known to QGIS that would be good for this, it might make sense
 to take advantage of this.

 Best regards,
 --
 ---+--
 I set the clouds in motion - turn up   | Frank Warmerdam,
 warmer...@pobox.com
 light and sound - activate the windows | http://pobox.com/~warmerdam
 and watch the world go round - Rush    | Geospatial Software Developer
 ___
 Qgis-developer mailing list
 Qgis-developer@lists.osgeo.org
 http://lists.osgeo.org/mailman/listinfo/qgis-developer






-- 
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==
Please do not email me off-list with technical
support questions. Using the lists will gain
more exposure for your issues and the knowledge
surrounding your issue will be shared with all.

Visit http://linfiniti.com to find out about:
 * QGIS programming and support services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux
Irc: timlinux on #qgis at freenode.net
==
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-10 Thread Tim Sutton
Hi

On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler
marco.hugentob...@sourcepole.ch wrote:
 Btw, is it possible to call computeMinMax() on demand, e.g. in
 QgsGDALProvider::minimumValue /maximumValue instead of the provider
 constructor?

 It probably does not matter much for the desktop experience. However, for
 server throughput, avoidance of opening and parsing the aux.xml file could 
 make
 a difference.


I think the overhead for reading the aux.xml file is really minimal
and the min/max population of that if its not there is quite quick and
a once off event so I wouldnt think it is going to impact the server
much. I'd suggest I first run some benchmarks to see how much overhead
it adds.

Regards

Tim

 Regards,
 Marco


 Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton:
 Hi Paolo

 Does it do that every time you load that layer or only the first time?
 Does the filesystem permissions allow writing of gdal .aux.xml files?

 My raster stats branch merge changed some behaviours and it is
 probably a side effect of that. One of the changes is that gdal gets
 full stats when trying to determine the min max of a band. It will
 fetch those stats quickly from aux.xml if present of if not its going
 to do a full stats gather just for the basic min max stats which is
 where you probably experience the slowness. I will see if I can find a
 better way to do the basic min/max stats gathering.

 Regards

 Tim

 On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it
 wrote:
  Hi all.
  With current master, I get bad performances in loading rasters (a 12Mb
  takes 5 sec); when double-clicking it from the QGIS browser, it is far
  slower (20 sec). From the console:
  Warning: Object::connect: No such signal QgsGdalProvider::statusChanged(
  QString ) in
  /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any
  hint?
  All the best.
  --
  Paolo Cavallini: http://www.faunalia.it/pc
  ___
  Qgis-developer mailing list
  Qgis-developer@lists.osgeo.org
  http://lists.osgeo.org/mailman/listinfo/qgis-developer


 --
 Dr. Marco Hugentobler
 Sourcepole -  Linux  Open Source Solutions
 Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland
 marco.hugentob...@sourcepole.ch http://www.sourcepole.ch
 Technical Advisor QGIS Project Steering Committee




-- 
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==
Please do not email me off-list with technical
support questions. Using the lists will gain
more exposure for your issues and the knowledge
surrounding your issue will be shared with all.

Visit http://linfiniti.com to find out about:
 * QGIS programming and support services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux
Irc: timlinux on #qgis at freenode.net
==
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-10 Thread Marco Hugentobler
 I think the overhead for reading the aux.xml file is really minimal
 and the min/max population of that if its not there is quite quick and
 a once off event so I wouldnt think it is going to impact the server
 much. I'd suggest I first run some benchmarks to see how much overhead
 it adds.

But is there any downside from populating min/max on demand (e.g. like in 
QgsVectorDataProvider::minimumValue / maximum Value)? 

While it's nice to have a minimal overhead (probably buffered by the qgis 
server layer cache in many cases), it would be even better to have 0 overhead.

Regards,
Marco

 Am Mittwoch, 10. August 2011, 11.50:16 schrieb Tim Sutton:
 Hi
 
 On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler
 
 marco.hugentob...@sourcepole.ch wrote:
  Btw, is it possible to call computeMinMax() on demand, e.g. in
  QgsGDALProvider::minimumValue /maximumValue instead of the provider
  constructor?
  
  It probably does not matter much for the desktop experience. However, for
  server throughput, avoidance of opening and parsing the aux.xml file
  could make a difference.
 
 I think the overhead for reading the aux.xml file is really minimal
 and the min/max population of that if its not there is quite quick and
 a once off event so I wouldnt think it is going to impact the server
 much. I'd suggest I first run some benchmarks to see how much overhead
 it adds.
 
 Regards
 
 Tim
 
  Regards,
  Marco
  
  Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton:
  Hi Paolo
  
  Does it do that every time you load that layer or only the first time?
  Does the filesystem permissions allow writing of gdal .aux.xml files?
  
  My raster stats branch merge changed some behaviours and it is
  probably a side effect of that. One of the changes is that gdal gets
  full stats when trying to determine the min max of a band. It will
  fetch those stats quickly from aux.xml if present of if not its going
  to do a full stats gather just for the basic min max stats which is
  where you probably experience the slowness. I will see if I can find a
  better way to do the basic min/max stats gathering.
  
  Regards
  
  Tim
  
  On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it
  
  wrote:
   Hi all.
   With current master, I get bad performances in loading rasters (a
   12Mb takes 5 sec); when double-clicking it from the QGIS browser,
   it is far slower (20 sec). From the console:
   Warning: Object::connect: No such signal
   QgsGdalProvider::statusChanged( QString ) in
   /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any
   hint?
   All the best.
   --
   Paolo Cavallini: http://www.faunalia.it/pc
   ___
   Qgis-developer mailing list
   Qgis-developer@lists.osgeo.org
   http://lists.osgeo.org/mailman/listinfo/qgis-developer
  
  --
  Dr. Marco Hugentobler
  Sourcepole -  Linux  Open Source Solutions
  Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland
  marco.hugentob...@sourcepole.ch http://www.sourcepole.ch
  Technical Advisor QGIS Project Steering Committee


-- 
Dr. Marco Hugentobler
Sourcepole -  Linux  Open Source Solutions
Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland
marco.hugentob...@sourcepole.ch http://www.sourcepole.ch
Technical Advisor QGIS Project Steering Committee
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-10 Thread Tim Sutton
Hi

On Wed, Aug 10, 2011 at 12:25 PM, Marco Hugentobler
marco.hugentob...@sourcepole.ch wrote:
 I think the overhead for reading the aux.xml file is really minimal
 and the min/max population of that if its not there is quite quick and
 a once off event so I wouldnt think it is going to impact the server
 much. I'd suggest I first run some benchmarks to see how much overhead
 it adds.

 But is there any downside from populating min/max on demand (e.g. like in
 QgsVectorDataProvider::minimumValue / maximum Value)?

 While it's nice to have a minimal overhead (probably buffered by the qgis
 server layer cache in many cases), it would be even better to have 0 overhead.


Ya you make a good point. I will try update it to calculate on demand.

Regards

Tim

 Regards,
 Marco

 Am Mittwoch, 10. August 2011, 11.50:16 schrieb Tim Sutton:
 Hi

 On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler

 marco.hugentob...@sourcepole.ch wrote:
  Btw, is it possible to call computeMinMax() on demand, e.g. in
  QgsGDALProvider::minimumValue /maximumValue instead of the provider
  constructor?
 
  It probably does not matter much for the desktop experience. However, for
  server throughput, avoidance of opening and parsing the aux.xml file
  could make a difference.

 I think the overhead for reading the aux.xml file is really minimal
 and the min/max population of that if its not there is quite quick and
 a once off event so I wouldnt think it is going to impact the server
 much. I'd suggest I first run some benchmarks to see how much overhead
 it adds.

 Regards

 Tim

  Regards,
  Marco
 
  Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton:
  Hi Paolo
 
  Does it do that every time you load that layer or only the first time?
  Does the filesystem permissions allow writing of gdal .aux.xml files?
 
  My raster stats branch merge changed some behaviours and it is
  probably a side effect of that. One of the changes is that gdal gets
  full stats when trying to determine the min max of a band. It will
  fetch those stats quickly from aux.xml if present of if not its going
  to do a full stats gather just for the basic min max stats which is
  where you probably experience the slowness. I will see if I can find a
  better way to do the basic min/max stats gathering.
 
  Regards
 
  Tim
 
  On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it
 
  wrote:
   Hi all.
   With current master, I get bad performances in loading rasters (a
   12Mb takes 5 sec); when double-clicking it from the QGIS browser,
   it is far slower (20 sec). From the console:
   Warning: Object::connect: No such signal
   QgsGdalProvider::statusChanged( QString ) in
   /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any
   hint?
   All the best.
   --
   Paolo Cavallini: http://www.faunalia.it/pc
   ___
   Qgis-developer mailing list
   Qgis-developer@lists.osgeo.org
   http://lists.osgeo.org/mailman/listinfo/qgis-developer
 
  --
  Dr. Marco Hugentobler
  Sourcepole -  Linux  Open Source Solutions
  Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland
  marco.hugentob...@sourcepole.ch http://www.sourcepole.ch
  Technical Advisor QGIS Project Steering Committee


 --
 Dr. Marco Hugentobler
 Sourcepole -  Linux  Open Source Solutions
 Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland
 marco.hugentob...@sourcepole.ch http://www.sourcepole.ch
 Technical Advisor QGIS Project Steering Committee




-- 
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==
Please do not email me off-list with technical
support questions. Using the lists will gain
more exposure for your issues and the knowledge
surrounding your issue will be shared with all.

Visit http://linfiniti.com to find out about:
 * QGIS programming and support services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux
Irc: timlinux on #qgis at freenode.net
==
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-10 Thread Alexander Bruy
Hi all,

may be it is time to add to the GdalTools plugin new utility
that will calculate raster statistics? Something similar to
Assign projection or Build overviews.

Bye
-- 
Alexander Bruy
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-10 Thread Paolo Cavallini
Il 10/08/2011 12:49, Alexander Bruy ha scritto:

 may be it is time to add to the GdalTools plugin new utility
 that will calculate raster statistics? Something similar to
 Assign projection or Build overviews.

Good idea; ideally this should be done in batch, also recursively, so in most
situations it should be done shortly after installation.
One tip should be added suggesting users to run this tool, and possibly also 
adding
indexes to large shapefiles, to prepare all their data for fast loading.
All the best.
-- 
Paolo Cavallini: http://www.faunalia.it/pc
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-10 Thread Peter Borissow
Hi Tim-
    I understand that the stats are only gathered when needed. Still, I think 
users should have the option to compute stats and possibly other support files 
on load. This could be accomplished via a preference/setting or a pop-up 
dialog. In theory, you could have a thread pool to generate the support files 
in the background. The size of the thread pool could be determined by a user 
preference/setting.


Just a thought.

Take Care,

Peter




From: Tim Sutton li...@linfiniti.com
To: Peter Borissow peter.boris...@yahoo.com
Cc: Frank Warmerdam warmer...@pobox.com; qgis-developer 
qgis-developer@lists.osgeo.org
Sent: Wednesday, August 10, 2011 5:46 AM
Subject: Re: [Qgis-developer] slow raster loading

Hi

On Wed, Aug 10, 2011 at 12:20 AM, Peter Borissow
peter.boris...@yahoo.com wrote:
 Also, might be a good idea to spawn a separate thread to generate the aux
 file in the background...


It could be a possibility but the way things are set up now, stats are
only gathered when really needed, and if you need then for the
activity you are doing you would have to wait for them anyway. The
only merrit of using a separate thread would be if we decided to start
gathering complete stats on first load of a raster in the background
always so that they are preemptively there when the user does
something that needs stats. However I don't know if that is a good
idea - if you have many rasters in a project that need stats it will
spawn many threads and the IO / CPU overhead may be unpleasant.

Regards

Tim


 
 From: Frank Warmerdam warmer...@pobox.com
 To: Tim Sutton li...@linfiniti.com
 Cc: qgis-developer qgis-developer@lists.osgeo.org
 Sent: Tuesday, August 9, 2011 5:51 PM
 Subject: Re: [Qgis-developer] slow raster loading

 On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote:

 Yes thats why I was asking if the file permissions exist to write the
 aux.xml. If they don't behaviour should be the same as prior to my
 native raster stats commits i.e. stats will be recalced each time you
 open a project / add a layer and trigger something that needs the
 stats.

 Tim, and others,

 By the way, there is a mechanism in GDAL to support storing .aux.xml
 information in an alternative directory when the file cannot be
 created on the target directory due to permissions, read-only media, etc.

 This is enabled by setting the GDAL_PAM_PROXY_DIR option to
 point to a directory where files can be written.  If there is a place
 known to QGIS that would be good for this, it might make sense
 to take advantage of this.

 Best regards,
 --
 ---+--
 I set the clouds in motion - turn up   | Frank Warmerdam,
 warmer...@pobox.com
 light and sound - activate the windows | http://pobox.com/~warmerdam
 and watch the world go round - Rush    | Geospatial Software Developer
 ___
 Qgis-developer mailing list
 Qgis-developer@lists.osgeo.org
 http://lists.osgeo.org/mailman/listinfo/qgis-developer






-- 
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==
Please do not email me off-list with technical
support questions. Using the lists will gain
more exposure for your issues and the knowledge
surrounding your issue will be shared with all.

Visit http://linfiniti.com to find out about:
 * QGIS programming and support services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux
Irc: timlinux on #qgis at freenode.net
==___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-10 Thread Etienne Tourigny
Hi all,

I had noticed the slow loading you refer to, when I got a copy of the master 
branch last week,  but it seems to be fixed now.

If you opened a raster from the command-line (i.e. qgis file.tif), the raster 
stats were fetched (and stored in the .aux.xml file) automatically, hence the 
delay.


Now the proper behaviour is happening, i.e. the stats are only calculated upon 
demand (such as when opening the histogram tab).


The Object::connect: No such signal problem is unrelated, and I have been 
seeing it ever since using 1.7.  This should definitely get fixed...




- Original Message -
From: Giovanni Manghi giovanni.man...@gmail.com
To: Tim Sutton li...@linfiniti.com
Cc: qgis-developer qgis-developer@lists.osgeo.org
Sent: Tuesday, August 9, 2011 8:57:48 PM
Subject: Re: [Qgis-developer] slow raster loading

Hi Tim,


 This may be fixed via a patch given to me by Etienne Sky - please let
 me know if it still does it.


just tested your last commit and I still see

Warning: Object::connect: No such signal
QgsGdalProvider::statusChanged( QString )
in /home/gio/qgis-Quantum-GIS-4b4820e/src/core/raster/qgsrasterlayer.cpp:2446


 Yes thats why I was asking if the file permissions exist to write the
 aux.xml. If they don't behaviour should be the same as prior to my
 native raster stats commits i.e. stats will be recalced each time you
 open a project / add a layer and trigger something that needs the
 stats.


I copied a geotiff in a folder with no write permissions and it took a
lot of time to open the raster, much more than it used to take when
opening the same image the first time in a folder with write
permissions.

Then I compiled QGIS with your last commit and now it seems it is not
collecting stats/creating the aux.xml file anymore.

cheers

-- Giovanni --

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-09 Thread Giovanni Manghi
Hi Paolo, Tim

unfortunately in the last months I had a very little time to follow the
developments, so maybe I will say obvious things.

I just compiled QGIS master under Ubuntu against GDAL 1.8 and when
loading a raster I see the same message

Warning: Object::connect: No such signal
QgsGdalProvider::statusChanged( QString )
in /home/gio/qgis-Quantum-GIS-6a1073f/src/core/raster/qgsrasterlayer.cpp:2446

the practical difference I see is that, as Tim describes, that rasters
are slower to open the first time, but then are faster as before, if not
more. This seems to me not a big deal when opening quite small images,
but it can be when opening multiple images at one time or when adding
very big rasters.

For instance I have 3GB landsat images with overviews that before these
changes opened in a nick of time, now -the first time- they take a few
seconds (10/15).

The same images without overviews before the changes took 10/15 seconds
to open, now -the first time- it takes much more.

Another issue I guess can be if the rasters are stored in a place where
the aux.xml cannot be written.

I guess that the right behaviour is the new one, but maybe it is worth
add an option to avoid collecting stats on raster load?

cheers

-- Giovanni --



On Tue, 2011-08-09 at 15:49 +0200, Paolo Cavallini wrote:
 Hi all.
 With current master, I get bad performances in loading rasters (a 12Mb takes 
 5
 sec); when double-clicking it from the QGIS browser, it is far slower (20 
 sec). From
 the console:
 Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( 
 QString ) in
 /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446
 Any hint?
 All the bes

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-09 Thread Frank Warmerdam
On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote:

 Yes thats why I was asking if the file permissions exist to write the
 aux.xml. If they don't behaviour should be the same as prior to my
 native raster stats commits i.e. stats will be recalced each time you
 open a project / add a layer and trigger something that needs the
 stats.

Tim, and others,

By the way, there is a mechanism in GDAL to support storing .aux.xml
information in an alternative directory when the file cannot be
created on the target directory due to permissions, read-only media, etc.

This is enabled by setting the GDAL_PAM_PROXY_DIR option to
point to a directory where files can be written.   If there is a place
known to QGIS that would be good for this, it might make sense
to take advantage of this.

Best regards,
-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-09 Thread Peter Borissow
Also, might be a good idea to spawn a separate thread to generate the aux file 
in the background...





From: Frank Warmerdam warmer...@pobox.com
To: Tim Sutton li...@linfiniti.com
Cc: qgis-developer qgis-developer@lists.osgeo.org
Sent: Tuesday, August 9, 2011 5:51 PM
Subject: Re: [Qgis-developer] slow raster loading

On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote:

 Yes thats why I was asking if the file permissions exist to write the
 aux.xml. If they don't behaviour should be the same as prior to my
 native raster stats commits i.e. stats will be recalced each time you
 open a project / add a layer and trigger something that needs the
 stats.

Tim, and others,

By the way, there is a mechanism in GDAL to support storing .aux.xml
information in an alternative directory when the file cannot be
created on the target directory due to permissions, read-only media, etc.

This is enabled by setting the GDAL_PAM_PROXY_DIR option to
point to a directory where files can be written.   If there is a place
known to QGIS that would be good for this, it might make sense
to take advantage of this.

Best regards,
-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


Re: [Qgis-developer] slow raster loading

2011-08-09 Thread Giovanni Manghi
Hi Tim,


 This may be fixed via a patch given to me by Etienne Sky - please let
 me know if it still does it.


just tested your last commit and I still see

Warning: Object::connect: No such signal
QgsGdalProvider::statusChanged( QString )
in /home/gio/qgis-Quantum-GIS-4b4820e/src/core/raster/qgsrasterlayer.cpp:2446


 Yes thats why I was asking if the file permissions exist to write the
 aux.xml. If they don't behaviour should be the same as prior to my
 native raster stats commits i.e. stats will be recalced each time you
 open a project / add a layer and trigger something that needs the
 stats.


I copied a geotiff in a folder with no write permissions and it took a
lot of time to open the raster, much more than it used to take when
opening the same image the first time in a folder with write
permissions.

Then I compiled QGIS with your last commit and now it seems it is not
collecting stats/creating the aux.xml file anymore.

cheers

-- Giovanni --

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer