Re: [Qgis-developer] Slow raster loading
Hi Paolo, Tim Marco, any idea about when this will happen? I've pushed the raster_resampler branch into the qgis/Quantum-GIS repository. Like this, developers can directly make changes in it. I think it is important to do enough testing before integrating into master branch. Tim already found and fixed an issue, I'm sure there are more issues around. On the other hand, once people are used to the average resampling for zoomed-out rasters, I'm sure they don't like to go back to 'nearest neighbour'. Regards, Marco On 15.02.2012 08:25, Paolo Cavallini wrote: Il 09/02/2012 10:03, Tim Sutton ha scritto: I would suggest that we wait until after Marco's resampling branch is merged before looking at these things as many parts of raster implementation are changing with it. quite sensible - Marco, any idea about when this will happen? Thanks. -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Slow raster loading
Il 09/02/2012 10:03, Tim Sutton ha scritto: I would suggest that we wait until after Marco's resampling branch is merged before looking at these things as many parts of raster implementation are changing with it. quite sensible - Marco, any idea about when this will happen? Thanks. -- Paolo Cavallini - Faunalia www.faunalia.eu Full contact details at www.faunalia.eu/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Slow raster loading
Hi Tim, *) computing of stats takes a long time, it is easy to see that also with not that big rasters it takes a lot, especially if multi-band, hurting the user experience. The proposal is to make computing of stats optional. Computing stats should be a once off event since I adapted the gdal provider to use gdal native stats. Is your experience on first load or on subsequent loads too? subsequent loads are fast, with the exception of GRASS rasters, probably because stats are not written anywhere. computing rasterlite stats is very slow if compared with all other formats. I agree with others that when computing stats should at least show a progress bar, but (also) having them computed optionally would be even better. When stretching is active standard deviation is enabled is set to 2 by default. In my experience this is causing to render rasters with strange colormaps, while a smaller value (1.7) seems to give better results. I am not sure it is so easy to avoid collecting stats. We used to have some logic that set stats to be only collected when they were needed, I'll need to check if that is still the case or not (but I suspect not). *) in the raster properties there are two properties (contrast enhancement and standard deviation) that have a button to save the option in a way to allow the setting be persistent among qgis sessions. This obviously should be moved to the general options, allowing to override at a layer level. That makes sense to me. ok cheers -- Giovanni -- ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Slow raster loading
Hi On Mon, Jan 30, 2012 at 5:07 PM, Giovanni Manghi giovanni.man...@gmail.com wrote: Hi all, On Mon, 2012-01-30 at 13:16 +0100, Paolo Cavallini wrote: Hi all. Currently, loading coloured rasters with stretching to minmax on is extremely slow (almost unusable): http://hub.qgis.org/issues/4421 E.g. I could load a large RasterLite in 1 sec instead of minutes (!). The trick, from the user point of view, is to disable stretching by default, and enable it only for rasters where the option makes sense (e.g. a monoband, greyscale). Obviously we could do it better, disabling the stretch for the rasters where it does not make sense. In addition, Giovanni noticed several other small bugs. my notes are in the ticket. There a few bugs (see ticket) and a couple of things need to be changed: *) computing of stats takes a long time, it is easy to see that also with not that big rasters it takes a lot, especially if multi-band, hurting the user experience. The proposal is to make computing of stats optional. Computing stats should be a once off event since I adapted the gdal provider to use gdal native stats. Is your experience on first load or on subsequent loads too? I am not sure it is so easy to avoid collecting stats. We used to have some logic that set stats to be only collected when they were needed, I'll need to check if that is still the case or not (but I suspect not). *) in the raster properties there are two properties (contrast enhancement and standard deviation) that have a button to save the option in a way to allow the setting be persistent among qgis sessions. This obviously should be moved to the general options, allowing to override at a layer level. That makes sense to me. I would suggest that we wait until after Marco's resampling branch is merged before looking at these things as many parts of raster implementation are changing with it. Regards Tim cheers -- Giovanni -- ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Slow raster loading
On Thu, Feb 9, 2012 at 7:03 PM, Tim Sutton li...@linfiniti.com wrote: Computing stats should be a once off event since I adapted the gdal provider to use gdal native stats. Is your experience on first load or on subsequent loads too? I am not sure it is so easy to avoid collecting stats. We used to have some logic that set stats to be only collected when they were needed, I'll need to check if that is still the case or not (but I suspect not). Tim, IMO we would should really thread these operations, and report the progress to the user in a friendly cancel-able way. If the user doesn't care about stats, which to me when loading something that is just for viewing is most of the time, QGIS should just say stats not loaded and show the file anyway. If you want to get the stats again we should have a recalculate stats button, with threading and progress bar. I had a person email me today (a council trying to me to QGIS) about trying to load a 2gb ecw and it just locks up the UI with no idea of how much is left to be done or what is going on. I have experienced the same thing but managed to get around it by making a fake stats file, QGIS then just opens thinking the stats are fine. - Nathan ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Slow raster loading
Le 09/02/2012 10:38, Nathan Woodrow a écrit : I had a person email me today (a council trying to me to QGIS) about trying to load a 2gb ecw and it just locks up the UI with no idea of how much is left to be done or what is going on. I have experienced the same thing but managed to get around it by making a fake stats file, QGIS then just opens thinking the stats are fine. Oh I'm often getting a similar kind of report : the user opens the properties, clicks on the wrong tab - too bad the file is a huge pct raster, kill your project. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Slow raster loading
I'm having the same problem when loading large MrSID maps. I don't recall QGIS reading/building the stats file in earlier versions. Is this necessary? On 02/09/2012 11:51 AM, MORREALE Jean Roc wrote: Le 09/02/2012 10:38, Nathan Woodrow a crit : I had a person email me today (a council trying to me to QGIS) about trying to load a 2gb ecw and it just locks up the UI with no idea of how much is left to be done or what is going on. I have experienced the same thing but managed to get around it by making a fake stats file, QGIS then just opens thinking the stats are fine. Oh I'm often getting a similar kind of report : the user opens the properties, clicks on the wrong tab - too bad the file is a huge pct raster, kill your project. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Slow raster loading
Hi all, On Mon, 2012-01-30 at 13:16 +0100, Paolo Cavallini wrote: Hi all. Currently, loading coloured rasters with stretching to minmax on is extremely slow (almost unusable): http://hub.qgis.org/issues/4421 E.g. I could load a large RasterLite in 1 sec instead of minutes (!). The trick, from the user point of view, is to disable stretching by default, and enable it only for rasters where the option makes sense (e.g. a monoband, greyscale). Obviously we could do it better, disabling the stretch for the rasters where it does not make sense. In addition, Giovanni noticed several other small bugs. my notes are in the ticket. There a few bugs (see ticket) and a couple of things need to be changed: *) computing of stats takes a long time, it is easy to see that also with not that big rasters it takes a lot, especially if multi-band, hurting the user experience. The proposal is to make computing of stats optional. *) in the raster properties there are two properties (contrast enhancement and standard deviation) that have a button to save the option in a way to allow the setting be persistent among qgis sessions. This obviously should be moved to the general options, allowing to override at a layer level. cheers -- Giovanni -- ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi Tim, They should be. Try going to the histogram tab of your raster properties or do something else that requires stats. In many cases, stats are not needed so you may not be seeing thm gathered. Also for read only medai, the stats should land up in the proxy dir which is set to ~/.qgis/gdal_pam. Otherwise can you give me a specific procedure to follow to replicate? sorry for the noise, yes it works ok. Cheers -- Giovanni -- ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi Tim, I'm wrong or after the last commits QGIS is not gathering raster stats anymore? At least in my box the aux.xml files are not created anywhere. Cheers -- Giovanni -- On Wed, 2011-08-10 at 11:50 +0200, Tim Sutton wrote: Hi On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: Btw, is it possible to call computeMinMax() on demand, e.g. in QgsGDALProvider::minimumValue /maximumValue instead of the provider constructor? It probably does not matter much for the desktop experience. However, for server throughput, avoidance of opening and parsing the aux.xml file could make a difference. I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. Regards Tim Regards, Marco Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton: Hi Paolo Does it do that every time you load that layer or only the first time? Does the filesystem permissions allow writing of gdal .aux.xml files? My raster stats branch merge changed some behaviours and it is probably a side effect of that. One of the changes is that gdal gets full stats when trying to determine the min max of a band. It will fetch those stats quickly from aux.xml if present of if not its going to do a full stats gather just for the basic min max stats which is where you probably experience the slowness. I will see if I can find a better way to do the basic min/max stats gathering. Regards Tim On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Btw, is it possible to call computeMinMax() on demand, e.g. in QgsGDALProvider::minimumValue /maximumValue instead of the provider constructor? It probably does not matter much for the desktop experience. However, for server throughput, avoidance of opening and parsing the aux.xml file could make a difference. Regards, Marco Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton: Hi Paolo Does it do that every time you load that layer or only the first time? Does the filesystem permissions allow writing of gdal .aux.xml files? My raster stats branch merge changed some behaviours and it is probably a side effect of that. One of the changes is that gdal gets full stats when trying to determine the min max of a band. It will fetch those stats quickly from aux.xml if present of if not its going to do a full stats gather just for the basic min max stats which is where you probably experience the slowness. I will see if I can find a better way to do the basic min/max stats gathering. Regards Tim On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi On Wed, Aug 10, 2011 at 12:20 AM, Peter Borissow peter.boris...@yahoo.com wrote: Also, might be a good idea to spawn a separate thread to generate the aux file in the background... It could be a possibility but the way things are set up now, stats are only gathered when really needed, and if you need then for the activity you are doing you would have to wait for them anyway. The only merrit of using a separate thread would be if we decided to start gathering complete stats on first load of a raster in the background always so that they are preemptively there when the user does something that needs stats. However I don't know if that is a good idea - if you have many rasters in a project that need stats it will spawn many threads and the IO / CPU overhead may be unpleasant. Regards Tim From: Frank Warmerdam warmer...@pobox.com To: Tim Sutton li...@linfiniti.com Cc: qgis-developer qgis-developer@lists.osgeo.org Sent: Tuesday, August 9, 2011 5:51 PM Subject: Re: [Qgis-developer] slow raster loading On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote: Yes thats why I was asking if the file permissions exist to write the aux.xml. If they don't behaviour should be the same as prior to my native raster stats commits i.e. stats will be recalced each time you open a project / add a layer and trigger something that needs the stats. Tim, and others, By the way, there is a mechanism in GDAL to support storing .aux.xml information in an alternative directory when the file cannot be created on the target directory due to permissions, read-only media, etc. This is enabled by setting the GDAL_PAM_PROXY_DIR option to point to a directory where files can be written. If there is a place known to QGIS that would be good for this, it might make sense to take advantage of this. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | Geospatial Software Developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: Btw, is it possible to call computeMinMax() on demand, e.g. in QgsGDALProvider::minimumValue /maximumValue instead of the provider constructor? It probably does not matter much for the desktop experience. However, for server throughput, avoidance of opening and parsing the aux.xml file could make a difference. I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. Regards Tim Regards, Marco Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton: Hi Paolo Does it do that every time you load that layer or only the first time? Does the filesystem permissions allow writing of gdal .aux.xml files? My raster stats branch merge changed some behaviours and it is probably a side effect of that. One of the changes is that gdal gets full stats when trying to determine the min max of a band. It will fetch those stats quickly from aux.xml if present of if not its going to do a full stats gather just for the basic min max stats which is where you probably experience the slowness. I will see if I can find a better way to do the basic min/max stats gathering. Regards Tim On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. But is there any downside from populating min/max on demand (e.g. like in QgsVectorDataProvider::minimumValue / maximum Value)? While it's nice to have a minimal overhead (probably buffered by the qgis server layer cache in many cases), it would be even better to have 0 overhead. Regards, Marco Am Mittwoch, 10. August 2011, 11.50:16 schrieb Tim Sutton: Hi On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: Btw, is it possible to call computeMinMax() on demand, e.g. in QgsGDALProvider::minimumValue /maximumValue instead of the provider constructor? It probably does not matter much for the desktop experience. However, for server throughput, avoidance of opening and parsing the aux.xml file could make a difference. I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. Regards Tim Regards, Marco Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton: Hi Paolo Does it do that every time you load that layer or only the first time? Does the filesystem permissions allow writing of gdal .aux.xml files? My raster stats branch merge changed some behaviours and it is probably a side effect of that. One of the changes is that gdal gets full stats when trying to determine the min max of a band. It will fetch those stats quickly from aux.xml if present of if not its going to do a full stats gather just for the basic min max stats which is where you probably experience the slowness. I will see if I can find a better way to do the basic min/max stats gathering. Regards Tim On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi On Wed, Aug 10, 2011 at 12:25 PM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. But is there any downside from populating min/max on demand (e.g. like in QgsVectorDataProvider::minimumValue / maximum Value)? While it's nice to have a minimal overhead (probably buffered by the qgis server layer cache in many cases), it would be even better to have 0 overhead. Ya you make a good point. I will try update it to calculate on demand. Regards Tim Regards, Marco Am Mittwoch, 10. August 2011, 11.50:16 schrieb Tim Sutton: Hi On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: Btw, is it possible to call computeMinMax() on demand, e.g. in QgsGDALProvider::minimumValue /maximumValue instead of the provider constructor? It probably does not matter much for the desktop experience. However, for server throughput, avoidance of opening and parsing the aux.xml file could make a difference. I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. Regards Tim Regards, Marco Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton: Hi Paolo Does it do that every time you load that layer or only the first time? Does the filesystem permissions allow writing of gdal .aux.xml files? My raster stats branch merge changed some behaviours and it is probably a side effect of that. One of the changes is that gdal gets full stats when trying to determine the min max of a band. It will fetch those stats quickly from aux.xml if present of if not its going to do a full stats gather just for the basic min max stats which is where you probably experience the slowness. I will see if I can find a better way to do the basic min/max stats gathering. Regards Tim On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi all, may be it is time to add to the GdalTools plugin new utility that will calculate raster statistics? Something similar to Assign projection or Build overviews. Bye -- Alexander Bruy ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Il 10/08/2011 12:49, Alexander Bruy ha scritto: may be it is time to add to the GdalTools plugin new utility that will calculate raster statistics? Something similar to Assign projection or Build overviews. Good idea; ideally this should be done in batch, also recursively, so in most situations it should be done shortly after installation. One tip should be added suggesting users to run this tool, and possibly also adding indexes to large shapefiles, to prepare all their data for fast loading. All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi Tim- I understand that the stats are only gathered when needed. Still, I think users should have the option to compute stats and possibly other support files on load. This could be accomplished via a preference/setting or a pop-up dialog. In theory, you could have a thread pool to generate the support files in the background. The size of the thread pool could be determined by a user preference/setting. Just a thought. Take Care, Peter From: Tim Sutton li...@linfiniti.com To: Peter Borissow peter.boris...@yahoo.com Cc: Frank Warmerdam warmer...@pobox.com; qgis-developer qgis-developer@lists.osgeo.org Sent: Wednesday, August 10, 2011 5:46 AM Subject: Re: [Qgis-developer] slow raster loading Hi On Wed, Aug 10, 2011 at 12:20 AM, Peter Borissow peter.boris...@yahoo.com wrote: Also, might be a good idea to spawn a separate thread to generate the aux file in the background... It could be a possibility but the way things are set up now, stats are only gathered when really needed, and if you need then for the activity you are doing you would have to wait for them anyway. The only merrit of using a separate thread would be if we decided to start gathering complete stats on first load of a raster in the background always so that they are preemptively there when the user does something that needs stats. However I don't know if that is a good idea - if you have many rasters in a project that need stats it will spawn many threads and the IO / CPU overhead may be unpleasant. Regards Tim From: Frank Warmerdam warmer...@pobox.com To: Tim Sutton li...@linfiniti.com Cc: qgis-developer qgis-developer@lists.osgeo.org Sent: Tuesday, August 9, 2011 5:51 PM Subject: Re: [Qgis-developer] slow raster loading On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote: Yes thats why I was asking if the file permissions exist to write the aux.xml. If they don't behaviour should be the same as prior to my native raster stats commits i.e. stats will be recalced each time you open a project / add a layer and trigger something that needs the stats. Tim, and others, By the way, there is a mechanism in GDAL to support storing .aux.xml information in an alternative directory when the file cannot be created on the target directory due to permissions, read-only media, etc. This is enabled by setting the GDAL_PAM_PROXY_DIR option to point to a directory where files can be written. If there is a place known to QGIS that would be good for this, it might make sense to take advantage of this. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | Geospatial Software Developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net ==___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi all, I had noticed the slow loading you refer to, when I got a copy of the master branch last week, but it seems to be fixed now. If you opened a raster from the command-line (i.e. qgis file.tif), the raster stats were fetched (and stored in the .aux.xml file) automatically, hence the delay. Now the proper behaviour is happening, i.e. the stats are only calculated upon demand (such as when opening the histogram tab). The Object::connect: No such signal problem is unrelated, and I have been seeing it ever since using 1.7. This should definitely get fixed... - Original Message - From: Giovanni Manghi giovanni.man...@gmail.com To: Tim Sutton li...@linfiniti.com Cc: qgis-developer qgis-developer@lists.osgeo.org Sent: Tuesday, August 9, 2011 8:57:48 PM Subject: Re: [Qgis-developer] slow raster loading Hi Tim, This may be fixed via a patch given to me by Etienne Sky - please let me know if it still does it. just tested your last commit and I still see Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /home/gio/qgis-Quantum-GIS-4b4820e/src/core/raster/qgsrasterlayer.cpp:2446 Yes thats why I was asking if the file permissions exist to write the aux.xml. If they don't behaviour should be the same as prior to my native raster stats commits i.e. stats will be recalced each time you open a project / add a layer and trigger something that needs the stats. I copied a geotiff in a folder with no write permissions and it took a lot of time to open the raster, much more than it used to take when opening the same image the first time in a folder with write permissions. Then I compiled QGIS with your last commit and now it seems it is not collecting stats/creating the aux.xml file anymore. cheers -- Giovanni -- ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi Paolo, Tim unfortunately in the last months I had a very little time to follow the developments, so maybe I will say obvious things. I just compiled QGIS master under Ubuntu against GDAL 1.8 and when loading a raster I see the same message Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /home/gio/qgis-Quantum-GIS-6a1073f/src/core/raster/qgsrasterlayer.cpp:2446 the practical difference I see is that, as Tim describes, that rasters are slower to open the first time, but then are faster as before, if not more. This seems to me not a big deal when opening quite small images, but it can be when opening multiple images at one time or when adding very big rasters. For instance I have 3GB landsat images with overviews that before these changes opened in a nick of time, now -the first time- they take a few seconds (10/15). The same images without overviews before the changes took 10/15 seconds to open, now -the first time- it takes much more. Another issue I guess can be if the rasters are stored in a place where the aux.xml cannot be written. I guess that the right behaviour is the new one, but maybe it is worth add an option to avoid collecting stats on raster load? cheers -- Giovanni -- On Tue, 2011-08-09 at 15:49 +0200, Paolo Cavallini wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the bes ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote: Yes thats why I was asking if the file permissions exist to write the aux.xml. If they don't behaviour should be the same as prior to my native raster stats commits i.e. stats will be recalced each time you open a project / add a layer and trigger something that needs the stats. Tim, and others, By the way, there is a mechanism in GDAL to support storing .aux.xml information in an alternative directory when the file cannot be created on the target directory due to permissions, read-only media, etc. This is enabled by setting the GDAL_PAM_PROXY_DIR option to point to a directory where files can be written. If there is a place known to QGIS that would be good for this, it might make sense to take advantage of this. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | Geospatial Software Developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Also, might be a good idea to spawn a separate thread to generate the aux file in the background... From: Frank Warmerdam warmer...@pobox.com To: Tim Sutton li...@linfiniti.com Cc: qgis-developer qgis-developer@lists.osgeo.org Sent: Tuesday, August 9, 2011 5:51 PM Subject: Re: [Qgis-developer] slow raster loading On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote: Yes thats why I was asking if the file permissions exist to write the aux.xml. If they don't behaviour should be the same as prior to my native raster stats commits i.e. stats will be recalced each time you open a project / add a layer and trigger something that needs the stats. Tim, and others, By the way, there is a mechanism in GDAL to support storing .aux.xml information in an alternative directory when the file cannot be created on the target directory due to permissions, read-only media, etc. This is enabled by setting the GDAL_PAM_PROXY_DIR option to point to a directory where files can be written. If there is a place known to QGIS that would be good for this, it might make sense to take advantage of this. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | Geospatial Software Developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi Tim, This may be fixed via a patch given to me by Etienne Sky - please let me know if it still does it. just tested your last commit and I still see Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /home/gio/qgis-Quantum-GIS-4b4820e/src/core/raster/qgsrasterlayer.cpp:2446 Yes thats why I was asking if the file permissions exist to write the aux.xml. If they don't behaviour should be the same as prior to my native raster stats commits i.e. stats will be recalced each time you open a project / add a layer and trigger something that needs the stats. I copied a geotiff in a folder with no write permissions and it took a lot of time to open the raster, much more than it used to take when opening the same image the first time in a folder with write permissions. Then I compiled QGIS with your last commit and now it seems it is not collecting stats/creating the aux.xml file anymore. cheers -- Giovanni -- ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
