Re: [QGIS-Developer] Caching of raster images

[Mathieu Pellerin]

You can also build your VRT using QGIS' processing toolbox.

On Oct 5, 2017 7:26 PM, "Régis Haubourg"  wrote:

> GDAL is probably installed on your system, you need it for QGIS :)
>
> something like this should do the trick in a linux bash or from the
> osgeo4w embedded shells (either windows or msys )
> gdalbuildvrt all_ecw.vrt *.tiff -a_srs "EPSG:yourSridHere"
> See http://geoserver.geo-solutions.it/edu/en/raster_
> data/advanced_gdal/example1.html for instance
>
>
>
>
> 2017-10-05 14:19 GMT+02:00 Patrick Dunford :
>
>> I probably don't have the technical means to use that.
>>
>> On 06/10/17 01:12, Régis Haubourg wrote:
>>
>> here it is: http://www.gdal.org/gdal_vrttut.html
>>
>> you can assemble raster of vector files, apply filters, reprojections,
>> build pyramids to avoid fetching the lowest resolution data. This is pretty
>> efficient.
>>
>> Cheers
>> Régis
>>
>> 2017-10-05 14:10 GMT+02:00 Patrick Dunford :
>>
>>> I don't know what this is.
>>>
>>> On 06/10/17 01:08, Régis Haubourg wrote:
>>>
>>> Hi Patrick ,
>>>  did you consider using GDAL VRT to avoid opening so much files?
>>> In my experience, this works well.
>>> Régis
>>>
>>>
>>> 2017-10-05 13:48 GMT+02:00 Patrick Dunford :
>>>
 Some time ago in a discussion of a particular bug a contributor
 expressed concern that the refresh of background rasters (aerial
 photography) in Windows was too slow.

 Maybe this is the reason that recent versions of master appear to be
 loading all of the background imagery into memory (I use a master from
 January this year to work around issues with later ones, and that master
 does not have this feature).

 Unfortunately if there are a lot of rasters then the memory demand is
 excessive and unsurprisingly slows down the computer negating any purported
 benefit of caching.

 As an example a project I am currently working on has about 900 aerial
 photo images (GeoJpeg). When the layer is turned on for display, Qgis
 requires about 46 GB of virtual memory. Since my computer only has 24 GB of
 physical memory, it is required to dip into the swap space considerably.
 Even with 60 GB of swap space on an SSD, the swapping needed to refresh the
 canvas is substantial and dramatically reduces performance resulting in
 substantial delays. Compare with the January master referred to above which
 only requires about 7 GB of virtual memory total with the aerial photo
 layer displayed. The time needed to refresh the canvas is less than 1
 second, most of the time.

 I know that the canvas refresh in Windows with aerial photos can be
 substantially slower than in Linux. This does not affect me, because I
 don't use Windows now that I have a stable platform for running an older
 Linux master alongside the most recent one. What I do know is that the
 memory demands are making it difficult to evaluate the recent masters. I
 need some kind of setting to turn this caching off. With the aerial photo
 layer turned off, the memory usage of the current master is about the same
 as the old one, and it's much quicker to update.


 ___
 QGIS-Developer mailing list
 QGIS-Developer@lists.osgeo.org
 List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
 Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>>
>>>
>>>
>>>
>>
>>
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Caching of raster images

[Régis Haubourg]

GDAL is probably installed on your system, you need it for QGIS :)

something like this should do the trick in a linux bash or from the osgeo4w
embedded shells (either windows or msys )
gdalbuildvrt all_ecw.vrt *.tiff -a_srs "EPSG:yourSridHere"
See
http://geoserver.geo-solutions.it/edu/en/raster_data/advanced_gdal/example1.html
for instance




2017-10-05 14:19 GMT+02:00 Patrick Dunford :

> I probably don't have the technical means to use that.
>
> On 06/10/17 01:12, Régis Haubourg wrote:
>
> here it is: http://www.gdal.org/gdal_vrttut.html
>
> you can assemble raster of vector files, apply filters, reprojections,
> build pyramids to avoid fetching the lowest resolution data. This is pretty
> efficient.
>
> Cheers
> Régis
>
> 2017-10-05 14:10 GMT+02:00 Patrick Dunford :
>
>> I don't know what this is.
>>
>> On 06/10/17 01:08, Régis Haubourg wrote:
>>
>> Hi Patrick ,
>>  did you consider using GDAL VRT to avoid opening so much files?
>> In my experience, this works well.
>> Régis
>>
>>
>> 2017-10-05 13:48 GMT+02:00 Patrick Dunford :
>>
>>> Some time ago in a discussion of a particular bug a contributor
>>> expressed concern that the refresh of background rasters (aerial
>>> photography) in Windows was too slow.
>>>
>>> Maybe this is the reason that recent versions of master appear to be
>>> loading all of the background imagery into memory (I use a master from
>>> January this year to work around issues with later ones, and that master
>>> does not have this feature).
>>>
>>> Unfortunately if there are a lot of rasters then the memory demand is
>>> excessive and unsurprisingly slows down the computer negating any purported
>>> benefit of caching.
>>>
>>> As an example a project I am currently working on has about 900 aerial
>>> photo images (GeoJpeg). When the layer is turned on for display, Qgis
>>> requires about 46 GB of virtual memory. Since my computer only has 24 GB of
>>> physical memory, it is required to dip into the swap space considerably.
>>> Even with 60 GB of swap space on an SSD, the swapping needed to refresh the
>>> canvas is substantial and dramatically reduces performance resulting in
>>> substantial delays. Compare with the January master referred to above which
>>> only requires about 7 GB of virtual memory total with the aerial photo
>>> layer displayed. The time needed to refresh the canvas is less than 1
>>> second, most of the time.
>>>
>>> I know that the canvas refresh in Windows with aerial photos can be
>>> substantially slower than in Linux. This does not affect me, because I
>>> don't use Windows now that I have a stable platform for running an older
>>> Linux master alongside the most recent one. What I do know is that the
>>> memory demands are making it difficult to evaluate the recent masters. I
>>> need some kind of setting to turn this caching off. With the aerial photo
>>> layer turned off, the memory usage of the current master is about the same
>>> as the old one, and it's much quicker to update.
>>>
>>>
>>> ___
>>> QGIS-Developer mailing list
>>> QGIS-Developer@lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>>
>>
>>
>
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Caching of raster images

[Patrick Dunford]

I probably don't have the technical means to use that.


On 06/10/17 01:12, Régis Haubourg wrote:

here it is: http://www.gdal.org/gdal_vrttut.html

you can assemble raster of vector files, apply filters, reprojections, 
build pyramids to avoid fetching the lowest resolution data. This is 
pretty efficient.


Cheers
Régis

2017-10-05 14:10 GMT+02:00 Patrick Dunford >:


I don't know what this is.


On 06/10/17 01:08, Régis Haubourg wrote:

Hi Patrick ,
 did you consider using GDAL VRT to avoid opening so much files?
In my experience, this works well.
Régis


2017-10-05 13:48 GMT+02:00 Patrick Dunford
mailto:blackwhite...@gmail.com>>:

Some time ago in a discussion of a particular bug a
contributor expressed concern that the refresh of background
rasters (aerial photography) in Windows was too slow.

Maybe this is the reason that recent versions of master
appear to be loading all of the background imagery into
memory (I use a master from January this year to work around
issues with later ones, and that master does not have this
feature).

Unfortunately if there are a lot of rasters then the memory
demand is excessive and unsurprisingly slows down the
computer negating any purported benefit of caching.

As an example a project I am currently working on has about
900 aerial photo images (GeoJpeg). When the layer is turned
on for display, Qgis requires about 46 GB of virtual memory.
Since my computer only has 24 GB of physical memory, it is
required to dip into the swap space considerably. Even with
60 GB of swap space on an SSD, the swapping needed to refresh
the canvas is substantial and dramatically reduces
performance resulting in substantial delays. Compare with the
January master referred to above which only requires about 7
GB of virtual memory total with the aerial photo layer
displayed. The time needed to refresh the canvas is less than
1 second, most of the time.

I know that the canvas refresh in Windows with aerial photos
can be substantially slower than in Linux. This does not
affect me, because I don't use Windows now that I have a
stable platform for running an older Linux master alongside
the most recent one. What I do know is that the memory
demands are making it difficult to evaluate the recent
masters. I need some kind of setting to turn this caching
off. With the aerial photo layer turned off, the memory usage
of the current master is about the same as the old one, and
it's much quicker to update.


___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org

List info:
https://lists.osgeo.org/mailman/listinfo/qgis-developer

Unsubscribe:
https://lists.osgeo.org/mailman/listinfo/qgis-developer








___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Caching of raster images

[Régis Haubourg]

here it is: http://www.gdal.org/gdal_vrttut.html

you can assemble raster of vector files, apply filters, reprojections,
build pyramids to avoid fetching the lowest resolution data. This is pretty
efficient.

Cheers
Régis

2017-10-05 14:10 GMT+02:00 Patrick Dunford :

> I don't know what this is.
>
> On 06/10/17 01:08, Régis Haubourg wrote:
>
> Hi Patrick ,
>  did you consider using GDAL VRT to avoid opening so much files?
> In my experience, this works well.
> Régis
>
>
> 2017-10-05 13:48 GMT+02:00 Patrick Dunford :
>
>> Some time ago in a discussion of a particular bug a contributor expressed
>> concern that the refresh of background rasters (aerial photography) in
>> Windows was too slow.
>>
>> Maybe this is the reason that recent versions of master appear to be
>> loading all of the background imagery into memory (I use a master from
>> January this year to work around issues with later ones, and that master
>> does not have this feature).
>>
>> Unfortunately if there are a lot of rasters then the memory demand is
>> excessive and unsurprisingly slows down the computer negating any purported
>> benefit of caching.
>>
>> As an example a project I am currently working on has about 900 aerial
>> photo images (GeoJpeg). When the layer is turned on for display, Qgis
>> requires about 46 GB of virtual memory. Since my computer only has 24 GB of
>> physical memory, it is required to dip into the swap space considerably.
>> Even with 60 GB of swap space on an SSD, the swapping needed to refresh the
>> canvas is substantial and dramatically reduces performance resulting in
>> substantial delays. Compare with the January master referred to above which
>> only requires about 7 GB of virtual memory total with the aerial photo
>> layer displayed. The time needed to refresh the canvas is less than 1
>> second, most of the time.
>>
>> I know that the canvas refresh in Windows with aerial photos can be
>> substantially slower than in Linux. This does not affect me, because I
>> don't use Windows now that I have a stable platform for running an older
>> Linux master alongside the most recent one. What I do know is that the
>> memory demands are making it difficult to evaluate the recent masters. I
>> need some kind of setting to turn this caching off. With the aerial photo
>> layer turned off, the memory usage of the current master is about the same
>> as the old one, and it's much quicker to update.
>>
>>
>> ___
>> QGIS-Developer mailing list
>> QGIS-Developer@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
>
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Caching of raster images

[Patrick Dunford]

I don't know what this is.


On 06/10/17 01:08, Régis Haubourg wrote:

Hi Patrick ,
 did you consider using GDAL VRT to avoid opening so much files?
In my experience, this works well.
Régis


2017-10-05 13:48 GMT+02:00 Patrick Dunford >:


Some time ago in a discussion of a particular bug a contributor
expressed concern that the refresh of background rasters (aerial
photography) in Windows was too slow.

Maybe this is the reason that recent versions of master appear to
be loading all of the background imagery into memory (I use a
master from January this year to work around issues with later
ones, and that master does not have this feature).

Unfortunately if there are a lot of rasters then the memory demand
is excessive and unsurprisingly slows down the computer negating
any purported benefit of caching.

As an example a project I am currently working on has about 900
aerial photo images (GeoJpeg). When the layer is turned on for
display, Qgis requires about 46 GB of virtual memory. Since my
computer only has 24 GB of physical memory, it is required to dip
into the swap space considerably. Even with 60 GB of swap space on
an SSD, the swapping needed to refresh the canvas is substantial
and dramatically reduces performance resulting in substantial
delays. Compare with the January master referred to above which
only requires about 7 GB of virtual memory total with the aerial
photo layer displayed. The time needed to refresh the canvas is
less than 1 second, most of the time.

I know that the canvas refresh in Windows with aerial photos can
be substantially slower than in Linux. This does not affect me,
because I don't use Windows now that I have a stable platform for
running an older Linux master alongside the most recent one. What
I do know is that the memory demands are making it difficult to
evaluate the recent masters. I need some kind of setting to turn
this caching off. With the aerial photo layer turned off, the
memory usage of the current master is about the same as the old
one, and it's much quicker to update.


___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org 
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Unsubscribe:
https://lists.osgeo.org/mailman/listinfo/qgis-developer





___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Caching of raster images

[Régis Haubourg]

Hi Patrick ,
 did you consider using GDAL VRT to avoid opening so much files?
In my experience, this works well.
Régis


2017-10-05 13:48 GMT+02:00 Patrick Dunford :

> Some time ago in a discussion of a particular bug a contributor expressed
> concern that the refresh of background rasters (aerial photography) in
> Windows was too slow.
>
> Maybe this is the reason that recent versions of master appear to be
> loading all of the background imagery into memory (I use a master from
> January this year to work around issues with later ones, and that master
> does not have this feature).
>
> Unfortunately if there are a lot of rasters then the memory demand is
> excessive and unsurprisingly slows down the computer negating any purported
> benefit of caching.
>
> As an example a project I am currently working on has about 900 aerial
> photo images (GeoJpeg). When the layer is turned on for display, Qgis
> requires about 46 GB of virtual memory. Since my computer only has 24 GB of
> physical memory, it is required to dip into the swap space considerably.
> Even with 60 GB of swap space on an SSD, the swapping needed to refresh the
> canvas is substantial and dramatically reduces performance resulting in
> substantial delays. Compare with the January master referred to above which
> only requires about 7 GB of virtual memory total with the aerial photo
> layer displayed. The time needed to refresh the canvas is less than 1
> second, most of the time.
>
> I know that the canvas refresh in Windows with aerial photos can be
> substantially slower than in Linux. This does not affect me, because I
> don't use Windows now that I have a stable platform for running an older
> Linux master alongside the most recent one. What I do know is that the
> memory demands are making it difficult to evaluate the recent masters. I
> need some kind of setting to turn this caching off. With the aerial photo
> layer turned off, the memory usage of the current master is about the same
> as the old one, and it's much quicker to update.
>
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer