Hi, See https://docs.qgis.org/3.28/en/docs/user_manual/working_with_raster/raster_properties.html#pyramids-properties (for a QGIS based solution) or https://gdal.org/programs/gdaladdo.html (for a command line or python version).
Andreas On Tue, 30 May 2023 at 16:00, afernandez via QGIS-Developer < qgis-developer@lists.osgeo.org> wrote: > Hello Hannes, > Sorry to say that I'm at a loss here. I had not previously heard of > 'overviews' and am having a hard time finding much information. I couldn't > find anything in the PyQGIS (v3.28) developer cookbook, which is my main > reference in this QGIS journey. The found information is at > https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html > but didn't exactly clarify how to implement overviews with a script or why > this would sidestep reading the file several times. I don't quite > understand this last point either, as the file is large but the size of the > band (after everything is prepped and before rendering) is relatively > small, and should fit amply into the available memory. > Thanks, > Arturo > > > Johannes Kröger (WhereGroup) via QGIS-Developer wrote: > > > Hi Arturo, > > make sure you create overviews on your big rasters or QGIS will re-read > the whole file all the time to render an appropriate representation in the > map. > > Cheers, Hannes > Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer: > > Hello Martin, > I'm trying to understand what you're saying but my background is in > numerical methods where tasks are split into threads for acceleration > purposes. If a single layer is rendered by a single core (as you write and > I was observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be > handled? Wouldn't it take too long? (However, you want to define 'too > long'). > As far as more details about the process, the tensor 'var' (n x m x index > where I have tried different combinations but let's say n x m is 100 x 200) > of floating point numbers is manipulated according to the following > pseudo-code: > # Initial manipulations > dims = var.dimensions > shape = var.shape > driver_name = 'GTIFF' > driver = gdal.GetDriverByName(driver_name) > np_dtype = var.dtype > type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype) > gdal_ds = driver.Create(_my_path_, cols, rows, 1, > gdal.GDT_UInt16) > gdal_ds.SetProjection(_my_projection_) > gdal_ds.SetGeoTransform(_my_transformation_) > # Creation of the bands and scaled matrix > band = gdal_ds.GetRasterBand(1) > data = var[_chosen_index_] > data = ma.getdata(data) > data_scaled = np.interp(data, (data.min(), data.max()), (0, > 255)) > data_scaled2 = data_scaled.astype(int) # This is to rescale > into integers so that it can color the layer > # *** Lines to set up the color palette *** > # Write the array to band once everything has been rescaled > band.WriteArray(data_scaled2) > gdal_ds.FlushCache() > > This procedure seems to work well and takes a reasonable amount of time. > Then, the time is consumed when doing the rendering itself. I'm not > entirely sure what you mean by 'sub-optimal raster format' or 'thre are > overviews missing' so I cannot address them specifically. > Thanks, > Arturo > > > > > Martin Dobias wrote: > > > Hi > > On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer < > qgis-developer@lists.osgeo.org> wrote: > >> Hello, >> One of my project would greatly benefit from a much faster rendering. >> After some testing, the bulk (>99%) of the time is spent at rendering the >> layer with QgsRasterLayer(...,...). What I have also noticed is that >> rendering time does not change with the number of cores, which suggests >> that the renderer might be using a single thread. I checked the API >> documentation but couldn't find anything about multithreading or some other >> procedure to accelerate rendering. I was wondering if anyone has any >> suggestion or has faced a similar difficulty. >> > > Multi-threaded rendering is done at the granularity of whole map layers, > so a single layer is rendered by a single CPU core. Therefore if you have > just one layer that is slow to render, the multi-threading is not going to > help. It is best to do some testing how you can improve the layer's > rendering - for example, it could be that you are using some sub-optimal > raster format, or there are overviews missing, or something else... one > would need more details. > > Regards > Martin > > > _______________________________________________ > QGIS-Developer mailing listqgis-develo...@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > -- > Johannes Kröger / GIS-Entwickler/-Berater > > --------------------------------------------- > Aufwind durch Wissen! > Web-Seminare und Online-Schulungen > bei der www.foss-academy.com > --------------------------------------------- > > WhereGroup GmbH > c/o KK03 GmbH > Lange Reihe 29 > 20099 Hamburg > Germany > > Tel: +49 (0)228 / 90 90 38 - 36 > Fax: +49 (0)228 / 90 90 38 - 11 > johannes.kroe...@wheregroup.comwww.wheregroup.com > Geschäftsführer: > Olaf Knopp, Peter Stamm > Amtsgericht Bonn, HRB 9885 > ------------------------------- > > Hi Arturo, > > make sure you create overviews on your big rasters or QGIS will re-read > the whole file all the time to render an appropriate representation in the > map. > > Cheers, Hannes > Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer: > > Hello Martin, > I'm trying to understand what you're saying but my background is in > numerical methods where tasks are split into threads for acceleration > purposes. If a single layer is rendered by a single core (as you write and > I was observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be > handled? Wouldn't it take too long? (However, you want to define 'too > long'). > As far as more details about the process, the tensor 'var' (n x m x index > where I have tried different combinations but let's say n x m is 100 x 200) > of floating point numbers is manipulated according to the following > pseudo-code: > # Initial manipulations > dims = var.dimensions > shape = var.shape > driver_name = 'GTIFF' > driver = gdal.GetDriverByName(driver_name) > np_dtype = var.dtype > type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype) > gdal_ds = driver.Create(_my_path_, cols, rows, 1, > gdal.GDT_UInt16) > gdal_ds.SetProjection(_my_projection_) > gdal_ds.SetGeoTransform(_my_transformation_) > # Creation of the bands and scaled matrix > band = gdal_ds.GetRasterBand(1) > data = var[_chosen_index_] > data = ma.getdata(data) > data_scaled = np.interp(data, (data.min(), data.max()), (0, > 255)) > data_scaled2 = data_scaled.astype(int) # This is to rescale > into integers so that it can color the layer > # *** Lines to set up the color palette *** > # Write the array to band once everything has been rescaled > band.WriteArray(data_scaled2) > gdal_ds.FlushCache() > > This procedure seems to work well and takes a reasonable amount of time. > Then, the time is consumed when doing the rendering itself. I'm not > entirely sure what you mean by 'sub-optimal raster format' or 'thre are > overviews missing' so I cannot address them specifically. > Thanks, > Arturo > > > > > Martin Dobias wrote: > > > Hi > > On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer < > qgis-developer@lists.osgeo.org> wrote: > >> Hello, >> One of my project would greatly benefit from a much faster rendering. >> After some testing, the bulk (>99%) of the time is spent at rendering the >> layer with QgsRasterLayer(...,...). What I have also noticed is that >> rendering time does not change with the number of cores, which suggests >> that the renderer might be using a single thread. I checked the API >> documentation but couldn't find anything about multithreading or some other >> procedure to accelerate rendering. I was wondering if anyone has any >> suggestion or has faced a similar difficulty. >> > > Multi-threaded rendering is done at the granularity of whole map layers, > so a single layer is rendered by a single CPU core. Therefore if you have > just one layer that is slow to render, the multi-threading is not going to > help. It is best to do some testing how you can improve the layer's > rendering - for example, it could be that you are using some sub-optimal > raster format, or there are overviews missing, or something else... one > would need more details. > > Regards > Martin > > > _______________________________________________ > QGIS-Developer mailing listqgis-develo...@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > -- > Johannes Kröger / GIS-Entwickler/-Berater > > --------------------------------------------- > Aufwind durch Wissen! > Web-Seminare und Online-Schulungen > bei der www.foss-academy.com > --------------------------------------------- > > WhereGroup GmbH > c/o KK03 GmbH > Lange Reihe 29 > 20099 Hamburg > Germany > > Tel: +49 (0)228 / 90 90 38 - 36 > Fax: +49 (0)228 / 90 90 38 - 11 > johannes.kroe...@wheregroup.comwww.wheregroup.com > Geschäftsführer: > Olaf Knopp, Peter Stamm > Amtsgericht Bonn, HRB 9885 > ------------------------------- > > _______________________________________________ > QGIS-Developer mailing list > QGIS-Developer@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer > -- -- Andreas Neumann QGIS.ORG board member (treasurer)
_______________________________________________ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
