Re: [gdal-dev] Should we make also GTiff to use TILED=YES as default?
Really, that issue with "TILED" geotiffs on overview is a problem of selecting cache size for GDAL. I wish that GDAL_CACHEMAX will be automatically expanded to contain at least one row of unpacked tiles, then the performance issue talked about here will go away. On Tue, Nov 30, 2021 at 9:44 PM Javier Jimenez Shaw wrote: > Yes, that is true. It happened to me once. > What I mean is that the plain vanilla without all defaults (no overviews, > no tiles) is currently manageable. Changing only the "tiled" default would > make the vanilla case unmanageable. And some not expert users will complain. > > Maybe it is good to document that behavior. > > Btw, how spread is the support for tiled TIFFs among other viewers? > > Cheers > > El mar., 30 nov. 2021 19:29, Rahkonen Jukka (MML) < > jukka.rahko...@maanmittauslaitos.fi> escribió: > >> Hi, >> >> >> >> On the other hand those QGIS users pay the price every time when they >> zoom close to the same image because then QGIS must read many super wide >> rows of data instead of just the tiles that intersect the view. Not even >> overviews which are created on top of striped image do not help when user >> hit the native resolution. But perhaps it is not so important what is the >> default, experienced users maybe know what they want. >> >> >> >> -Jukka Rahkonen- >> >> >> >> *Lähettäjä:* Javier Jimenez Shaw >> *Lähetetty:* tiistai 30. marraskuuta 2021 11.09 >> *Vastaanottaja:* Rahkonen Jukka (MML) < >> jukka.rahko...@maanmittauslaitos.fi> >> *Kopio:* gdal-dev@lists.osgeo.org >> *Aihe:* Re: [gdal-dev] Should we make also GTiff to use TILED=YES as >> default? >> >> >> >> Hi Jukka >> >> >> >> I don't think it is a good option. >> >> TL;DR: tiled without overviews is extremely slow for a global view. >> >> >> >> If you have a (big) image with TILED=YES but without overviews, it is >> really really slow in QGIS (from seconds to many minutes). >> >> The reason (I guess) is that QGIS is by default sub-sampling the image >> using nearest neighbor. If the image is not tiled, then it has to read only >> every nth line. However, when tiled, at the end has to read the full image >> from file. >> >> The bigger the zoom level, the faster it goes. But waiting 5 minutes to >> see the global view of your image to know where to zoom in... you close >> QGIS and blame somebody. >> >> >> >> That changes dramatically if there are (proper) overviews available, of >> course. >> >> >> >> Cheers >> >> .___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... ._ .__ >> Entre dos pensamientos racionales >> hay infinitos pensamientos irracionales. >> >> >> >> >> >> On Tue, 30 Nov 2021 at 08:04, Rahkonen Jukka (MML) < >> jukka.rahko...@maanmittauslaitos.fi> wrote: >> >> Hi, >> >> >> >> There are loads of questions in the stack.exchange site where users have >> problems with performance when they do something with reading and writing >> very large image files. Big files have obviously become much more common >> now due to clouds. Users try to solve the problem by increasing >> GDAL_CACHEMAX and using multithreading etc. but they use the default >> GeoTIFF settings and keep on writing huge striped TIFFs. I wonder if we >> should make tiling the default also for the GTiff driver like it is in the >> COG driver. >> >> >> >> -Jukka Rahkonen- >> >> ___ >> gdal-dev mailing list >> gdal-dev@lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev >> >> ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Should we make also GTiff to use TILED=YES as default?
Hi Javier, Btw, how spread is the support for tiled TIFFs among other viewers? Anecdata, but at Koordinates we've been producing tiled GeoTIFFs for imagery datasets since mid-2014 (and to all our clients' users) — hundreds of thousands of GeoTIFF exports with millions of tiles to a very diverse user group. From a quick search in our support systems, it doesn't appear we've ever received a question from someone having trouble with a tiled GeoTIFF as far back as we have data for. I certainly don't recall any. Rob :) ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Should we make also GTiff to use TILED=YES as default?
Yes, that is true. It happened to me once. What I mean is that the plain vanilla without all defaults (no overviews, no tiles) is currently manageable. Changing only the "tiled" default would make the vanilla case unmanageable. And some not expert users will complain. Maybe it is good to document that behavior. Btw, how spread is the support for tiled TIFFs among other viewers? Cheers El mar., 30 nov. 2021 19:29, Rahkonen Jukka (MML) < jukka.rahko...@maanmittauslaitos.fi> escribió: > Hi, > > > > On the other hand those QGIS users pay the price every time when they zoom > close to the same image because then QGIS must read many super wide rows of > data instead of just the tiles that intersect the view. Not even overviews > which are created on top of striped image do not help when user hit the > native resolution. But perhaps it is not so important what is the default, > experienced users maybe know what they want. > > > > -Jukka Rahkonen- > > > > *Lähettäjä:* Javier Jimenez Shaw > *Lähetetty:* tiistai 30. marraskuuta 2021 11.09 > *Vastaanottaja:* Rahkonen Jukka (MML) > > *Kopio:* gdal-dev@lists.osgeo.org > *Aihe:* Re: [gdal-dev] Should we make also GTiff to use TILED=YES as > default? > > > > Hi Jukka > > > > I don't think it is a good option. > > TL;DR: tiled without overviews is extremely slow for a global view. > > > > If you have a (big) image with TILED=YES but without overviews, it is > really really slow in QGIS (from seconds to many minutes). > > The reason (I guess) is that QGIS is by default sub-sampling the image > using nearest neighbor. If the image is not tiled, then it has to read only > every nth line. However, when tiled, at the end has to read the full image > from file. > > The bigger the zoom level, the faster it goes. But waiting 5 minutes to > see the global view of your image to know where to zoom in... you close > QGIS and blame somebody. > > > > That changes dramatically if there are (proper) overviews available, of > course. > > > > Cheers > > .___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... ._ .__ > Entre dos pensamientos racionales > hay infinitos pensamientos irracionales. > > > > > > On Tue, 30 Nov 2021 at 08:04, Rahkonen Jukka (MML) < > jukka.rahko...@maanmittauslaitos.fi> wrote: > > Hi, > > > > There are loads of questions in the stack.exchange site where users have > problems with performance when they do something with reading and writing > very large image files. Big files have obviously become much more common > now due to clouds. Users try to solve the problem by increasing > GDAL_CACHEMAX and using multithreading etc. but they use the default > GeoTIFF settings and keep on writing huge striped TIFFs. I wonder if we > should make tiling the default also for the GTiff driver like it is in the > COG driver. > > > > -Jukka Rahkonen- > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Should we make also GTiff to use TILED=YES as default?
Hi, On the other hand those QGIS users pay the price every time when they zoom close to the same image because then QGIS must read many super wide rows of data instead of just the tiles that intersect the view. Not even overviews which are created on top of striped image do not help when user hit the native resolution. But perhaps it is not so important what is the default, experienced users maybe know what they want. -Jukka Rahkonen- Lähettäjä: Javier Jimenez Shaw Lähetetty: tiistai 30. marraskuuta 2021 11.09 Vastaanottaja: Rahkonen Jukka (MML) Kopio: gdal-dev@lists.osgeo.org Aihe: Re: [gdal-dev] Should we make also GTiff to use TILED=YES as default? Hi Jukka I don't think it is a good option. TL;DR: tiled without overviews is extremely slow for a global view. If you have a (big) image with TILED=YES but without overviews, it is really really slow in QGIS (from seconds to many minutes). The reason (I guess) is that QGIS is by default sub-sampling the image using nearest neighbor. If the image is not tiled, then it has to read only every nth line. However, when tiled, at the end has to read the full image from file. The bigger the zoom level, the faster it goes. But waiting 5 minutes to see the global view of your image to know where to zoom in... you close QGIS and blame somebody. That changes dramatically if there are (proper) overviews available, of course. Cheers .___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... ._ .__ Entre dos pensamientos racionales hay infinitos pensamientos irracionales. On Tue, 30 Nov 2021 at 08:04, Rahkonen Jukka (MML) mailto:jukka.rahko...@maanmittauslaitos.fi>> wrote: Hi, There are loads of questions in the stack.exchange site where users have problems with performance when they do something with reading and writing very large image files. Big files have obviously become much more common now due to clouds. Users try to solve the problem by increasing GDAL_CACHEMAX and using multithreading etc. but they use the default GeoTIFF settings and keep on writing huge striped TIFFs. I wonder if we should make tiling the default also for the GTiff driver like it is in the COG driver. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org<mailto:gdal-dev@lists.osgeo.org> https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Should we make also GTiff to use TILED=YES as default?
Hi Jukka I don't think it is a good option. TL;DR: tiled without overviews is extremely slow for a global view. If you have a (big) image with TILED=YES but without overviews, it is really really slow in QGIS (from seconds to many minutes). The reason (I guess) is that QGIS is by default sub-sampling the image using nearest neighbor. If the image is not tiled, then it has to read only every nth line. However, when tiled, at the end has to read the full image from file. The bigger the zoom level, the faster it goes. But waiting 5 minutes to see the global view of your image to know where to zoom in... you close QGIS and blame somebody. That changes dramatically if there are (proper) overviews available, of course. Cheers .___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... ._ .__ Entre dos pensamientos racionales hay infinitos pensamientos irracionales. On Tue, 30 Nov 2021 at 08:04, Rahkonen Jukka (MML) < jukka.rahko...@maanmittauslaitos.fi> wrote: > Hi, > > > > There are loads of questions in the stack.exchange site where users have > problems with performance when they do something with reading and writing > very large image files. Big files have obviously become much more common > now due to clouds. Users try to solve the problem by increasing > GDAL_CACHEMAX and using multithreading etc. but they use the default > GeoTIFF settings and keep on writing huge striped TIFFs. I wonder if we > should make tiling the default also for the GTiff driver like it is in the > COG driver. > > > > -Jukka Rahkonen- > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Should we make also GTiff to use TILED=YES as default?
Hi, There are loads of questions in the stack.exchange site where users have problems with performance when they do something with reading and writing very large image files. Big files have obviously become much more common now due to clouds. Users try to solve the problem by increasing GDAL_CACHEMAX and using multithreading etc. but they use the default GeoTIFF settings and keep on writing huge striped TIFFs. I wonder if we should make tiling the default also for the GTiff driver like it is in the COG driver. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
