Per Frank's and Umberto's suggestion, I repeated my experiment on much
less beefy hardware: a dual-core laptop with one hard disk, running in
VMWare. I'll put the conclusion first, for folks like myself who are
impatient. :)
*** Conclusion:
Both tiling and overviews are useful, but only on machines with resource
shortages, such as slow disks or a lack of spare RAM for caching.
Hardly surprising is the conclusion that throwing more RAM into the
machine makes a bigger difference than any of these data-tweaking
approaches.
Although it's intuitive that faster hardware makes for better results,
I was surprised to find a level of "fast enough hardware" that makes
these performance enhancements useless.
--- And now the more boring experimental design stuff. :)
This test compares the effect of overviews and internal tiling on the
performance of a TIFF raster data source,
on a low-end system where disk performance is substandard and filesystem
caching is minimal due to low RAM.
*** PLATFORM
The hardware is a mid-budget laptop: Core Duo 1.7GHz, single SATA hard
disk, 1 GB RAM.
The software is HostGIS Linux 4.1 running under VMWare Server. The VM
was given 512 MB of memory, and the disk file (virtual IDE) was created
entirely before booting the machine.
(Hey, if the goal is a system that's slow enough to need performance
enhancement, VMWare is as good as it gets <grin>)
*** DATA SETUP
The data is 48 USGS DOQQs (black-n-white), downloaded and merged into a
single GeoTIFF using gdal_merge.py. I then created three copies of the
TIFF: with overviews, with internal tiling, and with both.
All images use the same projection: UTM zone 10N with NAD83 datum (EPSG
26910).
The spatial extent observed is:
543502 4150151 577642 4206427
The commands used were as follows:
gdal_merge.py -o original.tif o37122e2ne.tif o37122e2nw.tif
o37122e2se.tif o37122e2sw.tif o37122e3se.tif o37122e3sw.tif \
o37122e4ne.tif o37122e4nw.tif o37122e4se.tif
o37122e4sw.tif o37122f2ne.tif o37122f2nw.tif o37122f2se.tif o37122f3ne.tif \
o37122f3nw.tif o37122f3sw.tif o37122f4ne.tif
o37122f4nw.tif o37122f4se.tif o37122f4sw.tif o37122g2ne.tif o37122g2nw.tif \
o37122g2se.tif o37122g2sw.tif o37122g3ne.tif
o37122g3nw.tif o37122g3se.tif o37122g3sw.tif o37122g4ne.tif o37122g4nw.tif \
o37122g4se.tif o37122g4sw.tif o37122h2ne.tif
o37122h2nw.tif o37122h2se.tif o37122h2sw.tif o37122h3ne.tif o37122h3nw.tif \
o37122h3se.tif o37122h3sw.tif o37122h4ne.tif
o37122h4nw.tif o37122h4se.tif o37122h4sw.tif o37122e3ne.tif o37122e3nw.tif
gdal_translate -co "TILED=YES" original.tif tiled.tif
cp original.tif overviews.tif
gdaladdo overviews.tif 2 4 8 16 32
cp tiled.tif tilesandoverviews.tif
gdaladdo tilesandoverviews.tif 2 4 8 16 32
The resulting "original" image is 34140 x 56276 pixels, 1.79 GB in size.
The ones with overviews have them are 2.04 GB and have these overviews:
17070x28163, 8535x14082, 4268x7041, 2134x3521, 1067x1761
*** TESTING
A mapfile is created specifying three layers, named BARE, OVERVIEW,
TILE, BOTH, each one reading from the corresponding raster. No
reprojection is being done in the mapfile.
The mapfile specifies an output image size of 1000 x 1000 pixels.
The virtual machine was rebooted after each test, to ensure that
filesystem caching was not occurring, and each test was repeated several
times in random order to ensure that no unknown time-sensitive variable
was affecting the first or last in each series.
# full view of the region
shp2img -m mapfile.map -l BARE -o output.png
Time: 40.196s 27.812s 28.473s
shp2img -m mapfile.map -l OVERVIEW -o output.png
Time: 1.787s 2.947s 2.905s
shp2img -m mapfile.map -l TILE -o output.png
Time: 78.947s 77.175s 75.600s
shp2img -m mapfile.map -l BOTH -o output.png
Time: 2.572s 3.647s 1.828s
Ah-ha! Now that's what we're expecting!
Tiling hurts performance in a case where you end up showing the whole
thing anyway.
Overviews have dramatic (20 to 1) effect on rendering the full view.
# 1 square kilometer pulled from the map
shp2img -m mapfile.map -l BARE -e 549561 4150150 550561 4151150 -o
output.png
Time: 4.514s 5.903s 5.458s
shp2img -m mapfile.map -l OVERVIEW -e 549561 4150150 550561 4151150 -o
output.png
Time: 8.225s 2.980s 3.556s
shp2img -m mapfile.map -l TILE -e 549561 4150150 550561 4151150 -o
output.png
Time: 2.529s 1.852s 1.868s
shp2img -m mapfile.map -l BOTH -e 549561 4150150 550561 4151150 -o
output.png
Time: 2.533s 1.956s 1.970s
Again, not too surprising, though the margin is a lot less dramatic than
the full-view tests. Use of tiles
gave a 2:1 improvement over not using tiles, and use of overviews gave a
lesser but still useful boost.
