Klokan Petr Pridal <mailto:klo...@klokan.cz>
February 3, 2015 at 2:18 AM
Hi Jorge,
rendering of your sample file down to zoomlevel 19 would generate over
2 billion tiles - probably tens of terabytes of data. I guess it is
not what you want to do... Typically the overzooming is done on client
side in a JavaScript viewer or on the tileserver hosting.
The tiling utilities suggests you the native zoom level automatically
- based on the raster resolution of the input.
In case you want to render large datasets composed of multiple files
efficiently I would recommend to try our MapTiler Pro command line
utility.
It is significantly faster then GDAL2Tiles and produces optimised and
correct tiles, while handling merging of multiple files, solving
issues with dateline crossing, reprojection and merging of data in
different srs, direct output to MBTiles, OGC WMTS compatible export,
and many more features.
The usage on command line is quite easy:
$ maptiler -o merged_output_dir input1.tif input2.tif input3.tif
Input files can be in the original srs - every extra reprojection
degrades visual quality of the output.
Merging with VRT is not recommend - it slow down the tile rendering,
as there are artificial blocks introduced - it is better for
performance to pass the original files directly.
MapTiler is a complete C/C++ reimplementation of my GDAL2Tiles
utility. There are several significant improvements in the merging and
internal tiling process including multi-thread CPU parallelization and
automatic file size optimisation (PNG color quantization and JPEG tweaks).
Installation is possible with the .deb or .rpm packages or binaries
for Mac OS X or Windows or Docker.
The small file, like your sample mentioned above, can be rendered
directly with the MapTiler Free in the graphical user interface which
is available for download at: http://www.maptiler.com/download/
If you drop us an email we send you also the Pro version with command
line for your free testing.
Best regards,
Petr Pridal, PhD
Klokan Technologies GmbH
--
http://blog.klokan.cz/
http://www.maptiler.org/
http://www.oldmapsonline.org/
Jorge Arévalo <mailto:jo...@cartodb.com>
January 29, 2015 at 1:13 AM
El jueves, 29 de enero de 2015, Even Rouault
<even.roua...@spatialys.com <mailto:even.roua...@spatialys.com>> escribió:
Le jeudi 29 janvier 2015 00:05:32, Jorge Arévalo a écrit :
> > Even Rouault <mailto:even.roua...@spatialys.com <javascript:;>>
> > January 28, 2015 at 9:28 PM
> >
> > Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
> >> Hi,
> >>
> >> I'm working with a patched version of gdal2tiles, which makes
use of
> >> parallelization:
> >>
http://gis.stackexchange.com/questions/7743/performance-of-google-map-ti
> >> le- creation-processes/74446#74446
> >>
> >> I want to create a complete TMS cache from raster imagery. No
> >> assumptions about SRS of data type for input data.
> >>
> >> I want the tiling process to be as fast as possible
(gdal2tiles is
> >> really heavy process), do you have any recomendations about
the data or
> >> the parameters used?
> >>
> >> Right now, I'm doing this
> >>
> >> Step 1: build vrt from input images
> >>
> >> gdal_vrtmerge.py -o merged.vrt<list of input tif files>
> >>
> >> Step 2: build tiles from vrt file
> >>
> >> gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt
tms_dir
> >>
> >> Even with parallelization, process still feels really slow.
Would it be
> >> faster if, for example, I convert all my input files to
epsg:3857? Or if
> >> I scale them to 8-bit? Or if I use near resampling method
instead of
> >> cubic? (I'm using cubic because I'm working with continuous data:
> >> satellite images, am I doing it right?).
> >>
> > From a quick look at the source, it seems that there's an
optimization
> > if the
> >
> > input SRS == output SRS that avoids going through the warped
VRT path.
> >
> > That said, we definitely need one or several maintainers for
gdal2tiles.
> > There are quite a faw patches floating around in Trac that
would need
> > someone to review, test, fix, apply them, as well as writing
tests (no
> > autotest for gdal2tiles yet), etc...
>
> Ok. But the applications is taking hours to generate a complete tile
> cache (zoom levels 0-19) for a 3MB tiff file, in epsg:3857. A 4
cores
> machine with 8GB of RAM. The file is this one
>
>
https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif
>
> Taking so much time for a 3MB file sounds ridiculous. I'm
probably doing
> something wrong. This is the line
>
> gdal2tiles.py -s epsg:3857 -z 0-19 -r cubic katrina-3857.tif
tiles_dir
>
> Do you see something wrong in this approach?
The performance seems to be deeply impacted by "-r cubic", but
even without
it, it still sucks. The reason is the zoom level 19. Your dataset
has a
resolution of 3469 m. zoom 19 corresponds to a resolution of ~ 0.3
m. So you
basically try to generate a TMS that is rougly 12000 x 12000
larger than your
source dataset...
Wow, thanks for the enlightment. So, with that resolution, I could
easily create TMS cache until levels 5-6, according to this
https://msdn.microsoft.com/en-us/library/aa940990.aspx
If I'd want higher zoom levels... That could mean
high processing cost. Worse if cúbic resampling method is used.
Conclussion: better guessing the highest zoom level for
that resolution using the table on the previous link, and building the
TMS cache with that constraint in mind.
Thanks again, Even!
>
> Anyway, if this is a problem of gdal2tiles and it needs fine
tunning or
> maintenance, we could talk. I don't know if there's any other
method to
> generate a complete TMS cache using GDAL.
>
> >> Any other tips?
> >>
> >> Many thanks in advance
> >>
> >> Best regards
--
Spatialys - Geospatial professional services
http://www.spatialys.com
--
Jorge Arévalo
http://cartodb.com/
Skype ID: jorgecartodb
Even Rouault <mailto:even.roua...@spatialys.com>
January 29, 2015 at 12:27 AM
Le jeudi 29 janvier 2015 00:05:32, Jorge Arévalo a écrit :
Even Rouault<mailto:even.roua...@spatialys.com>
January 28, 2015 at 9:28 PM
Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
Hi,
I'm working with a patched version of gdal2tiles, which makes use of
parallelization:
http://gis.stackexchange.com/questions/7743/performance-of-google-map-ti
le- creation-processes/74446#74446
I want to create a complete TMS cache from raster imagery. No
assumptions about SRS of data type for input data.
I want the tiling process to be as fast as possible (gdal2tiles is
really heavy process), do you have any recomendations about the data or
the parameters used?
Right now, I'm doing this
Step 1: build vrt from input images
gdal_vrtmerge.py -o merged.vrt<list of input tif files>
Step 2: build tiles from vrt file
gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt tms_dir
Even with parallelization, process still feels really slow. Would it be
faster if, for example, I convert all my input files to epsg:3857? Or if
I scale them to 8-bit? Or if I use near resampling method instead of
cubic? (I'm using cubic because I'm working with continuous data:
satellite images, am I doing it right?).
From a quick look at the source, it seems that there's an optimization
if the
input SRS == output SRS that avoids going through the warped VRT path.
That said, we definitely need one or several maintainers for gdal2tiles.
There are quite a faw patches floating around in Trac that would need
someone to review, test, fix, apply them, as well as writing tests (no
autotest for gdal2tiles yet), etc...
Ok. But the applications is taking hours to generate a complete tile
cache (zoom levels 0-19) for a 3MB tiff file, in epsg:3857. A 4 cores
machine with 8GB of RAM. The file is this one
https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif
Taking so much time for a 3MB file sounds ridiculous. I'm probably doing
something wrong. This is the line
gdal2tiles.py -s epsg:3857 -z 0-19 -r cubic katrina-3857.tif tiles_dir
Do you see something wrong in this approach?
The performance seems to be deeply impacted by "-r cubic", but even without
it, it still sucks. The reason is the zoom level 19. Your dataset has a
resolution of 3469 m. zoom 19 corresponds to a resolution of ~ 0.3 m. So you
basically try to generate a TMS that is rougly 12000 x 12000 larger than your
source dataset...
Anyway, if this is a problem of gdal2tiles and it needs fine tunning or
maintenance, we could talk. I don't know if there's any other method to
generate a complete TMS cache using GDAL.
Any other tips?
Many thanks in advance
Best regards
Jorge Arévalo <mailto:jo...@cartodb.com>
January 29, 2015 at 12:05 AM
Even Rouault <mailto:even.roua...@spatialys.com>
January 28, 2015 at 9:28 PM
Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
Hi,
I'm working with a patched version of gdal2tiles, which makes use of
parallelization:
http://gis.stackexchange.com/questions/7743/performance-of-google-map-tile-
creation-processes/74446#74446
I want to create a complete TMS cache from raster imagery. No
assumptions about SRS of data type for input data.
I want the tiling process to be as fast as possible (gdal2tiles is
really heavy process), do you have any recomendations about the data or
the parameters used?
Right now, I'm doing this
Step 1: build vrt from input images
gdal_vrtmerge.py -o merged.vrt<list of input tif files>
Step 2: build tiles from vrt file
gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt tms_dir
Even with parallelization, process still feels really slow. Would it be
faster if, for example, I convert all my input files to epsg:3857? Or if
I scale them to 8-bit? Or if I use near resampling method instead of
cubic? (I'm using cubic because I'm working with continuous data:
satellite images, am I doing it right?).
> From a quick look at the source, it seems that there's an optimization if the
input SRS == output SRS that avoids going through the warped VRT path.
That said, we definitely need one or several maintainers for gdal2tiles. There
are quite a faw patches floating around in Trac that would need someone to
review, test, fix, apply them, as well as writing tests (no autotest for
gdal2tiles yet), etc...
Ok. But the applications is taking hours to generate a complete tile
cache (zoom levels 0-19) for a 3MB tiff file, in epsg:3857. A 4 cores
machine with 8GB of RAM. The file is this one
https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif
Taking so much time for a 3MB file sounds ridiculous. I'm probably
doing something wrong. This is the line
gdal2tiles.py -s epsg:3857 -z 0-19 -r cubic katrina-3857.tif tiles_dir
Do you see something wrong in this approach?
Anyway, if this is a problem of gdal2tiles and it needs fine tunning
or maintenance, we could talk. I don't know if there's any other
method to generate a complete TMS cache using GDAL.
Any other tips?
Many thanks in advance
Best regards
Even Rouault <mailto:even.roua...@spatialys.com>
January 28, 2015 at 9:28 PM
Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
Hi,
I'm working with a patched version of gdal2tiles, which makes use of
parallelization:
http://gis.stackexchange.com/questions/7743/performance-of-google-map-tile-
creation-processes/74446#74446
I want to create a complete TMS cache from raster imagery. No
assumptions about SRS of data type for input data.
I want the tiling process to be as fast as possible (gdal2tiles is
really heavy process), do you have any recomendations about the data or
the parameters used?
Right now, I'm doing this
Step 1: build vrt from input images
gdal_vrtmerge.py -o merged.vrt<list of input tif files>
Step 2: build tiles from vrt file
gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt tms_dir
Even with parallelization, process still feels really slow. Would it be
faster if, for example, I convert all my input files to epsg:3857? Or if
I scale them to 8-bit? Or if I use near resampling method instead of
cubic? (I'm using cubic because I'm working with continuous data:
satellite images, am I doing it right?).
From a quick look at the source, it seems that there's an optimization if the
input SRS == output SRS that avoids going through the warped VRT path.
That said, we definitely need one or several maintainers for gdal2tiles. There
are quite a faw patches floating around in Trac that would need someone to
review, test, fix, apply them, as well as writing tests (no autotest for
gdal2tiles yet), etc...
Any other tips?
Many thanks in advance
Best regards
