Re: [Geoserver-users] Serving SRTM1 data with GeoServer
Thank you, Ian (and Peter). I had actually considered using gdal_merge.py to combine the GeoTiffs I'd created, but thought the resulting mega-TIFF would be too large. However: gdal_merge.py -o elevation.tif -co BIGTIFF=YES -co TILED=YES -co COMPRESS=DEFLATE *.hgt.tif worked fine. GeoServer accepted this store and maps it slowly, but without generating errors. For reference, the resulting TIFF, elevation.tif is just over 100 gigabytes in size, and has a resolution of 1296001 x 417601, which is slightly less than I expected because it turns out SRTM1 doesn't cover the entire world. On Tue, 21 May 2019, Ian Turton wrote: Date: Tue, 21 May 2019 08:18:23 +0100 From: Ian Turton To: Barry Carter Cc: geoserver-users Subject: Re: [Geoserver-users] Serving SRTM1 data with GeoServer Have a look at https://www.ianturton.com/tutorials/bluemarble.html where I have example scripts and work flow to create a base image from a set of tiles and how to make a pyramid from it. Ian On Tue, 21 May 2019 at 03:16, Barry Carter wrote: I'm trying to provide SRTM1 (1 arcsecond resolution elevation) images and data using GeoServer, but can't figure out how to serve such a large dataset efficiently. Here's what I've done so far: - Downloaded the 1 degree squared HGT files. The Earth is 360*180 = 64800 square degrees, but only 14295 HGT files since the other square degrees are 100% water. The HGT files are uncompressed and 25,934,402 bytes each. - I then used commands like: gdal_translate -co TILED=YES -co COMPRESS=DEFLATE N58W122.hgt N58W122.hgt.tif to convert these HGT files into TIFFs. The TIFFs vary in size, but are much smaller. - I then imported the directory with the TIFFs as a store using ImageMosaic. This took a while, but appears to have worked, since "srtm1" (the name I gave it) now appears in my list of stores. - I then created a layer using this source, but when I use "OpenLayers" preview, the server hangs for a while and ultimately dies with "too many open files" (or similar). This seems odd, since my Linux OS had fs.file-max set to 13063650. I even tried doubling it to 26127300, but that didn't help. At this point, I suspect I need to create an image pyramid or something so GeoServer can handle the data efficiently, but I'm not sure exactly what to do. I've used 'gdal_retile.py' previously to create pyramids from one large image into tiles at various zoom levels, but I think I want to do the opposite here (merge smaller images for lower zoom levels). The gdal_retile.py option -pyramidsonly seems to say it does that, but I can't get it work (it just hangs and creates no files) ___ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users -- Ian Turton ___ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
Re: [Geoserver-users] Serving SRTM1 data with GeoServer
Have a look at https://www.ianturton.com/tutorials/bluemarble.html where I have example scripts and work flow to create a base image from a set of tiles and how to make a pyramid from it. Ian On Tue, 21 May 2019 at 03:16, Barry Carter wrote: > I'm trying to provide SRTM1 (1 arcsecond resolution elevation) images and > data > using GeoServer, but can't figure out how to serve such a large dataset > efficiently. Here's what I've done so far: > >- Downloaded the 1 degree squared HGT files. The Earth is 360*180 = > 64800 > square degrees, but only 14295 HGT files since the other square > degrees > are 100% water. The HGT files are uncompressed and 25,934,402 bytes > each. > >- I then used commands like: > > gdal_translate -co TILED=YES -co COMPRESS=DEFLATE N58W122.hgt > N58W122.hgt.tif > > to convert these HGT files into TIFFs. The TIFFs vary in size, but are much > smaller. > >- I then imported the directory with the TIFFs as a store using > ImageMosaic. This took a while, but appears to have worked, since > "srtm1" > (the name I gave it) now appears in my list of stores. > >- I then created a layer using this source, but when I use "OpenLayers" > preview, the server hangs for a while and ultimately dies with "too > many > open files" (or similar). This seems odd, since my Linux OS had > fs.file-max set to 13063650. I even tried doubling it to 26127300, but > that didn't help. > > At this point, I suspect I need to create an image pyramid or something so > GeoServer can handle the data efficiently, but I'm not sure exactly what to > do. I've used 'gdal_retile.py' previously to create pyramids from one large > image into tiles at various zoom levels, but I think I want to do the > opposite > here (merge smaller images for lower zoom levels). The gdal_retile.py > option > -pyramidsonly seems to say it does that, but I can't get it work (it just > hangs and creates no files) > > > > ___ > Geoserver-users mailing list > > Please make sure you read the following two resources before posting to > this list: > - Earning your support instead of buying it, but Ian Turton: > http://www.ianturton.com/talks/foss4g.html#/ > - The GeoServer user list posting guidelines: > http://geoserver.org/comm/userlist-guidelines.html > > If you want to request a feature or an improvement, also see this: > https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer > > > Geoserver-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geoserver-users > -- Ian Turton ___ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
Re: [Geoserver-users] Serving SRTM1 data with GeoServer
Hi Barry I have recently done something similar and had a slightly different problem, but still related to too many files, with GeoServer performing very slowly when zoomed out. Our solution, which works very well, was to create a layer group with the original (TIFF) ImageMosaic store from scale 5 - 260 and another top layer ImageMosaic store from scale 260 - 4. This top layer was created by downsampling each DEM image to 6.25% with gdal_translate and then combining them into 16x16 degree tiles with gdalwarp. The same gdaldem hillshade command can then be run on these top layer tiles. By splitting the layer like this, the number of files being accessed was reduced to a level that GeoServer can very easily manage and it works perfectly for us. The top layer does not consume too much extra disk space. I'm not sure if this is the best way to do it, so I'm always willing to hear from others if there's a better solution. Regards Peter AfriGIS On Tue, 21 May 2019 at 04:18, Barry Carter wrote: > I'm trying to provide SRTM1 (1 arcsecond resolution elevation) images and > data > using GeoServer, but can't figure out how to serve such a large dataset > efficiently. Here's what I've done so far: > >- Downloaded the 1 degree squared HGT files. The Earth is 360*180 = > 64800 > square degrees, but only 14295 HGT files since the other square > degrees > are 100% water. The HGT files are uncompressed and 25,934,402 bytes > each. > >- I then used commands like: > > gdal_translate -co TILED=YES -co COMPRESS=DEFLATE N58W122.hgt > N58W122.hgt.tif > > to convert these HGT files into TIFFs. The TIFFs vary in size, but are much > smaller. > >- I then imported the directory with the TIFFs as a store using > ImageMosaic. This took a while, but appears to have worked, since > "srtm1" > (the name I gave it) now appears in my list of stores. > >- I then created a layer using this source, but when I use "OpenLayers" > preview, the server hangs for a while and ultimately dies with "too > many > open files" (or similar). This seems odd, since my Linux OS had > fs.file-max set to 13063650. I even tried doubling it to 26127300, but > that didn't help. > > At this point, I suspect I need to create an image pyramid or something so > GeoServer can handle the data efficiently, but I'm not sure exactly what to > do. I've used 'gdal_retile.py' previously to create pyramids from one large > image into tiles at various zoom levels, but I think I want to do the > opposite > here (merge smaller images for lower zoom levels). The gdal_retile.py > option > -pyramidsonly seems to say it does that, but I can't get it work (it just > hangs and creates no files) > > > > ___ > Geoserver-users mailing list > > Please make sure you read the following two resources before posting to > this list: > - Earning your support instead of buying it, but Ian Turton: > http://www.ianturton.com/talks/foss4g.html#/ > - The GeoServer user list posting guidelines: > http://geoserver.org/comm/userlist-guidelines.html > > If you want to request a feature or an improvement, also see this: > https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer > > > Geoserver-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geoserver-users > ___ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
[Geoserver-users] Serving SRTM1 data with GeoServer
I'm trying to provide SRTM1 (1 arcsecond resolution elevation) images and data using GeoServer, but can't figure out how to serve such a large dataset efficiently. Here's what I've done so far: - Downloaded the 1 degree squared HGT files. The Earth is 360*180 = 64800 square degrees, but only 14295 HGT files since the other square degrees are 100% water. The HGT files are uncompressed and 25,934,402 bytes each. - I then used commands like: gdal_translate -co TILED=YES -co COMPRESS=DEFLATE N58W122.hgt N58W122.hgt.tif to convert these HGT files into TIFFs. The TIFFs vary in size, but are much smaller. - I then imported the directory with the TIFFs as a store using ImageMosaic. This took a while, but appears to have worked, since "srtm1" (the name I gave it) now appears in my list of stores. - I then created a layer using this source, but when I use "OpenLayers" preview, the server hangs for a while and ultimately dies with "too many open files" (or similar). This seems odd, since my Linux OS had fs.file-max set to 13063650. I even tried doubling it to 26127300, but that didn't help. At this point, I suspect I need to create an image pyramid or something so GeoServer can handle the data efficiently, but I'm not sure exactly what to do. I've used 'gdal_retile.py' previously to create pyramids from one large image into tiles at various zoom levels, but I think I want to do the opposite here (merge smaller images for lower zoom levels). The gdal_retile.py option -pyramidsonly seems to say it does that, but I can't get it work (it just hangs and creates no files) ___ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
