Re: [Geoserver-users] Serving a large number of Geotiffs
Hi, It may be good to know that older Geoserver versions can't utilize external overviews which are saved into separate files (image_name.ovr) if gdaladdo is run with -ro option. I believe that external overviews are supported nowadays but I could not find since which version. Internal overviews are added into the image files and they cannot be removed or updated easily if one wants to test with different resampling methods so I recommend to take a backup first. -Jukka Rahkonen- Andrea Aime wrote: On Fri, Mar 18, 2016 at 10:07 AM, Max mailto:basili...@gmail.com>> wrote: If I create overviews for the images, do the Geoserver layers associated with it need to be recreated? Going by memory, to be sure the overviews are recognized, you have to restart the server, or reload the configuration (the reload button in the server configuration panel) Cheers Andrea -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. --- -- Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more. http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140___ Geoserver-users mailing list Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
Re: [Geoserver-users] Serving a large number of Geotiffs
Those sound like small tiffs so you would be better combining them together to avoid opening too many files at a time. Have a look at gdalbuildvrt to make a virtual raster catalogue that you can then convert into a tiled compressed geotiff that is in your main output projection. Have a read through the annual "GeoServer on Steroids" presentation ( http://www.slideshare.net/geosolutions/geoserver-on-steroids-foss4g-2015 ) for more ideas.# Ian On 17 March 2016 at 09:51, Max wrote: > I have a large number - more than 60 thousand - of relatively small > Geotiffs, usually from 2 to 12 Mb. I have a web client that uses > Leaflet to view them, but things are quite slow even inside our own > network. I have a hunch that both Geoserver and these Geotiffs are not > configured in the best way possible. > > The tiffs were generated from a bunch of EPSG:3035 Ascii grid files > using this command: > > for f in *.rsl; do gdal_translate -a_srs EPSG:3035 -co "TILED=YES" > -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "COMPRESS=DEFLATE" -co > "ZLEVEL=9" -co "BIGTIFF=YES" $f $f.tif; done > > Then I wrote a Python script that created a store and a layer for each > of the tiffs. > > Here is the output from gdalinfo for a typical one: > > Driver: GTiff/GeoTIFF > Files: se502c_R9_C1_3_T_2531_sec.rsl.tif > Size is 3396, 2271 > Coordinate System is: > PROJCS["ETRS89 / LAEA Europe", > GEOGCS["ETRS89", > DATUM["European_Terrestrial_Reference_System_1989", > SPHEROID["GRS 1980",6378137,298.2572221010002, > AUTHORITY["EPSG","7019"]], > TOWGS84[0,0,0,0,0,0,0], > AUTHORITY["EPSG","6258"]], > PRIMEM["Greenwich",0], > UNIT["degree",0.0174532925199433], > AUTHORITY["EPSG","4258"]], > PROJECTION["Lambert_Azimuthal_Equal_Area"], > PARAMETER["latitude_of_center",52], > PARAMETER["longitude_of_center",10], > PARAMETER["false_easting",4321000], > PARAMETER["false_northing",321], > > As you can see they are EPSG:3035. They should be internally tiled. In > the Coordinate Reference Systems for Geoserver, it says that both > Native SRS and Declared SRS are EPSG:3035, and that the handling > should be to reproject native to declared. > > My web client overlays these Geotiffs on a standard OpenStreetMap > layer in Web Mercator. All the geotiffs we have tried appear > correctly, so I guess reprojection is still happening at some stage. > > My gut feeling is that I might gain some speed by reprojecting the > original Geotiffs to Web Mercator, or at least changing the declared > SRS to Web Mercator in Geoserver. Not keen on it, it's going to take > days. What other properties of the Geotiffs could I tinker with? > > Would caching with GeoWebCache help, given the file size? > > Bonus question - if a Geotiff is changed, by reprojecting or retiling > or what have you, does the Geoserver layer associated with it get > invalidated? It took days to create them all. > > I'd really appreciate your feeback. > > Thanks in advance, > Max > > > -- > Transform Data into Opportunity. > Accelerate data analysis in your applications with > Intel Data Analytics Acceleration Library. > Click to learn more. > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > ___ > Geoserver-users mailing list > Geoserver-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geoserver-users > -- Ian Turton -- Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more. http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140___ Geoserver-users mailing list Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
Re: [Geoserver-users] Serving a large number of Geotiffs
If I create overviews for the images, do the Geoserver layers associated with it need to be recreated? Max On 17 March 2016 at 14:21, Rahkonen Jukka (MML) wrote: > Hi, > > > > If you want to keep all 6 layers which is not optimal as Andrea just > wrote, the most obvious thing to do is to create overviews for the images. > > gdaladdo –r average 2 4 8 16 32 64 image.tif --config COMPRESS_OVERVIEW > DEFLATE > > > > It is worth making a test before reprojecting images physically into > EPSG:3857. On-the-fly reprojecting is not necessarily so slow. > > I saw that you have used the BIGTIFF=YES option. It is not needed for your > small images and it probably makes reading the images a little bit slower > but I guess that the difference is marginal. Tile size 256x256 could be a > bit faster than 512x512 but that’t also marginal. > > > > > > -Jukka Rahkonen- > > > > basiliosz wrote: > > > > As 6 layers. They are not contiguous, are completely separate and are > viewed strictly one at a time per user. > > The scenario: user selects a city and a couple parameters, the JS code > figures out what layer to request, then submits the request for that layer > to Geoserver. Eventually the layer is added to the map. > > Max > > On 17 Mar 2016 13:44, "Andrea Aime" wrote: > > Hi, > > just to be sure, how have you published them? > > As a single image mosaic or as 6 separate layers? > > > > Cheers > > Andrea > > > > > > On Thu, Mar 17, 2016 at 12:56 PM, Max wrote: > > Thanks - because of the origin of the data the Geotiffs are not > adjacent, there's a little group for each of about 500 cities, but > strictly one of them at a time is shown, so I'm not sure how much > that's going to help. If there's something computationally intensive > going on at the server's end, that's going to be a drag in any case. > > Going through the Geoserver on Steroids presentation now, thanks! > > Max > > > On 17 March 2016 at 11:16, Ian Turton wrote: >> Those sound like small tiffs so you would be better combining them >> together >> to avoid opening too many files at a time. Have a look at gdalbuildvrt to >> make a virtual raster catalogue that you can then convert into a tiled >> compressed geotiff that is in your main output projection. >> >> Have a read through the annual "GeoServer on Steroids" presentation >> (http://www.slideshare.net/geosolutions/geoserver-on-steroids-foss4g-2015 >> ) >> for more ideas.# >> >> Ian >> >> On 17 March 2016 at 09:51, Max wrote: >>> >>> I have a large number - more than 60 thousand - of relatively small >>> Geotiffs, usually from 2 to 12 Mb. I have a web client that uses >>> Leaflet to view them, but things are quite slow even inside our own >>> network. I have a hunch that both Geoserver and these Geotiffs are not >>> configured in the best way possible. >>> >>> The tiffs were generated from a bunch of EPSG:3035 Ascii grid files >>> using this command: >>> >>> for f in *.rsl; do gdal_translate -a_srs EPSG:3035 -co "TILED=YES" >>> -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "COMPRESS=DEFLATE" -co >>> "ZLEVEL=9" -co "BIGTIFF=YES" $f $f.tif; done >>> >>> Then I wrote a Python script that created a store and a layer for each >>> of the tiffs. >>> >>> Here is the output from gdalinfo for a typical one: >>> >>> Driver: GTiff/GeoTIFF >>> Files: se502c_R9_C1_3_T_2531_sec.rsl.tif >>> Size is 3396, 2271 >>> Coordinate System is: >>> PROJCS["ETRS89 / LAEA Europe", >>> GEOGCS["ETRS89", >>> DATUM["European_Terrestrial_Reference_System_1989", >>> SPHEROID["GRS 1980",6378137,298.2572221010002, >>> AUTHORITY["EPSG","7019"]], >>> TOWGS84[0,0,0,0,0,0,0], >>> AUTHORITY["EPSG","6258"]], >>> PRIMEM["Greenwich",0], >>> UNIT["degree",0.0174532925199433], >>> AUTHORITY["EPSG","4258"]], >>> PROJECTION["Lambert_Azimuthal_Equal_Area"], >>> PARAMETER["latitude_of_center",52], >>> PARAMETER["longitude_of_center",10], >>> PARAMETER["false_easting",4321000], >>> PARAMETER["false_northing",321], >>> >>> As you can see they are EPSG:3035. They should be internally tiled. In >>> the Coordinate Reference Systems for Geoserver, it says that both >>> Native SRS and Declared SRS are EPSG:3035, and that the handling >>> should be to reproject native to declared. >>> >>> My web client overlays these Geotiffs on a standard OpenStreetMap >>> layer in Web Mercator. All the geotiffs we have tried appear >>> correctly, so I guess reprojection is still happening at some stage. >>> >>> My gut feeling is that I might gain some speed by reprojecting the >>> original Geotiffs to Web Mercator, or at least changing the declared >>> SRS to Web Mercator in Geoserver. Not keen on it, it's going to take >>> days. What other properties of the Geotiffs could I tinker with? >>> >>> Would caching with GeoWebCache help, given the file size? >>> >>> Bonus question - if a Geotiff is changed, by reprojecting or retiling >>> or what have you, does the Geoserve
Re: [Geoserver-users] Serving a large number of Geotiffs
On Thu, Mar 17, 2016 at 3:00 PM, Max wrote: > As 6 layers. They are not contiguous, are completely separate and are > viewed strictly one at a time per user. > Ah, that is a factory in the slowness you're seeing... GeoServer is not currently designed to handle efficiently that amount of layers, there are linear scans in the catalog code. It could be improved with some development effort (to make it use hashed lookups instead of linear scans), and there is also a community (aka unsupported) module called jdbcconfig that does not even load the config in memory and makes direct queries against the db that works better with so many layers, but it's still really slow in some cases (it's being improved over time, but not quite ready for all use cases yet). Suggestion: if possible, set them up a single image mosaic, and then use the cql_filter against the mosaic index attributes to just get the one image you need Cheers Andrea -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. --- -- Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more. http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140___ Geoserver-users mailing list Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
Re: [Geoserver-users] Serving a large number of Geotiffs
On Fri, Mar 18, 2016 at 10:07 AM, Max wrote: > If I create overviews for the images, do the Geoserver layers > associated with it need to be recreated? > Going by memory, to be sure the overviews are recognized, you have to restart the server, or reload the configuration (the reload button in the server configuration panel) Cheers Andrea -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. --- -- Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more. http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140___ Geoserver-users mailing list Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
Re: [Geoserver-users] Serving a large number of Geotiffs
Hi, If you want to keep all 6 layers which is not optimal as Andrea just wrote, the most obvious thing to do is to create overviews for the images. gdaladdo –r average 2 4 8 16 32 64 image.tif --config COMPRESS_OVERVIEW DEFLATE It is worth making a test before reprojecting images physically into EPSG:3857. On-the-fly reprojecting is not necessarily so slow. I saw that you have used the BIGTIFF=YES option. It is not needed for your small images and it probably makes reading the images a little bit slower but I guess that the difference is marginal. Tile size 256x256 could be a bit faster than 512x512 but that’t also marginal. -Jukka Rahkonen- basiliosz wrote: As 6 layers. They are not contiguous, are completely separate and are viewed strictly one at a time per user. The scenario: user selects a city and a couple parameters, the JS code figures out what layer to request, then submits the request for that layer to Geoserver. Eventually the layer is added to the map. Max On 17 Mar 2016 13:44, "Andrea Aime" mailto:andrea.a...@geo-solutions.it>> wrote: Hi, just to be sure, how have you published them? As a single image mosaic or as 6 separate layers? Cheers Andrea On Thu, Mar 17, 2016 at 12:56 PM, Max mailto:basili...@gmail.com>> wrote: Thanks - because of the origin of the data the Geotiffs are not adjacent, there's a little group for each of about 500 cities, but strictly one of them at a time is shown, so I'm not sure how much that's going to help. If there's something computationally intensive going on at the server's end, that's going to be a drag in any case. Going through the Geoserver on Steroids presentation now, thanks! Max On 17 March 2016 at 11:16, Ian Turton mailto:ijtur...@gmail.com>> wrote: > Those sound like small tiffs so you would be better combining them together > to avoid opening too many files at a time. Have a look at gdalbuildvrt to > make a virtual raster catalogue that you can then convert into a tiled > compressed geotiff that is in your main output projection. > > Have a read through the annual "GeoServer on Steroids" presentation > (http://www.slideshare.net/geosolutions/geoserver-on-steroids-foss4g-2015 ) > for more ideas.# > > Ian > > On 17 March 2016 at 09:51, Max > mailto:basili...@gmail.com>> wrote: >> >> I have a large number - more than 60 thousand - of relatively small >> Geotiffs, usually from 2 to 12 Mb. I have a web client that uses >> Leaflet to view them, but things are quite slow even inside our own >> network. I have a hunch that both Geoserver and these Geotiffs are not >> configured in the best way possible. >> >> The tiffs were generated from a bunch of EPSG:3035 Ascii grid files >> using this command: >> >> for f in *.rsl; do gdal_translate -a_srs EPSG:3035 -co "TILED=YES" >> -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "COMPRESS=DEFLATE" -co >> "ZLEVEL=9" -co "BIGTIFF=YES" $f $f.tif; done >> >> Then I wrote a Python script that created a store and a layer for each >> of the tiffs. >> >> Here is the output from gdalinfo for a typical one: >> >> Driver: GTiff/GeoTIFF >> Files: se502c_R9_C1_3_T_2531_sec.rsl.tif >> Size is 3396, 2271 >> Coordinate System is: >> PROJCS["ETRS89 / LAEA Europe", >> GEOGCS["ETRS89", >> DATUM["European_Terrestrial_Reference_System_1989", >> SPHEROID["GRS 1980",6378137,298.2572221010002, >> AUTHORITY["EPSG","7019"]], >> TOWGS84[0,0,0,0,0,0,0], >> AUTHORITY["EPSG","6258"]], >> PRIMEM["Greenwich",0], >> UNIT["degree",0.0174532925199433], >> AUTHORITY["EPSG","4258"]], >> PROJECTION["Lambert_Azimuthal_Equal_Area"], >> PARAMETER["latitude_of_center",52], >> PARAMETER["longitude_of_center",10], >> PARAMETER["false_easting",4321000], >> PARAMETER["false_northing",321], >> >> As you can see they are EPSG:3035. They should be internally tiled. In >> the Coordinate Reference Systems for Geoserver, it says that both >> Native SRS and Declared SRS are EPSG:3035, and that the handling >> should be to reproject native to declared. >> >> My web client overlays these Geotiffs on a standard OpenStreetMap >> layer in Web Mercator. All the geotiffs we have tried appear >> correctly, so I guess reprojection is still happening at some stage. >> >> My gut feeling is that I might gain some speed by reprojecting the >> original Geotiffs to Web Mercator, or at least changing the declared >> SRS to Web Mercator in Geoserver. Not keen on it, it's going to take >> days. What other properties of the Geotiffs could I tinker with? >> >> Would caching with GeoWebCache help, given the file size? >> >> Bonus question - if a Geotiff is changed, by reprojecting or retiling >> or what have you, does the Geoserver layer associated with it get >> invalidated? It took days to create them all. >> >> I'd really appreciate your feeback. >> >> Thanks in advance, >> Max >> >> >> --
Re: [Geoserver-users] Serving a large number of Geotiffs
As 6 layers. They are not contiguous, are completely separate and are viewed strictly one at a time per user. The scenario: user selects a city and a couple parameters, the JS code figures out what layer to request, then submits the request for that layer to Geoserver. Eventually the layer is added to the map. Max On 17 Mar 2016 13:44, "Andrea Aime" wrote: > Hi, > just to be sure, how have you published them? > As a single image mosaic or as 6 separate layers? > > Cheers > Andrea > > > On Thu, Mar 17, 2016 at 12:56 PM, Max wrote: > >> Thanks - because of the origin of the data the Geotiffs are not >> adjacent, there's a little group for each of about 500 cities, but >> strictly one of them at a time is shown, so I'm not sure how much >> that's going to help. If there's something computationally intensive >> going on at the server's end, that's going to be a drag in any case. >> >> Going through the Geoserver on Steroids presentation now, thanks! >> >> Max >> >> On 17 March 2016 at 11:16, Ian Turton wrote: >> > Those sound like small tiffs so you would be better combining them >> together >> > to avoid opening too many files at a time. Have a look at gdalbuildvrt >> to >> > make a virtual raster catalogue that you can then convert into a tiled >> > compressed geotiff that is in your main output projection. >> > >> > Have a read through the annual "GeoServer on Steroids" presentation >> > ( >> http://www.slideshare.net/geosolutions/geoserver-on-steroids-foss4g-2015 >> ) >> > for more ideas.# >> > >> > Ian >> > >> > On 17 March 2016 at 09:51, Max wrote: >> >> >> >> I have a large number - more than 60 thousand - of relatively small >> >> Geotiffs, usually from 2 to 12 Mb. I have a web client that uses >> >> Leaflet to view them, but things are quite slow even inside our own >> >> network. I have a hunch that both Geoserver and these Geotiffs are not >> >> configured in the best way possible. >> >> >> >> The tiffs were generated from a bunch of EPSG:3035 Ascii grid files >> >> using this command: >> >> >> >> for f in *.rsl; do gdal_translate -a_srs EPSG:3035 -co "TILED=YES" >> >> -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "COMPRESS=DEFLATE" -co >> >> "ZLEVEL=9" -co "BIGTIFF=YES" $f $f.tif; done >> >> >> >> Then I wrote a Python script that created a store and a layer for each >> >> of the tiffs. >> >> >> >> Here is the output from gdalinfo for a typical one: >> >> >> >> Driver: GTiff/GeoTIFF >> >> Files: se502c_R9_C1_3_T_2531_sec.rsl.tif >> >> Size is 3396, 2271 >> >> Coordinate System is: >> >> PROJCS["ETRS89 / LAEA Europe", >> >> GEOGCS["ETRS89", >> >> DATUM["European_Terrestrial_Reference_System_1989", >> >> SPHEROID["GRS 1980",6378137,298.2572221010002, >> >> AUTHORITY["EPSG","7019"]], >> >> TOWGS84[0,0,0,0,0,0,0], >> >> AUTHORITY["EPSG","6258"]], >> >> PRIMEM["Greenwich",0], >> >> UNIT["degree",0.0174532925199433], >> >> AUTHORITY["EPSG","4258"]], >> >> PROJECTION["Lambert_Azimuthal_Equal_Area"], >> >> PARAMETER["latitude_of_center",52], >> >> PARAMETER["longitude_of_center",10], >> >> PARAMETER["false_easting",4321000], >> >> PARAMETER["false_northing",321], >> >> >> >> As you can see they are EPSG:3035. They should be internally tiled. In >> >> the Coordinate Reference Systems for Geoserver, it says that both >> >> Native SRS and Declared SRS are EPSG:3035, and that the handling >> >> should be to reproject native to declared. >> >> >> >> My web client overlays these Geotiffs on a standard OpenStreetMap >> >> layer in Web Mercator. All the geotiffs we have tried appear >> >> correctly, so I guess reprojection is still happening at some stage. >> >> >> >> My gut feeling is that I might gain some speed by reprojecting the >> >> original Geotiffs to Web Mercator, or at least changing the declared >> >> SRS to Web Mercator in Geoserver. Not keen on it, it's going to take >> >> days. What other properties of the Geotiffs could I tinker with? >> >> >> >> Would caching with GeoWebCache help, given the file size? >> >> >> >> Bonus question - if a Geotiff is changed, by reprojecting or retiling >> >> or what have you, does the Geoserver layer associated with it get >> >> invalidated? It took days to create them all. >> >> >> >> I'd really appreciate your feeback. >> >> >> >> Thanks in advance, >> >> Max >> >> >> >> >> >> >> -- >> >> Transform Data into Opportunity. >> >> Accelerate data analysis in your applications with >> >> Intel Data Analytics Acceleration Library. >> >> Click to learn more. >> >> http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 >> >> ___ >> >> Geoserver-users mailing list >> >> Geoserver-users@lists.sourceforge.net >> >> https://lists.sourceforge.net/lists/listinfo/geoserver-users >> > >> > >> > >> > >> > -- >> > Ian Turton >> >> >> --
[Geoserver-users] Serving a large number of Geotiffs
I have a large number - more than 60 thousand - of relatively small Geotiffs, usually from 2 to 12 Mb. I have a web client that uses Leaflet to view them, but things are quite slow even inside our own network. I have a hunch that both Geoserver and these Geotiffs are not configured in the best way possible. The tiffs were generated from a bunch of EPSG:3035 Ascii grid files using this command: for f in *.rsl; do gdal_translate -a_srs EPSG:3035 -co "TILED=YES" -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "COMPRESS=DEFLATE" -co "ZLEVEL=9" -co "BIGTIFF=YES" $f $f.tif; done Then I wrote a Python script that created a store and a layer for each of the tiffs. Here is the output from gdalinfo for a typical one: Driver: GTiff/GeoTIFF Files: se502c_R9_C1_3_T_2531_sec.rsl.tif Size is 3396, 2271 Coordinate System is: PROJCS["ETRS89 / LAEA Europe", GEOGCS["ETRS89", DATUM["European_Terrestrial_Reference_System_1989", SPHEROID["GRS 1980",6378137,298.2572221010002, AUTHORITY["EPSG","7019"]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY["EPSG","6258"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4258"]], PROJECTION["Lambert_Azimuthal_Equal_Area"], PARAMETER["latitude_of_center",52], PARAMETER["longitude_of_center",10], PARAMETER["false_easting",4321000], PARAMETER["false_northing",321], As you can see they are EPSG:3035. They should be internally tiled. In the Coordinate Reference Systems for Geoserver, it says that both Native SRS and Declared SRS are EPSG:3035, and that the handling should be to reproject native to declared. My web client overlays these Geotiffs on a standard OpenStreetMap layer in Web Mercator. All the geotiffs we have tried appear correctly, so I guess reprojection is still happening at some stage. My gut feeling is that I might gain some speed by reprojecting the original Geotiffs to Web Mercator, or at least changing the declared SRS to Web Mercator in Geoserver. Not keen on it, it's going to take days. What other properties of the Geotiffs could I tinker with? Would caching with GeoWebCache help, given the file size? Bonus question - if a Geotiff is changed, by reprojecting or retiling or what have you, does the Geoserver layer associated with it get invalidated? It took days to create them all. I'd really appreciate your feeback. Thanks in advance, Max -- Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more. http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 ___ Geoserver-users mailing list Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
Re: [Geoserver-users] Serving a large number of Geotiffs
Thanks - because of the origin of the data the Geotiffs are not adjacent, there's a little group for each of about 500 cities, but strictly one of them at a time is shown, so I'm not sure how much that's going to help. If there's something computationally intensive going on at the server's end, that's going to be a drag in any case. Going through the Geoserver on Steroids presentation now, thanks! Max On 17 March 2016 at 11:16, Ian Turton wrote: > Those sound like small tiffs so you would be better combining them together > to avoid opening too many files at a time. Have a look at gdalbuildvrt to > make a virtual raster catalogue that you can then convert into a tiled > compressed geotiff that is in your main output projection. > > Have a read through the annual "GeoServer on Steroids" presentation > (http://www.slideshare.net/geosolutions/geoserver-on-steroids-foss4g-2015 ) > for more ideas.# > > Ian > > On 17 March 2016 at 09:51, Max wrote: >> >> I have a large number - more than 60 thousand - of relatively small >> Geotiffs, usually from 2 to 12 Mb. I have a web client that uses >> Leaflet to view them, but things are quite slow even inside our own >> network. I have a hunch that both Geoserver and these Geotiffs are not >> configured in the best way possible. >> >> The tiffs were generated from a bunch of EPSG:3035 Ascii grid files >> using this command: >> >> for f in *.rsl; do gdal_translate -a_srs EPSG:3035 -co "TILED=YES" >> -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "COMPRESS=DEFLATE" -co >> "ZLEVEL=9" -co "BIGTIFF=YES" $f $f.tif; done >> >> Then I wrote a Python script that created a store and a layer for each >> of the tiffs. >> >> Here is the output from gdalinfo for a typical one: >> >> Driver: GTiff/GeoTIFF >> Files: se502c_R9_C1_3_T_2531_sec.rsl.tif >> Size is 3396, 2271 >> Coordinate System is: >> PROJCS["ETRS89 / LAEA Europe", >> GEOGCS["ETRS89", >> DATUM["European_Terrestrial_Reference_System_1989", >> SPHEROID["GRS 1980",6378137,298.2572221010002, >> AUTHORITY["EPSG","7019"]], >> TOWGS84[0,0,0,0,0,0,0], >> AUTHORITY["EPSG","6258"]], >> PRIMEM["Greenwich",0], >> UNIT["degree",0.0174532925199433], >> AUTHORITY["EPSG","4258"]], >> PROJECTION["Lambert_Azimuthal_Equal_Area"], >> PARAMETER["latitude_of_center",52], >> PARAMETER["longitude_of_center",10], >> PARAMETER["false_easting",4321000], >> PARAMETER["false_northing",321], >> >> As you can see they are EPSG:3035. They should be internally tiled. In >> the Coordinate Reference Systems for Geoserver, it says that both >> Native SRS and Declared SRS are EPSG:3035, and that the handling >> should be to reproject native to declared. >> >> My web client overlays these Geotiffs on a standard OpenStreetMap >> layer in Web Mercator. All the geotiffs we have tried appear >> correctly, so I guess reprojection is still happening at some stage. >> >> My gut feeling is that I might gain some speed by reprojecting the >> original Geotiffs to Web Mercator, or at least changing the declared >> SRS to Web Mercator in Geoserver. Not keen on it, it's going to take >> days. What other properties of the Geotiffs could I tinker with? >> >> Would caching with GeoWebCache help, given the file size? >> >> Bonus question - if a Geotiff is changed, by reprojecting or retiling >> or what have you, does the Geoserver layer associated with it get >> invalidated? It took days to create them all. >> >> I'd really appreciate your feeback. >> >> Thanks in advance, >> Max >> >> >> -- >> Transform Data into Opportunity. >> Accelerate data analysis in your applications with >> Intel Data Analytics Acceleration Library. >> Click to learn more. >> http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 >> ___ >> Geoserver-users mailing list >> Geoserver-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/geoserver-users > > > > > -- > Ian Turton -- Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more. http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 ___ Geoserver-users mailing list Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
Re: [Geoserver-users] Serving a large number of Geotiffs
Hi, just to be sure, how have you published them? As a single image mosaic or as 6 separate layers? Cheers Andrea On Thu, Mar 17, 2016 at 12:56 PM, Max wrote: > Thanks - because of the origin of the data the Geotiffs are not > adjacent, there's a little group for each of about 500 cities, but > strictly one of them at a time is shown, so I'm not sure how much > that's going to help. If there's something computationally intensive > going on at the server's end, that's going to be a drag in any case. > > Going through the Geoserver on Steroids presentation now, thanks! > > Max > > On 17 March 2016 at 11:16, Ian Turton wrote: > > Those sound like small tiffs so you would be better combining them > together > > to avoid opening too many files at a time. Have a look at gdalbuildvrt to > > make a virtual raster catalogue that you can then convert into a tiled > > compressed geotiff that is in your main output projection. > > > > Have a read through the annual "GeoServer on Steroids" presentation > > ( > http://www.slideshare.net/geosolutions/geoserver-on-steroids-foss4g-2015 ) > > for more ideas.# > > > > Ian > > > > On 17 March 2016 at 09:51, Max wrote: > >> > >> I have a large number - more than 60 thousand - of relatively small > >> Geotiffs, usually from 2 to 12 Mb. I have a web client that uses > >> Leaflet to view them, but things are quite slow even inside our own > >> network. I have a hunch that both Geoserver and these Geotiffs are not > >> configured in the best way possible. > >> > >> The tiffs were generated from a bunch of EPSG:3035 Ascii grid files > >> using this command: > >> > >> for f in *.rsl; do gdal_translate -a_srs EPSG:3035 -co "TILED=YES" > >> -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "COMPRESS=DEFLATE" -co > >> "ZLEVEL=9" -co "BIGTIFF=YES" $f $f.tif; done > >> > >> Then I wrote a Python script that created a store and a layer for each > >> of the tiffs. > >> > >> Here is the output from gdalinfo for a typical one: > >> > >> Driver: GTiff/GeoTIFF > >> Files: se502c_R9_C1_3_T_2531_sec.rsl.tif > >> Size is 3396, 2271 > >> Coordinate System is: > >> PROJCS["ETRS89 / LAEA Europe", > >> GEOGCS["ETRS89", > >> DATUM["European_Terrestrial_Reference_System_1989", > >> SPHEROID["GRS 1980",6378137,298.2572221010002, > >> AUTHORITY["EPSG","7019"]], > >> TOWGS84[0,0,0,0,0,0,0], > >> AUTHORITY["EPSG","6258"]], > >> PRIMEM["Greenwich",0], > >> UNIT["degree",0.0174532925199433], > >> AUTHORITY["EPSG","4258"]], > >> PROJECTION["Lambert_Azimuthal_Equal_Area"], > >> PARAMETER["latitude_of_center",52], > >> PARAMETER["longitude_of_center",10], > >> PARAMETER["false_easting",4321000], > >> PARAMETER["false_northing",321], > >> > >> As you can see they are EPSG:3035. They should be internally tiled. In > >> the Coordinate Reference Systems for Geoserver, it says that both > >> Native SRS and Declared SRS are EPSG:3035, and that the handling > >> should be to reproject native to declared. > >> > >> My web client overlays these Geotiffs on a standard OpenStreetMap > >> layer in Web Mercator. All the geotiffs we have tried appear > >> correctly, so I guess reprojection is still happening at some stage. > >> > >> My gut feeling is that I might gain some speed by reprojecting the > >> original Geotiffs to Web Mercator, or at least changing the declared > >> SRS to Web Mercator in Geoserver. Not keen on it, it's going to take > >> days. What other properties of the Geotiffs could I tinker with? > >> > >> Would caching with GeoWebCache help, given the file size? > >> > >> Bonus question - if a Geotiff is changed, by reprojecting or retiling > >> or what have you, does the Geoserver layer associated with it get > >> invalidated? It took days to create them all. > >> > >> I'd really appreciate your feeback. > >> > >> Thanks in advance, > >> Max > >> > >> > >> > -- > >> Transform Data into Opportunity. > >> Accelerate data analysis in your applications with > >> Intel Data Analytics Acceleration Library. > >> Click to learn more. > >> http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > >> ___ > >> Geoserver-users mailing list > >> Geoserver-users@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/geoserver-users > > > > > > > > > > -- > > Ian Turton > > > -- > Transform Data into Opportunity. > Accelerate data analysis in your applications with > Intel Data Analytics Acceleration Library. > Click to learn more. > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > ___ > Geoserver-users mailing list > Geoserver-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geoserver-users > -- == GeoServer Profe
