Re: [mapserver-users] Hello from Finland
On 3/16/21 10:23 PM, Rahkonen Jukka (MML) wrote: Gdalinfo -stats reports the min/max values but not their locations. As far as I know there is no GDAL tool or Python script in the GDAL samples that can do that. Users have been asking for such tool in the mailing lists and in gis.stackexchange so if you find of create such utility please let me know. I'm fairly sure, I ran across something some time ago when I was googling for a solution for the H and L issue. I'll try to find it. For finding the local min/max areas you could perhaps try the gdaldem tools https://gdal.org/programs/gdaldem.html. Areas with slope=0 should be what you are searching. I'll look into it, thanks! -Stefan -- (o_ Stefan Gofferje| SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Hello from Finland
Hi, -Alkuperäinen viesti- Stefan Gofferje wrote: > Terve Jukka, > On 3/16/21 12:54 PM, Rahkonen Jukka (MML) wrote: >> When it comes to memory usage with raster files, Mapserver is reading >> rasters with GDAL so in this case with >> https://gdal.org/drivers/raster/grib.html. I have never been playing with >> GRIB files and I do not know how well the format is optimized for features >> that are needed for making fast map services: fast access to subregions and >> overview levels. I have no idea how well GRIB files are indexed for random >> access, it there is difference between png and JPEG2000 encoding, and how >> well the GDAL GRIB driver can utilize the possibilities that the format >> offer. > That sounds good. IIRC GDAL is pretty mature with its memory management. >At least I haven't manged to crash it yet :D. >> The most common raster format that Mapserver services are using is certainly >> GeoTIFF and when they are tiled, compressed, and have overviews they work >> pretty well. It is certainly an advantage if you can use the GRIB files >> directly without conversions. I see that you are using Mapcache in front of >> Mapserver and it will help a lot if you pre-seed the tiles. If you do not >> pre-seed then each time when the user is the first visitor on some site the >> bottle neck will be Mapserver. > That is a good point! Especially as the forecast data is updated every 6 > hours. I will have to integrate that into my update script somewhow once I > got there. >> The Ls/Hs idea is interesting. I do not believe there is anything ready >> available but I know that my colleagues at the Finnish Geodetic Institute >> have made quite nice things with Mapserver and shell scripts running some >> GDAL commands. Should they be computed on-the-fly for the area that is >> visible on the map? > I'm doing some GDAL stuff in shell scripts for beta.saakeskus.fi at the > moment. You can indeed do some fun stuff with the GDAL tools. In the long > run, I was thinking of changing from bash scripts to Python, though, because > then I can make some more calculations than are possible just with the GDAL > tools. > Yes, I was thinking if it was possible to do that inside of mapserver. Finding _the_ highest and lowest point is fairly trivial with GDAL. IIRC, gdalinfo can output that info by default. A little more challenging is finding all points which are highest or lowest among their neighbors. That's what's needed for a weather map because usually, you have more than one high and one low area on one map. Gdalinfo -stats reports the min/max values but not their locations. As far as I know there is no GDAL tool or Python script in the GDAL samples that can do that. Users have been asking for such tool in the mailing lists and in gis.stackexchange so if you find of create such utility please let me know. For finding the local min/max areas you could perhaps try the gdaldem tools https://gdal.org/programs/gdaldem.html. Areas with slope=0 should be what you are searching. -Jukka- > -Stefan -- (o_ Stefan Gofferje| SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Hello from Finland
Terve Jukka, On 3/16/21 12:54 PM, Rahkonen Jukka (MML) wrote: When it comes to memory usage with raster files, Mapserver is reading rasters with GDAL so in this case with https://gdal.org/drivers/raster/grib.html. I have never been playing with GRIB files and I do not know how well the format is optimized for features that are needed for making fast map services: fast access to subregions and overview levels. I have no idea how well GRIB files are indexed for random access, it there is difference between png and JPEG2000 encoding, and how well the GDAL GRIB driver can utilize the possibilities that the format offer. That sounds good. IIRC GDAL is pretty mature with its memory management. At least I haven't manged to crash it yet :D. The most common raster format that Mapserver services are using is certainly GeoTIFF and when they are tiled, compressed, and have overviews they work pretty well. It is certainly an advantage if you can use the GRIB files directly without conversions. I see that you are using Mapcache in front of Mapserver and it will help a lot if you pre-seed the tiles. If you do not pre-seed then each time when the user is the first visitor on some site the bottle neck will be Mapserver. That is a good point! Especially as the forecast data is updated every 6 hours. I will have to integrate that into my update script somewhow once I got there. The Ls/Hs idea is interesting. I do not believe there is anything ready available but I know that my colleagues at the Finnish Geodetic Institute have made quite nice things with Mapserver and shell scripts running some GDAL commands. Should they be computed on-the-fly for the area that is visible on the map? I'm doing some GDAL stuff in shell scripts for beta.saakeskus.fi at the moment. You can indeed do some fun stuff with the GDAL tools. In the long run, I was thinking of changing from bash scripts to Python, though, because then I can make some more calculations than are possible just with the GDAL tools. Yes, I was thinking if it was possible to do that inside of mapserver. Finding _the_ highest and lowest point is fairly trivial with GDAL. IIRC, gdalinfo can output that info by default. A little more challenging is finding all points which are highest or lowest among their neighbors. That's what's needed for a weather map because usually, you have more than one high and one low area on one map. -Stefan -- (o_ Stefan Gofferje| SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Hello from Finland
Hi Jukka, You might want to look at: https://www.pydap.org/en/latest/index.html It is the backend of ifm.fcoo.dk using netcdf files. grib can be converted to netcdf. Søren FORSVARSMINISTERIETS MATERIEL- OG INDKØBSSTYRELSE DANISH MINISTRY OF DEFENCE ACQUISITION AND LOGISTICS ORGANISATION Joint GEOMETOC Support Centre Lautrupbjerg 1-5, DK-2750 Ballerup TLF: +45 7281 5627 MOBIL: +45 2961 9652 E-mail: fmi-ma-op...@mil.dk<mailto:fmi-ma-op...@mil.dk> www.fmi.dk <https://mail.fcoo.dk/owa/s...@fcoo.dk/redir.aspx?C=XRPK8XZJebk6vtukXbinjL73cjoSMG7ceUiX8edX2NLcBHhZKovVCA..&URL=http%3a%2f%2fWWW.FCOO.DK%2f> Fra: mapserver-users på vegne af Rahkonen Jukka (MML) Sendt: 16. marts 2021 11:54:13 Til: Stefan Gofferje; mapserver-users@lists.osgeo.org Emne: Re: [mapserver-users] Hello from Finland Hi Stefan, When it comes to memory usage with raster files, Mapserver is reading rasters with GDAL so in this case with https://gdal.org/drivers/raster/grib.html. I have never been playing with GRIB files and I do not know how well the format is optimized for features that are needed for making fast map services: fast access to subregions and overview levels. I have no idea how well GRIB files are indexed for random access, it there is difference between png and JPEG2000 encoding, and how well the GDAL GRIB driver can utilize the possibilities that the format offer. The most common raster format that Mapserver services are using is certainly GeoTIFF and when they are tiled, compressed, and have overviews they work pretty well. It is certainly an advantage if you can use the GRIB files directly without conversions. I see that you are using Mapcache in front of Mapserver and it will help a lot if you pre-seed the tiles. If you do not pre-seed then each time when the user is the first visitor on some site the bottle neck will be Mapserver. The Ls/Hs idea is interesting. I do not believe there is anything ready available but I know that my colleagues at the Finnish Geodetic Institute have made quite nice things with Mapserver and shell scripts running some GDAL commands. Should they be computed on-the-fly for the area that is visible on the map? I wonder if it could be possible to query the CONTOUR layer that you already has https://www.mapserver.org/input/vector/contour.html and search the min/max values from there. Or then to improve the contouring algorithm a bit because it is already processing all the pixels. -Jukka Rahkonen- -Alkuperäinen viesti- Lähettäjä: mapserver-users Puolesta Stefan Gofferje Lähetetty: tiistai 16. maaliskuuta 2021 12.11 Vastaanottaja: mapserver-users@lists.osgeo.org Aihe: Re: [mapserver-users] Hello from Finland Hi Jeff, On 3/15/21 2:35 PM, Jeff McKenna wrote: > Thanks Stefan for this wonderful introduction, it's nice to hear of > > positive experiences to give us all more motivation to continue to > > helping to improve MapServer! Welcome to both you and Iliya to the > > community. Keep us posted on your activities! > > Oh, once your MapServer backend is plugged-in be sure to add your > weather > site to the MapServer gallery, so we can help new users see > what can be > done... Thanks for your nice words! I now have mapserver and mapcache running successfully and talking to each other in my home lab. I have set up a number of test layers from different input sources pretty successfully. The documentation and the configuration required some adaption of my thinking processes because it's pretty different from anything else I have ever done before but I love this kind of challenges! Currently I have only two issues which prevent me from doing the swap right now. Thanks to NOAA using a 0-360° grid instead of (-180)-180°, there's some artefacts when reprojecting to epsg:3857. For raster maps this is already addressed in #6106. I ran into an issue which LINE maps with a CONTOUR source which probably is related and opened #6261. Then I have a question... Although it's not strictly necessary, I would like to paint those Hs and Ls in my atmospheric pressure map. Is there a way to run scripts or something like that in mapserver. I could probably run some GDAL stuff externally over the grid files in my download script and create some point overlays which could be in a GROUP with the isoline layer but doing this inside of mapserver would kinda be cooler. And a second question: How is mapserver handling raster files memory-wise? Does it try to load the whole file to the memory at once? My biggest issue during the whole 14 years I run this website always has been balancing preprocessing and map rendering. The more preprocessing I have to do, the longer the whole process of map generation takes but of course I couldn't afford to buy a $100k top-of-the-line server with terabytes of RAM and plenty of CPU co
Re: [mapserver-users] Hello from Finland
Hi Stefan, When it comes to memory usage with raster files, Mapserver is reading rasters with GDAL so in this case with https://gdal.org/drivers/raster/grib.html. I have never been playing with GRIB files and I do not know how well the format is optimized for features that are needed for making fast map services: fast access to subregions and overview levels. I have no idea how well GRIB files are indexed for random access, it there is difference between png and JPEG2000 encoding, and how well the GDAL GRIB driver can utilize the possibilities that the format offer. The most common raster format that Mapserver services are using is certainly GeoTIFF and when they are tiled, compressed, and have overviews they work pretty well. It is certainly an advantage if you can use the GRIB files directly without conversions. I see that you are using Mapcache in front of Mapserver and it will help a lot if you pre-seed the tiles. If you do not pre-seed then each time when the user is the first visitor on some site the bottle neck will be Mapserver. The Ls/Hs idea is interesting. I do not believe there is anything ready available but I know that my colleagues at the Finnish Geodetic Institute have made quite nice things with Mapserver and shell scripts running some GDAL commands. Should they be computed on-the-fly for the area that is visible on the map? I wonder if it could be possible to query the CONTOUR layer that you already has https://www.mapserver.org/input/vector/contour.html and search the min/max values from there. Or then to improve the contouring algorithm a bit because it is already processing all the pixels. -Jukka Rahkonen- -Alkuperäinen viesti- Lähettäjä: mapserver-users Puolesta Stefan Gofferje Lähetetty: tiistai 16. maaliskuuta 2021 12.11 Vastaanottaja: mapserver-users@lists.osgeo.org Aihe: Re: [mapserver-users] Hello from Finland Hi Jeff, On 3/15/21 2:35 PM, Jeff McKenna wrote: > Thanks Stefan for this wonderful introduction, it's nice to hear of > > positive experiences to give us all more motivation to continue to > > helping to improve MapServer! Welcome to both you and Iliya to the > > community. Keep us posted on your activities! > > Oh, once your MapServer backend is plugged-in be sure to add your > weather > site to the MapServer gallery, so we can help new users see > what can be > done... Thanks for your nice words! I now have mapserver and mapcache running successfully and talking to each other in my home lab. I have set up a number of test layers from different input sources pretty successfully. The documentation and the configuration required some adaption of my thinking processes because it's pretty different from anything else I have ever done before but I love this kind of challenges! Currently I have only two issues which prevent me from doing the swap right now. Thanks to NOAA using a 0-360° grid instead of (-180)-180°, there's some artefacts when reprojecting to epsg:3857. For raster maps this is already addressed in #6106. I ran into an issue which LINE maps with a CONTOUR source which probably is related and opened #6261. Then I have a question... Although it's not strictly necessary, I would like to paint those Hs and Ls in my atmospheric pressure map. Is there a way to run scripts or something like that in mapserver. I could probably run some GDAL stuff externally over the grid files in my download script and create some point overlays which could be in a GROUP with the isoline layer but doing this inside of mapserver would kinda be cooler. And a second question: How is mapserver handling raster files memory-wise? Does it try to load the whole file to the memory at once? My biggest issue during the whole 14 years I run this website always has been balancing preprocessing and map rendering. The more preprocessing I have to do, the longer the whole process of map generation takes but of course I couldn't afford to buy a $100k top-of-the-line server with terabytes of RAM and plenty of CPU cores for the rendering... I'm thinking about my input file strategy at the moment... 1.) One GRIB file for everything (around 1.9GB) 2.) One GRIB file per parameter for all time steps 3.) One GRIB file per time step including all parameters 4.) One GRIB file per parameter for each time step 4. would be the most complex to configure and also would be fun (irony!) to make local forecasts from but probably take the least memory. 2. and 3. are a fairly good compromise with 2 being easier for calculating local forecasts and 3 easier for calculating additional stuff like my "severe weather warning level" 1. would be easiest for rendering and calculating but even at the moment with GrADS I every now and then run into memory issues (leaks?). This project really turns out to be a fun challenge! Stefan -- (o_ Stefan Gofferje
Re: [mapserver-users] Hello from Finland
Hi Jeff, On 3/15/21 2:35 PM, Jeff McKenna wrote: > Thanks Stefan for this wonderful introduction, it's nice to hear of > positive experiences to give us all more motivation to continue to > helping to improve MapServer! Welcome to both you and Iliya to the > community. Keep us posted on your activities! > > Oh, once your MapServer backend is plugged-in be sure to add your > weather site to the MapServer gallery, so we can help new users see > what can be done... Thanks for your nice words! I now have mapserver and mapcache running successfully and talking to each other in my home lab. I have set up a number of test layers from different input sources pretty successfully. The documentation and the configuration required some adaption of my thinking processes because it's pretty different from anything else I have ever done before but I love this kind of challenges! Currently I have only two issues which prevent me from doing the swap right now. Thanks to NOAA using a 0-360° grid instead of (-180)-180°, there's some artefacts when reprojecting to epsg:3857. For raster maps this is already addressed in #6106. I ran into an issue which LINE maps with a CONTOUR source which probably is related and opened #6261. Then I have a question... Although it's not strictly necessary, I would like to paint those Hs and Ls in my atmospheric pressure map. Is there a way to run scripts or something like that in mapserver. I could probably run some GDAL stuff externally over the grid files in my download script and create some point overlays which could be in a GROUP with the isoline layer but doing this inside of mapserver would kinda be cooler. And a second question: How is mapserver handling raster files memory-wise? Does it try to load the whole file to the memory at once? My biggest issue during the whole 14 years I run this website always has been balancing preprocessing and map rendering. The more preprocessing I have to do, the longer the whole process of map generation takes but of course I couldn't afford to buy a $100k top-of-the-line server with terabytes of RAM and plenty of CPU cores for the rendering... I'm thinking about my input file strategy at the moment... 1.) One GRIB file for everything (around 1.9GB) 2.) One GRIB file per parameter for all time steps 3.) One GRIB file per time step including all parameters 4.) One GRIB file per parameter for each time step 4. would be the most complex to configure and also would be fun (irony!) to make local forecasts from but probably take the least memory. 2. and 3. are a fairly good compromise with 2 being easier for calculating local forecasts and 3 easier for calculating additional stuff like my "severe weather warning level" 1. would be easiest for rendering and calculating but even at the moment with GrADS I every now and then run into memory issues (leaks?). This project really turns out to be a fun challenge! Stefan -- (o_ Stefan Gofferje| SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Hello from Finland
Thanks Stefan for this wonderful introduction, it's nice to hear of positive experiences to give us all more motivation to continue to helping to improve MapServer! Welcome to both you and Iliya to the community. Keep us posted on your activities! Oh, once your MapServer backend is plugged-in be sure to add your weather site to the MapServer gallery, so we can help new users see what can be done... https://github.com/mapserver/mapserver/wiki/MapServer-Site-Gallery Hello to Finland from the east coast of Canada! -jeff -- Jeff McKenna GatewayGeo: Developers of MS4W, MapServer Consulting and Training co-founder of FOSS4G http://gatewaygeo.com/ On 2021-03-14 7:43 a.m., Stefan Gofferje wrote: Hello all! I just wanted to quickly introduce myself. I just started tinkering with mapserver to see if I can make it work as a new backend for my weather website. I'm running www.saakeskus.fi as a free website for stormchasers in Finland since 2007. At the moment, it's tons of pre-rendered maps which are done with GrADS in a Joomla website. A while ago I started reworking the web part based on Leaflet and and started writing new map creation scripts in bash with GDAL. That works pretty nicely but results in one huge mapfile for each parameter at each time... And then I stumbled over mapserver... :). I have a test server running on docker on my home server at the moment and I'm making (I think) pretty good progress understanding how it works. I don't really know a lot about GIS stuff but I got my first working WMS maps from grib files out of it after about one and a half day of tinkering. On the old version of my website, I published almost all my GrADS scripts because there wasn't much information on the web on how to do nice weather maps with GrADS when I started. If I can make mapserver work for me, I'm planning to do the same with mapserver map files. Looking forward to the community! Have a beautiful Sunday! Stefan ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Hello from Finland
Welcome Stefan! Your plan sounds great and I look forward to it. I'm also somewhat new here :) Iljya On Sun., Mar. 14, 2021, 11:53 Stefan Gofferje, wrote: > Hello all! > > I just wanted to quickly introduce myself. I just started tinkering with > mapserver to see if I can make it work as a new backend for my weather > website. I'm running www.saakeskus.fi as a free website for stormchasers > in Finland since 2007. At the moment, it's tons of pre-rendered maps > which are done with GrADS in a Joomla website. A while ago I started > reworking the web part based on Leaflet and and started writing new map > creation scripts in bash with GDAL. That works pretty nicely but results > in one huge mapfile for each parameter at each time... > And then I stumbled over mapserver... :). > I have a test server running on docker on my home server at the moment > and I'm making (I think) pretty good progress understanding how it > works. I don't really know a lot about GIS stuff but I got my first > working WMS maps from grib files out of it after about one and a half > day of tinkering. > On the old version of my website, I published almost all my GrADS > scripts because there wasn't much information on the web on how to do > nice weather maps with GrADS when I started. If I can make mapserver > work for me, I'm planning to do the same with mapserver map files. > > Looking forward to the community! > Have a beautiful Sunday! > Stefan > > -- > (o_ Stefan Gofferje| SCLT, MCP, CCSA > //\ Reg'd Linux User #247167 | VCP #2263 > V_/_ https://www.gofferje.net | https://www.saakeskus.fi > > ___ > mapserver-users mailing list > mapserver-users@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users > ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Hello from Finland
Hello all! I just wanted to quickly introduce myself. I just started tinkering with mapserver to see if I can make it work as a new backend for my weather website. I'm running www.saakeskus.fi as a free website for stormchasers in Finland since 2007. At the moment, it's tons of pre-rendered maps which are done with GrADS in a Joomla website. A while ago I started reworking the web part based on Leaflet and and started writing new map creation scripts in bash with GDAL. That works pretty nicely but results in one huge mapfile for each parameter at each time... And then I stumbled over mapserver... :). I have a test server running on docker on my home server at the moment and I'm making (I think) pretty good progress understanding how it works. I don't really know a lot about GIS stuff but I got my first working WMS maps from grib files out of it after about one and a half day of tinkering. On the old version of my website, I published almost all my GrADS scripts because there wasn't much information on the web on how to do nice weather maps with GrADS when I started. If I can make mapserver work for me, I'm planning to do the same with mapserver map files. Looking forward to the community! Have a beautiful Sunday! Stefan -- (o_ Stefan Gofferje| SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
