[mapserver-users] RE: Raster rendering performance and zoom levels
Try this, it works in most cases 1- Fisrt enable internal tiling with the command gdal_translate -co "TILED=YES" original.tif tiled.tif 2-than add overviews gdaladdo -r average 2 4 8 16 32 64 128 3-create a tileindex gdaltindex myindex.shp *.tif 4-than create a spatial index shptree myindex.shp and finally add the layer to the map LAYER NAME myorthos TYPE RASTER DUMP TRUE STATUS ON TILEINDEX ../data/myindex TILEITEM Location -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-rendering-performance-and-zoom-levels-tp7091875p7092803.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster dataset recommendations
Hello Jean, You can use the gdal_retile.py script to generate the tiles and the zoom levels, than use the minscale and maxscale for showing and hiding the appropriate level. The raster data are not suggested to have them in a database or in your case in postgis. If they are file system it is faster, it is easier to maintain, to update tiles ect. You can see also other posts that give suggestion about file system vs database for raster data. -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-dataset-recommendations-tp6064491p6183710.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster dataset recommendations
Jean, thats my problem too. I haven't resolved it yet the perfect way, but i have created mosaics of the data with lowered resolution, and a lower number of tiles . The number of tiles that are requested to see the map increases when you zoom out, so it needs to seek and read the tiles. Having them in different HD will make the displaying of the map faster because you will have more than one hd that does the seek/read process. The idea is that in your zoom levels you dont need more than four tiles to display the image. So youll have to calculate the mapserver resolution, image resolution and map control size. I have more than 3 terabytes of data in 15000 tiles and i created another mosaic with 1.5 terabytes and ~ 4000 tiles. And so on. My main problem is determining the scale when to switch from one mosaic to the other and i resolved it with testing the speed of the response. Maybe this time someone will have an idea to this post. -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-dataset-recommendations-tp6064491p6066256.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster dataset recommendations
Since it as less than 4gb, if you have the image in different tiles just merge them together. The image format should be tif. 1- Fisrt enable internal tiling with the command gdal_translate -co "TILED=YES" original.tif tiled.tif 2-than add overviews gdaladdo -r average 2 4 8 16 32 64 128 3-create a tileindex gdaltindex myindex.shp *.tif 4-than create a spatial index shptree myindex.shp and finally add the layer to the map LAYER NAME myorthos TYPE RASTER DUMP TRUE STATUS ON TILEINDEX ../data/myindex TILEITEM Location -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-dataset-recommendations-tp6064491p6064777.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster Display Performance
one issue may be that you should not use .shp in the tileindex property, in this way mapserver dont uses the index.qix created by the shptree command. you should put only the name "index" -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-Display-Performance-tp4691679p5549558.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster Strategies Examples
Sorry but im still not understanding this: > That remains a complicated question. I believe I suggested 1/32 as a > good overview level since beyond that point a normal map view will tend > to involve a significant number of files. My only problem now is to find the correct scale for each layer. I use minscale and maxscale for the original layer and the mosaiced layer. So since my data will be shown from 1:150 to 1:100 the minscale of the original layer will be 100 and maxscale X. The minscale of the mosaic overview will be X and the maxscale 150 original: minscale 100 maxscale X mosaic: minscale X maxscale 150 i cant find X? -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440312.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster Strategies Examples
Sorry but im still not understanding this: > That remains a complicated question. I believe I suggested 1/32 as a > good overview level since beyond that point a normal map view will tend > to involve a significant number of files. My only problem now is to find the correct scale for each layer. I use minscale and maxscale for the original layer and the mosaiced layer. So since my data will be shown from 1:150 to 1:100 the minscale of the original layer will be 100 and maxscale X. The minscale of the mosaic overview will be X and the maxscale 150 original: minscale 100 maxscale X mosaic: minscale X maxscale 150 i cant find X? On Thu, Aug 19, 2010 at 3:15 PM, Frank Warmerdam [via OSGeo.org] < ml-node+5440161-2089753629-182...@n2.nabble.com > wrote: > Edi.Karadumi wrote: > > Thanx to you my questions are comming to an end > > > > Three more last things > > > > Should i retile the new mosaiced overview? Meaning should i create > smaller > > size tiles of the new mosaic? or 2Gb is not a problem > > Edi, > > Generally speaking, as long as you enable internal tiling when building > the mosaic there is no need to break it down into separate files. > Add '-co TILED=YES' to the gdal_merge.py command to produce an output > GeoTIFF file that is internally tiled. > > > And how can i find in which scale should i switch layers? So what will be > > > the maxscale of the original layer? > > That remains a complicated question. I believe I suggested 1/32 as a > good overview level since beyond that point a normal map view will tend > to involve a significant number of files. > > > if ill still have performance problems, or ill get 1-2 terabytes of data > > more should i make another mosaic with 0.08*16 or 0.08*8, or is there > > another strategy? > > I presume as you add data, your area is growing. If that is so, the 1/32nd > > overview level is still appropriate, and you just need to regenerate the > overview for the larger area. > > If you are using MapServer and GDAL then there is no real problem with > using > one large geotiff file for the overview as long as it has internal tiling, > even if it grows to many GB in size. > > Best regards, > -- > ---+-- > > I set the clouds in motion - turn up | Frank Warmerdam, [hidden > email]<http://user/SendEmail.jtp?type=node&node=5440161&i=0> > light and sound - activate the windows | http://pobox.com/~warmerdam > and watch the world go round - Rush| Geospatial Programmer for Rent > > ___ > mapserver-users mailing list > [hidden email] <http://user/SendEmail.jtp?type=node&node=5440161&i=1> > http://lists.osgeo.org/mailman/listinfo/mapserver-users > > > -- > View message @ > http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440161.html > To unsubscribe from Re: Raster Strategies Examples, click > here<http://osgeo-org.1803224.n2.nabble.com/template/NodeServlet.jtp?tpl=unsubscribe_by_code&node=5440104&code=ZWRpa2FyYWR1bWlAZ21haWwuY29tfDU0NDAxMDR8LTE4NTQwODgxMTM=>. > > > -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440310.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster Strategies Examples
Thanx to you my questions are comming to an end Three more last things Should i retile the new mosaiced overview? Meaning should i create smaller size tiles of the new mosaic? or 2Gb is not a problem And how can i find in which scale should i switch layers? So what will be the maxscale of the original layer? if ill still have performance problems, or ill get 1-2 terabytes of data more should i make another mosaic with 0.08*16 or 0.08*8, or is there another strategy? On Thu, Aug 19, 2010 at 2:46 PM, Frank Warmerdam [via OSGeo.org] wrote: Edi.Karadumi wrote: > Sorry to bother again, when i use gdal_merge.py the part -ps 160 160, > thats > the pixel dimension and i get an image 1kb. The pixel size of the original > file is 0.08 meters. So i think i should use 0.08x32 in the gdal_merge.py > command? Edi, If the original resolution is 8cm then the 1/32nd overview should indeed have a pixel size of 0.08x32 or 2.56m. > And how many tiles should i merge together? Now i have 2T images > and 2000Gb/32 = 62Gb the rezult image. Should i merge them all together and > than recreate tiles? Even if i had 1200gb, 1200/32=37.5gb for an image not > 1gb. That is still an image size that cant be handled Note that the 1/32nd overview is reduced by a factor of 32 in x and y so overall it will be original_size/(32*32). So 2TB would reduce to about 1.95GB. Best regards, -- -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440104.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster Strategies Examples
Thanx to you my questions are comming to an end [?] Three more last things Should i retile the new mosaiced overview? Meaning should i create smaller size tiles of the new mosaic? And how can i find in which scale should i switch layers? So what will be the maxscale of the original layer? if ill still have performance problems, or ill get 1-2 terabytes of data more should i make another mosaic with 0.08*16 or 0.08*8, or is there another strategy? On Thu, Aug 19, 2010 at 2:46 PM, Frank Warmerdam [via OSGeo.org] < ml-node+5440051-1962745825-182...@n2.nabble.com > wrote: > Edi.Karadumi wrote: > > Sorry to bother again, when i use gdal_merge.py the part -ps 160 160, > thats > > the pixel dimension and i get an image 1kb. The pixel size of the > original > > file is 0.08 meters. So i think i should use 0.08x32 in the gdal_merge.py > > > command? > > Edi, > > If the original resolution is 8cm then the 1/32nd overview should indeed > have a pixel size of 0.08x32 or 2.56m. > > > And how many tiles should i merge together? Now i have 2T images > > and 2000Gb/32 = 62Gb the rezult image. Should i merge them all together > and > > than recreate tiles? Even if i had 1200gb, 1200/32=37.5gb for an image > not > > 1gb. That is still an image size that cant be handled > > Note that the 1/32nd overview is reduced by a factor of 32 in x and y > so overall it will be original_size/(32*32). So 2TB would reduce to > about 1.95GB. > > Best regards, > -- > ---+-- > > I set the clouds in motion - turn up | Frank Warmerdam, [hidden > email]<http://user/SendEmail.jtp?type=node&node=5440051&i=0> > light and sound - activate the windows | http://pobox.com/~warmerdam > and watch the world go round - Rush| Geospatial Programmer for Rent > > ___ > mapserver-users mailing list > [hidden email] <http://user/SendEmail.jtp?type=node&node=5440051&i=1> > http://lists.osgeo.org/mailman/listinfo/mapserver-users > > > -- > View message @ > http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440051.html > To unsubscribe from Re: Raster Strategies Examples, click > here<http://osgeo-org.1803224.n2.nabble.com/template/NodeServlet.jtp?tpl=unsubscribe_by_code&node=5439959&code=ZWRpa2FyYWR1bWlAZ21haWwuY29tfDU0Mzk5NTl8LTE4NTQwODgxMTM=>. > > > -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440101.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster Strategies Examples
Sorry to bother again, when i use gdal_merge.py the part -ps 160 160, thats the pixel dimension and i get an image 1kb. The pixel size of the original file is 0.08 meters. So i think i should use 0.08x32 in the gdal_merge.py command? And how many tiles should i merge together? Now i have 2T images and 2000Gb/32 = 62Gb the rezult image. Should i merge them all together and than recreate tiles? Even if i had 1200gb, 1200/32=37.5gb for an image not 1gb. That is still an image size that cant be handled On Tue, Aug 3, 2010 at 2:13 PM, Frank Warmerdam [via OSGeo.org] wrote: Folks, I'm sorry for speaking up on this without having read the other replies. Forgive me for repeating information. Edi.Karadumi wrote: > as you may know, i have very slow performance when i zoom out and im > stucked > here. As i have read i should make a copy of the tiles with reduced > resolution. Merge the tiles together and use min/max scale to show > different > layers in different scales. the min/max scales i zoom in/out are > 100/120. Now my questions are > > - How can i calculate the scale where i should create another layer of the > tiles, or i shoud see it with some tests? I find the MapServer scale values very confusing myself. If I want to compute the scale for particular request (for instance to establish breakpoints for MINSCALE/MAXSCALE), I do a request at the target resolution with shp2img and debug output on and examine the debug output for the scale reported. > - how much should be the resolution of the new layer? You basically want to move to a new mosaiced overview layer at the point where a map request is likely to request several tiles at once. If we, roughly, assume a map request is on the order of 500x500 then at 32 times reduction in resolution would give a request over an area of 16000 x 16000 which is certain to touch several of your 6000 x 6000 base files. So, I'd aim to have this merged mosaic at approximately 1/32nd of the resolution of the original imagery. 1200GB of full res imagery reduced to 1/32nd resolution is only about 1GB so it is a very managable size for an overview layer. > - is there any tools or program to merge the tiles? merging 6000 tiles > with > the gdalwarp by writing the command by myself is frustrating gdalwarp can take multiple input files in one run, but there are some issues with how gdalwarp is implemented that make it not scale well to many input files without great care. Since the input files are quite a modest size I would suggest using gdal_merge.py to mosaic them at a reduced resolution. If your original imagery were 5m pixels something like: gdal_merge.py -o overview_mosaic.tif -ps 160 160 */*.tif might do the trick. > - how many tiles should i merge together to create the new layer? (how > many > tiles should have the new layer) i know that in each zoomscale its better > to > appear only one tile but i dont know how to calculate it > > - the tiles that i should merge are the originals or those with internal > tiling and overviews? If you use gdal_merge.py, it is best to use input files that have overviews already built. This will ensure that the mosaic built uses the same downsampling technique (ie. averaging) that you used to build the overviews. It will also make the mosaicing much faster. Best regards, -- -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5439959.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster Strategies Examples
On Tue, Aug 3, 2010 at 2:12 PM, Frank Warmerdam wrote: I find the MapServer scale values very confusing myself. If I want to compute the scale for particular request (for instance to establish breakpoints for MINSCALE/MAXSCALE), I do a request at the target resolution with shp2img and debug output on and examine the debug output for the scale reported. you mean to use a command like shp2img -m All.map -o test.png -map_debug 3 and see if the response time is good? what do you mean by examine the debug output for the scale reported? -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5371817.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Raster Strategies Examples
i dont have problems to calculate the scale of an image, because im using one of mapserver apps, fusion to show the map and it has widgets do display the scale of a zoom in/out, or i just use map.scaledenom to show the current scale. My problem is how to find the scale where i should create the new mosaic overview. I have read that we should not use more than 4 tiles to display a request from the client, the Quadtree Structure or Q-Tree. As i have read we should start with max zoom than zooming out till performance decreases. That will be the breakpoint of the new scale. But i cant calculate it cuz i get different responses and im hosting the server so practically i have more performance than other users in other places or countries. Thnx all for the replies -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5371539.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Raster Strategies Examples
I have read many strategies for raster performance, but i still have problems with my case. The posts that i have read explain the strategie, but are not very detailed. Im new to mapserver so i have problems implementing them. My case is: -about 6000 tiles that form the map -170mb each tile -aproximately 1.2T of iamges -i have 10T HD and i think disk space is not a problem for me -tile format is TIF -tile size 9375x6250 pixels The strategy i have implemented and the problems i have -First i divided the tiles in 60 folders, to increase the performance in disk seek/read. -Than i transformed source files into internally tiles with the command gdal_translate -of GTiff -co "TILED=YES" original.tif tiled.tif -i used gdaladdo to add internal pyramids gdaladdo -r average 2 4 8 16 32 64 128 -Created a tileindex using gdaltindex -write_absolute_path MapAll.shp //server/Maps/Subfolder1/*.tif -Created a spatial index .qix file shptree MapAll.shp -than added the layer to the mapfile without the .shp extension so the application can use the .qix as you may know, i have very slow performance when i zoom out and im stucked here. As i have read i should make a copy of the tiles with reduced resolution. Merge the tiles together and use min/max scale to show different layers in different scales. the min/max scales i zoom in/out are 100/120. Now my questions are - How can i calculate the scale where i should create another layer of the tiles, or i shoud see it with some tests? - how much should be the resolution of the new layer? - is there any tools or program to merge the tiles? merging 6000 tiles with the gdalwarp by writing the command by myself is frustrating - how many tiles should i merge together to create the new layer? (how many tiles should have the new layer) i know that in each zoomscale its better to appear only one tile but i dont know how to calculate it - the tiles that i should merge are the originals or those with internal tiling and overviews? -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5363321.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
