[mapserver-users] Re: Raster Strategies Examples

[Edi.Karadumi]

i dont have problems to calculate the scale of an image, because im using one
of mapserver apps, fusion to show the map and it has widgets do display the
scale of a zoom in/out, or i just use map.scaledenom to show the current
scale. My problem is how to find the scale where i should create the new
mosaic overview. I have read that we should not use more than 4 tiles to
display a request from the client, the Quadtree Structure or Q-Tree. As i
have read we should start with max zoom than zooming out till performance
decreases. That will be the breakpoint of the new scale. But i cant
calculate it cuz i get different responses and im hosting the server so
practically i have more performance than other users in other places or
countries. Thnx all for the replies
-- 
View this message in context: 
http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5371539.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Re: Raster Strategies Examples

[Edi.Karadumi]

On Tue, Aug 3, 2010 at 2:12 PM, Frank Warmerdam wrote:


I find the MapServer scale values very confusing myself.  If I want
to compute the scale for particular request (for instance to establish
breakpoints for MINSCALE/MAXSCALE), I do a request at the target resolution
with shp2img and debug output on and examine the debug output for the scale
reported.

you mean to use a command like 

shp2img -m All.map -o test.png -map_debug 3

and see if the response time is good?

what do you mean by examine the debug output for the scale 
reported? 

-- 
View this message in context: 
http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5371817.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Re: Raster Strategies Examples

[Edi.Karadumi]

Sorry to bother again, when i use gdal_merge.py the part -ps 160 160, thats
the pixel dimension and i get an image 1kb. The pixel size of the original
file is 0.08 meters. So i think i should use 0.08x32 in the gdal_merge.py
command? And how many tiles should i merge together? Now i have 2T images
and 2000Gb/32 = 62Gb the rezult image. Should i merge them all together and
than recreate tiles? Even if i had 1200gb, 1200/32=37.5gb for an image not
1gb. That is still an image size that cant be handled




On Tue, Aug 3, 2010 at 2:13 PM, Frank Warmerdam [via OSGeo.org]
 wrote:

Folks, 

I'm sorry for speaking up on this without having read the other replies. 
Forgive me for repeating information. 

Edi.Karadumi wrote: 
> as you may know, i have very slow performance when i zoom out and im
> stucked 
> here. As i have read i should make a copy of the tiles with reduced 
> resolution. Merge the tiles together and use min/max scale to show
> different 
> layers in different scales. the min/max scales i zoom in/out are 
> 100/120. Now my questions are 
> 
> - How can i calculate the scale where i should create another layer of the 
> tiles, or i shoud see it with some tests? 

I find the MapServer scale values very confusing myself.  If I want 
to compute the scale for particular request (for instance to establish 
breakpoints for MINSCALE/MAXSCALE), I do a request at the target resolution 
with shp2img and debug output on and examine the debug output for the scale 
reported. 

> - how much should be the resolution of the new layer? 

You basically want to move to a new mosaiced overview layer at the point 
where a map request is likely to request several tiles at once.  If 
we, roughly, assume a map request is on the order of 500x500 then at 
32 times reduction in resolution would give a request over an area of 
16000 x 16000 which is certain to touch several of your 6000 x 6000 
base files.  So, I'd aim to have this merged mosaic at approximately 
1/32nd of the resolution of the original imagery. 

1200GB of full res imagery reduced to 1/32nd resolution is only about 1GB 
so it is a very managable size for an overview layer. 

> - is there any tools or program to merge the tiles? merging 6000 tiles
> with 
> the gdalwarp by writing the command by myself is frustrating 

gdalwarp can take multiple input files in one run, but there are 
some issues with how gdalwarp is implemented that make it not scale 
well to many input files without great care.  Since the input files 
are quite a modest size I would suggest using gdal_merge.py to 
mosaic them at a reduced resolution.  If your original imagery were 
5m pixels something like: 

gdal_merge.py -o overview_mosaic.tif -ps 160 160 */*.tif 

might do the trick. 

> - how many tiles should i merge together to create the new layer? (how
> many 
> tiles should have the new layer) i know that in each zoomscale its better
> to 
> appear only one tile but i dont know how to calculate it 
> 
> - the tiles that i should merge are the originals or those with internal 
> tiling and overviews? 

If you use gdal_merge.py, it is best to use input files that have overviews 
already built.  This will ensure that the mosaic built uses the same 
downsampling technique (ie. averaging) that you used to build the overviews. 
It will also make the mosaicing much faster. 

Best regards, 
-- 
-- 
View this message in context: 
http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5439959.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Re: Raster Strategies Examples

[Edi.Karadumi]

Thanx to you my questions are comming to an end [?]

Three more last things

Should i retile the new mosaiced overview? Meaning should i create smaller
size tiles of the new mosaic?

And how can i find in which scale should i switch layers? So what will be
the maxscale of the original layer?

if ill still have performance problems, or ill get 1-2 terabytes of data
more should i make another mosaic with 0.08*16 or 0.08*8, or is there
another strategy?



On Thu, Aug 19, 2010 at 2:46 PM, Frank Warmerdam [via OSGeo.org] <
ml-node+5440051-1962745825-182...@n2.nabble.com
> wrote:

> Edi.Karadumi wrote:
> > Sorry to bother again, when i use gdal_merge.py the part -ps 160 160,
> thats
> > the pixel dimension and i get an image 1kb. The pixel size of the
> original
> > file is 0.08 meters. So i think i should use 0.08x32 in the gdal_merge.py
>
> > command?
>
> Edi,
>
> If the original resolution is 8cm then the 1/32nd overview should indeed
> have a pixel size of 0.08x32 or 2.56m.
>
>  > And how many tiles should i merge together? Now i have 2T images
>  > and 2000Gb/32 = 62Gb the rezult image. Should i merge them all together
> and
>  > than recreate tiles? Even if i had 1200gb, 1200/32=37.5gb for an image
> not
>  > 1gb. That is still an image size that cant be handled
>
> Note that the 1/32nd overview is reduced by a factor of 32 in x and y
> so overall it will be original_size/(32*32).  So 2TB would reduce to
> about 1.95GB.
>
> Best regards,
> --
> ---+--
>
> I set the clouds in motion - turn up   | Frank Warmerdam, [hidden 
> email]
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush| Geospatial Programmer for Rent
>
> ___
> mapserver-users mailing list
> [hidden email] 
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
>
> --
>  View message @
> http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440051.html
> To unsubscribe from Re: Raster Strategies Examples, click 
> here.
>
>
>

 
-- 
View this message in context: 
http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440101.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Re: Raster Strategies Examples

[Edi.Karadumi]

Thanx to you my questions are comming to an end  

Three more last things

Should i retile the new mosaiced overview? Meaning should i create smaller
size tiles of the new mosaic? or 2Gb is not a problem

And how can i find in which scale should i switch layers? So what will be
the maxscale of the original layer?

if ill still have performance problems, or ill get 1-2 terabytes of data
more should i make another mosaic with 0.08*16 or 0.08*8, or is there
another strategy?



On Thu, Aug 19, 2010 at 2:46 PM, Frank Warmerdam [via OSGeo.org]
 wrote:
Edi.Karadumi wrote: 
> Sorry to bother again, when i use gdal_merge.py the part -ps 160 160,
> thats 
> the pixel dimension and i get an image 1kb. The pixel size of the original 
> file is 0.08 meters. So i think i should use 0.08x32 in the gdal_merge.py 
> command? 

Edi, 

If the original resolution is 8cm then the 1/32nd overview should indeed 
have a pixel size of 0.08x32 or 2.56m. 

 > And how many tiles should i merge together? Now i have 2T images 
 > and 2000Gb/32 = 62Gb the rezult image. Should i merge them all together
and 
 > than recreate tiles? Even if i had 1200gb, 1200/32=37.5gb for an image
not 
 > 1gb. That is still an image size that cant be handled 

Note that the 1/32nd overview is reduced by a factor of 32 in x and y 
so overall it will be original_size/(32*32).  So 2TB would reduce to 
about 1.95GB. 

Best regards, 
-- 


-- 
View this message in context: 
http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440104.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Re: Raster Strategies Examples

[Edi.Karadumi]

Sorry but im still not understanding this:

> That remains a complicated question.  I believe I suggested 1/32 as a
> good overview level since beyond that point a normal map view will tend
> to involve a significant number of files.

My only problem now is to find the correct scale for each layer. I use
minscale and maxscale for the original layer and the mosaiced layer. So
since my data will be shown from 1:150 to 1:100 the minscale of the
original layer will be 100 and maxscale X. The minscale of the mosaic
overview will be X and the maxscale 150

original:
minscale 100
maxscale X

mosaic:
minscale X
maxscale 150

i cant find X?

On Thu, Aug 19, 2010 at 3:15 PM, Frank Warmerdam [via OSGeo.org] <
ml-node+5440161-2089753629-182...@n2.nabble.com
> wrote:

> Edi.Karadumi wrote:
> > Thanx to you my questions are comming to an end
> >
> > Three more last things
> >
> > Should i retile the new mosaiced overview? Meaning should i create
> smaller
> > size tiles of the new mosaic? or 2Gb is not a problem
>
> Edi,
>
> Generally speaking, as long as you enable internal tiling when building
> the mosaic there is no need to break it down into separate files.
> Add '-co TILED=YES' to the gdal_merge.py command to produce an output
> GeoTIFF file that is internally tiled.
>
> > And how can i find in which scale should i switch layers? So what will be
>
> > the maxscale of the original layer?
>
> That remains a complicated question.  I believe I suggested 1/32 as a
> good overview level since beyond that point a normal map view will tend
> to involve a significant number of files.
>
> > if ill still have performance problems, or ill get 1-2 terabytes of data
> > more should i make another mosaic with 0.08*16 or 0.08*8, or is there
> > another strategy?
>
> I presume as you add data, your area is growing.  If that is so, the 1/32nd
>
> overview level is still appropriate, and you just need to regenerate the
> overview for the larger area.
>
> If you are using MapServer and GDAL then there is no real problem with
> using
> one large geotiff file for the overview as long as it has internal tiling,
> even if it grows to many GB in size.
>
> Best regards,
> --
> ---+--
>
> I set the clouds in motion - turn up   | Frank Warmerdam, [hidden 
> email]
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush| Geospatial Programmer for Rent
>
> ___
> mapserver-users mailing list
> [hidden email] 
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
>
> --
>  View message @
> http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440161.html
> To unsubscribe from Re: Raster Strategies Examples, click 
> here.
>
>
>

-- 
View this message in context: 
http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440310.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Re: Raster Strategies Examples

[Edi.Karadumi]

Sorry but im still not understanding this:

> That remains a complicated question.  I believe I suggested 1/32 as a 
> good overview level since beyond that point a normal map view will tend 
> to involve a significant number of files. 

My only problem now is to find the correct scale for each layer. I use
minscale and maxscale for the original layer and the mosaiced layer. So
since my data will be shown from 1:150 to 1:100 the minscale of the
original layer will be 100 and maxscale X. The minscale of the mosaic
overview will be X and the maxscale 150

original:
minscale 100
maxscale X

mosaic:
minscale X
maxscale 150

i cant find X?



-- 
View this message in context: 
http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440312.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Re: Raster Strategies Examples

[mwkorver]

One simple trick that I don't think has been mentioned is to add the shape
indexes of each of your raster layers to your mapfile to be able to see how
many raster tiles were used for each request.  This makes it clear how many
files were used to create your output image.  

-- 
View this message in context: 
http://osgeo-org.1803224.n2.nabble.com/Raster-Strategies-Examples-tp5363321p5440532.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Re: Raster Strategies Examples

[Frank Warmerdam]

Edi.Karadumi wrote:

Sorry to bother again, when i use gdal_merge.py the part -ps 160 160, thats
the pixel dimension and i get an image 1kb. The pixel size of the original
file is 0.08 meters. So i think i should use 0.08x32 in the gdal_merge.py
command? 


Edi,

If the original resolution is 8cm then the 1/32nd overview should indeed
have a pixel size of 0.08x32 or 2.56m.

> And how many tiles should i merge together? Now i have 2T images
> and 2000Gb/32 = 62Gb the rezult image. Should i merge them all together and
> than recreate tiles? Even if i had 1200gb, 1200/32=37.5gb for an image not
> 1gb. That is still an image size that cant be handled

Note that the 1/32nd overview is reduced by a factor of 32 in x and y
so overall it will be original_size/(32*32).  So 2TB would reduce to
about 1.95GB.

Best regards,
--
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Programmer for Rent

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Re: Raster Strategies Examples

[Frank Warmerdam]

Edi.Karadumi wrote:
Thanx to you my questions are comming to an end  


Three more last things

Should i retile the new mosaiced overview? Meaning should i create smaller
size tiles of the new mosaic? or 2Gb is not a problem


Edi,

Generally speaking, as long as you enable internal tiling when building
the mosaic there is no need to break it down into separate files.
Add '-co TILED=YES' to the gdal_merge.py command to produce an output
GeoTIFF file that is internally tiled.


And how can i find in which scale should i switch layers? So what will be
the maxscale of the original layer?


That remains a complicated question.  I believe I suggested 1/32 as a
good overview level since beyond that point a normal map view will tend
to involve a significant number of files.


if ill still have performance problems, or ill get 1-2 terabytes of data
more should i make another mosaic with 0.08*16 or 0.08*8, or is there
another strategy?


I presume as you add data, your area is growing.  If that is so, the 1/32nd
overview level is still appropriate, and you just need to regenerate the
overview for the larger area.

If you are using MapServer and GDAL then there is no real problem with using
one large geotiff file for the overview as long as it has internal tiling,
even if it grows to many GB in size.

Best regards,
--
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Programmer for Rent

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users