[mapserver-users] Re: Hiding overviews
Hello Daniel,
before submitting a "patch" there are just some things to note. But it's
nice that there's so much interest on it.
> What was involved in setting up an Apache module? There was talk about
> doing that as an experiment a few weeks ago on the mapserver-dev list.
> If you are interested in sharing what you've done then I'm sure that
> would generate some interest. Perhaps you could file a ticket and
attach
> a patch and some notes?
Well, looked at the mail archive back till Jan 07 but could not find
something about it in a first quick look. Maybe you can point me to the
thread?
Currently I call the dirty thing mod_wms since that's what it currently
is. The main target was to get a faster WMS mainly because MapServer got
a little bit slow especially with large map files. I looked at the
mapserv.c and saw, that even in fastcgi mode it loads and parses the
hole map file for _every_ request. This seems to be a bad thing but
can't be done much different with CGI I think.
Well, if we would use the Apache module mechanism we can do it better.
We can load and parse the map file and ideally prepare everthing
required for the requests. When receiving a request we would do nearly
the same things that are currently done in the CGI - take the URI, break
it into name/value pairs, hash the things for fast lookups and take all
other things out of the already prepared MapObj.
This speeds up and simplifies much of the things but are only applicable
since we know that the configuration remains constant which is something
the CGI can't assume.
Currently the baby is mostly a hack... Some simple steps were involved
till now:
1. generate a template with: apxs2 -g -n wms
2. create a function taking the (currently) only config option we
provide which is taking the filename of the map file, try to load it as
usual with LoadMap() and store the result in a per directory config.
3. adopt the fastcgi output redirection code from mapio.c to redirect to
the Apache output functions
4. copy over most of the things from mapserv.c to the module's request
handler
Sounds easy, doesn't it? Well, there's a last step which makes it IMHO
being a real hack: the MapServer functions output the content type
directly to the output stream via things like
'msIO_printf("Content-type: text/xml%c%c",10,10);' which have to be
modified to set it in the Apache's request_rec.
So currently I can compile it under Debian Testing for Apache 2.2 and it
works quite well. Last week I've done a first successful build under
RedHat Enterprise Linux 5 with some additional packages from Fedora Core
(gdal, geos f.e.) but had not much time for testing.
There are still some things to improve I believe but that would require
a deeper look into the source. Mainly this would involve a modification
of the cleanup code to prevent the destruction of label, font and symbol
caches. Also, in that environment it doesn't make much sense to close
database connections so we could keep them open from Apache startup to
shutdown and so would be nice to prepare the queries in the startup
sequence for faster handling at runtime...
Daniel, I would prefer to send a commented patch directly to you first
and let you decide what to do with it. Remember that I already missed
some things with the metadata thing... May be tried to improve the world
but really created the hell on earth ;-)
So that's for now. Will now comment the patch...
Greetings!
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Re: Hiding overviews
Gregor Mosheh wrote: Daniel Morissette wrote: Well, in the meantime I patched the MapServer code a little bit and made it being an Apache module, reading and parsing the map file only once on startup. Are you serious, mod_mapserver? Holy cow, that would be awesome! I would be VERY interested to hear about stability with prefork MPM, whether it seems production grade, etc. I'd be glad to try it out as a patch to some of our systems (production but willing) and provide feedback. +1 Bring it on. :) Colin ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Re: Hiding overviews
Daniel Morissette wrote: Well, in the meantime I patched the MapServer code a little bit and made it being an Apache module, reading and parsing the map file only once on startup. Are you serious, mod_mapserver? Holy cow, that would be awesome! I would be VERY interested to hear about stability with prefork MPM, whether it seems production grade, etc. I'd be glad to try it out as a patch to some of our systems (production but willing) and provide feedback. -- Gregor Mosheh / Greg Allensworth, BS, A+ System Administrator HostGIS cartographic development & hosting services http://www.HostGIS.com/ "Remember that no one cares if you can back up, only if you can restore." - AMANDA ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Re: Hiding overviews
Eichner, Andreas - SID-NLK wrote: [...] Well, in the meantime I patched the MapServer code a little bit and made it being an Apache module, reading and parsing the map file only once on startup. This is much faster than the good old CGI and seems to work fine... Andreas, What was involved in setting up an Apache module? There was talk about doing that as an experiment a few weeks ago on the mapserver-dev list. If you are interested in sharing what you've done then I'm sure that would generate some interest. Perhaps you could file a ticket and attach a patch and some notes? Daniel -- Daniel Morissette http://www.mapgears.com/ ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Hiding overviews
Hello Armin, > If you don't mind using Python MapScript instead of the Mapserver CGI then you > could try to use this Python class that uses the OWSRequest > functionality of Mapserver/Mapscript: although this is a nice approach, I can't go with Python. We need a high performance WMS and I believe that the snake can not fulfill our expectations. Well, in the meantime I patched the MapServer code a little bit and made it being an Apache module, reading and parsing the map file only once on startup. This is much faster than the good old CGI and seems to work fine... > It modifies the map object before the output of the GetCapabilities > response. For a group of layers always the first one in the map file is > used as the one visible to the clients for the capabilities. In > addition, only the layers that have a 'ows_title' metadata tag are > included in the response. In an earlier mail in this thread I stated that IMHO there's a problem with the MapServer code. I modified the stated things and now it does the same thing: only giving a if 'ows_title' metadata is given. > And with an Apache rewrite rule the URL's can > be set to look a bit nicer. Some more details are in the README file. Well, another annoying thing which is resolved by the module solution. Now the Apache setup is something like: SetHandler "wms" WMS_Map "/srv/wms/wms.map" resulting in a simple URI like http://localhost/wms?service=WMS&request=GetCapabilities. So with little work I solved three issues for me: speed, URI niceness and overview layer grouping. Well, several nice ideas came to daylight but I will go with mine. Thanks to you and all helpers involved. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Re: Hiding overviews
Jukka I was working a bit on this issue since we face the same problem. If you don't mind using Python MapScript instead of the Mapserver CGI then you could try to use this Python class that uses the OWSRequest functionality of Mapserver/Mapscript: http://svn.pmapper.net/repos/owspy/trunk/ It modifies the map object before the output of the GetCapabilities response. For a group of layers always the first one in the map file is used as the one visible to the clients for the capabilities. In addition, only the layers that have a 'ows_title' metadata tag are included in the response. And with an Apache rewrite rule the URL's can be set to look a bit nicer. Some more details are in the README file. Armin On 06/03/2008 09:31, Rahkonen Jukka wrote: Hi, I would say that if it is not possible at the moment to make a logical WMS layer that automatically swithes between several external overviews and the original imagery through tileindex according to scale settings, then it would be very useful to have. However, it feels so obvious demand that I would be a bit surprised if there is not such an option. Anyway, if it is missing then I will give +1 for feature request. There seems to be an option called wms_layer_group in the layer metadata section of mapfile, but I must admid that I do not understand what it stands for. -Jukka Rahkonen- -Alkuperäinen viesti- Lähettäjä: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Puolesta Eichner, Andreas - SID-NLK Lähetetty: 6. maaliskuuta 2008 9:26 Vastaanottaja: mapserver-users@lists.osgeo.org Aihe: [mapserver-users] Re: Hiding overviews Hello Jacob, thanks for your try but this didn't help, sorry. IMHO the problem is that there seems to be no way to declare multiple, scale dependend data sources for one layer (in a WMS sense). Since the only reason of using of overviews for me is the (nearly) scale-independent performance. But optimization is _really_ a private thing of the service. It is not of the client's interest that MapServer internally organizes overviews as multiple layers... ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Re: Hiding overviews
Hi, I would say that if it is not possible at the moment to make a logical WMS layer that automatically swithes between several external overviews and the original imagery through tileindex according to scale settings, then it would be very useful to have. However, it feels so obvious demand that I would be a bit surprised if there is not such an option. Anyway, if it is missing then I will give +1 for feature request. There seems to be an option called wms_layer_group in the layer metadata section of mapfile, but I must admid that I do not understand what it stands for. -Jukka Rahkonen- > -Alkuperäinen viesti- > Lähettäjä: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] Puolesta > Eichner, Andreas - SID-NLK > Lähetetty: 6. maaliskuuta 2008 9:26 > Vastaanottaja: mapserver-users@lists.osgeo.org > Aihe: [mapserver-users] Re: Hiding overviews > > > Hello Jacob, > > thanks for your try but this didn't help, sorry. IMHO the > problem is that there seems to be no way to declare multiple, > scale dependend data sources for one layer (in a WMS sense). > Since the only reason of using of overviews for me is the > (nearly) scale-independent performance. But optimization is > _really_ a private thing of the service. It is not of the > client's interest that MapServer internally organizes > overviews as multiple layers... > > ___ > mapserver-users mailing list > mapserver-users@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/mapserver-users > ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Hiding overviews
Hello Jacob, thanks for your try but this didn't help, sorry. IMHO the problem is that there seems to be no way to declare multiple, scale dependend data sources for one layer (in a WMS sense). Since the only reason of using of overviews for me is the (nearly) scale-independent performance. But optimization is _really_ a private thing of the service. It is not of the client's interest that MapServer internally organizes overviews as multiple layers... ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
RE: [mapserver-users] Re: Hiding overviews
Hi Andreas, I use a similar approach, but not under WMS circumstances. But basically I take advantage of the fact that layers in MapServer that have the same name, will switch on/off at the same time. For example, I can create a layer called "Satellite Image", which is a tile-index. I can then add another layer called "Satellite Image", which actually points to a single raster image (overview). As long as MapServer is told to switch on "Satellite Image", it will draw all layers by that name (but still respecting minscale and maxscale). I am not sure how this will affect a "getCapabilities" document, though. You *could* look into using the "requires" attribute, but I think this may not work well in combination with minscale/maxscale. The last option (perhaps the most logical one) is to put them into the same group. I believe that should cause MapServer to draw them at the same time. Hope this helps, Jacob > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Eichner, Andreas - SID-NLK > Sent: 6 March 2008 15:41 > To: mapserver-users@lists.osgeo.org > Subject: [mapserver-users] Re: Hiding overviews > > > Hmm, seems my description wasn't clear enough, so I'll try to > precise... > > For a layer with raster data I created external overviews. To make use > of them I copied the LAYER portion of the map file, changed the > TILEINDEX statement to point to the index of the appropriate overview, > set MINSCALE/MAXSCALE to form a continues range and added a GROUP > statement to all of them to form a single layer again as it was before > without the overviews added. Now I can take a WMS client, enable or > disable the layer group which collects the detail layer and > all of it's > overviews which kick in if you increase the scale. > > So instead of having a single layer consisting of huge tiled > which have > all to be touched and scaled down for a request at large scales I now > have a group consisting of several layers where only one is active at > each scale. Doing so was a suggestion from the raster data > (optimizing) > howto and they called this solution "external overviews". > > So if I have a layer "raster" (scale 10...500) I want to split it > into "detail" (scale 10...1000) and "overview" (scale 1000...500) > and put both in a _group_ "raster" for example. Now you can go and > request the layer "raster" as before and what you get depends on the > chosen scale and is either "detail" below 1000 or "overview" > above 1000. > > The problem is that you can explicitly request the layer "detail", but > if you are above 1000 you will see... Absolutely nothing. > Same with the > "overview" layer below 1000. This is because those layers have a name > which enables them to be requested in a WMS "GetMap" request. If they > wouldn't you could only request the named group layer "raster". So I > thought it could be done by ommitting the "wms_name" part of the > METADATA but if you do and issue a "GetCapabilities" request you will > see that MapServer outputs a warning about the missing mandatory name > attribute and inserts the layer name from the NAME attribute > of the map > file. > > We have a WMS client which displays nested named layers as > folders. The > result isn't user friendly... So I wondered if there's a way to hide > those external overviews from the user since the only thing he can > decide is either to display the hole group (which ever layer may be > affected) or none of those layers at all. > > Hope I could make the situation a little bit clearer. I believe it's > only a little thing I miss but there are too much trees in > the wood... > ___ > mapserver-users mailing list > mapserver-users@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/mapserver-users > This email and any attachments are intended solely for the named recipient and are confidential. The copying or distribution of them or any information they contain by anyone other than the named recipient is prohibited. If you have received this document in error, please notify the sender and delete all copies from your computer system and destroy all hard copies. It is the recipient's responsibility to check this email and any attachments to this email for viruses before use. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Re: Hiding overviews
Hmm, seems my description wasn't clear enough, so I'll try to precise... For a layer with raster data I created external overviews. To make use of them I copied the LAYER portion of the map file, changed the TILEINDEX statement to point to the index of the appropriate overview, set MINSCALE/MAXSCALE to form a continues range and added a GROUP statement to all of them to form a single layer again as it was before without the overviews added. Now I can take a WMS client, enable or disable the layer group which collects the detail layer and all of it's overviews which kick in if you increase the scale. So instead of having a single layer consisting of huge tiled which have all to be touched and scaled down for a request at large scales I now have a group consisting of several layers where only one is active at each scale. Doing so was a suggestion from the raster data (optimizing) howto and they called this solution "external overviews". So if I have a layer "raster" (scale 10...500) I want to split it into "detail" (scale 10...1000) and "overview" (scale 1000...500) and put both in a _group_ "raster" for example. Now you can go and request the layer "raster" as before and what you get depends on the chosen scale and is either "detail" below 1000 or "overview" above 1000. The problem is that you can explicitly request the layer "detail", but if you are above 1000 you will see... Absolutely nothing. Same with the "overview" layer below 1000. This is because those layers have a name which enables them to be requested in a WMS "GetMap" request. If they wouldn't you could only request the named group layer "raster". So I thought it could be done by ommitting the "wms_name" part of the METADATA but if you do and issue a "GetCapabilities" request you will see that MapServer outputs a warning about the missing mandatory name attribute and inserts the layer name from the NAME attribute of the map file. We have a WMS client which displays nested named layers as folders. The result isn't user friendly... So I wondered if there's a way to hide those external overviews from the user since the only thing he can decide is either to display the hole group (which ever layer may be affected) or none of those layers at all. Hope I could make the situation a little bit clearer. I believe it's only a little thing I miss but there are too much trees in the wood... ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
