[mapserver-users] Production mapserver
What do we need to do to get Mapserver running in a production configuration? There are packages out there to help new users get mapserver up and running in a development/playing-around-in mode, ms4w, etc. But these are designed for ease of installation, not performance on a production server. I have a set of shapefiles, and some mapfiles I've built to display them. I have them running on a dev machine, just fine. So what do I need to do to go into production. I have a dedicated Ubuntu 10.04 box, that will be doing nothing but serving maps through WMS requests, using Mapserver. It currently has both Apache and MapServer installed, using the current apt-get packages. Should I be using Apache? Or should I use Lighttpd? Standard CGI? FastCGI? Some other alternative? (WSGI?) Does Mapserver install on Ubuntu by default to use FastCGI? How can I tell whether MapServer is running under FastCGI? How do I configure it to do so, if it's not already? ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Production mapserver
On 11-03-21 1:27 PM, Jeff Dege wrote: There are packages out there to help new users get mapserver up and running in a development/playing-around-in mode, ms4w, etc. But these are designed for ease of installation, not performance on a production server. Quick comment: MS4W is in fact ready to use in a production environment, and it is definitely used for production purposes by thousands of organizations: take the time to review the steps to enable FastCGI, for example (http://www.maptools.org/ms4w/index.phtml?page=README_INSTALL.html#f-fastcgi). Long-time MS4W users might remember that the MS4W opening page used to state that it was not for production purposes; but that is no longer the case. Thanks. -jeff -- Jeff McKenna MapServer Consulting and Training Services http://www.gatewaygeomatics.com/ ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
RE: [mapserver-users] Production mapserver
Useful to know, but I don't want to let your comment distract from my original question. I listed MS4W only as an example of an easy-to-install MapServer configuration. What I'm actually trying to get working is MapServer on Linux. -Original Message- From: mapserver-users-boun...@lists.osgeo.org [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Jeff McKenna Sent: Monday, March 21, 2011 11:34 AM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Production mapserver On 11-03-21 1:27 PM, Jeff Dege wrote: There are packages out there to help new users get mapserver up and running in a development/playing-around-in mode, ms4w, etc. But these are designed for ease of installation, not performance on a production server. Quick comment: MS4W is in fact ready to use in a production environment, and it is definitely used for production purposes by thousands of organizations: take the time to review the steps to enable FastCGI, for example (http://www.maptools.org/ms4w/index.phtml?page=README_INSTALL.html#f-fastcgi). Long-time MS4W users might remember that the MS4W opening page used to state that it was not for production purposes; but that is no longer the case. Thanks. -jeff -- Jeff McKenna MapServer Consulting and Training Services http://www.gatewaygeomatics.com/ ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Production mapserver
On 3/21/2011 11:27 AM, Jeff Dege wrote: What do we need to do to get Mapserver running in a production configuration? I guess that's up to you. Do you need 100% uptime? Mirrors and backups and redundancy? You'll also need an ISP that lets you host data. Many residential plans forbid it. You'll need a commercial plan. Or are you renting a VSP, or something? Your question is a little vague. Do you mean internet connections? Hardware? Just mapserver? There are packages out there to help new users get mapserver up and running in a development/playing-around-in mode, ms4w, etc. But these are designed for ease of installation, not performance on a production server. Do you know the level of performance you'll need? How many concurrent users? Have you put a load on it? Since you said the word performance, I'd say go for FastCGI. If you need more performance than that, well, that's another ballgame. I have a set of shapefiles, and some mapfiles I’ve built to display them. I have them running on a dev machine, just fine. So what do I need to do to go into production. I have a test box, and a live box... and I try to keep them as exact as possible. That way when you make changes, and do load tests, and what not, you know what to expect from the live site. Need to test something? The more identical your live and test boxes are, the more sure you can be about your testing. (Note, however, my live box is running off raid 5, where as my test is just a single HD. CPU's are close. Software wise, they are the same though) I have a dedicated Ubuntu 10.04 box, that will be doing nothing but serving maps through WMS requests, using Mapserver. It currently has both Apache and MapServer installed, using the current apt-get packages. Should I be using Apache? Or should I use Lighttpd? Use the one you are most comfortable with. Speed wise it wont matter. You'll be spending all your time in mapserver code, not apache/lighty code. (unless, of course, you have huge amounts of static files along with the maps... but those should all be marked as cacheable, so still, not gonna make a performance difference) Cant answer anything about Ubuntu, never used it, run Slackware myself. I prefer to build mapserver from source. I'm also using mapscript (with perl), so I build that as well. Standard CGI? FastCGI? Some other alternative? (WSGI?) CGI is slow. FastCGI if you want performance. Never used WSGI. As background: I run on a dual core amd64 3800, 4 green HD's in software raid 5, 6 gig of ram, running Slackware64. I think the box cost me around $400 to build. I get 100K hits a day (about 2 requests a second), and the box sits 75% idle. I have about 350 gig of arial imagery, and 25 gig of shapefiles in PostGIS. It's run on apache/mapscript/mod_perl. -Andy ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Production mapserver
On Mon, Mar 21, 2011 at 18:28, Andy Colson a...@squeakycode.net wrote: On 3/21/2011 11:27 AM, Jeff Dege wrote: What do we need to do to get Mapserver running in a production configuration? I guess that's up to you. Do you need 100% uptime? Mirrors and backups and redundancy? You'll also need an ISP that lets you host data. Many residential plans forbid it. You'll need a commercial plan. Or are you renting a VSP, or something? Your question is a little vague. Do you mean internet connections? Hardware? Just mapserver? There are packages out there to help new users get mapserver up and running in a development/playing-around-in mode, ms4w, etc. But these are designed for ease of installation, not performance on a production server. Do you know the level of performance you'll need? How many concurrent users? Have you put a load on it? Since you said the word performance, I'd say go for FastCGI. If you need more performance than that, well, that's another ballgame. I have a set of shapefiles, and some mapfiles I’ve built to display them. I have them running on a dev machine, just fine. So what do I need to do to go into production. I have a test box, and a live box... and I try to keep them as exact as possible. That way when you make changes, and do load tests, and what not, you know what to expect from the live site. Need to test something? The more identical your live and test boxes are, the more sure you can be about your testing. (Note, however, my live box is running off raid 5, where as my test is just a single HD. CPU's are close. Software wise, they are the same though) I have a dedicated Ubuntu 10.04 box, that will be doing nothing but serving maps through WMS requests, using Mapserver. It currently has both Apache and MapServer installed, using the current apt-get packages. Should I be using Apache? Or should I use Lighttpd? Use the one you are most comfortable with. Speed wise it wont matter. You'll be spending all your time in mapserver code, not apache/lighty code. (unless, of course, you have huge amounts of static files along with the maps... but those should all be marked as cacheable, so still, not gonna make a performance difference) Note that mapserver as fastcgi has issues when running on lighty's fastcgi implementation, so until that is fixed I would stick with apache. Cant answer anything about Ubuntu, never used it, run Slackware myself. I prefer to build mapserver from source. I'm also using mapscript (with perl), so I build that as well. compiling stuff from source also leaves out all the parts of mapserver that can cause some marginal overhead. for example, if you are only serving shapefiles, you can leave out the gdal/ogr and postgis dependencies Standard CGI? FastCGI? Some other alternative? (WSGI?) CGI is slow. FastCGI if you want performance. Never used WSGI. As background: I run on a dual core amd64 3800, 4 green HD's in software raid 5, 6 gig of ram, running Slackware64. I think the box cost me around $400 to build. I get 100K hits a day (about 2 requests a second), and the box sits 75% idle. I have about 350 gig of arial imagery, and 25 gig of shapefiles in PostGIS. It's run on apache/mapscript/mod_perl. -Andy ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Production mapserver
On 3/21/2011 12:33 PM, thomas bonfort wrote: On Mon, Mar 21, 2011 at 18:28, Andy Colsona...@squeakycode.net wrote: Cant answer anything about Ubuntu, never used it, run Slackware myself. I prefer to build mapserver from source. I'm also using mapscript (with perl), so I build that as well. compiling stuff from source also leaves out all the parts of mapserver that can cause some marginal overhead. for example, if you are only serving shapefiles, you can leave out the gdal/ogr and postgis dependencies Indeed. When I first started playing with mapserver, I went totally OCD trying to compile in every possible format it would support. Was really kinda fun. (Dont know what hdf5 is, never seen it, never needed it, cant say as I ever will... but I have support for it! Yeah!) Its amazing how many formats that mapserver/gdal/ogr support. (Dude! Its like Pokemon! Gotta get 'em all!) Huh.. I'm gonna need to lay off the crack again. :-) -Andy ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Production mapserver
In the benchmark testing we did last year, see http://wiki.osgeo.org/wiki/Benchmarking_2010, I tested MapServer with minimal formats (just PostGIS) compiled it vs MapServer with Oracle, PostGIS, GDAL/OGR. There was no difference affect on performance by having a minimal build. Mike -- Michael Smith Remote Sensing/GIS Center US Army Corps of Engineers On 3/21/11 1:43 PM, Andy Colson a...@squeakycode.net wrote: On 3/21/2011 12:33 PM, thomas bonfort wrote: On Mon, Mar 21, 2011 at 18:28, Andy Colsona...@squeakycode.net wrote: Cant answer anything about Ubuntu, never used it, run Slackware myself. I prefer to build mapserver from source. I'm also using mapscript (with perl), so I build that as well. compiling stuff from source also leaves out all the parts of mapserver that can cause some marginal overhead. for example, if you are only serving shapefiles, you can leave out the gdal/ogr and postgis dependencies Indeed. When I first started playing with mapserver, I went totally OCD trying to compile in every possible format it would support. Was really kinda fun. (Dont know what hdf5 is, never seen it, never needed it, cant say as I ever will... but I have support for it! Yeah!) Its amazing how many formats that mapserver/gdal/ogr support. (Dude! Its like Pokemon! Gotta get 'em all!) Huh.. I'm gonna need to lay off the crack again. :-) -Andy ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Production mapserver
On 3/21/2011 1:52 PM, Smith, Michael ERDC-CRREL-NH wrote: In the benchmark testing we did last year, see http://wiki.osgeo.org/wiki/Benchmarking_2010, I tested MapServer with minimal formats (just PostGIS) compiled it vs MapServer with Oracle, PostGIS, GDAL/OGR. There was no difference affect on performance by having a minimal build. Mike, While this might be true, having a minimal built does potentially require less memory for the processes running which more available for file cache performance that might have an impact in a production environment, especially if you are trying to push a lot of map draws through the system. Another trick you can do on Linux, is to disable the swap partition if you do not need it, and if you are just serving mapserver requests you do not. This will improve performance by 1-2% on the system, because Linux is what is called a swappy OS, ie it has a how propensity to swap stuff to disk in anticipation that it might need the memory for something else. -Steve W ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
RE: [mapserver-users] Production mapserver
So, from what I'm hearing Apache vs. Lighttpd isn't going to make a significant difference. Cgi vs. FastCGI very definitely will. Great. So how do I get FastCGI working? There are docs at: http://mapserver.org/optimization/fastcgi.html They are only getting me part of the way there. 1. I'm seeing two packages, that are available for Apache, mod_fastcgi and mod_fcgi. Are these both needed? Only one needed? Are they replacements, alternatives, or entirely unrelated to each other? 2. In the Configuration section of the above doc, it says to modify http.conf to load the module, and to add a handler. The Ubuntu packages placed fastcgi.conf and fastcgi.load in /etc/apache2/mods-available. I expect that I simply need to create a link to these in /etcs/apache2/mods-enabled, the way the others are. 3. The Configuration section also says to add a FastCgiConfig command. Where should this be? I see nothing in that configuration that indicates that it's the mapserv executable I want to run via fastcgi. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Production mapserver
Jeff, all the documentation is online, the folks producing the open source software you are intending to use have gone to some trouble to produce it. If that isn't sufficient, I highly doubt that the answers you'll get from a mailing list where people don't know your requirements will get you to what you are expecting. I would suggest you either invest some time in learning yourself how to make these projects work together, or contract someone from those communities to do it for you. Expecting folks to do the work for you for free on community mailing lists is, imho, a bit inappropriate. regards, thomas On Mon, Mar 21, 2011 at 20:30, Jeff Dege jeff.d...@korterra.com wrote: So, from what I'm hearing Apache vs. Lighttpd isn't going to make a significant difference. Cgi vs. FastCGI very definitely will. Great. So how do I get FastCGI working? There are docs at: http://mapserver.org/optimization/fastcgi.html They are only getting me part of the way there. 1. I'm seeing two packages, that are available for Apache, mod_fastcgi and mod_fcgi. Are these both needed? Only one needed? Are they replacements, alternatives, or entirely unrelated to each other? 2. In the Configuration section of the above doc, it says to modify http.conf to load the module, and to add a handler. The Ubuntu packages placed fastcgi.conf and fastcgi.load in /etc/apache2/mods-available. I expect that I simply need to create a link to these in /etcs/apache2/mods-enabled, the way the others are. 3. The Configuration section also says to add a FastCgiConfig command. Where should this be? I see nothing in that configuration that indicates that it's the mapserv executable I want to run via fastcgi. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Production mapserver
Hi, There can be much more important things to optimise before changing from CGI to FastCGI. I mean optimising the data. Take care of the indexes. Limit the numbers of features to be rendered, drawing more than a couple of thousands is usually unnecassary. Use simplified vector layers when you do not need all the zillions of vertices in the original top quality data, and so on. You said you are going to use shapefiles and some imagery. CGI can take something like 50 milliseconds more time for opening those than FastCGI. It can be even 100 milliseconds, but as long as the limits of CGI are not met your users can hardly feel the difference. Naturally the situation changes when you reach the limits of throughput from CGI. In our environment CGI can do 10 requests per second with aerial imagery while FastCGI takes 20 per second. However, we have seen that with multicore processors CGI is extremely reliable and till the 10 images per second the user experiance is good. When feeding some 50 million tiles into cache we actually switched from FastCGI back to CGI because FastCGI failed every now and then (once per 5 or 10 GetMaps). -Jukka Rahkonen- -Alkuperäinen viesti- Lähettäjä: mapserver-users-boun...@lists.osgeo.org puolesta: Jeff Dege Lähetetty: ma 21.3.2011 21:30 Vastaanottaja: mapserver-users@lists.osgeo.org Aihe: RE: [mapserver-users] Production mapserver So, from what I'm hearing Apache vs. Lighttpd isn't going to make a significant difference. Cgi vs. FastCGI very definitely will. Great. So how do I get FastCGI working? There are docs at: http://mapserver.org/optimization/fastcgi.html They are only getting me part of the way there. 1. I'm seeing two packages, that are available for Apache, mod_fastcgi and mod_fcgi. Are these both needed? Only one needed? Are they replacements, alternatives, or entirely unrelated to each other? 2. In the Configuration section of the above doc, it says to modify http.conf to load the module, and to add a handler. The Ubuntu packages placed fastcgi.conf and fastcgi.load in /etc/apache2/mods-available. I expect that I simply need to create a link to these in /etcs/apache2/mods-enabled, the way the others are. 3. The Configuration section also says to add a FastCgiConfig command. Where should this be? I see nothing in that configuration that indicates that it's the mapserv executable I want to run via fastcgi. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Production mapserver
Jeff, For the WMS Benchmarking effort, we used the FastCGI library from http://www.fastcgi.com. It is the mapserv executable that you will call via fastcgi. We set up a separate fcgi-bin path for mapserver with fastcgi. That is not actually required. The URL here would be http://server/fcgi-bin/mapserv?map=XXX Our apache setup looked like this ScriptAlias /fcgi-bin/ /opt/mapserver/fcgi-bin/ Directory /opt/mapserver/fcgi-bin/ SetHandler fcgid-script Options +ExecCGI +FollowSymLinks # Customize the next two directives for your requirements. Order allow,deny Allow from all /Directory IfModule fcgid_module FcgidInitialEnv LD_LIBRARY_PATH /usr/lib64:/usr/lib:/usr/local/lib:/opt/mapserver/lib:/oracle/instantclient_ 11_2 FcgidMinProcessesPerClass 0 FcgidIdleScanInterval 1 FcgidProcessLifeTime 10 FcgidIOTimeout 100 FcgidMinProcessesPerClass 0 /IfModule Mike -- Michael Smith Remote Sensing/GIS Center US Army Corps of Engineers On 3/21/11 3:30 PM, Jeff Dege jeff.d...@korterra.com wrote: So, from what I'm hearing Apache vs. Lighttpd isn't going to make a significant difference. Cgi vs. FastCGI very definitely will. Great. So how do I get FastCGI working? There are docs at: http://mapserver.org/optimization/fastcgi.html They are only getting me part of the way there. 1. I'm seeing two packages, that are available for Apache, mod_fastcgi and mod_fcgi. Are these both needed? Only one needed? Are they replacements, alternatives, or entirely unrelated to each other? 2. In the Configuration section of the above doc, it says to modify http.conf to load the module, and to add a handler. The Ubuntu packages placed fastcgi.conf and fastcgi.load in /etc/apache2/mods-available. I expect that I simply need to create a link to these in /etcs/apache2/mods-enabled, the way the others are. 3. The Configuration section also says to add a FastCgiConfig command. Where should this be? I see nothing in that configuration that indicates that it's the mapserv executable I want to run via fastcgi. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
