RE: [OSGeo-Discuss] WMS Performance Shootout presentation/results
Great to see the shootout results. Also interesting to see the Amazon RDS announcement (MySQL based) with possibility of using quadruple extra large EC2 instances: db.m2.4xlarge - 68 GB of RAM http://aws.typepad.com/aws/2009/10/introducing-rds-the-amazon-relational-dat abase-service-.html http://aws.typepad.com/aws/2009/10/two-new-ec2-instance-types-additional-mem ory.html Maybe next shootout the DB layer could look at Amazon RDS(mySQL) and PostgreSQL/PostGIS using a quadruple extra large instance 68Gb RAM. After reading Todd Hoff's blog I'd be curious to see if PostGIS could be configured to make use of large memory capacities and how it affects performance: http://highscalability.com/are-cloud-based-memory-architectures-next-big-thi ng Thanks Randy From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Craig Miller Sent: Friday, October 23, 2009 8:38 PM To: 'OSGeo Discussions' Subject: RE: [OSGeo-Discuss] WMS Performance Shootout presentation/results I agree wholeheartedly. It looks like the bottleneck was the database. I've been privy to some MapServer tests done by testing teams over several months and the result there was always deploying the data with long update cycles to the middle tier disks instead of using the database. Only then could the performance of the actual map servers be evaluated. Performance shootouts/testing take time to do correctly as each run teaches you more and more about how your deployment architecture affects the results. Craig Geospatial Software Engineer Spatial Minds, LLC http://spatialminds.com/ From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of antti roppola Sent: Friday, October 23, 2009 7:34 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] WMS Performance Shootout presentation/results It was really interesting. The very close results suggests to me that the bottlenecks were external to the WMS and more related to external limitations like the ability to supply things like I/O. It would be interesting to have profiling data on where the response time was spent. For Mapserver it'd be a simple case of running Valgrinf and KCacheGrind: http://kcachegrind.sourceforge.net/cgi-bin/show.cgi/KcacheGrindIndex Case point. We had an in house app for crunching big raster and KCacheGrind showed us that an external library was the biggest bottleneck. A. On Sat, Oct 24, 2009 at 10:39 AM, Jeff McKenna jmcke...@gatewaygeomatics.com wrote: For those that did not make it to Sydney, here is the WMS Performance Shootout presentation with results (GeoServer vs MapServer): http://www.slideshare.net/gatewaygeomatics.com/wms-performance-shootout MapServer: power users who manage MapServer sites with high loads/map draws should take note of the results of MapServer CGI vs MapServer FastCGI, even in the case of Shapefiles and Rasters (yes, quite surprising). All: a lot of credit should go to Andrea Aime from GeoServer who worked very hard in bringing the MapServer team up to speed to learn the testing process. It was a great experience and we're already looking forward to next year. -jeff -- Jeff McKenna FOSS4G Consulting and Training Services http://www.gatewaygeomatics.com/ ___ mapserver-users mailing list mapserver-us...@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users -- Jeff McKenna FOSS4G Consulting and Training Services http://www.gatewaygeomatics.com/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: Comparison between MapServer/OpenLayers and ESRI ArcIMS
Cloud options are looking interesting. http://aws.amazon.com/ec2/ Windows, Linux, Solaris options I imagine ESRI license entanglement with virtual servers could be a problem. But no problem at all with Open Source GIS stacks. No license to get tangled with load balancing and auto scaling where servers come and go as needed. Mostly I've seen small business interest since they tend to take overhead costs more seriously. It might be useful to include a Cloud based server solution addendum, because that would be less optimal for an ESRI vendor and could look good compared to in-house hardware. Unfortunately, medium and large organizations seem to have budget allocations already in place for the big ticket approach. But then in this economy even that could be changing. AWS now includes Load Balancing and Auto Scaling options as well as S3 Backup, multiple offsite elastic block store duplication, edge cache, and elastic IP. http://aws.amazon.com/about-aws/whats-new/2009/05/17/monitoring-auto-scaling-elastic-load-balancing/ And for the real bleeding edge http://aws.amazon.com/elasticmapreduce/ (Not a selling point to small, medium, or large organizations, unless academically oriented :-) rkgeorge -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Jason Birch Sent: Friday, May 29, 2009 5:49 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Re: Comparision between MapServer/OpenLayers and ESRI ArcIMS I think that it's generally less fear of the unknown or job security than it is the cost of adding complexity to what is often an already over-extended support load. In many cases it just makes sense to spend $1000 for a server OS that doesn't require additional training, is easy to get qualified techs for, and just works with the existing systems. It doesn't matter how easy Linux is; it's one more thing to keep track of and one more thing to go wrong. If you want to win the open source battle at small organisations that don't already have OS operating system tendencies, focus on the application level where you can make a strong business case on a feature-by-feature level, and with additional arguments about truly open data being more sustainable and less risky. Personally I think that an open source or bust attitude is not very pragmatic. Sell open source software where it is the best tool for the job, but pick your battles. Jason -Original Message- From: Alex Mandel Sent: Friday, May 29, 2009 4:25 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Re: Comparision between MapServer/OpenLayers and ESRI ArcIMS That would be fear of the unknown(non gui) and job security at work. Wouldn't want someone else in the org who knows more about running servers. Maybe you can get them to throw a bone to demo something on a virtual machine hosted elsewhere(Amazon) just to show how easy it is. Welcome to the land of small to medium government agencies, etc. The best thing here is showing examples from equivalent groups, of which there are plenty online now. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as with ESRI)?
I know this is not OS but GoogleChart is easy to use: http://code.google.com/apis/chart/ http://chart.apis.google.com/chart?cht=p3chd=t:20,40,30,10chs=250x100chl= Hello|World|of|Google and it can be used to add chart icons for use in online mapping interfaces, not necessarily Google's, and not just pie charts. However, as with all of Google stuff the license is a gotcha for most businesses. Adding your own similar chart tool using svg or wpf/Silverlight interfaces would not be much of a problem. Plus you get better scaling with vector graphics along with click/rollover/tooltip... capabilities. Raster Icon charts seem to be a useable approach for map data as discussed here: http://blog.thematicmapping.org/2008/04/using-google-charts-with-kml.html I'd favor vector if you want drill-in capability though. randy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mohamed Ghareeb Sent: Tuesday, April 29, 2008 5:14 AM To: 'OSGeo Discussions' Subject: RE: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as with ESRI)? Concerning OpenJump it is really good but I couldn't make a pie chart map with it as I could do with Arcview 3.x I added the charts plugin but I think It doesn't classify the size of charts symbol. Does anyone can do it by any open-source GIS? Mohamed Mostafa -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Landon Blake Sent: Thursday, April 24, 2008 10:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as with ESRI)? A convert! Welcome Jennifer. I can't speak for GRASS, but I know that OpenJUMP (http://jump-pilot.sourceforge.net/OpenJUMP.html) could be compared to the old 3.X Arcview. It has limited printing abilities at this point in time, but I don't think there is a better cross-platform tool for basic ESRI Shapefile manipulation. (I'm a volunteer on the project and therefore biased in my opinion on this matter.) I think you will find you can do 95% of what you could with ESRI software, you'll just have to do it with an assortment of tools instead of a single tool. Landon -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jennifer Horsman Sent: Thursday, April 24, 2008 12:41 PM To: OSGeo Discussions Subject: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as with ESRI)? The thread that was started today with the subject Your open source career got me thinking about asking a question that has been rolling around in my head. This is pointed at those people who have experience with ESRI products as well as OS GIS products. I have been a long-time user of ESRI products, but I want to start my own contract business and will not be able to afford the license for ArcGIS/ArcInfo. So I recently set up a Linux box with GRASS installed, but it has been over 10 years since I have used GRASS (it has probably changed since then too!) Does GRASS have the same analysis and display capabilities as ArcGIS? I know this is a very general question, so perhaps another question would be where does GRASS fall short and where does it excel in comparison to the ESRI products? Thanks, Jennifer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] RE: Comparison between Proprietary and OS
Hi, This brings to mind an additional point. Even though OS GIS tends to be a patchwork of projects that demand a good deal of experience from its users, it also gives you infinite extensibility. From a business perspective, this affords a proficient user of OS the ability to exploit automated workflow processes. Many use cases involve repetitive drudge work that can be automated if you have access to the code and/or knowledge of scripting. This is often possible with commercial products too, but OS usually has more hooks to feed a workflow and you can resort to the code if required. Since we are in the internet era, workflows tend to get pushed further out into the cloud which is why WPS has a lot of promise and means that the push button= result will be mandatory. The public use of GIS workflow demands simplicity, possibly over simplicity. randy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andre Grobler Sent: Saturday, April 26, 2008 5:39 AM To: discuss@lists.osgeo.org Subject: [OSGeo-Discuss] RE: Comparison between Proprietary and OS I'm very new at OS, have worked 10 years on AV3.x and a little Idrisi, but only from a vegetation point of view, so we're talking basic end-user digitizer mapper (and georef images), bit of topographical modeling and extracting underlying information. I wanted to do OS when starting my own company, but time constraints had me take a loan to get Windows Office and ArcView, just because I knew what I would get, and be able to get up and running on it within a week. I now realize most of the things I need to do in day to day, I could do in OS Q-GIS albeit with a little less sleekness. But I could not risk letting work slide (or causing clients, who use ESRI, hassles) to get up and running in Q-GIS, GRASS and possibly OSSIM. So the hurdles for me to OS were acceptance specifically for the following reasons: Free and easy access and training of ESRI at varsity. Autocad did the same and look where that got them. Linux, just mentioning command lines has me a little nervous. (I know this is changing, but the field calculator is enough programming for me, thanks) I'm not suggesting anybody is obliged to fix these things, I mean there's no such thing as a free lunch. I'm just saying what OS inherently is, is probably going to keep it out of the mainstream (non-programming) end-users easy reach. So in short I am doing what somebody already suggested, get ESRI for day to day soft landing and learn OS GRASS and OSSIM meanwhile for real work;-) Hopefully in a while I'll wonder what the fuss was about... and possibly contribute, if only to the dummies FAQ section. André Grobler -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: 26 April 2008 10:49 AM To: discuss@lists.osgeo.org Subject: [!! SPAM] Discuss Digest, Vol 16, Issue 28 Send Discuss mailing list submissions to discuss@lists.osgeo.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.osgeo.org/mailman/listinfo/discuss or, via email, send a message with subject or body 'help' to [EMAIL PROTECTED] You can reach the person managing the list at [EMAIL PROTECTED] When replying, please edit your Subject line so it is more specific than Re: Contents of Discuss digest... Today's Topics: 1. Re: Can I do the same GIS tasks with OS (as with ESRI)? (RAVI KUMAR) 2. Re: Can I do the same GIS tasks with OS (as with ESRI)? (Cameron Shorter) 3. Re: Can I do the same GIS tasks with OS (as with ESRI)? (Wolf Bergenheim) 4. Re: Can I do the same GIS tasks with OS (as with ESRI)? (RAVI KUMAR) 5. Re: Can I do the same GIS tasks with OS (as with ESRI)? (Malte Halbey-Martin) -- Message: 1 Date: Fri, 25 Apr 2008 22:31:47 -0700 (PDT) From: RAVI KUMAR [EMAIL PROTECTED] Subject: Re: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as withESRI)? To: OSGeo Discussions discuss@lists.osgeo.org Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=iso-8859-1 Hi, this is the kind of question I face when in my lectures evangelising OS GIS. ArcGIS has many tools, though some prefer to call it a deluge of tools, which almost distance the user from understanding the concept of GIS. Auto Complete Polygon: In Qgis which is a very userfriendly OS GIS you have 'Cut polygon', do try and find the difference. Polygonising from lines: Open JUMP has one of the most userfriendly approaches. Create lines and polygonise in OpenJUMP and the software automatically creates a folder for Dangles (un-wanted line pieces) The query is more for Vector GIS, I suppose. GRASS GIS: It has so many features for Image analysis and Raster GIS, the commercial GIS need a barge pole to even touch it. The vector Part of GRASS is robust too. Ravi
RE: [OSGeo-Discuss] Your open source career
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Landon Blake Sent: Thursday, April 24, 2008 12:57 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Your open source career Tyler, You know I can't pass up an opportunity to talk about myself. :] I don't have much time because of an impending Friday deadline, but I will share a couple of thoughts with you. I could share more next week if you want. From my personal experience, skills as an open source programming can give employees a definite advantage in their career, especially in two types of organizations. The first type of organization is a small organization that can't afford the cost of proprietary software. This can be a huge cost, especially for software that is geared towards specialized professions, and not the mass of typical computer users. Employees with an open source skill set can offer solutions to this type of organization that wouldn't otherwise be possible. For example, we are able to use an enterprise database system and GIS software at my surveying and engineering company that are open source because of the skill set that a coworker and I have acquired, largely on our own time. This isn't software that my company would have purchased from a commercial vendor. The second type of organization is one in which a specialized trade or profession is practiced. These organizations are often not served by the out-of-the-box software that is suitable for more generic types of businesses. Any type of programming skills, including open source programming skills, enables an employee to develop custom solutions that assist their organization. These applications are generally better suited to the specialized tasks because they are written by an individual with a unique knowledge of the problem domain. This is not typically something you get from software developed by a third party. For example, I'm in the process of developing a tool that will process thousands of points collected during the course of a bathymetric survey, something we currently do in a spreadsheet. The tool, when completed, could save my company several man hours and hundreds of dollars on every bathymetric survey we perform. Landon -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tyler Mitchell (OSGeo) Sent: Thursday, April 24, 2008 11:12 AM To: OSGeo Discussions Subject: [OSGeo-Discuss] Your open source career Hi everyone, We've probably all heard of the typical business models for open source companies, but I'm working on a few slides for a presentation that highlight the benefit of open source for employees - and would love to hear from some of you. I have some personal examples where open source made me a more valuable employee, and how other colleagues who were into open source were considered invaluable. I also believe the many employers who value open source are able to attract talented staff that traditional or proprietary employers cannot. Do you have a story about how embracing open source geospatial applications helped broaden the opportunities for your future or helped you bust out of a mundane box? Maybe you learned on your own time and brought your new skills into the office? I'm particularly interested in your personal stories about how open source may have motivated you to grow, learn and extend your career or professional set of tools. Anyone want to share? Best wishes, Tyler ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as with ESRI)?
Sorry for the previous blank post. Open source is a great boon to small business innovation, as others have pointed out. Anyone dependent on small business consulting/contracting will have plenty of uses for open source tools. I have also used Jump in place of ArcView for shp viewing (as well as FWTools, Tatuk, AutoCAD, uDig, Geoserver ...). Shp seems to have become largely a lowest common denominator interchange format with support available all over. Of course ArcGIS/ArcInfo is a much different beast. It would take a good bit of sophistication to beat it in breadth of spectrum with just a single tool. There seems to be some kind of open source tool for most of the Arc bandwidth so you can probably get by without an expensive license with a little extra work. randy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Landon Blake Sent: Thursday, April 24, 2008 2:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as with ESRI)? A convert! Welcome Jennifer. I can't speak for GRASS, but I know that OpenJUMP (http://jump-pilot.sourceforge.net/OpenJUMP.html) could be compared to the old 3.X Arcview. It has limited printing abilities at this point in time, but I don't think there is a better cross-platform tool for basic ESRI Shapefile manipulation. (I'm a volunteer on the project and therefore biased in my opinion on this matter.) I think you will find you can do 95% of what you could with ESRI software, you'll just have to do it with an assortment of tools instead of a single tool. Landon -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jennifer Horsman Sent: Thursday, April 24, 2008 12:41 PM To: OSGeo Discussions Subject: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as with ESRI)? The thread that was started today with the subject Your open source career got me thinking about asking a question that has been rolling around in my head. This is pointed at those people who have experience with ESRI products as well as OS GIS products. I have been a long-time user of ESRI products, but I want to start my own contract business and will not be able to afford the license for ArcGIS/ArcInfo. So I recently set up a Linux box with GRASS installed, but it has been over 10 years since I have used GRASS (it has probably changed since then too!) Does GRASS have the same analysis and display capabilities as ArcGIS? I know this is a very general question, so perhaps another question would be where does GRASS fall short and where does it excel in comparison to the ESRI products? Thanks, Jennifer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as with ESRI)?
It might be good to add a geoserver layer into the stack between PostGIS and client. Then you can publish into Google Earth, Google Maps, Virtual Earth/LiveMaps, or your own homegrown html, SVG, WPF, Silverlight whatever ... as well as OpenLayers. Paper can be the clients choice if you add a pdf/fop option. randy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Josh Livni Sent: Thursday, April 24, 2008 4:41 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Can I do the same GIS tasks with OS (as with ESRI)? If you want nice cartographic output, ArcMap on a totally different level than ArcView. I think I paid about 1k a few years ago for my copy (no extensions) through some reseller, which I really don't think is outlandish at all for what you get. That said, I rarely do paper-based cartography anymore, and one thing to consider when starting out a new venture is those moving goalposts. For example, there's tons of demand (as far as I can tell) for putting stuff on the web. One spot I see lots of work for is somewhere just above what the average google map hobbyists can offer (eg the postgis/mapserver/openlayers wrapped in a decent web framework or CMS kind of a thing). Or do you prefer to focus on modeling? cartography? I think if you have an idea of the kinds of projects you want to work on, that might help you decide which of the many open source gis software stacks might be the best to start studying. Without knowing more, my .02 is you just can't go wrong spending a little time learning PostGIS, and if you do want to put stuff on the web, you may as well start by playing around with OpenLayers. Cheers, -Josh Paul Ramsey wrote: I'd buck up for a copy of ArcView (much cheaper than ArcGIS), and use GRASS / PostGIS / etc tools for things like analysis. You can use ArcView to generate the paper and do some quick low-end analytics and the other tools for more involved stuff. My general synopsis: for server-side, for scriptability, for automation, for web-based, open source wins for most use cases, given a technically savvy user; for ad hoc, for cartographic production, for a user who is used to a point-n-click experience end to end, proprietary still wins. This equation hasn't changed much in the 10 years I've been running it. The goal posts have moved, open source is better at adhoc now than before, but still not at the level of ESRI, and ESRI is better at the server stuff now, but still not at the level of open source. P. On Thu, Apr 24, 2008 at 12:41 PM, Jennifer Horsman [EMAIL PROTECTED] wrote: The thread that was started today with the subject Your open source career got me thinking about asking a question that has been rolling around in my head. This is pointed at those people who have experience with ESRI products as well as OS GIS products. I have been a long-time user of ESRI products, but I want to start my own contract business and will not be able to afford the license for ArcGIS/ArcInfo. So I recently set up a Linux box with GRASS installed, but it has been over 10 years since I have used GRASS (it has probably changed since then too!) Does GRASS have the same analysis and display capabilities as ArcGIS? I know this is a very general question, so perhaps another question would be where does GRASS fall short and where does it excel in comparison to the ESRI products? Thanks, Jennifer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] RE: OGC WPS and Amazon SQS
I noticed OGC finalized the WPS spec: http://www.opengeospatial.org/pressroom/pressreleases/843 Does anyone know of projects working on WPS implementations? The goal of WPS is apparently to provide a consistent framework for interchangeable service process algorithms that can potentially be chained together into answers to higher level questions than the typical 'what', 'when', and 'where.' Dealing with 'why', 'how much', and 'what if' modeling usually requires a process pipeline for convolutions, boolean band operations, and summary pixel calculations, all of which are cpu cycle intense, especially for large imagery sets. In fact cpu usage issues would make the usual service approach prohibitive. Even the little I have worked on JAI pipelines shows me the futility of a one cpu to many service requests approach for WPS. However, looking at the AWS Simple Queue Service, SQS http://www.amazon.com/Simple-Queue-Service-home-page/b/ref=sc_fe_l_2?ie=UTF8 http://www.amazon.com/Simple-Queue-Service-home-page/b/ref=sc_fe_l_2?ie=UTF 8node=13584001no=3435361me=A36L942TSJ2AJA node=13584001no=3435361me=A36L942TSJ2AJA, some interesting possibilities come to mind. Locking message queues with AMI instance pools is essentially a poor man's supercomputer. It would be interesting to look at harnessing the utility computing concept with instance pools available for each stage in a process pipeline connected using the asynchronous SQS service. This is a more or less controlled 'distributed computing model' applied to WPS. Ref here for some examples of existing distributed computing projects: http://distributedcomputing.info/projects.html Here are a couple possible approaches to a WPS service model that might overcome the cpu bottle neck: 1) Sequential SQS pipeline with dedicated instance for each process node - this would work best for operations amenable to a streaming pipeline - Boolean band operations or pixel summary operations for instance 2) Distributed computing model with a chunk server feeding a pipeline and an array pool of instances processing the chunks coming down the SQS queue - this would be better suited to tiled operations WPS is great when someone else provides the service. I imagine it would be very interesting to the academic scientific world and government groups tasked with providing access to all the myriad imagery coming off space sensor platforms. Just thinking out loud. More thoughts here: http://www.cadmaps.com/gisblog/?p=28 randy ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Image Management in an RDBMS...(was OS Spatial environment 'sizing')
Hi Bruce, What approaches are people using with large Lidar datasets? You might take a look at the WeoGeo group. They are a commercial operation, not FOSS, but they are throwing dedicated AWS instances at the issue of lidar file serving. The dedicated instance, I gather, is for the sole use of the download client with any clip, re-project, or re-format processing requested. Paul Bisset would be happy to clarify. http://www.weogeo.com/ Some more details here: http://www.cadmaps.com/gisblog/?p=25 The lidar itself is I assume a raw file, probably split into a set of S3 objects which are stream concatenated and run through the processing instance and out to the requestor. Since the data sets are very large the dedicated assignment of a temporary cpu or two is justified. Beyond that I would like to see at least some lidar sets treated like the JPL srtm which is made accessible via WMS with pixel coded elevation as grayscale. Paul Ramsey's inimitable advice would work well for lidar too. The end client can turn a WMS request on a lidar image pyramid behind say Geoserver into whatever is desired, in my case I would turn it into 3D xaml meshes. I imagine this could be done in a more standards compliant manner through the WCS spec. I believe that the JPL srtm was made public before an implementation of WCS was accessible. http://onearth.jpl.nasa.gov/index.html The default styles for the elevation layers are a version of the elevation maps scaled to 8 bit. The full elevation values can be retrieved from this server by requesting the short_int or feet_short_int styles in combination with the image/png image/geotiff or image/tiff value for the format argument. The result of such a request will be an image where the signed short integer values contained in the image file for each pixel are the elevation of the respective point on the map, in either meters or feet. The base data is in meters. The us_ned layer base data is floating point real numbers in meters, data which can be retried in tiff or geotiff format when using the real or feet_real as styles. I like the interchange of comments on this subject . In the end I'm inclined to stick with the simplicity of file storage for imagery. Thanks Randy From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Thursday, February 21, 2008 6:25 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Image Management in an RDBMS...(was OS Spatial environment 'sizing') IMO: Paul, On Feb 21, 2008, at 4:19 PM, [EMAIL PROTECTED] wrote: What it comes down to is what is appropriate for your use case. Indeed! However, there seem to be vanishingly few use cases for which raster-in-database is actually the more appropriate solution. ;-) I beg to differ. (BTW, point-in-time recovery, a nice example of a place where database semantics have an upper hand. Although more modern file systems and enterprise backup systems are pretty competitive now... even a relatively simple hack like the OS/X Time Machine feature solves that problem for-all-practical-purposes.) Trying to manage very large regional datasets via a file based solution is problematic as described earlier with tile based approach to vector data in particular. Again for my use case the DB is better. Just to throw in another related issue: Lidar systems are throwing out an enormous amount of data. I had one dataset of only around 17 million odd records several years ago (of course stored in our corporate db ;-) ) that we could not handle with ArcGIS Desktop (v9.1). From memory it was a 32bit issue. What approaches are people using with large Lidar datasets? Bruce Notice: This email and any attachments may contain information that is personal, confidential, legally privileged and/or copyright. No part of it should be reproduced, adapted or communicated without the prior written consent of the copyright owner. It is the responsibility of the recipient to check for and remove viruses. If you have received this email in error, please notify the sender by return email, delete it from your system and destroy any copies. You are not authorised to use, communicate or rely on the information contained in this email. Please consider the environment before printing this email. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] OS Spatial environment 'sizing'
Hi Bruce, On the scale relatively quickly front, you should look at Amazon's EC2/S3 services. I've recently worked with it and find it an attractive platform for scaling http://www.cadmaps.com/gisblog The stack I like is Ubuntu+Java+ Postgresql/PostGIS + Apache2 mod_jk Tomcat + Geoserver + custom SVG or XAML clients run out of Tomcat If you use the larger instances the cost is higher but it sounds like you plan on some heavy raster services (WMS,WCS) and lots of memory will help. Small EC2 instance provides $0.10/hr: 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform Large EC2 instances provide $0.40/hr: 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform Extra large EC2 instances $0.80/hr: 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform Note: that the instances do not need to be permanent. Some people (WeoGeo) have been using a couple of failover small instances and then starting new large instances for specific requirements. The idea is to start and stop instances as required rather than having ongoing infrastructure costs. It only takes a minute or so to start an ec2 instance. If you are running a corporate service there may be parts of the day with very little use so you just schedule your heavy duty instances for peak times. If you can connect your raster to S3 buckets rather than instance storage you have built in replicated backup. I know that Java JAI can easily eat up memory and is core to Geoserver WMS/WCS so you probably want to look at large memory footprint for any platform with lots of raster service. I'm partial to Geoserver because of its Java foundation. I think I would try to keep the Apache2 mod_jk Tomcat Geoserver on a separate server instance from PostGIS. This might avoid problems for instance startup since your database would need to be loaded separately. The instance ami resides in a 10G partition the balance of data will probably reside on a /mnt partition separate from ec2-run-instances. You may be able to avoid datadir problems by adding something like Elastra to the mix. Elastra beta is a wrapper for PostgreSql that puts the datadir on S3 rather than local to an instance. I suppose they still keep indices(GIST et al) on the local instance. (I still think it an interesting exercise to see what could be done connecting PostGIS to AWS SimpleDB services.) So thinking out loud here is a possible architecture- Basic permanent setup put raster in S3 - this may require some customization of Geoserver, build a datadir in a PostGIS and backup to S3 create a private ami for Postgresql/PostGIS create a private ami for the load balancer instance create a private ami with your service stack for both a small and large instance for flexibility, Startup services start a balancer instance point your DNS CNAME to this balancer instance start a PostGis instance (you could have more than one if necessary but it would be easier to just scale to a larger instance type if the load demands it) have a scripted download from an S3 BU to your PostGIS datadir (I'm assuming a relatively static data resource) Variable services start service stack instance and connect to PostGIS update balancer to see new instance - this could be tricky repeat previous two steps as needed at night scale back - cron scaling for a known cycle or use a controller like weoceo to detect and respond to load fluctuation By the way the public AWS ami with the best resources that I have found is Ubuntu 7.10 Gutsy. The debian dependency tools are much easier to use and the resources are plentiful. I've been toying with using an AWS stack adapted for serving some larger Postgis vector sets such as fully connected census demographic data and block polygons here in US. The idea would be to populate the data directly from the census SF* and TIGER with a background Java bot. There are some potentially novel 3D viewing approaches possible with xaml. Anyway lots of fun to have access to virtual systems like this. As you can see I'm excited anyway. randy From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Monday, February 18, 2008 6:35 PM To: OSGeo Discussions Subject: [OSGeo-Discuss] OS Spatial environment 'sizing' IMO: Hello everyone, I'm trying to get a feel for server 'sizing' for a **hypothetical** Corporate environment to support OS Spatial apps. Assume that: - this is a dedicated environment to allow the use of OS Spatial applications to serve Corporate OGC Services. - the applications of interest are GeoServer, Deegree, GeoNetwork, MapServer, MapGuide and Postgres/PostGIS. - the environment may need to scale relatively quickly.
RE: [OSGeo-Discuss] OS Spatial environment 'sizing'
Hi Ivan, The most common advice I've seen says to leave raster out of the DB. Of course footprints and meta data could be there, but you would want to point Geoserver coverage to the image/image pyramid url somewhere in the directory hierarchy. Brent has a nice writeup here: http://docs.codehaus.org/display/GEOSDOC/Load+NASA+Blue+Marble+Data In an AWS sense my idea is to Java proxy the Geoserver Coverage Data URL to S3 buckets and park the imagery over on the S3 side to take advantage of stability and replication. Performance, though, might not be as good as a local directory. Maybe a one time cache to a local directory would work better. Note: Amazon doesn't charge for inside AWS data transfers. So in theory: PostGIS holds the footprint geometry + metadata EC2 Geoserver WFS handles footprint queries into an Svg/Xaml client, just stick it on top of something like JPL BMNG. Once a user picks a coverage switch to the Geoserver WMS/WCS service for zooming around in the selected image pyramid S3 buckets contain the tiffs, pyramids ... EC2 Geoserver handles WMS/WCS service EC2 proxy pulls the imagery from the S3 side as needed Sorry I haven't had time to try this so it is just theoretical. Of course you can go traditional and just keep the coverage imagery files on the local instance avoiding the S3 proxy idea. The reason I don't like that idea is the imagery has to be loaded with every instance creation while an S3 approach would need only one copy. randy -Original Message- From: Lucena, Ivan [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 19, 2008 2:59 PM To: [EMAIL PROTECTED]; OSGeo Discussions Subject: Re: [OSGeo-Discuss] OS Spatial environment 'sizing' Hi Randy, Bruce, That is a nice piece of advise Randy. I am sorry to intrude the conversation but I would like to ask how that heavy raster manipulation would be treated by PostgreSQL/PostGIS, managed or unmanaged? Best regards, Ivan Randy George wrote: Hi Bruce, On the scale relatively quickly front, you should look at Amazon's EC2/S3 services. I've recently worked with it and find it an attractive platform for scaling http://www.cadmaps.com/gisblog The stack I like is Ubuntu+Java+ Postgresql/PostGIS + Apache2 mod_jk Tomcat + Geoserver + custom SVG or XAML clients run out of Tomcat If you use the larger instances the cost is higher but it sounds like you plan on some heavy raster services (WMS,WCS) and lots of memory will help. Small EC2 instance provides $0.10/hr: 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform Large EC2 instances provide $0.40/hr: 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform Extra large EC2 instances $0.80/hr: 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform Note: that the instances do not need to be permanent. Some people (WeoGeo) have been using a couple of failover small instances and then starting new large instances for specific requirements. The idea is to start and stop instances as required rather than having ongoing infrastructure costs. It only takes a minute or so to start an ec2 instance. If you are running a corporate service there may be parts of the day with very little use so you just schedule your heavy duty instances for peak times. If you can connect your raster to S3 buckets rather than instance storage you have built in replicated backup. I know that Java JAI can easily eat up memory and is core to Geoserver WMS/WCS so you probably want to look at large memory footprint for any platform with lots of raster service. I'm partial to Geoserver because of its Java foundation. I think I would try to keep the Apache2 mod_jk Tomcat Geoserver on a separate server instance from PostGIS. This might avoid problems for instance startup since your database would need to be loaded separately. The instance ami resides in a 10G partition the balance of data will probably reside on a /mnt partition separate from ec2-run-instances. You may be able to avoid datadir problems by adding something like Elastra to the mix. Elastra beta is a wrapper for PostgreSql that puts the datadir on S3 rather than local to an instance. I suppose they still keep indices(GIST et al) on the local instance. (I still think it an interesting exercise to see what could be done connecting PostGIS to AWS SimpleDB services.) So thinking out loud here is a possible architecture- Basic permanent setup put raster in S3 - this may require some customization of Geoserver, build a datadir in a PostGIS and backup to S3 create a private ami for Postgresql/PostGIS create a private ami for the load
