Re: [OSGeo-Discuss] Raster data on a DBMS
On 11/4/08, Chris Puttick [EMAIL PROTECTED] wrote: It is not necessary to store the image file itself in the database to get concurrency control, data protection, integrity and management features. There are a number of good document management systems (Alfresco, KnowledgeTree) that offer all the above for files, and the Zimbra collaboration system makes use of database for emails much the same reason. None of these actually store the files in database; the database is used to provide all the controls, and the access to the files is only via an interface that references the database and the additional functionality it provides. That doesn't make much sense to *me*. It is one thing to not want to deploy an rdbms to store images as it allegedly creates unnecessary complexity. It makes no sense to replace that complexity with some other complexity, that of a document management system in this case. Given that I have personally never heard of Alfresco or Knowledge Tree or other such document management systems, but I *have* heard of PostGres/MySQL/SQLite, I would much rather deal with known complexity than with unknown complexity. OTOH Microsoft put all their Exchange emails into the database and anyone who has ever managed an Exchange installation of any size can tell you just how many problems that can cause you... Yeah, but one allegedly bad or problematic approach doesn't necessarily speak for all other such approaches. Besides, while I may not like MS, for whatever reason, Exchange seems to be doing quite well in the marketplace. In any case, that is not the argument -- the argument is simply this -- does keeping images in a db make sense to you, the implementer? That is the only thing that matters. The users won't give a rip... all they care is that they can get to the images in their own known and intuitive ways. Your project manager will not give a rip as long as the project is under budget and on time (of course, you could be the project manager as well, as is the case in many FOSS projects). The only one who should give a rip is you, the developer, the implementer of the solution. I do believe that there may be cases where a raster-in-db makes sense. I personally don't want to recreate the mechanism for storing images on the filesystem... one can't just dump thousands of images in a folder. They have to be named uniquely, stored so too many don't fill a single folder, and so on... the db can do all these tasks for the developer happily. Nevertheless, the original premise still stands -- the world is big enough for both approaches, and the best marketing for any approach is the implementation of the approach. Hang it on the wall for the world to see, and if they don't like it, they will ask you to take it down. Chris - Gilberto Camara [EMAIL PROTECTED] wrote: Dear OSGEO Jim Gray´s paper and much more on this issue is on his site at MS Research. Storing images on a database gives much more benefits that simple retrieval of metadata. Databases offer concurrency control, data protection, integrity and management features that simple file systems are lacking. If you have hundreds of images scattered around as files, you lack data management. Your metadata may point to a file that could have been deleted. In a multi-user environment, file systems do not prevent different users from updating the same image. The result may be a data which is inconsistent. Allow me to reiterate my earlier argument, which is that FOSS4G should **allow** users the option of storing raster data in a database. Storing images in a database is not recommended in each and every situation. The user should have the option, according to his needs. The current debate on whether images should be stored on an RDBMS reminds me of a similar debate during the early 90s, concerning whether vector data should be stored in an RDBMS. Remember the days of ARC-INFO? In mid 90s, our team at INPE tried to use the Postgres-95 RDBMS to store vector data. The result was a system with a very slow performance. The concept was right, but the implementation was lacking. It was only when PostgreSQL and PostGIS came of age that we could develop a multi-user spatial database with good performance. By the same argument, these are early days of storing raster data in RDBMS. There are missing features on the database and the performance may be slower than file systems. But the concept is fundamentally correct. I predict that five years hence this debate will be solved and we will look at it as a relique of the past. Best Regards Gilberto Christopher Schmidt said: I don't see anything in that paragraph that indicates that storing the *image data* in the database is important. (A link to the paper online or something could change that, of course.)
Re: [OSGeo-Discuss] Raster data on a DBMS
Lucena, Ivan wrote: That would reduce the complexity a lot. It would be *just* 80 files. I probably don't even need a separate then in folder. I could also generate pyramids overlay on those. That would increase the data storage a little: 80 files 80x720x360x250x2x2=2073600 Does anybody advocate Geoserver, Mapserver, ArcServer, Imageserver, tile cache, openlayers or others? Or should I use raster or database for that? I think the subject of this thread is the real problem? Raster DATA. Raster information is only images. They require some sort of processing to extract any data from them? Data overlays for raster images will help to provide methods of searching for features and then displaying the correct area of the raster image? Timestamp data to identify a sequence of images would also be best stored in a database, with features and locations extracted from particular images stored along with links to the correct raster image. So the raster images are stored as files, and the data relating to them in a database? -- Lester Caine - G8HFL - Contact - http://lsces.co.uk/lsces/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk// Firebird - http://www.firebirdsql.org/index.php ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Raster data on a DBMS
On Mon, Nov 03, 2008 at 08:57:53PM -0200, Gilberto Camara wrote: Jim Gray´s paper and much more on this issue is on his site at MS Research. Gray has hundreds of papers listed on his Microsoft Research page. As I said, I'm not claiming that Gray's paper said or did not say something, merely that the section you quoted did not. Allow me to reiterate my earlier argument, which is that FOSS4G should **allow** users the option of storing raster data in a database. Storing images in a database is not recommended in each and every situation. The user should have the option, according to his needs. I'm not sure if you feel that someone is preventing this from happening in some way. It sounds like you think that there is some blocker here other than someone investing the time and effort to make this happen. Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Raster data on a DBMS
On 11/3/08, Gilberto Camara [EMAIL PROTECTED] wrote: .. Allow me to reiterate my earlier argument, which is that FOSS4G should **allow** users the option of storing raster data in a database. Storing images in a database is not recommended in each and every situation. The user should have the option, according to his needs. .. There are two kinds of users in this world when it comes to storing raster data... and, fortunately, the world is big enough for both of them. The FOSS in FOSS4G gives each kind to develop what they need/want. There is no concept of allowing anyone the option. Gilberto, my sense is that instead of convincing someone else that storing rasters in a db can/may be a better option in some cases, energy and resources might be better deployed in making options available, and making existing options easier and more powerful. In another thread you wrote -- INPE´s FOSS4G developement of raster data on RDBMS using the TerraLib library is a tangible proof of concept. TerraAmazon (built using TerraLib) is INPE's OS solution for monitoring tropical forests operationally. While I have an earlier version of TerraLib/TerraView that I got when I visited INPE last year, I am not familiar with its raster-in-db capabilities. Making and distributing a product that can work on different platforms, Windows, Macs, *nix, will be the best marketing for such capabilities, and provide the options that you advocate. Fwiw, O'Reilly's database war stories series has an article on Flickr that store metadata in a db and the photos themselves on directly in the filesystem. See http://radar.oreilly.com/archives/2006/04/database_war_stories_3_flickr.html. Other stories in that series are also very instructive. Apple's Aperture does the same thing... all the metadata are in SQLite (Coredata on Mac OS X) while the images themselves are stored in folders on the disk -- albeit hidden from the user. See http://www.bagelturf.com/aparticles/tips/tiplib/index.php and http://www.mungosmash.com/archives/2005/12/the_aperture_da.php. Of course, both of the above examples are of the high number of small size images variety while the raster/geographic applications are of the relatively fewer number of very large size images kind. Still, the case studies are instructive. I think the best argument for any approach is the approach itself. The free and open part in FOSS4G allows any and all approaches to exist. Scratch your own itch, meet your own need, then put it out for others to use. If they like it, they will adopt it and improve it. If they don't like it, it will die a timely death. Darwinian at its best. -- Puneet Kishor http://punkish.eidesis.org/ Nelson Institute for Environmental Studies http://www.nelson.wisc.edu/ Open Source Geospatial Foundation (OSGeo) http://www.osgeo.org/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
