David, I think I understand how DataStores work now, at least in theory. I appreciate you taking the time to explain, as I know you must be very busy with work at Vivid Solutions.
I don't think the DataStore framework is the solution to OpenJUMP's design problem, although it makes sense if your working with a small set of feature from a much larger data source, like a remote PostgreSQL database for example. However, I am interested in allowing a user to view and manipulate all of the features in a spatial data set, within reasonable limits, without restrictions placed by the size of the computer's RAM. I will try to work with Michael and any of the other interested developer's to devise a solution to this. If Vivid is interested, I will track the changes I make to the core so the modifications can be incorporated into Vivid's JUMP. I know a few changes will have to take place to the core source code, but I would like to encapsulate most of the functionality as a plug-in. If Vivid was willing to make the changes to their core I would be willing to make sure the plug-in was compatible with Vivid's JUMP. That would give both Vivid's JUMP and OpenJUMP the ability to work with very large spatial datasets. Michael, I am glad I am not the only one who got a headache after trying to understand some of GeoTools code. Overall it seems like a great project but I've always found the lack of developer documentation a frustration. :] Before I talk about the memory problem, I had a couple of questions for you: [1] How do you pronounce your first name? (This is probably a funny question from a guy on a mailing list, but I am just curious.) :] [2] Which JUMP project are you involved in, if any? Is it SIGLE? [3] Are you still actively using or developing with OpenJUMP? I was very pleased to hear from you again, and to learn that you had also given some serious thought to OpenJUMP's memory problem. It looks like you even implemented some solutions. I am very eager to hear about what you learned, as this will be a great aid in my own efforts. You mentioned that you "successfully implemented the feature interface to read features on disk". Did this not solve the problem? I am curious because it is a solution that I considered. My only hesitation was that a Feature interface implemented in this way would still require some level of representation in RAM. It this representation was significant, and the spatial dataset large enough, you will still hit the RAM restriction, though not as soon. How did this solution work out, and should I consider it? If this is not a workable solution I think my only option will be to follow Alvaro's suggestion and change the classes in OpenJUMP's source code that depend on the getFeatures() and query() methods. (Although this looks like it will be a bear of a task. The Agile code won't be of much use to me, as Alvaro writes his source code comments in Spanish...) :] Are you interested in working with me on this solution? If you are working with the SIGLE project and would like to incorporate my changes/plug-in I'd like to stay intouch with you about my design for the solution to this memory problem and how I can make sure it is compatible with SIGLE. Let me know. The Sunburned Surveyor On 8/22/06, David Zwiers <[EMAIL PROTECTED]> wrote: > Michaël, > > We are planning to include an API for writing to DataStores at some point in > the future, but we do not have a target date yet. > > I should also remind you, this API is not intended to feed a streaming > renderer (like Geotools), so there will still be cases where the size of the > data will preclude the user from viewing all the data at one time. > > David Zwiers > Vivid Solutions > Telephone : (250) 385-6040 > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michaël Michaud > Sent: August 22, 2006 4:03 PM > To: List for discussion of JPP development and use. > Subject: Re: [JPP-Devel] Refactoring JUMP fpr support oflarge > spatialdatasets... > > > >P.S. - If I can't get this figured out I will give you a call, but I > >wanted to post this to the list for the benefit of the other > >developers. > > > > > I am very pleased to follow this discussion. > I already tested shapefile-scalable from agile, unsuccessufully tried to > implement a hsqldb-database connection to avoid the memory pb, > successfully implemented the feature interface to read features on disk, > and catched headaches trying to understand the GeoTools datastore api. > So I am pleased to follow this interesting discussion on how will jump > soon allow large dataset reading. > > The next question is... is the new datastore api planned for scalable > readers only or does it include data modification capabilities for > scalable disk-based readers/writers ? > > Michaël > > >On 8/22/06, David Zwiers <[EMAIL PROTECTED]> wrote: > > > > > >>Sunburned, > >> > >>Most of the required components are readily available. I would look into > >>leveraging the DataStore framework (Jump CVS) in conjunction with an > >>indexed SHP reader (GeoTools port / interface?). You need only write the > >>'glue code' between the DataStore framework (already does caching based > >>on the viewing area) and your SHP reader of choice. > >> > >>If you need a few more directed hints to get started, feel free to drop > >>me a line. > >> > >>David Zwiers > >>Vivid Solutions > >>Telephone : (250) 385-6040 > >> > >> > >>-----Original Message----- > >>From: [EMAIL PROTECTED] > >>[mailto:[EMAIL PROTECTED] On Behalf Of > >>Sunburned Surveyor > >>Sent: August 22, 2006 8:26 AM > >>To: List for discussion of JPP development and use. > >>Subject: [JPP-Devel] Refactoring JUMP fpr support of large > >>spatialdatasets... > >> > >>I've been taking a look at the OpenJUMP source code, as well as the > >>work done by Alvaro on Agiles version of OpenJUMP, in an effort to > >>come up with a relatively simple solution to the RAM restriction > >>OpenJUMP has when reading large ESRI Shapefiles or other spatial > >>datasets. I've hit an obstacle of sorts, and I was hoping some of the > >>more experienced developers would have a solution. > >> > >>Alvaro overcame the RAM restriction by creating an object that > >>implemented the FeatureCollection interface, but stored its features > >>on disk, and not in RAM. I had the same idea, and was initially > >>pleased when I saw Alvaro had pioneered this technique. I didn't > >>understand why Alvaro had created his own version of the OpenJUMP > >>core, instead of encapsulating his support for large spatial datasets > >>in a plug-in. > >> > >>Then I read Alvaro's note: > >> > >>" In package org.agil.dao, you could see a FeatureCollection > >>implementation > >>(FeatureCollectionOnDemand) which read features directly from a > >>database. > >>After use it, you must change all JUMP code where get a List from a > >>FeatureCollection, and change it for an Iterator." > >> > >>I took a look at the source code and found that Alvaro was indeed > >>correct. The FeatureCollection interface has two methods that return a > >>generic list of all the Feature objects in the FeatureCollection. This > >>creates a problem, as all of the Feature objects then need to exist in > >>RAM, which is what we are trying to avoid in the first place. > >> > >>One of these two methods, the getFeatures() method, is called 26 times > >>in other parts of the source code. > >> > >>What does this mean? It means both JUMP and OpenJUMP are currently > >>wired in a way that prevents RAM independent FeatureCollection > >>implementations without some significant changes to the core source > >>code. (It would require that all of the methods that expect a list of > >>all the Features in a FeatureCollection be changed to work with an > >>Iterator object, as Alvaro suggested.) > >> > >>There are a couple of ways to fix this. I want to know which way will > >>be the simplest and most effective. > >> > >>[1] Refactor the OpenJUMP source code to use Iterators instead of > >>lists when calling the two problem methods of the FeatureCollection > >>interface. > >> > >>[2] Create a "RAM" independent object that implements the Feature > >>interface instead of a RAM independent implementation of an object the > >>Feature Collection interface. We could then return a list of these RAM > >>independent Features. (I'm not sure how this would reduce memory usage > >>though. Probably not as well as a RAM independent FeatureCollection > >>implementation. There might also be a pretty severe performance > >>penalty. ???) > >> > >>[3] Create a custom implementation of the Java.Util.List interface > >>that doesn't require its member objects to be in RAM. The only real > >>problem I saw with this technique initially is the toArray() method. I > >>suppose we could return an empty array, because I don't know another > >>way to implement this method without placing all the Feature Objects > >>into the array, and thus into RAM. This technique will also ivolve a > >>bit of work, as the List interface contains quite a few methods. > >> > >>I'm really looking for the simplest solution. I have planned a > >>"spatial database" for OpenJUMP-Ex that will solve this problem in > >>OpenJUMP's design. So I don't really want to invest a lot of energy > >>into what I hope will be a temporary "band-ais" solution. > >> > >>Thank you for your thoughts. > >> > >>The Sunburned Surveyor > >> > >>------------------------------------------------------------------------ > >>- > >>Using Tomcat but need to do more? Need to support web services, > >>security? > >>Get stuff done quickly with pre-integrated technology to make your job > >>easier > >>Download IBM WebSphere Application Server v.1.0.1 based on Apache > >>Geronimo > >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > >>_______________________________________________ > >>Jump-pilot-devel mailing list > >>[email protected] > >>https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > >> > >> > >> > >>------------------------------------------------------------------------- > >>Using Tomcat but need to do more? Need to support web services, security? > >>Get stuff done quickly with pre-integrated technology to make your job > >>easier > >>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > >>_______________________________________________ > >>Jump-pilot-devel mailing list > >>[email protected] > >>https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > >> > >> > >> > > > >------------------------------------------------------------------------- > >Using Tomcat but need to do more? Need to support web services, security? > >Get stuff done quickly with pre-integrated technology to make your job easier > >Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > >_______________________________________________ > >Jump-pilot-devel mailing list > >[email protected] > >https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Jump-pilot-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Jump-pilot-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Jump-pilot-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
