Yup. GeoTools is a nightmare to use and understand. It makes me wonder how they ever built something as functional as UDig on top of it.
However, the alternative was to write a new Shapefile parser from scratch, and that seemed like an awful lot of work. I have no doubt I could do it, but it is probably a notch above my current skill level as a hobby programmer. I thought it would be best to instead work with the GeoTools programmers to get something usable from their code. I wonder if Deegree or Kosmo is using a streaming Shapefile parser? The Sunburned Surveyor On 8/30/07, Larry Becker <[EMAIL PROTECTED]> wrote: > Yes, I have looked at the various newer versions of the GeoTools > Shapefile readers. They seemed to be too complicated to be a drop in > replacement for our older GeoTools version (which also has the > capability to randomly read, but that is useless in OJ's > implementation). > > Larry > > > On 8/30/07, Sunburned Surveyor <[EMAIL PROTECTED]> wrote: > > Larry, > > > > I don't know if it is of interest to you, but GeoTools has a streaming > > ESRI Shapefile parser. They actually have the ability to randomly > > access an indexed Shapefile. I plan on using their Shapefile code in > > my FeatureCache. The idea is to use the GeoTools code to access one > > Feature at a time from the Shapefile, storing this Feature in the > > FeatureCache. > > > > SS > > > > On 8/30/07, Larry Becker <[EMAIL PROTECTED]> wrote: > > > >At one point I actually rewrote the Shapefile parser to be streaming as > > > >well... > > > > > > I guess I hadn't actually realized it wasn't until now. Do you > > > remember what happened to the code, or why you didn't stay with the > > > streaming version? A very large shape file seems like a more likely > > > scenario that I actually care about. > > > > > > Larry > > > On 8/30/07, Martin Davis <[EMAIL PROTECTED]> wrote: > > > > This would be especially important if you had multiple > > > > FeatureCollections stored in one file, and you wanted to load only one > > > > of them. > > > > > > > > Sunburned Surveyor wrote: > > > > > Paul is correct. The pull parser does not reduce the memory of the > > > > > parsing results, but it does reduce the memory used during the parsing > > > > > process. That is because an in-memory representation of the entire XML > > > > > document is not constructed. > > > > > > > > > > One advantage of this is using the parser to select only data within > > > > > the XML file that meets specific criteria. For example, if we had a > > > > > 50MB SGF file representing the city of Stockton, I could parse the > > > > > file and create only building features, even thought the file might > > > > > contain road features, landmark features, park features. etc. > > > > > In fact, I could even parse the file and only create features for > > > > > buildings whose "building type" attribute was set to "Public". This > > > > > allows me to extract the information I want without reading all 50 MB > > > > > into memory. > > > > > > > > > > The Sunburned Surveyor > > > > > > > > > > On 8/30/07, Paul Austin <[EMAIL PROTECTED]> wrote: > > > > > > > > > >> Hi Larry, > > > > >> > > > > >> You are correct that the resulting data set will take up a lot of > > > > >> memory > > > > >> at the end, the advantage with the pull parser is that you don't > > > > >> take up > > > > >> a whole bunch of extra memory for the XML DOM structures which > > > > >> typically > > > > >> get loaded into memory for the whole document. So with the pull > > > > >> parser > > > > >> there is little memory overhead where as for DOM you have probably at > > > > >> least 2x memory required to load if not more > > > > >> > > > > >> Paul > > > > >> > > > > >> Larry Becker wrote: > > > > >> > > > > >>> It isn't the parser that takes up the memory except temporarily), > > > > >>> but > > > > >>> the memory resident dataset after loading. This will still limit > > > > >>> the > > > > >>> size. > > > > >>> > > > > >>> Larry > > > > >>> > > > > >>> On 8/30/07, Sunburned Surveyor <[EMAIL PROTECTED]> wrote: > > > > >>> > > > > >>> > > > > >>>> Yup. It makes you wonder why they didn't use pull parsers from the > > > > >>>> very beginning, doesn't it. > > > > >>>> > > > > >>>> SS > > > > >>>> > > > > >>>> On 8/30/07, Paul Austin <[EMAIL PROTECTED]> wrote: > > > > >>>> > > > > >>>> > > > > >>>>> Agreed the pull parser is the only way to go for large XML files > > > > >>>>> > > > > >>>>> Paul > > > > >>>>> > > > > >>>>> Sunburned Surveyor wrote: > > > > >>>>> > > > > >>>>> > > > > >>>>>> Martin, > > > > >>>>>> > > > > >>>>>> If we decide to support a restricted form of GML 2 we could > > > > >>>>>> build our > > > > >>>>>> reader and writer on top of the XML Pull Parser from Sun. This > > > > >>>>>> would > > > > >>>>>> help us to avoid memory problems when reading in large files. > > > > >>>>>> > > > > >>>>>> https://sjsxp.dev.java.net/ > > > > >>>>>> > > > > >>>>>> Just a thought. > > > > >>>>>> > > > > >>>>>> The Sunburned Surveyor > > > > >>>>>> > > > > >>>>>> ------------------------------------------------------------------------- > > > > >>>>>> This SF.net email is sponsored by: Splunk Inc. > > > > >>>>>> Still grepping through log files to find problems? Stop. > > > > >>>>>> Now Search log events and configuration files using AJAX and a > > > > >>>>>> browser. > > > > >>>>>> Download your FREE copy of Splunk now >> http://get.splunk.com/ > > > > >>>>>> _______________________________________________ > > > > >>>>>> Jump-pilot-devel mailing list > > > > >>>>>> Jump-pilot-devel@lists.sourceforge.net > > > > >>>>>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>> ------------------------------------------------------------------------- > > > > >>>>> This SF.net email is sponsored by: Splunk Inc. > > > > >>>>> Still grepping through log files to find problems? Stop. > > > > >>>>> Now Search log events and configuration files using AJAX and a > > > > >>>>> browser. > > > > >>>>> Download your FREE copy of Splunk now >> http://get.splunk.com/ > > > > >>>>> _______________________________________________ > > > > >>>>> Jump-pilot-devel mailing list > > > > >>>>> Jump-pilot-devel@lists.sourceforge.net > > > > >>>>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>> ------------------------------------------------------------------------- > > > > >>>> This SF.net email is sponsored by: Splunk Inc. > > > > >>>> Still grepping through log files to find problems? Stop. > > > > >>>> Now Search log events and configuration files using AJAX and a > > > > >>>> browser. > > > > >>>> Download your FREE copy of Splunk now >> http://get.splunk.com/ > > > > >>>> _______________________________________________ > > > > >>>> Jump-pilot-devel mailing list > > > > >>>> Jump-pilot-devel@lists.sourceforge.net > > > > >>>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>> > > > > >>> > > > > >> ------------------------------------------------------------------------- > > > > >> This SF.net email is sponsored by: Splunk Inc. > > > > >> Still grepping through log files to find problems? Stop. > > > > >> Now Search log events and configuration files using AJAX and a > > > > >> browser. > > > > >> Download your FREE copy of Splunk now >> http://get.splunk.com/ > > > > >> _______________________________________________ > > > > >> Jump-pilot-devel mailing list > > > > >> Jump-pilot-devel@lists.sourceforge.net > > > > >> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > >> > > > > >> > > > > > > > > > > ------------------------------------------------------------------------- > > > > > This SF.net email is sponsored by: Splunk Inc. > > > > > Still grepping through log files to find problems? Stop. > > > > > Now Search log events and configuration files using AJAX and a > > > > > browser. > > > > > Download your FREE copy of Splunk now >> http://get.splunk.com/ > > > > > _______________________________________________ > > > > > Jump-pilot-devel mailing list > > > > > Jump-pilot-devel@lists.sourceforge.net > > > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > > > > > > > > > -- > > > > Martin Davis > > > > Senior Technical Architect > > > > Refractions Research, Inc. > > > > (250) 383-3022 > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > This SF.net email is sponsored by: Splunk Inc. > > > > Still grepping through log files to find problems? Stop. > > > > Now Search log events and configuration files using AJAX and a browser. > > > > Download your FREE copy of Splunk now >> http://get.splunk.com/ > > > > _______________________________________________ > > > > Jump-pilot-devel mailing list > > > > Jump-pilot-devel@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > > > > -- > > > http://amusingprogrammer.blogspot.com/ > > > > > > ------------------------------------------------------------------------- > > > This SF.net email is sponsored by: Splunk Inc. > > > Still grepping through log files to find problems? Stop. > > > Now Search log events and configuration files using AJAX and a browser. > > > Download your FREE copy of Splunk now >> http://get.splunk.com/ > > > _______________________________________________ > > > Jump-pilot-devel mailing list > > > Jump-pilot-devel@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. > > Still grepping through log files to find problems? Stop. > > Now Search log events and configuration files using AJAX and a browser. > > Download your FREE copy of Splunk now >> http://get.splunk.com/ > > _______________________________________________ > > Jump-pilot-devel mailing list > > Jump-pilot-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > -- > http://amusingprogrammer.blogspot.com/ > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Jump-pilot-devel mailing list > Jump-pilot-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
