I'm retesting with Arnd's new XYZ-data which contained a 62MB shape file and a 33MB dbf file.
1. OJ loaded the file in 15 seconds and finished drawing it in 55 seconds. 335MB committed memory. 2. Finished selecting about 2/3 of the points in 75 seconds Finished drawing the selection handles in another two minutes. Up to 452MB of committed memory. Right clicked on the view and it took about 2 minutes to respond. Clearly the selection of 600000+ objects is causing havoc in the UI EnableChecks. Also, the selection handle rendering optimization that I made is not effective for points, so it must draw 600000 tiny yellow rectangles each redraw. So is it about memory? I increased OJ's memory allocation to 750MB and reloaded. Same load and redraw stats. 3. I used the Edit->Select layer items. It took 2 minutes to display the selected items message (1000000) on the status bar. While selection handles were drawing, committed memory cycled between 650 and 700 MB. Clearly, frequent pauses in drawing were caused by the JVM garbage collector. I waited for the handles to finish drawing so that I could test the menu response time. Finally, it finished. I clicked on the Edit menu. After a few seconds the workbench window went totally blank for 3 minutes, then came back. I didn't try it again. Conclusion: I concur with Arnd that OpenJump is unworkable for point files in excess of 500000. The redraw of a million points is bad, redraw of a million selection handles is worse, and the UI is totally unresponsive with large numbers of objects selected. I believe that we may have reached the design limits of the UI feedback mechanisms. In particular EnableCheck class methods make multiple references to the number of features. The selection feedback on the status bar even computes the number of points (requiring a million iterations for each access). I repeated the experiment with JUMP 1.2 (750MB memory) which does not have some of the UI enhancements like the status bar display of number of selected items and points. The first thing I noticed is that JUMP renders much slower than OpenJump. No way am I going to wait for it to finish. I did a drag select around the whole view (since there is no select layer items on the Edit menu). It is taking a long time. I'm not timing any of this since the only thing I'm interested in is the menu response time. I found that an easy test to see if the UI is responsive is to mouse over a toolbar icon and see if it highlights. Oops, JUMP just gave an Out of Memory Error. End of that experiment. I still believe the main problem is the UI checks. The UI is totally responsive until you make a large selection. Then it gets progressively slower proportional to the size of the selection. Fixing this is the number one priority. I loaded the file in SkyJUMP which has metrics for screen redraw. It rendered the million points in 50 seconds. There is room for improvement here, but this is not really the main problem. Any optimization done to optimize point drawing would also apply to selection handles. Fixing this is the number two priority. regards, Larry On 10/4/07, Larry Becker <[EMAIL PROTECTED]> wrote: > > OK, for my PC (P4 3.2Ghz with OJ set to use 256MB) it takes 5 seconds to > load your 351k points u1 layer, and 17 seconds to draw. Pretty bad draw > performance compared with other geometry types, but not really relevant to > the discussion since waiting for the redraw is not necessary. > > 1. I managed to copy and paste about 100k of these to a new layer in about > 10 seconds. Replicate gives similar results. > > 2. Selecting about 1/3 of the points takes about 5 seconds to do, and > about 8 more to render the handles, although as I mentioned before, you can > proceed as soon as the the status bar indicates the number selected. > > 3. Selecting all points takes about 20 seconds, and another 30 to render. > > Preliminary results for 350000 points: slow but definitely workable. > > I loaded 3 copies of the u1 layer giving a total of over 1 million > points. Memory usage is at 200 MB committed. I suspect it would be quite a > bit more if Stefan's data had any attributes. > > 4. Redraw is now about 45 seconds, but just ignoring it works for me. > > 5. Selecting about 1/3 failed after 90 seconds and resulted in an Out of > Memory Error. > > After allocating 512MB of memory to OJ, I restarted. 390000 points > selected after 20 seconds. The selection handles finished rendering 40 > seconds later. 388MB in use. > > 6. Right clicking on the selection takes about 5 seconds to bring up the > menu. > > 7. Replicate took about 20 seconds to replicate to a new layer. > > These are my current findings. I'll update the post when I test with > Arnd's data. So far I don't see anything too bad. About what you might > expect really. The rendering code is not optimized for points, and the > selection code is dealing with its worst case since the bounding box for > points is a point. > > Larry > On 10/4/07, Stefan Steiniger <[EMAIL PROTECTED]> wrote: > > > > oha.. > > that is interesting. Accidentaly i stored the points as multipoints. > > Now i updated my jump @office and stored the points as point (and wrote > > the file again to the ftp). The file size is now already a bit smaller, > > and also the commited memory... > > > > stefan > > > > Stefan Steiniger wrote: > > > > > Good if Arnd provides the data, but here the link to my data in case > > > it fails: > > > ftp://ftp.geo.uzh.ch/pub/sstein/data/u1_points.zip > > > > > > an option to get one million points is to copy the 350k points and > > > then to move them (if that works ;). > > > Btw. the lascers can data have no attributes (may be therefore smaller > > > > > than Arnd's; the covered area is relatively small < 200m) > > > (shapefile size is 30mb and zip xyz-ascii was 15mb) > > > > > > stefan > > > > > > Arnd Kielhorn wrote: > > > > > >> Hello Larry, > > >> > > >> first of all, thank You very much! > > >> > > >> I work with delaunay triangulation and building a 3D-Model with our > > >> plugin. > > >> > > >> If we both could make a direct connection (via FTP or a free tool > > >> called teamviewer) I could sent You one file with laserscanning data > > >> with 1 million points (XYZ). It is about 30 MB in the PIROL-CSV > > >> format or about 100 MB as shape file. > > >> > > >> Kindly regards > > >> Arnd > > >> > > >> Larry Becker schrieb: > > >> > > >>> I'm going to start investigating the point performance problem as > > >>> soon as I get some test data. I have been trying to locate a large > > >>> point dataset on the web, but no luck. Perhaps the best approach > > >>> might be to add a new item to the Generate menu, "Random Points". > > >>> > > >>> @Arnd: One thing that people often forget about JUMP is that > > >>> rendering is a background process. You don't have to wait for it > > >>> to finish to start the next operation. > > >>> What kind of terrain maps are you generating? Are you generating > > >>> contour lines from a grid of points? You mentioned deleting > > >>> selected points being impossibly slow. I don't see why this should > > >>> be so. If I can duplicate the problem, it should be an easy fix. > > >>> > > >>> regards, > > >>> Larry > > >>> > > >>> On 10/2/07, *Larry Becker* <[EMAIL PROTECTED] > > >>> <mailto: [EMAIL PROTECTED]>> wrote: > > >>> > > >>> HI Arnd, > > >>> > > >>> Would it be helpful to work around the problem by saving the > > >>> points as a shapefile and using a utility like shp2tile to break > > >>> the point data up into multiple files? > > >>> > > >>> see: http://imaptools.com/download-software.html > > >>> > > >>> regards, > > >>> Larry > > >>> > > >>> > > >>> On 10/2/07, *Arnd Kielhorn* < [EMAIL PROTECTED] > > >>> <mailto:[EMAIL PROTECTED]>> wrote: > > >>> > > >>> Hello Paul, > > >>> > > >>> thank You for the tip with the z-value, but how I can use it > > > > >>> as a > > >>> z-value on the point itself? > > >>> > > >>> The file fomrat is txt, csv (from PIROL CSV-Format) like > > >>> following example > > >>> $Rechts Hoch heigh > > >>> $grad grad m > > >>> $double double double > > >>> 3434000.00 5800000.00 93.78 > > >>> 3434001.00 5800000.00 93.84 > > >>> ... ... ... > > >>> > > >>> or the ESRI ascii grid format asc (reading plugin from > > SIGLE). > > >>> The most files I got have 1 million points and such csv > > files > > >>> have about > > >>> 30 MB. > > >>> In my start file for OJ I got 768 MB, because I have 1 GB > > RAM. > > >>> > > >>> Kindly regards > > >>> Arnd > > >>> > > >>> > > >>> Paul Austin schrieb: > > >>> > Arnd, > > >>> > > > >>> > What file format are you using? > > >>> > > > >>> > As JUMP loads the whole file into memory I would recommend > > > > >>> setting the > > >>> > heap size on the JAVA VM to be -Xmx512M if you are dealing > > >>> with large > > >>> > datasets. > > >>> > > > >>> > You mentioned that the height is an attribute on the > > feature, > > >>> if you use > > >>> > the height as a z-value on the point itself there will be > > >>> some memory > > >>> > savings. > > >>> > > > >>> > Paul > > >>> > > > >>> > Arnd Kielhorn wrote: > > >>> > > > >>> >> Hello Larry, > > >>> >> > > >>> >> as in the further discussion just named the main problema > > >>> are (for > > >>> >> example I just have to work with point layers with only a > > > > >>> heigh > > >>> >> attribute but with 1 million points; but also e.g. > > 300,000 > > >>> points are > > >>> >> enough): > > >>> >> - (re)drawing of the points > > >>> >> - strongly restricted or impossible operation (e.g. > > deleting > > >>> some > > >>> >> points/vertices after marking them; e.g. copying hundreds > > or > > >>> a few > > >>> >> thousand in a new layer to have layer with lesser points) > > >>> >> Very often OJ hangs on and I have to restart it. > > >>> >> I use such point layers for creating a digital terrain > > maps. > > >>> >> > > >>> >> Kindly regards > > >>> >> Arnd > > >>> >> > > >>> >> Larry Becker schrieb: > > >>> >> > > >>> >>> Hi Arnd, > > >>> >>> > > >>> >>> I haven't worked with large point datasets much. What > > >>> it is > > >>> >>> exactly that is slow? Is it redraw time? Or perhaps > > the > > >>> particular > > >>> >>> operation that you are doing? > > >>> >>> > > >>> >>> I know that point layers do not benefit from the > > recent > > >>> decimation > > >>> >>> optimization, so they take longer to draw than > > equivalent > > >>> polygons > > >>> >>> and polyline layers. I think Michaël tried a point > > >>> decimation > > >>> >>> optimization, but wasn't satisfied with the results. > > >>> >>> > > >>> >>> regards, > > >>> >>> Larry Becker > > >>> >>> > > >>> >>> On 10/1/07, *Stefan Steiniger* < [EMAIL PROTECTED] > > >>> <mailto: [EMAIL PROTECTED]> > > >>> >>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>> > > >>> wrote: > > >>> >>> > > >>> >>> mhm > > >>> >>> from my point of view it is both OJ and JRE. It > > would > > >>> be probably > > >>> >>> better > > >>> >>> when the points are stored and processes in a > > database. > > >>> >>> > > >>> >>> stefan > > >>> >>> > > >>> >>> Arnd Kielhorn schrieb: > > >>> >>> > Hello, > > >>> >>> > > > >>> >>> > yes I know, first of all OJ is a GIS for vector > > data. > > >>> But the > > >>> >>> > functionality is also very good for point > > datasets. > > >>> But working > > >>> >>> with > > >>> >>> > great point datasets (more 500,000 points) bring > > >>> problems with the > > >>> >>> > memory and the garbage collector works not > > satisfied. > > >>> Every > > >>> >>> command take > > >>> >>> > a lot of time or could not carried out. (CPU P4 > > 2.4 > > >>> GHz, 1 GB RAM) > > >>> >>> > Is it only a thing of the JRE or also from OJ? > > >>> >>> > So, it is useful to know if there is a chance to > > >>> increase the > > >>> >>> > performence of OJ in this case and what could be > > the > > >>> possible > > >>> >>> milestones > > >>> >>> > to reach this aim? > > >>> >>> > I am very eager to read everyones opion. > > >>> >>> > > > >>> >>> > Kindly regards > > >>> >>> > Arnd > > >>> >>> > _______________________________________________ > > >>> >>> > jump-users mailing list > > >>> >>> > [email protected] > > >>> <mailto: [email protected]> > > >>> >>> <mailto: [email protected] > > >>> <mailto:[email protected]>> > > >>> >>> > > > >>> http://lists.refractions.net/mailman/listinfo/jump-users > > >>> <http://lists.refractions.net/mailman/listinfo/jump-users> > > >>> >>> > > > >>> > > >>> <http://amusingprogrammer.blogspot.com/> > > >>> > > ------------------------------------------------------------------------ > > >>> > > >>> > > >>> _______________________________________________ > > >>> jump-users mailing list > > >>> [email protected] > > >>> http://lists.refractions.net/mailman/listinfo/jump-users > > >>> > > >> > > >> > > >> > > >> _______________________________________________ > > >> jump-users mailing list > > >> [email protected] > > >> http://lists.refractions.net/mailman/listinfo/jump-users > > >> > > >> > > > > > > _______________________________________________ > > > jump-users mailing list > > > [email protected] > > > http://lists.refractions.net/mailman/listinfo/jump-users > > > > > > > > > > _______________________________________________ > > jump-users mailing list > > [email protected] > > http://lists.refractions.net/mailman/listinfo/jump-users > > > > > > -- > http://amusingprogrammer.blogspot.com/ -- http://amusingprogrammer.blogspot.com/
_______________________________________________ jump-users mailing list [email protected] http://lists.refractions.net/mailman/listinfo/jump-users
