well done!
i am not sure if this makes sense, but for drawing alone one could use a
"skip n points" mechanism or use an indexing scheme (e.g. quadtrees with
skipping the lower(?) levels to show). I guess this would need some
testing to identify the number of points that should be drawn maximally
and that need to be drawn. But for the selection this is not possible
(except the drawing). Not sure if modifying "enable check" is a solution
I guess Martin has some nice thoughts on that ;)
stefan
btw: @Martin: is you FOSS4G - DTM-TIN paper available? (I read he used
also massive data - millions of heigh points - with postgis)
PS: have you seen this presentation on high res image browsing, which
comes quite close to our problem, although a it is bit different domain:
http://www.ted.com/index.php/talks/view/id/129
(sorry don't want to hijack this topic but it is related)
Larry Becker schrieb:
I'm retesting with Arnd's new XYZ-data which contained a 62MB shape file
and a 33MB dbf file.
1. OJ loaded the file in 15 seconds and finished drawing it in 55
seconds. 335MB committed memory.
2. Finished selecting about 2/3 of the points in 75 seconds Finished
drawing the selection handles in another two minutes. Up to 452MB of
committed memory. Right clicked on the view and it took about 2 minutes
to respond.
Clearly the selection of 600000+ objects is causing havoc in the UI
EnableChecks. Also, the selection handle rendering optimization that I
made is not effective for points, so it must draw 600000 tiny yellow
rectangles each redraw.
So is it about memory? I increased OJ's memory allocation to 750MB and
reloaded. Same load and redraw stats.
3. I used the Edit->Select layer items. It took 2 minutes to display
the selected items message (1000000) on the status bar. While selection
handles were drawing, committed memory cycled between 650 and 700 MB.
Clearly, frequent pauses in drawing were caused by the JVM garbage
collector. I waited for the handles to finish drawing so that I could
test the menu response time. Finally, it finished. I clicked on the
Edit menu. After a few seconds the workbench window went totally blank
for 3 minutes, then came back. I didn't try it again.
Conclusion: I concur with Arnd that OpenJump is unworkable for point
files in excess of 500000. The redraw of a million points is bad,
redraw of a million selection handles is worse, and the UI is totally
unresponsive with large numbers of objects selected.
I believe that we may have reached the design limits of the UI feedback
mechanisms. In particular EnableCheck class methods make multiple
references to the number of features. The selection feedback on the
status bar even computes the number of points (requiring a million
iterations for each access).
I repeated the experiment with JUMP 1.2 (750MB memory) which does not
have some of the UI enhancements like the status bar display of number
of selected items and points. The first thing I noticed is that JUMP
renders much slower than OpenJump. No way am I going to wait for it to
finish. I did a drag select around the whole view (since there is no
select layer items on the Edit menu). It is taking a long time. I'm
not timing any of this since the only thing I'm interested in is the
menu response time. I found that an easy test to see if the UI is
responsive is to mouse over a toolbar icon and see if it highlights.
Oops, JUMP just gave an Out of Memory Error. End of that experiment.
I still believe the main problem is the UI checks. The UI is totally
responsive until you make a large selection. Then it gets progressively
slower proportional to the size of the selection. Fixing this is the
number one priority.
I loaded the file in SkyJUMP which has metrics for screen redraw. It
rendered the million points in 50 seconds. There is room for improvement
here, but this is not really the main problem. Any optimization done to
optimize point drawing would also apply to selection handles. Fixing
this is the number two priority.
regards,
Larry
On 10/4/07, *Larry Becker* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
OK, for my PC (P4 3.2Ghz with OJ set to use 256MB) it takes 5
seconds to load your 351k points u1 layer, and 17 seconds to draw.
Pretty bad draw performance compared with other geometry types, but
not really relevant to the discussion since waiting for the redraw
is not necessary.
1. I managed to copy and paste about 100k of these to a new layer in
about 10 seconds. Replicate gives similar results.
2. Selecting about 1/3 of the points takes about 5 seconds to do,
and about 8 more to render the handles, although as I mentioned
before, you can proceed as soon as the the status bar indicates the
number selected.
3. Selecting all points takes about 20 seconds, and another 30 to
render.
Preliminary results for 350000 points: slow but definitely workable.
I loaded 3 copies of the u1 layer giving a total of over 1 million
points. Memory usage is at 200 MB committed. I suspect it would be
quite a bit more if Stefan's data had any attributes.
4. Redraw is now about 45 seconds, but just ignoring it works for me.
5. Selecting about 1/3 failed after 90 seconds and resulted in an
Out of Memory Error.
After allocating 512MB of memory to OJ, I restarted. 390000 points
selected after 20 seconds. The selection handles finished rendering
40 seconds later. 388MB in use.
6. Right clicking on the selection takes about 5 seconds to bring up
the menu.
7. Replicate took about 20 seconds to replicate to a new layer.
These are my current findings. I'll update the post when I test
with Arnd's data. So far I don't see anything too bad. About what
you might expect really. The rendering code is not optimized for
points, and the selection code is dealing with its worst case since
the bounding box for points is a point.
Larry
On 10/4/07, *Stefan Steiniger* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
oha..
that is interesting. Accidentaly i stored the points as multipoints.
Now i updated my jump @office and stored the points as point
(and wrote
the file again to the ftp). The file size is now already a bit
smaller,
and also the commited memory...
stefan
Stefan Steiniger wrote:
> Good if Arnd provides the data, but here the link to my data
in case
> it fails:
> ftp://ftp.geo.uzh.ch/pub/sstein/data/u1_points.zip
>
> an option to get one million points is to copy the 350k points and
> then to move them (if that works ;).
> Btw. the lascers can data have no attributes (may be therefore
smaller
> than Arnd's; the covered area is relatively small < 200m)
> (shapefile size is 30mb and zip xyz-ascii was 15mb)
>
> stefan
>
> Arnd Kielhorn wrote:
>
> > Hello Larry,
> >
> > first of all, thank You very much!
> >
> > I work with delaunay triangulation and building a 3D-Model
with our
> > plugin.
> >
> > If we both could make a direct connection (via FTP or a free
tool
> > called teamviewer) I could sent You one file with
laserscanning data
> > with 1 million points (XYZ). It is about 30 MB in the PIROL-CSV
> > format or about 100 MB as shape file.
> >
> > Kindly regards
> > Arnd
> >
> > Larry Becker schrieb:
> >
> >> I'm going to start investigating the point performance
problem as
> >> soon as I get some test data. I have been trying to locate
a large
> >> point dataset on the web, but no luck. Perhaps the best
approach
> >> might be to add a new item to the Generate menu, "Random
Points".
> >>
> >> @Arnd: One thing that people often forget about JUMP is that
> >> rendering is a background process. You don't have to wait
for it
> >> to finish to start the next operation.
> >> What kind of terrain maps are you generating? Are you
generating
> >> contour lines from a grid of points? You mentioned deleting
> >> selected points being impossibly slow. I don't see why this
should
> >> be so. If I can duplicate the problem, it should be an easy
fix.
> >>
> >> regards,
> >> Larry
> >>
> >> On 10/2/07, *Larry Becker* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
> >> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>> wrote:
> >>
> >> HI Arnd,
> >>
> >> Would it be helpful to work around the problem by
saving the
> >> points as a shapefile and using a utility like shp2tile
to break
> >> the point data up into multiple files?
> >>
> >> see: http://imaptools.com/download-software.html
> >>
> >> regards,
> >> Larry
> >>
> >>
> >> On 10/2/07, *Arnd Kielhorn* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
> >> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>
wrote:
> >>
> >> Hello Paul,
> >>
> >> thank You for the tip with the z-value, but how I
can use it
> >> as a
> >> z-value on the point itself?
> >>
> >> The file fomrat is txt, csv (from PIROL CSV-Format) like
> >> following example
> >> $Rechts Hoch heigh
> >> $grad grad m
> >> $double double double
> >> 3434000.00 5800000.00 93.78
> >> 3434001.00 5800000.00 93.84
> >> ... ... ...
> >>
> >> or the ESRI ascii grid format asc (reading plugin
from SIGLE).
> >> The most files I got have 1 million points and such
csv files
> >> have about
> >> 30 MB.
> >> In my start file for OJ I got 768 MB, because I have
1 GB RAM.
> >>
> >> Kindly regards
> >> Arnd
> >>
> >>
> >> Paul Austin schrieb:
> >> > Arnd,
> >> >
> >> > What file format are you using?
> >> >
> >> > As JUMP loads the whole file into memory I would
recommend
> >> setting the
> >> > heap size on the JAVA VM to be -Xmx512M if you are
dealing
> >> with large
> >> > datasets.
> >> >
> >> > You mentioned that the height is an attribute on
the feature,
> >> if you use
> >> > the height as a z-value on the point itself there
will be
> >> some memory
> >> > savings.
> >> >
> >> > Paul
> >> >
> >> > Arnd Kielhorn wrote:
> >> >
> >> >> Hello Larry,
> >> >>
> >> >> as in the further discussion just named the main
problema
> >> are (for
> >> >> example I just have to work with point layers
with only a
> >> heigh
> >> >> attribute but with 1 million points; but also
e.g. 300,000
> >> points are
> >> >> enough):
> >> >> - (re)drawing of the points
> >> >> - strongly restricted or impossible operation
(e.g. deleting
> >> some
> >> >> points/vertices after marking them; e.g. copying
hundreds or
> >> a few
> >> >> thousand in a new layer to have layer with lesser
points)
> >> >> Very often OJ hangs on and I have to restart it.
> >> >> I use such point layers for creating a digital
terrain maps.
> >> >>
> >> >> Kindly regards
> >> >> Arnd
> >> >>
> >> >> Larry Becker schrieb:
> >> >>
> >> >>> Hi Arnd,
> >> >>>
> >> >>> I haven't worked with large point datasets
much. What
> >> it is
> >> >>> exactly that is slow? Is it redraw time? Or
perhaps the
> >> particular
> >> >>> operation that you are doing?
> >> >>>
> >> >>> I know that point layers do not benefit from
the recent
> >> decimation
> >> >>> optimization, so they take longer to draw than
equivalent
> >> polygons
> >> >>> and polyline layers. I think Michaël tried a point
> >> decimation
> >> >>> optimization, but wasn't satisfied with the results.
> >> >>>
> >> >>> regards,
> >> >>> Larry Becker
> >> >>>
> >> >>> On 10/1/07, *Stefan Steiniger* <
[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> >> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> >> >>> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>>
> >> wrote:
> >> >>>
> >> >>> mhm
> >> >>> from my point of view it is both OJ and JRE.
It would
> >> be probably
> >> >>> better
> >> >>> when the points are stored and processes in
a database.
> >> >>>
> >> >>> stefan
> >> >>>
> >> >>> Arnd Kielhorn schrieb:
> >> >>> > Hello,
> >> >>> >
> >> >>> > yes I know, first of all OJ is a GIS for
vector data.
> >> But the
> >> >>> > functionality is also very good for point
datasets.
> >> But working
> >> >>> with
> >> >>> > great point datasets (more 500,000 points)
bring
> >> problems with the
> >> >>> > memory and the garbage collector works not
satisfied.
> >> Every
> >> >>> command take
> >> >>> > a lot of time or could not carried out.
(CPU P4 2.4
> >> GHz, 1 GB RAM)
> >> >>> > Is it only a thing of the JRE or also from
OJ?
> >> >>> > So, it is useful to know if there is a
chance to
> >> increase the
> >> >>> > performence of OJ in this case and what
could be the
> >> possible
> >> >>> milestones
> >> >>> > to reach this aim?
> >> >>> > I am very eager to read everyones opion.
> >> >>> >
> >> >>> > Kindly regards
> >> >>> > Arnd
> >> >>> >
_______________________________________________
> >> >>> > jump-users mailing list
> >> >>> > [email protected]
<mailto:[email protected]>
> >> <mailto: [email protected]
<mailto:[email protected]>>
> >> >>> <mailto: [email protected]
<mailto:[email protected]>
> >> <mailto:[email protected]
<mailto:[email protected]>>>
> >> >>> >
> >> http://lists.refractions.net/mailman/listinfo/jump-users
> >> <
http://lists.refractions.net/mailman/listinfo/jump-users>
> >> >>> >
> >>
> >> <http://amusingprogrammer.blogspot.com/
<http://amusingprogrammer.blogspot.com/>>
> >>
------------------------------------------------------------------------
> >>
> >>
> >> _______________________________________________
> >> jump-users mailing list
> >> [email protected]
<mailto:[email protected]>
> >> http://lists.refractions.net/mailman/listinfo/jump-users
> >>
> >
> >
> >
> > _______________________________________________
> > jump-users mailing list
> > [email protected]
<mailto:[email protected]>
> > http://lists.refractions.net/mailman/listinfo/jump-users
<http://lists.refractions.net/mailman/listinfo/jump-users>
> >
> >
>
> _______________________________________________
> jump-users mailing list
> [email protected]
<mailto:[email protected]>
> http://lists.refractions.net/mailman/listinfo/jump-users
>
>
_______________________________________________
jump-users mailing list
[email protected]
<mailto:[email protected]>
http://lists.refractions.net/mailman/listinfo/jump-users
--
http://amusingprogrammer.blogspot.com/
<http://amusingprogrammer.blogspot.com/>
--
http://amusingprogrammer.blogspot.com/
------------------------------------------------------------------------
_______________________________________________
jump-users mailing list
[email protected]
http://lists.refractions.net/mailman/listinfo/jump-users
_______________________________________________
jump-users mailing list
[email protected]
http://lists.refractions.net/mailman/listinfo/jump-users
