To follow up:
I downloaded the nightly build (20071009) and retested with Arnd's
XYZ-data.
To select 624,000 points it took 40 seconds, plus an additional 50 to
render the handles.
It took 3 seconds for the menu to appear after a right click on the
layer view. Same for the Edit menu. The Layer context menu responded
immediately.
Committed memory went up to 586 MB.
It would appear that the performance for selecting points and drawing
handles has improved by a factor of 2. The GUI responsiveness problem
has been fixed. It is now possible to select all million points
without an out of memory error (in 750MB).
I will wait for Arnd to do his own tests and see if OpenJump is now
suitable for working with large point datasets.
regards,
Larry Becker
On 10/8/07, * Larry Becker* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
Hi Stefan,
Actually since the selection handle is slightly larger than a
point, I just skipped rendering the point. I don't know if you
noticed, but when something is selected, JUMP renderes it again so
that selected items will be "on top" of the other features. For
linestrings and polygons, this is nice, but for points it is
redundant.
Larry
On 10/8/07, *Stefan Steiniger* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
Hei Larry,
thank you for making the changes to the source :)
And I am glad you did it, as you are more familar with the
code then I am.
Just one question on the change in the
AbstractSelectionRenderer. What
is meant with "obscured" handle? Is that you look if already a
selection-point is drawn nearby and you check if it is
obscured using
the screen resultion?
stefan
Stefan Steiniger wrote:
>
>> All we have to do is save a few class variables (like
numberSelected)
>> and all of those computations and memory use go away. We
update the
>> variables each time the selection really changes. New
methods like
>> countSelectedItems() will replace getSelectedItems() in the
>> EnableChecks.
>
>
> this sounds reasonable - great work Larry
> stefan
>
>>
>> regards,
>> Larry
>>
>> On 10/5/07, *Stefan Steiniger* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>> wrote:
>>
>> well done!
>>
>> i am not sure if this makes sense, but for drawing alone one
>> could use a
>> "skip n points" mechanism or use an indexing scheme ( e.g.
>> quadtrees with
>> skipping the lower(?) levels to show). I guess this
would need some
>> testing to identify the number of points that should be
drawn
>> maximally
>> and that need to be drawn. But for the selection this is
not
>> possible
>> (except the drawing). Not sure if modifying "enable
check" is a
>> solution
>>
>> I guess Martin has some nice thoughts on that ;)
>> stefan
>>
>> btw: @Martin: is you FOSS4G - DTM-TIN paper available?
(I read he
>> used
>> also massive data - millions of heigh points - with postgis)
>>
>> PS: have you seen this presentation on high res image
browsing,
>> which
>> comes quite close to our problem, although a it is bit
different
>> domain:
>> http://www.ted.com/index.php/talks/view/id/129
<http://www.ted.com/index.php/talks/view/id/129>
>> (sorry don't want to hijack this topic but it is related)
>>
>> Larry Becker schrieb:
>> > I'm retesting with Arnd's new XYZ-data which
contained a 62MB
>> shape file
>> > and a 33MB dbf file.
>> >
>> > 1. OJ loaded the file in 15 seconds and finished
drawing it in 55
>> > seconds. 335MB committed memory.
>> >
>> > 2. Finished selecting about 2/3 of the points in 75
seconds
>> Finished
>> > drawing the selection handles in another two
minutes. Up to
>> 452MB of
>> > committed memory. Right clicked on the view and it
took about 2
>> minutes
>> > to respond.
>> >
>> > Clearly the selection of 600000+ objects is causing
havoc in
>> the UI
>> > EnableChecks. Also, the selection handle rendering
optimization
>> that I
>> > made is not effective for points, so it must draw
600000 tiny
>> yellow
>> > rectangles each redraw.
>> >
>> > So is it about memory? I increased OJ's memory
allocation to
>> 750MB and
>> > reloaded. Same load and redraw stats.
>> >
>> > 3. I used the Edit->Select layer items. It took 2
minutes to
>> display
>> > the selected items message (1000000) on the status
bar. While
>> selection
>> > handles were drawing, committed memory cycled between
650 and
>> 700 MB.
>> > Clearly, frequent pauses in drawing were caused by
the JVM
>> garbage
>> > collector. I waited for the handles to finish
drawing so that I
>> could
>> > test the menu response time. Finally, it
finished. I clicked
>> on the
>> > Edit menu. After a few seconds the workbench window
went totally
>> blank
>> > for 3 minutes, then came back. I didn't try it again.
>> >
>> > Conclusion: I concur with Arnd that OpenJump is
unworkable
>> for point
>> > files in excess of 500000. The redraw of a million
points is
>> bad,
>> > redraw of a million selection handles is worse, and
the UI is
>> totally
>> > unresponsive with large numbers of objects selected.
>> >
>> > I believe that we may have reached the design limits
of the UI
>> feedback
>> > mechanisms. In particular EnableCheck class methods make
>> multiple
>> > references to the number of features. The selection
feedback
>> on the
>> > status bar even computes the number of points
(requiring a
>> million
>> > iterations for each access).
>> >
>> > I repeated the experiment with JUMP 1.2 (750MB
memory) which
>> does not
>> > have some of the UI enhancements like the status bar
display of
>> number
>> > of selected items and points. The first thing I
noticed is that
>> JUMP
>> > renders much slower than OpenJump. No way am I going
to wait for
>> it to
>> > finish. I did a drag select around the whole view (since
>> there is no
>> > select layer items on the Edit menu). It is taking a
long
>> time. I'm
>> > not timing any of this since the only thing I'm
interested in
>> is the
>> > menu response time. I found that an easy test to see
if the
>> UI is
>> > responsive is to mouse over a toolbar icon and see if it
>> highlights.
>> > Oops, JUMP just gave an Out of Memory Error. End of that
>> experiment.
>> >
>> > I still believe the main problem is the UI
checks. The UI is
>> totally
>> > responsive until you make a large selection. Then it
gets
>> progressively
>> > slower proportional to the size of the
selection. Fixing this
>> is the
>> > number one priority.
>> >
>> > I loaded the file in SkyJUMP which has metrics for screen
>> redraw. It
>> > rendered the million points in 50 seconds. There is
room for
>> improvement
>> > here, but this is not really the main problem. Any
optimization
>> done to
>> > optimize point drawing would also apply to selection
handles.
>> Fixing
>> > this is the number two priority.
>> >
>> > regards,
>> > Larry
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On 10/4/07, *Larry Becker* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>
>> > <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> <mailto:
[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>>
>> wrote:
>> >
>> > OK, for my PC (P4 3.2Ghz with OJ set to use
256MB) it takes 5
>> > seconds to load your 351k points u1 layer, and 17
seconds to
>> draw.
>> > Pretty bad draw performance compared with other
geometry
>> types, but
>> > not really relevant to the discussion since
waiting for the
>> redraw
>> > is not necessary.
>> >
>> > 1. I managed to copy and paste about 100k of
these to a new
>> layer in
>> > about 10 seconds. Replicate gives similar results.
>> >
>> > 2. Selecting about 1/3 of the points takes about
5 seconds
>> to do,
>> > and about 8 more to render the handles, although as I
>> mentioned
>> > before, you can proceed as soon as the the status bar
>> indicates the
>> > number selected.
>> >
>> > 3. Selecting all points takes about 20 seconds, and
>> another 30 to
>> > render.
>> >
>> > Preliminary results for 350000 points: slow but
definitely
>> workable.
>> >
>> > I loaded 3 copies of the u1 layer giving a total
of over 1
>> million
>> > points. Memory usage is at 200 MB committed. I
suspect it
>> would be
>> > quite a bit more if Stefan's data had any attributes.
>> >
>> > 4. Redraw is now about 45 seconds, but just
ignoring it works
>> for me.
>> >
>> > 5. Selecting about 1/3 failed after 90 seconds and
>> resulted in an
>> > Out of Memory Error.
>> >
>> > After allocating 512MB of memory to OJ, I
restarted. 390000
>> points
>> > selected after 20 seconds. The selection handles
finished
>> rendering
>> > 40 seconds later. 388MB in use.
>> >
>> > 6. Right clicking on the selection takes about 5
seconds to
>> bring up
>> > the menu.
>> >
>> > 7. Replicate took about 20 seconds to replicate
to a new
>> layer.
>> >
>> > These are my current findings. I'll update the
post when
>> I test
>> > with Arnd's data. So far I don't see anything too
>> bad. About what
>> > you might expect really. The rendering code is not
>> optimized for
>> > points, and the selection code is dealing with
its worst case
>> since
>> > the bounding box for points is a point.
>> >
>> > Larry
>> >
>> > On 10/4/07, *Stefan Steiniger* <
[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>> > <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>> wrote:
>> >
>> > oha..
>> > that is interesting. Accidentaly i stored the
points as
>> multipoints.
>> > Now i updated my jump @office and stored the
points as
>> point
>> > (and wrote
>> > the file again to the ftp). The file size is
now already
>> a bit
>> > smaller,
>> > and also the commited memory...
>> >
>> > stefan
>> >
>> > Stefan Steiniger wrote:
>> >
>> > > Good if Arnd provides the data, but here
the link to
>> my data
>> > in case
>> > > it fails:
>> > >
ftp://ftp.geo.uzh.ch/pub/sstein/data/u1_points.zip
<ftp://ftp.geo.uzh.ch/pub/sstein/data/u1_points.zip>
>> > >
>> > > an option to get one million points is to
copy the
>> 350k points and
>> > > then to move them (if that works ;).
>> > > Btw. the lascers can data have no
attributes (may be
>> therefore
>> > smaller
>> > > than Arnd's; the covered area is
relatively small <
>> 200m)
>> > > (shapefile size is 30mb and zip xyz-ascii
was 15mb)
>> > >
>> > > stefan
>> > >
>> > > Arnd Kielhorn wrote:
>> > >
>> > > > Hello Larry,
>> > > >
>> > > > first of all, thank You very much!
>> > > >
>> > > > I work with delaunay triangulation and
building a
>> 3D-Model
>> > with our
>> > > > plugin.
>> > > >
>> > > > If we both could make a direct connection
(via FTP or
>> a free
>> > tool
>> > > > called teamviewer) I could sent You one
file with
>> > laserscanning data
>> > > > with 1 million points (XYZ). It is about
30 MB in the
>> PIROL-CSV
>> > > > format or about 100 MB as shape file.
>> > > >
>> > > > Kindly regards
>> > > > Arnd
>> > > >
>> > > > Larry Becker schrieb:
>> > > >
>> > > >> I'm going to start investigating the point
>> performance
>> > problem as
>> > > >> soon as I get some test data. I have
been trying to
>> locate
>> > a large
>> > > >> point dataset on the web, but no
luck. Perhaps
>> the best
>> > approach
>> > > >> might be to add a new item to the
Generate menu,
>> "Random
>> > Points".
>> > > >>
>> > > >> @Arnd: One thing that people often
forget about JUMP
>> is that
>> > > >> rendering is a background process. You
don't have
>> to wait
>> > for it
>> > > >> to finish to start the next operation.
>> > > >> What kind of terrain maps are you
generating?
>> Are you
>> > generating
>> > > >> contour lines from a grid of
points? You mentioned
>> deleting
>> > > >> selected points being impossibly
slow. I don't see
>> why this
>> > should
>> > > >> be so. If I can duplicate the problem,
it should be
>> an easy
>> > fix.
>> > > >>
>> > > >> regards,
>> > > >> Larry
>> > > >>
>> > > >> On 10/2/07, *Larry Becker* <
[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>
>> > <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>
>> > > >> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>
>> > <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>>> wrote:
>> > > >>
>> > > >> HI Arnd,
>> > > >>
>> > > >> Would it be helpful to work around
the
>> problem by
>> > saving the
>> > > >> points as a shapefile and using a
utility like
>> shp2tile
>> > to break
>> > > >> the point data up into multiple files?
>> > > >>
>> > > >> see:
http://imaptools.com/download-software.html
>> <http://imaptools.com/download-software.html
<http://imaptools.com/download-software.html>>
>> > > >>
>> > > >> regards,
>> > > >> Larry
>> > > >>
>> > > >>
>> > > >> On 10/2/07, *Arnd Kielhorn* <
[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>> > <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>
>> > > >> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
<mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
>> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>>>
>> > wrote:
>> > > >>
>> > > >> Hello Paul,
>> > > >>
>> > > >> thank You for the tip with the
z-value, but
>> how I
>> > can use it
>> > > >> as a
>> > > >> z-value on the point itself?
>> > > >>
>> > > >> The file fomrat is txt, csv
(from PIROL
>> CSV-Format) like
>> > > >> following example
>> > > >> $Rechts Hoch heigh
>> > > >> $grad grad m
>> > > >> $double double double
>> > > >> 3434000.00 5800000.00 93.78
>> > > >> 3434001.00 5800000.00 93.84
>> > > >> ... ... ...
>> > > >>
>> > > >> or the ESRI ascii grid format
asc (reading
>> plugin
>> > from SIGLE).
>> > > >> The most files I got have 1
million points
>> and such
>> > csv files
>> > > >> have about
>> > > >> 30 MB.
>> > > >> In my start file for OJ I got
768 MB,
>> because I have
>> > 1 GB RAM.
>> > > >>
>> > > >> Kindly regards
>> > > >> Arnd
>> > > >>
>> > > >>
>> > > >> Paul Austin schrieb:
>> > > >> > Arnd,
>> > > >> >
>> > > >> > What file format are you using?
>> > > >> >
>> > > >> > As JUMP loads the whole file
into memory I
>> would
>> > recommend
>> > > >> setting the
>> > > >> > heap size on the JAVA VM to be
-Xmx512M if
>> you are
>> > dealing
>> > > >> with large
>> > > >> > datasets.
>> > > >> >
>> > > >> > You mentioned that the height
is an
>> attribute on
>> > the feature,
>> > > >> if you use
>> > > >> > the height as a z-value on the
point
>> itself there
>> > will be
>> > > >> some memory
>> > > >> > savings.
>> > > >> >
>> > > >> > Paul
>> > > >> >
>> > > >> > Arnd Kielhorn wrote:
>> > > >> >
>> > > >> >> Hello Larry,
>> > > >> >>
>> > > >> >> as in the further discussion
just named
>> the main
>> > problema
>> > > >> are (for
>> > > >> >> example I just have to work
with point
>> layers
>> > with only a
>> > > >> heigh
>> > > >> >> attribute but with 1 million
points; but
>> also
>> > e.g. 300,000
>> > > >> points are
>> > > >> >> enough):
>> > > >> >> - (re)drawing of the points
>> > > >> >> - strongly restricted or
impossible
>> operation
>> > (e.g. deleting
>> > > >> some
>> > > >> >> points/vertices after marking
them; e.g.
>> copying
>> > hundreds or
>> > > >> a few
>> > > >> >> thousand in a new layer to
have layer
>> with lesser
>> > points)
>> > > >> >> Very often OJ hangs on and I
have to
>> restart it.
>> > > >> >> I use such point layers for
creating a
>> digital
>> > terrain maps.
>> > > >> >>
>> > > >> >> Kindly regards
>> > > >> >> Arnd
>> > > >> >>
>> > > >> >> Larry Becker schrieb:
>> > > >> >>
>> > > >> >>> Hi Arnd,
>> > > >> >>>
>> > > >> >>> I haven't worked with
large point
>> datasets
>> > much. What
>> > > >> it is
>> > > >> >>> exactly that is slow? Is it
redraw
>> time? Or
>> > perhaps the
>> > > >> particular
>> > > >> >>> operation that you are doing?
>> > > >> >>>
>> > > >> >>> I know that point layers
do not
>> benefit from
>> > the recent
>> > > >> decimation
>> > > >> >>> optimization, so they take
longer to
>> draw than
>> > equivalent
>> > > >> polygons
>> > > >> >>> and polyline layers. I
think Michaël
>> tried a point
>> > > >> decimation
>> > > >> >>> optimization, but wasn't
satisfied with
>> the results.
>> > > >> >>>
>> > > >> >>> regards,
>> > > >> >>> Larry Becker
>> > > >> >>>
>> > > >> >>> On 10/1/07, *Stefan Steiniger* <
>> > [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
<mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
<mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>
>> > > >> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
<mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>>
>> > > >> >>> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>> > <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>
>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
<mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>> > <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>>>>
>> > > >> wrote:
>> > > >> >>>
>> > > >> >>> mhm
>> > > >> >>> from my point of view it
is both OJ
>> and JRE.
>> > It would
>> > > >> be probably
>> > > >> >>> better
>> > > >> >>> when the points are
stored and
>> processes in
>> > a database.
>> > > >> >>>
>> > > >> >>> stefan
>> > > >> >>>
>> > > >> >>> Arnd Kielhorn schrieb:
>> > > >> >>> > Hello,
>> > > >> >>> >
>> > > >> >>> > yes I know, first of
all OJ is a
>> GIS for
>> > vector data.
>> > > >> But the
>> > > >> >>> > functionality is also
very good
>> for point
>> > datasets.
>> > > >> But working
>> > > >> >>> with
>> > > >> >>> > great point datasets
(more 500,000
>> points)
>> > bring
>> > > >> problems with the
>> > > >> >>> > memory and the garbage
collector
>> works not
>> > satisfied.
>> > > >> Every
>> > > >> >>> command take
>> > > >> >>> > a lot of time or could
not carried
>> out.
>> > (CPU P4 2.4
>> > > >> GHz, 1 GB RAM)
>> > > >> >>> > Is it only a thing of
the JRE or
>> also from
>> > OJ?
>> > > >> >>> > So, it is useful to
know if there
>> is a
>> > chance to
>> > > >> increase the
>> > > >> >>> > performence of OJ in
this case and
>> what
>> > could be the
>> > > >> possible
>> > > >> >>> milestones
>> > > >> >>> > to reach this aim?
>> > > >> >>> > I am very eager to
read everyones
>> opion.
>> > > >> >>> >
>> > > >> >>> > Kindly regards
>> > > >> >>> > Arnd
>> > > >> >>> >
>> >
_______________________________________________
jump-users mailing list
[email protected]
<mailto:[email protected]>
http://lists.refractions.net/mailman/listinfo/jump-users
_______________________________________________
jump-users mailing list
[email protected]
http://lists.refractions.net/mailman/listinfo/jump-users
