Larry - You are the performance guru. Need I say more?

The Sunburned Surveyor

On 10/10/07, Larry Becker <[EMAIL PROTECTED]> wrote:
> Or if Arnd's use case here is deleting selections, a shift key
> modifier for the Delete key would disable undo (as it does in
> Windows).
>
> I'm glad to see Arnd found some gains in performance even though we
> aren't there yet. :-)
>
> Larry
>
> On 10/10/07, Stefan Steiniger <[EMAIL PROTECTED]> wrote:
> >
> > > delete 624000 and rebuild the layer view   after 01:30 "Java heap space"
> > >
> > mhm.. maybe it is the "undo" function that consumes further time and space.
> > So one would need to introduce an option for disabling undo.
> >
> > just a thought
> >
> > stefan
> >
> > > After that I closed OJ and set now 1536 (1024 + 512) in the
> > > openjump:bat and restart OJ to test the last task again
> > >
> > > task    used time[mm:ss]   used memory after task[MB]
> > > select 624000   01:25   402
> > > delete 624000 and rebuild the layer view   after 45:00 I stopped the
> > > task withour any result
> > >
> > > Conclusion from my point of view:
> > > The performence after Larrys modification increased clearly!!
> > > Congratulation and many thanks for that!! But to do more than viewing
> > > or make a colour theming (also OK is calculating new attributes, under
> > > 02:00) with such big datasets such as interpolating the modification
> > > is not sufficient at the moment. I have no experiences whether and how
> > > fast other GIS software can do all these tasks. It will be very great
> > > if there could be further potential for higher performence in the
> > > future developments.
> > >
> > > Thank You again Larry!!
> > >
> > > Kindly regards
> > > Arnd
> > >
> > >
> > >
> > > Larry Becker schrieb:
> > >
> > >> To follow up:
> > >>
> > >> I downloaded the nightly build (20071009) and retested with Arnd's
> > >> XYZ-data.
> > >>
> > >> To select 624,000 points it took 40 seconds, plus an additional 50 to
> > >> render the handles.
> > >>
> > >> It took 3 seconds for the menu to appear after a right click on the
> > >> layer view.  Same for the Edit menu.  The Layer context menu
> > >> responded immediately.
> > >>
> > >> Committed memory went up to 586 MB.
> > >>
> > >> It would appear that the performance for selecting points and drawing
> > >> handles has improved by a factor of 2.  The GUI responsiveness
> > >> problem has been fixed.  It is now possible to select all million
> > >> points without an out of memory error (in 750MB).
> > >>
> > >> I will wait for Arnd to do his own tests and see if OpenJump is now
> > >> suitable for working with large point datasets.
> > >>
> > >> regards,
> > >> Larry Becker
> > >>
> > >> On 10/8/07, * Larry Becker* <[EMAIL PROTECTED]
> > >> <mailto:[EMAIL PROTECTED]>> wrote:
> > >>
> > >>     Hi Stefan,
> > >>
> > >>       Actually since the selection handle is slightly larger than a
> > >>     point, I just skipped rendering the point.  I don't know if you
> > >>     noticed, but when something is selected, JUMP renderes it again so
> > >>     that selected items will be "on top" of the other features.  For
> > >>     linestrings and polygons, this is nice, but for points it is
> > >> redundant.
> > >>
> > >>     Larry
> > >>
> > >>
> > >>     On 10/8/07, *Stefan Steiniger* < [EMAIL PROTECTED]
> > >>     <mailto:[EMAIL PROTECTED]>> wrote:
> > >>
> > >>         Hei Larry,
> > >>
> > >>         thank you for making the changes to the source :)
> > >>         And I am glad you did it, as you are more familar with the
> > >>         code then I am.
> > >>
> > >>         Just one question on the change in the
> > >>         AbstractSelectionRenderer. What
> > >>         is meant with "obscured" handle? Is that you look if already a
> > >>         selection-point is drawn nearby and you check if it is
> > >>         obscured using
> > >>         the screen resultion?
> > >>
> > >>         stefan
> > >>
> > >>         Stefan Steiniger wrote:
> > >>
> > >>         >
> > >>         >> All we have to do is save a few class variables (like
> > >>         numberSelected)
> > >>         >> and all of those computations and memory use go away.  We
> > >>         update the
> > >>         >> variables each time the selection really changes.  New
> > >>         methods like
> > >>         >> countSelectedItems() will replace getSelectedItems() in the
> > >>         >> EnableChecks.
> > >>         >
> > >>         >
> > >>         > this sounds reasonable - great work Larry
> > >>         > stefan
> > >>         >
> > >>         >>
> > >>         >> regards,
> > >>         >> Larry
> > >>         >>
> > >>         >> On 10/5/07, *Stefan Steiniger* <[EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>
> > >>         >> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>
> > >> wrote:
> > >>         >>
> > >>         >>     well done!
> > >>         >>
> > >>         >>     i am not sure if this makes sense, but for drawing
> > >> alone one
> > >>         >> could use a
> > >>         >>     "skip n points" mechanism or use an indexing scheme (
> > >> e.g.
> > >>         >> quadtrees with
> > >>         >>     skipping the lower(?) levels to show). I guess this
> > >>         would need some
> > >>         >>     testing to identify the number of points that should be
> > >>         drawn
> > >>         >> maximally
> > >>         >>     and that need to be drawn. But for the selection this is
> > >>         not
> > >>         >> possible
> > >>         >>     (except the drawing). Not sure if modifying "enable
> > >>         check" is a
> > >>         >>     solution
> > >>         >>
> > >>         >>     I guess Martin has some nice thoughts on that ;)
> > >>         >>     stefan
> > >>         >>
> > >>         >>     btw: @Martin: is you FOSS4G - DTM-TIN paper available?
> > >>         (I read he
> > >>         >> used
> > >>         >>     also massive data - millions of heigh points - with
> > >> postgis)
> > >>         >>
> > >>         >>     PS: have you seen this presentation on high res image
> > >>         browsing,
> > >>         >> which
> > >>         >>     comes quite close to our problem, although a it is bit
> > >>         different
> > >>         >> domain:
> > >>         >>             http://www.ted.com/index.php/talks/view/id/129
> > >>         <http://www.ted.com/index.php/talks/view/id/129>
> > >>         >>     (sorry don't want to hijack this topic but it is related)
> > >>         >>
> > >>         >>     Larry Becker schrieb:
> > >>         >>      > I'm retesting with Arnd's new XYZ-data which
> > >>         contained a 62MB
> > >>         >>     shape file
> > >>         >>      > and a 33MB dbf file.
> > >>         >>      >
> > >>         >>      > 1. OJ loaded the file in 15 seconds and finished
> > >>         drawing it in 55
> > >>         >>      > seconds.  335MB committed memory.
> > >>         >>      >
> > >>         >>      > 2.  Finished selecting about 2/3 of the points in 75
> > >>         seconds
> > >>         >> Finished
> > >>         >>      > drawing the selection handles in another two
> > >>         minutes.  Up to
> > >>         >> 452MB of
> > >>         >>      > committed memory.  Right clicked on the view and it
> > >>         took about 2
> > >>         >>     minutes
> > >>         >>      > to respond.
> > >>         >>      >
> > >>         >>      > Clearly the selection of 600000+ objects is causing
> > >>         havoc in
> > >>         >> the UI
> > >>         >>      > EnableChecks.  Also, the selection handle rendering
> > >>         optimization
> > >>         >>     that I
> > >>         >>      > made is not effective for points, so it must draw
> > >>         600000 tiny
> > >>         >> yellow
> > >>         >>      > rectangles each redraw.
> > >>         >>      >
> > >>         >>      > So is it about memory?  I increased OJ's memory
> > >>         allocation to
> > >>         >>     750MB and
> > >>         >>      > reloaded.  Same load and redraw stats.
> > >>         >>      >
> > >>         >>      > 3.  I used the Edit->Select layer items.  It took 2
> > >>         minutes to
> > >>         >>     display
> > >>         >>      > the selected items message (1000000) on the status
> > >>         bar.  While
> > >>         >>     selection
> > >>         >>      > handles were drawing, committed memory cycled between
> > >>         650 and
> > >>         >> 700 MB.
> > >>         >>      > Clearly, frequent pauses in drawing were caused by
> > >>         the JVM
> > >>         >> garbage
> > >>         >>      > collector.  I waited for the handles to finish
> > >>         drawing so that I
> > >>         >>     could
> > >>         >>      > test the menu response time.  Finally, it
> > >>         finished.  I clicked
> > >>         >> on the
> > >>         >>      > Edit menu.  After a few seconds the workbench window
> > >>         went totally
> > >>         >>     blank
> > >>         >>      > for 3 minutes, then came back.  I didn't try it again.
> > >>         >>      >
> > >>         >>      > Conclusion:  I concur with Arnd that OpenJump is
> > >>         unworkable
> > >>         >> for point
> > >>         >>      > files in excess of 500000.  The redraw of a million
> > >>         points is
> > >>         >> bad,
> > >>         >>      > redraw of a million selection handles is worse, and
> > >>         the UI is
> > >>         >> totally
> > >>         >>      > unresponsive with large numbers of objects selected.
> > >>         >>      >
> > >>         >>      > I believe that we may have reached the design limits
> > >>         of the UI
> > >>         >>     feedback
> > >>         >>      > mechanisms.  In particular EnableCheck class
> > >> methods make
> > >>         >> multiple
> > >>         >>      > references to the number of features.  The selection
> > >>         feedback
> > >>         >> on the
> > >>         >>      > status bar even computes the number of points
> > >>         (requiring a
> > >>         >> million
> > >>         >>      > iterations for each access).
> > >>         >>      >
> > >>         >>      > I repeated the experiment with JUMP 1.2 (750MB
> > >>         memory) which
> > >>         >> does not
> > >>         >>      > have some of the UI enhancements like the status bar
> > >>         display of
> > >>         >>     number
> > >>         >>      > of selected items and points.  The first thing I
> > >>         noticed is that
> > >>         >>     JUMP
> > >>         >>      > renders much slower than OpenJump.  No way am I going
> > >>         to wait for
> > >>         >>     it to
> > >>         >>      > finish.  I did a drag select around the whole view
> > >> (since
> > >>         >> there is no
> > >>         >>      > select layer items on the Edit menu).  It is taking a
> > >>         long
> > >>         >>     time.  I'm
> > >>         >>      > not timing any of this since the only thing I'm
> > >>         interested in
> > >>         >> is the
> > >>         >>      > menu response time.  I found that an easy test to see
> > >>         if the
> > >>         >> UI is
> > >>         >>      > responsive is to mouse over a toolbar icon and see
> > >> if it
> > >>         >> highlights.
> > >>         >>      > Oops, JUMP just gave an Out of Memory Error.  End
> > >> of that
> > >>         >> experiment.
> > >>         >>      >
> > >>         >>      > I still believe the main problem is the UI
> > >>         checks.  The UI is
> > >>         >> totally
> > >>         >>      > responsive until you make a large selection.  Then it
> > >>         gets
> > >>         >>     progressively
> > >>         >>      > slower proportional to the size of the
> > >>         selection.  Fixing this
> > >>         >> is the
> > >>         >>      > number one priority.
> > >>         >>      >
> > >>         >>      > I loaded the file in SkyJUMP which has metrics for
> > >> screen
> > >>         >> redraw.  It
> > >>         >>      > rendered the million points in 50 seconds. There is
> > >>         room for
> > >>         >>     improvement
> > >>         >>      > here, but this is not really the main problem.  Any
> > >>         optimization
> > >>         >>     done to
> > >>         >>      > optimize point drawing would also apply to selection
> > >>         handles.
> > >>         >> Fixing
> > >>         >>      > this is the number two priority.
> > >>         >>      >
> > >>         >>      > regards,
> > >>         >>      > Larry
> > >>         >>      >
> > >>         >>      >
> > >>         >>      >
> > >>         >>      >
> > >>         >>      >
> > >>         >>      >
> > >>         >>      >
> > >>         >>      > On 10/4/07, *Larry Becker* <[EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>>
> > >>         >>      > <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]> <mailto:
> > >>         [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>>
> > >>         >>     wrote:
> > >>         >>      >
> > >>         >>      >     OK, for my PC (P4 3.2Ghz with OJ set to use
> > >>         256MB) it takes 5
> > >>         >>      >     seconds to load your 351k points u1 layer, and 17
> > >>         seconds to
> > >>         >>     draw.
> > >>         >>      >     Pretty bad draw performance compared with other
> > >>         geometry
> > >>         >>     types, but
> > >>         >>      >     not really relevant to the discussion since
> > >>         waiting for the
> > >>         >>     redraw
> > >>         >>      >     is not necessary.
> > >>         >>      >
> > >>         >>      >     1. I managed to copy and paste about 100k of
> > >>         these to a new
> > >>         >>     layer in
> > >>         >>      >     about 10 seconds.  Replicate gives similar
> > >> results.
> > >>         >>      >
> > >>         >>      >     2. Selecting about 1/3 of the points takes about
> > >>         5 seconds
> > >>         >> to do,
> > >>         >>      >     and about 8 more to render the handles,
> > >> although as I
> > >>         >> mentioned
> > >>         >>      >     before, you can proceed as soon as the the
> > >> status bar
> > >>         >>     indicates the
> > >>         >>      >     number selected.
> > >>         >>      >
> > >>         >>      >     3. Selecting all points takes about 20 seconds,
> > >> and
> > >>         >> another 30 to
> > >>         >>      >     render.
> > >>         >>      >
> > >>         >>      >     Preliminary results for 350000 points: slow but
> > >>         definitely
> > >>         >>     workable.
> > >>         >>      >
> > >>         >>      >     I loaded 3 copies of the u1 layer giving a total
> > >>         of over 1
> > >>         >>     million
> > >>         >>      >     points.  Memory usage is at 200 MB committed.  I
> > >>         suspect it
> > >>         >>     would be
> > >>         >>      >     quite a bit more if Stefan's data had any
> > >> attributes.
> > >>         >>      >
> > >>         >>      >     4. Redraw is now about 45 seconds, but just
> > >>         ignoring it works
> > >>         >>     for me.
> > >>         >>      >
> > >>         >>      >     5. Selecting about 1/3 failed after 90 seconds and
> > >>         >> resulted in an
> > >>         >>      >     Out of Memory Error.
> > >>         >>      >
> > >>         >>      >     After allocating 512MB of memory to OJ, I
> > >>         restarted.  390000
> > >>         >>     points
> > >>         >>      >     selected after 20 seconds.  The selection handles
> > >>         finished
> > >>         >>     rendering
> > >>         >>      >     40 seconds later. 388MB in use.
> > >>         >>      >
> > >>         >>      >     6. Right clicking on the selection takes about 5
> > >>         seconds to
> > >>         >>     bring up
> > >>         >>      >     the menu.
> > >>         >>      >
> > >>         >>      >     7.  Replicate took about 20 seconds to replicate
> > >>         to a new
> > >>         >> layer.
> > >>         >>      >
> > >>         >>      >     These are my current findings.  I'll update the
> > >>         post when
> > >>         >> I test
> > >>         >>      >     with Arnd's data.  So far I don't see anything too
> > >>         >>     bad.  About what
> > >>         >>      >     you might expect really.  The rendering code is
> > >> not
> > >>         >> optimized for
> > >>         >>      >     points, and the selection code is dealing with
> > >>         its worst case
> > >>         >>     since
> > >>         >>      >     the bounding box for points is a point.
> > >>         >>      >
> > >>         >>      >     Larry
> > >>         >>      >
> > >>         >>      >     On 10/4/07, *Stefan Steiniger* <
> > >>         [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >>         >>      >     <mailto:[EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]> <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>>>> wrote:
> > >>         >>      >
> > >>         >>      >         oha..
> > >>         >>      >         that is interesting. Accidentaly i stored the
> > >>         points as
> > >>         >>     multipoints.
> > >>         >>      >         Now i updated my jump @office and stored the
> > >>         points as
> > >>         >> point
> > >>         >>      >         (and wrote
> > >>         >>      >         the file again to the ftp). The file size is
> > >>         now already
> > >>         >>     a bit
> > >>         >>      >         smaller,
> > >>         >>      >         and also the commited memory...
> > >>         >>      >
> > >>         >>      >         stefan
> > >>         >>      >
> > >>         >>      >         Stefan Steiniger wrote:
> > >>         >>      >
> > >>         >>      >         >  Good if Arnd provides the data, but here
> > >>         the link to
> > >>         >>     my data
> > >>         >>      >         in case
> > >>         >>      >         >  it fails:
> > >>         >>      >         >
> > >> ftp://ftp.geo.uzh.ch/pub/sstein/data/u1_points.zip
> > >>         <ftp://ftp.geo.uzh.ch/pub/sstein/data/u1_points.zip>
> > >>         >>      >         >
> > >>         >>      >         >  an option to get one million points is to
> > >>         copy the
> > >>         >>     350k points and
> > >>         >>      >         >  then to move them (if that works ;).
> > >>         >>      >         >  Btw. the lascers can data have no
> > >>         attributes (may be
> > >>         >>     therefore
> > >>         >>      >         smaller
> > >>         >>      >         >  than Arnd's; the covered area is
> > >>         relatively small <
> > >>         >> 200m)
> > >>         >>      >         >  (shapefile size is 30mb and zip xyz-ascii
> > >>         was 15mb)
> > >>         >>      >         >
> > >>         >>      >         >  stefan
> > >>         >>      >         >
> > >>         >>      >         >  Arnd Kielhorn wrote:
> > >>         >>      >         >
> > >>         >>      >         > > Hello Larry,
> > >>         >>      >         > >
> > >>         >>      >         > > first of all, thank You very much!
> > >>         >>      >         > >
> > >>         >>      >         > > I work with delaunay triangulation and
> > >>         building a
> > >>         >>     3D-Model
> > >>         >>      >         with our
> > >>         >>      >         > > plugin.
> > >>         >>      >         > >
> > >>         >>      >         > > If we both could make a direct connection
> > >>         (via FTP or
> > >>         >>     a free
> > >>         >>      >         tool
> > >>         >>      >         > > called teamviewer) I could sent You one
> > >>         file with
> > >>         >>      >         laserscanning data
> > >>         >>      >         > > with 1 million points (XYZ). It is about
> > >>         30 MB in the
> > >>         >>     PIROL-CSV
> > >>         >>      >         > > format or about 100 MB as shape file.
> > >>         >>      >         > >
> > >>         >>      >         > > Kindly regards
> > >>         >>      >         > > Arnd
> > >>         >>      >         > >
> > >>         >>      >         > > Larry Becker schrieb:
> > >>         >>      >         > >
> > >>         >>      >         > >> I'm going to start investigating the
> > >> point
> > >>         >> performance
> > >>         >>      >         problem as
> > >>         >>      >         > >> soon as I get some test data.  I have
> > >>         been trying to
> > >>         >>     locate
> > >>         >>      >         a large
> > >>         >>      >         > >> point dataset on the web, but no
> > >>         luck.  Perhaps
> > >>         >> the best
> > >>         >>      >         approach
> > >>         >>      >         > >> might be to add a new item to the
> > >>         Generate menu,
> > >>         >>     "Random
> > >>         >>      >         Points".
> > >>         >>      >         > >>
> > >>         >>      >         > >> @Arnd: One thing that people often
> > >>         forget about JUMP
> > >>         >>     is that
> > >>         >>      >         > >> rendering is a background  process.  You
> > >>         don't have
> > >>         >>     to wait
> > >>         >>      >         for it
> > >>         >>      >         > >> to finish to start the next operation.
> > >>         >>      >         > >> What kind of terrain maps are you
> > >>         generating?
> > >>         >> Are you
> > >>         >>      >         generating
> > >>         >>      >         > >> contour lines from a grid of
> > >>         points?  You mentioned
> > >>         >>     deleting
> > >>         >>      >         > >> selected points being impossibly
> > >>         slow.  I don't see
> > >>         >>     why this
> > >>         >>      >         should
> > >>         >>      >         > >> be so.  If I can duplicate the problem,
> > >>         it should be
> > >>         >>     an easy
> > >>         >>      >         fix.
> > >>         >>      >         > >>
> > >>         >>      >         > >> regards,
> > >>         >>      >         > >> Larry
> > >>         >>      >         > >>
> > >>         >>      >         > >> On 10/2/07, *Larry Becker* <
> > >>         [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>>
> > >>         >>      >         <mailto:[EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>>>
> > >>         >>      >         > >> <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>>
> > >>         >>      >         <mailto:[EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>>>>> wrote:
> > >>         >>      >         > >>
> > >>         >>      >         > >>     HI Arnd,
> > >>         >>      >         > >>
> > >>         >>      >         > >>       Would it be helpful to work around
> > >>         the
> > >>         >> problem by
> > >>         >>      >         saving the
> > >>         >>      >         > >>     points as a shapefile and using a
> > >>         utility like
> > >>         >>     shp2tile
> > >>         >>      >         to break
> > >>         >>      >         > >>     the point data up into multiple
> > >> files?
> > >>         >>      >         > >>
> > >>         >>      >         > >>     see:
> > >>         http://imaptools.com/download-software.html
> > >>         >>     <http://imaptools.com/download-software.html
> > >>         <http://imaptools.com/download-software.html>>
> > >>         >>      >         > >>
> > >>         >>      >         > >>     regards,
> > >>         >>      >         > >>     Larry
> > >>         >>      >         > >>
> > >>         >>      >         > >>
> > >>         >>      >         > >>     On 10/2/07, *Arnd Kielhorn* <
> > >>         [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >>         >>      >         <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]> <mailto:[EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>>>
> > >>         >>      >         > >>     <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >>         <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>>>
> > >>         >>      >         wrote:
> > >>         >>      >         > >>
> > >>         >>      >         > >>         Hello Paul,
> > >>         >>      >         > >>
> > >>         >>      >         > >>         thank You for the tip with the
> > >>         z-value, but
> > >>         >>     how I
> > >>         >>      >         can use it
> > >>         >>      >         > >> as a
> > >>         >>      >         > >>         z-value on the point itself?
> > >>         >>      >         > >>
> > >>         >>      >         > >>         The file fomrat is txt, csv
> > >>         (from PIROL
> > >>         >>     CSV-Format) like
> > >>         >>      >         > >>         following example
> > >>         >>      >         > >>         $Rechts    Hoch    heigh
> > >>         >>      >         > >>         $grad    grad    m
> > >>         >>      >         > >>         $double    double    double
> > >>         >>      >         > >>         3434000.00    5800000.00
> > >> 93.78
> > >>         >>      >         > >>         3434001.00    5800000.00    93.84
> > >>         >>      >         > >>         ...   ...   ...
> > >>         >>      >         > >>
> > >>         >>      >         > >>         or the ESRI ascii grid format
> > >>         asc (reading
> > >>         >>     plugin
> > >>         >>      >         from SIGLE).
> > >>         >>      >         > >>         The most files I got have 1
> > >>         million points
> > >>         >>     and such
> > >>         >>      >         csv files
> > >>         >>      >         > >>         have about
> > >>         >>      >         > >>         30 MB.
> > >>         >>      >         > >>         In my start file for OJ I got
> > >>         768 MB,
> > >>         >>     because I have
> > >>         >>      >         1 GB RAM.
> > >>         >>      >         > >>
> > >>         >>      >         > >>         Kindly regards
> > >>         >>      >         > >>         Arnd
> > >>         >>      >         > >>
> > >>         >>      >         > >>
> > >>         >>      >         > >>         Paul Austin schrieb:
> > >>         >>      >         > >>         > Arnd,
> > >>         >>      >         > >>         >
> > >>         >>      >         > >>         > What file format are you using?
> > >>         >>      >         > >>         >
> > >>         >>      >         > >>         > As JUMP loads the whole file
> > >>         into memory I
> > >>         >>     would
> > >>         >>      >         recommend
> > >>         >>      >         > >>         setting the
> > >>         >>      >         > >>         > heap size on the JAVA VM to be
> > >>         -Xmx512M if
> > >>         >>     you are
> > >>         >>      >         dealing
> > >>         >>      >         > >>         with large
> > >>         >>      >         > >>         > datasets.
> > >>         >>      >         > >>         >
> > >>         >>      >         > >>         > You mentioned that the height
> > >>         is an
> > >>         >>     attribute on
> > >>         >>      >         the feature,
> > >>         >>      >         > >>         if you use
> > >>         >>      >         > >>         > the height as a z-value on the
> > >>         point
> > >>         >>     itself there
> > >>         >>      >         will be
> > >>         >>      >         > >>         some memory
> > >>         >>      >         > >>         > savings.
> > >>         >>      >         > >>         >
> > >>         >>      >         > >>         > Paul
> > >>         >>      >         > >>         >
> > >>         >>      >         > >>         > Arnd Kielhorn wrote:
> > >>         >>      >         > >>         >
> > >>         >>      >         > >>         >> Hello Larry,
> > >>         >>      >         > >>         >>
> > >>         >>      >         > >>         >> as in the further discussion
> > >>         just named
> > >>         >>     the main
> > >>         >>      >         problema
> > >>         >>      >         > >>         are (for
> > >>         >>      >         > >>         >> example I just have to work
> > >>         with point
> > >>         >> layers
> > >>         >>      >         with only a
> > >>         >>      >         > >> heigh
> > >>         >>      >         > >>         >> attribute but with 1 million
> > >>         points; but
> > >>         >>     also
> > >>         >>      >         e.g. 300,000
> > >>         >>      >         > >>         points are
> > >>         >>      >         > >>         >> enough):
> > >>         >>      >         > >>         >> - (re)drawing of the points
> > >>         >>      >         > >>         >> - strongly restricted or
> > >>         impossible
> > >>         >>     operation
> > >>         >>      >         (e.g. deleting
> > >>         >>      >         > >>         some
> > >>         >>      >         > >>         >> points/vertices after marking
> > >>         them; e.g.
> > >>         >>     copying
> > >>         >>      >         hundreds or
> > >>         >>      >         > >>         a few
> > >>         >>      >         > >>         >> thousand in a new layer to
> > >>         have layer
> > >>         >>     with lesser
> > >>         >>      >         points)
> > >>         >>      >         > >>         >> Very often OJ hangs on and I
> > >>         have to
> > >>         >>     restart it.
> > >>         >>      >         > >>         >> I use such point layers for
> > >>         creating a
> > >>         >>     digital
> > >>         >>      >         terrain maps.
> > >>         >>      >         > >>         >>
> > >>         >>      >         > >>         >> Kindly regards
> > >>         >>      >         > >>         >> Arnd
> > >>         >>      >         > >>         >>
> > >>         >>      >         > >>         >> Larry Becker schrieb:
> > >>         >>      >         > >>         >>
> > >>         >>      >         > >>         >>> Hi Arnd,
> > >>         >>      >         > >>         >>>
> > >>         >>      >         > >>         >>>   I haven't worked with
> > >>         large point
> > >>         >> datasets
> > >>         >>      >         much.  What
> > >>         >>      >         > >> it is
> > >>         >>      >         > >>         >>> exactly that is slow?  Is it
> > >>         redraw
> > >>         >>     time?  Or
> > >>         >>      >         perhaps the
> > >>         >>      >         > >>         particular
> > >>         >>      >         > >>         >>> operation that you are doing?
> > >>         >>      >         > >>         >>>
> > >>         >>      >         > >>         >>>   I know that point layers
> > >>         do not
> > >>         >>     benefit from
> > >>         >>      >         the recent
> > >>         >>      >         > >>         decimation
> > >>         >>      >         > >>         >>> optimization, so they take
> > >>         longer to
> > >>         >>     draw than
> > >>         >>      >         equivalent
> > >>         >>      >         > >>         polygons
> > >>         >>      >         > >>         >>> and polyline layers.  I
> > >>         think Michaƫl
> > >>         >>     tried a point
> > >>         >>      >         > >> decimation
> > >>         >>      >         > >>         >>> optimization, but wasn't
> > >>         satisfied with
> > >>         >>     the results.
> > >>         >>      >         > >>         >>>
> > >>         >>      >         > >>         >>> regards,
> > >>         >>      >         > >>         >>> Larry Becker
> > >>         >>      >         > >>         >>>
> > >>         >>      >         > >>         >>> On 10/1/07, *Stefan
> > >> Steiniger* <
> > >>         >>      >         [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > >>         <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >>         >>     <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > >>         <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>
> > >>         >>      >         > >>         <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >>         <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>>
> > >>         >>      >         > >>         >>> <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>
> > >>         >>     <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >>         >>      >         <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]> <mailto:[EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>>>
> > >>         >>     <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> > >>         <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >>         >>      >         <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]> <mailto: [EMAIL PROTECTED]
> > >>         <mailto:[EMAIL PROTECTED]>>>>>>
> > >>         >>      >         > >> wrote:
> > >>         >>      >         > >>         >>>
> > >>         >>      >         > >>         >>>     mhm
> > >>         >>      >         > >>         >>>     from my point of view it
> > >>         is both OJ
> > >>         >>     and JRE.
> > >>         >>      >         It would
> > >>         >>      >         > >>         be probably
> > >>         >>      >         > >>         >>>     better
> > >>         >>      >         > >>         >>>     when the points are
> > >>         stored and
> > >>         >>     processes in
> > >>         >>      >         a database.
> > >>         >>      >         > >>         >>>
> > >>         >>      >         > >>         >>>     stefan
> > >>         >>      >         > >>         >>>
> > >>         >>      >         > >>         >>>     Arnd Kielhorn schrieb:
> > >>         >>      >         > >>         >>>     > Hello,
> > >>         >>      >         > >>         >>>     >
> > >>         >>      >         > >>         >>>     > yes I know, first of
> > >>         all OJ is a
> > >>         >>     GIS for
> > >>         >>      >         vector data.
> > >>         >>      >         > >>         But the
> > >>         >>      >         > >>         >>>     > functionality is also
> > >>         very good
> > >>         >>     for point
> > >>         >>      >         datasets.
> > >>         >>      >         > >>         But working
> > >>         >>      >         > >>         >>> with
> > >>         >>      >         > >>         >>>     > great point datasets
> > >>         (more 500,000
> > >>         >>     points)
> > >>         >>      >         bring
> > >>         >>      >         > >>         problems with the
> > >>         >>      >         > >>         >>>     > memory and the garbage
> > >>         collector
> > >>         >>     works not
> > >>         >>      >         satisfied.
> > >>         >>      >         > >>         Every
> > >>         >>      >         > >>         >>>     command take
> > >>         >>      >         > >>         >>>     > a lot of time or could
> > >>         not carried
> > >>         >>     out.
> > >>         >>      >         (CPU P4 2.4
> > >>         >>      >         > >>         GHz, 1 GB RAM)
> > >>         >>      >         > >>         >>>     > Is it only a thing of
> > >>         the JRE or
> > >>         >>     also from
> > >>         >>      >         OJ?
> > >>         >>      >         > >>         >>>     > So, it is useful to
> > >>         know if there
> > >>         >>     is a
> > >>         >>      >         chance to
> > >>         >>      >         > >>         increase the
> > >>         >>      >         > >>         >>>     > performence of OJ in
> > >>         this case and
> > >>         >>     what
> > >>         >>      >         could be the
> > >>         >>      >         > >>         possible
> > >>         >>      >         > >>         >>>     milestones
> > >>         >>      >         > >>         >>>     > to reach this aim?
> > >>         >>      >         > >>         >>>     > I am very eager to
> > >>         read everyones
> > >>         >>     opion.
> > >>         >>      >         > >>         >>>     >
> > >>         >>      >         > >>         >>>     > Kindly regards
> > >>         >>      >         > >>         >>>     > Arnd
> > >>         >>      >         > >>         >>>     >
> > >>         >>      >
> > >>         _______________________________________________
> > >>         jump-users mailing list
> > >>         [email protected]
> > >>         <mailto:[email protected]>
> > >>         http://lists.refractions.net/mailman/listinfo/jump-users
> > >>
> > >> _______________________________________________
> > >> jump-users mailing list
> > >> [email protected]
> > >> http://lists.refractions.net/mailman/listinfo/jump-users
> > >>
> > >
> > >
> > > _______________________________________________
> > > jump-users mailing list
> > > [email protected]
> > > http://lists.refractions.net/mailman/listinfo/jump-users
> > >
> > >
> >
> > _______________________________________________
> > jump-users mailing list
> > [email protected]
> > http://lists.refractions.net/mailman/listinfo/jump-users
> >
>
>
> --
> http://amusingprogrammer.blogspot.com/
> _______________________________________________
> jump-users mailing list
> [email protected]
> http://lists.refractions.net/mailman/listinfo/jump-users
>
_______________________________________________
jump-users mailing list
[email protected]
http://lists.refractions.net/mailman/listinfo/jump-users

Reply via email to