Re: [JPP-Devel] Slow parsing of JML files

Michaël Michaud Sun, 30 Jul 2017 23:20:08 -0700

Hi Ede,

I can confirm what you say.


On a file with 1M simple shapes + 6 string attributes + 1 date attribute

shp : 23s (same time with or without attributes, 19s if the file set hasno shx)

jml : *18s* (17s without the date and *5mn with the date and before youroptimization !!!*)

json : 8s (date is interpreted as string, I could not convert the stringformatted as yyyy-MM-dd to a date through View/Edit schema plugins*)


Michaël

Exception in thread "AWT-EventQueue-0"java.lang.IllegalArgumentException: Cannot format given Object as a Date

    at java.text.DateFormat.format(DateFormat.java:310)
    at java.text.Format.format(Format.java:157)

atcom.vividsolutions.jump.workbench.ui.AttributeTablePanel$MyTable$2.setValue(AttributeTablePanel.java:192)




Le 30/07/2017 à 18:46, edgar.sol...@web.de a écrit :

Jukka,

sorry, r5482 is the latest. commit msg stays the same. ..ede

On 7/30/2017 18:44, edgar.sol...@web.de wrote:

Jukka,

can you try r4682?

Log Message:
-----------
speeding up JML/GML reader when reading time attributes by parsing them lazy 
(during access later)
- adding FlexibleFeature, FlexibleFeatureSchema
- porting GMLReader to use the flexible classes

..ede

On 7/29/2017 22:50, Rahkonen Jukka (MML) wrote:

Hi,

I took some timings with todays nightly build by opening my dataset with 
1101817 polygons and quite many attributes.

Opening time
JML: 25 minutes 14 sec
Shapefile: 1 minute 28 sec
GeoJSON: 45 seconds

Seems that there must be something suboptimal in the JML driver and that Ede 
did really nice work with the GeoJSON driver.

-Jukka-


________________________________________
Lähettäjä: edgar.sol...@web.de <edgar.sol...@web.de>
Lähetetty: 29. heinäkuuta 2017 22:25
Vastaanottaja: OpenJump develop and use
Aihe: Re: [JPP-Devel] Slow parsing of JML files

just spent some time with GMLReader, which is actually our JMLReader and 
according to my finding it get's some magnitudes faster when the date parsing 
is commented out (in GMLInputTemplate.getColumnValue()).

had a short look at the FlexibleDateParser and have the impression that regex 
patterns are not precompiled but recompiled on every usage. btw. that would be 
a classic in terms of parsing slowdowns.

..ede

On 7/29/2017 13:03, Michaël Michaud wrote:

Hi,

I also remembered that jml reader was quite slow compared with shp, but my last 
tests show me only slight differences.

(Maybe shapefile reader has slowed down with my last commits - attempts to make 
it more robust with shp coming from esri or qgis)

Now, reading a big file (888000 features) with complex polygons and 3 
attributes is just a bit longer with jml (38 vs 32s)
For a file containing 1M of simple features (squares), it is even faster with 
jml (14 vs 23). Reading more attributes seems longer with jml.

I would say :

reading complex geometries is much longer with jml
reading attributes is longer with jml
reading simple geometries is longer with shx/shp

Jukka, have you examples where jml is much longer that shp ?

Michaël


Le 28/07/2017 à 17:25, edgar.sol...@web.de a écrit :

On 28.07.2017 17:06, Rahkonen Jukka (MML) wrote:

Hi,

I noticed with a dataset having 1.2 million features with lots of attributes 
that OpenJUMP is rather slow in parsing its own native JML format. It takes 
about 30 minutes to open the file. Shapefiles are much faster but they do not 
suit me because long strings are truncated. Can the hardcore developers guess 
what is the bottle neck with JML/XML parsing and if there could be some place 
for improvements?

hi Jukka,

could you provide said dataset privately?

does it make a difference when you cut down the number of attributes?

..ede

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

Re: [JPP-Devel] Slow parsing of JML files

Reply via email to