Jukka, thanks for the flowers :).. try the same w/o date attributes, that should speed up JML by magnitudes.
..ede On 7/29/2017 22:50, Rahkonen Jukka (MML) wrote: > Hi, > > I took some timings with todays nightly build by opening my dataset with > 1101817 polygons and quite many attributes. > > Opening time > JML: 25 minutes 14 sec > Shapefile: 1 minute 28 sec > GeoJSON: 45 seconds > > Seems that there must be something suboptimal in the JML driver and that Ede > did really nice work with the GeoJSON driver. > > -Jukka- > > > ________________________________________ > Lähettäjä: edgar.sol...@web.de <edgar.sol...@web.de> > Lähetetty: 29. heinäkuuta 2017 22:25 > Vastaanottaja: OpenJump develop and use > Aihe: Re: [JPP-Devel] Slow parsing of JML files > > just spent some time with GMLReader, which is actually our JMLReader and > according to my finding it get's some magnitudes faster when the date parsing > is commented out (in GMLInputTemplate.getColumnValue()). > > had a short look at the FlexibleDateParser and have the impression that regex > patterns are not precompiled but recompiled on every usage. btw. that would > be a classic in terms of parsing slowdowns. > > ..ede > > On 7/29/2017 13:03, Michaël Michaud wrote: >> Hi, >> >> I also remembered that jml reader was quite slow compared with shp, but my >> last tests show me only slight differences. >> >> (Maybe shapefile reader has slowed down with my last commits - attempts to >> make it more robust with shp coming from esri or qgis) >> >> Now, reading a big file (888000 features) with complex polygons and 3 >> attributes is just a bit longer with jml (38 vs 32s) >> For a file containing 1M of simple features (squares), it is even faster >> with jml (14 vs 23). Reading more attributes seems longer with jml. >> >> I would say : >> >> reading complex geometries is much longer with jml >> reading attributes is longer with jml >> reading simple geometries is longer with shx/shp >> >> Jukka, have you examples where jml is much longer that shp ? >> >> Michaël >> >> >> Le 28/07/2017 à 17:25, edgar.sol...@web.de a écrit : >>> On 28.07.2017 17:06, Rahkonen Jukka (MML) wrote: >>>> Hi, >>>> >>>> I noticed with a dataset having 1.2 million features with lots of >>>> attributes that OpenJUMP is rather slow in parsing its own native JML >>>> format. It takes about 30 minutes to open the file. Shapefiles are much >>>> faster but they do not suit me because long strings are truncated. Can the >>>> hardcore developers guess what is the bottle neck with JML/XML parsing and >>>> if there could be some place for improvements? >>>> >>> hi Jukka, >>> >>> could you provide said dataset privately? >>> >>> does it make a difference when you cut down the number of attributes? >>> >>> ..ede >>> >>> ------------------------------------------------------------------------------ >>> Check out the vibrant tech community on one of the world's most >>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>> _______________________________________________ >>> Jump-pilot-devel mailing list >>> Jump-pilot-devel@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel >>> >> >> >> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> _______________________________________________ >> Jump-pilot-devel mailing list >> Jump-pilot-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Jump-pilot-devel mailing list > Jump-pilot-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Jump-pilot-devel mailing list > Jump-pilot-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
