[JPP-Devel] [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of FlexibleDateParser

[ede via Jump-pilot-devel]

On 19.08.2020 21:45, michael michaud wrote:
> Finally found it : you were right when you said that the database loader 
> could be at fault if, by design or flaw, it did not read dates as dates.
> This comes from r6129, when you added FlexibleFeature in database loader. At 
> the same time, the DateConverter has been changed to always read dates 
> (indeed timestamp) as Strings. The comment is : "speedup loading datasets w/ 
> date/time columns utilizing flex feature lazy conversion"
>
> I propose to revert this change, but I let you check it first if you remember 
> another reason to replace the previous code of DateConverter.

there is a fallback to FlexDateParser that was commented as well to speed up 
things. how about we keep

ret = rs.getTimestamp(columnIndex);

but leave the FlexDateParser fallback commented? as you seem to have a 
spatialite dataset how about you patch & test it? ..ede


---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 07:45 PM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] #497 Shapefile export slowed down because of FlexibleDateParser

[michael michaud via Jump-pilot-devel]

Finally found it : you were right when you said that the database loader could 
be at fault if, by design or flaw, it did not read dates as dates.
This comes from r6129, when you added FlexibleFeature in database loader. At 
the same time, the DateConverter has been changed to always read dates (indeed 
timestamp) as Strings. The comment is : "speedup loading datasets w/ date/time 
columns utilizing flex feature lazy conversion"

I propose to revert this change, but I let you check it first if you remember 
another reason to replace the previous code of DateConverter.


---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 12:06 PM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

Re: [JPP-Devel] Offset in the digitizing tools

[Rahkonen Jukka (MML)]

All the time, it was just more surprising and easier to notice with snap to 
grid on. By experiencing with the snapping tolerance the offset seems to be 
about 15 pixels.

-Jukka-

-Alkuperäinen viesti-
Lähettäjä: edgar.sol...@web.de  
Lähetetty: keskiviikko 19. elokuuta 2020 17.56
Vastaanottaja: jump-pilot-devel@lists.sourceforge.net
Aihe: Re: [JPP-Devel] Offset in the digitizing tools

do the vertices only get placed wrongly when 'snap to grid' is enabled or all 
the time or ?.. ede

On 19.08.2020 14:28, Rahkonen Jukka (MML) wrote:
> Hi,
>
> Tested with OpenJUMP-20200818-r6382-PLUS with java versions Corretto 8 and 
> JDK 11 on Windows. The issue appears also with OpenJUMP-20200415-r6252-PLUS. 
> What happens is that the tip of the digitizing tool has some sort of offset 
> compared with the map canvas. As a result the digitized point is inserted 
> above the tooltip and effectively it is impossible to digitize new features 
> properly. In addition, tools like Delete vertex or Move vertex do not seem to 
> find the item to edit. Move selected features tool works and it does not that 
> offset issue but it works in a bit different way and click does not need to 
> hit any vertex or feature.
>
> -Jukka Rahkonen-
>
>
>
> ___
> Jump-pilot-devel mailing list
> Jump-pilot-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
>



___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel


___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

Re: [JPP-Devel] Offset in the digitizing tools

[edgar . soldin]

do the vertices only get placed wrongly when 'snap to grid' is enabled or all 
the time or ?.. ede

On 19.08.2020 14:28, Rahkonen Jukka (MML) wrote:
> Hi,
>
> Tested with OpenJUMP-20200818-r6382-PLUS with java versions Corretto 8 and 
> JDK 11 on Windows. The issue appears also with OpenJUMP-20200415-r6252-PLUS. 
> What happens is that the tip of the digitizing tool has some sort of offset 
> compared with the map canvas. As a result the digitized point is inserted 
> above the tooltip and effectively it is impossible to digitize new features 
> properly. In addition, tools like Delete vertex or Move vertex do not seem to 
> find the item to edit. Move selected features tool works and it does not that 
> offset issue but it works in a bit different way and click does not need to 
> hit any vertex or feature.
>
> -Jukka Rahkonen-
>
>
>
> ___
> Jump-pilot-devel mailing list
> Jump-pilot-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
>



___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of FlexibleDateParser

[ede via Jump-pilot-devel]

i just noticed that FlexFeature uses AttributeType.DATE, which is 
java.sql.Date.class to determine if conversion is needed or not. that might 
lead to conversion of java.util.Date objects which is definitely unwanted.

btw. found this commit
  https://sourceforge.net/p/jump-pilot/code/6129/
where i disabled parsing of spatialite dates (at request ;). maybe we should at 
least reenable

  ret = rs.getTimestamp(columnIndex);

and only keep expensive FlexDateParsing disabled?

to reproduce/check i need a test dataset though. maybe an sqlite file? ..ede

ps. here are the two threads wrt. slow date attrib loading
  
https://www.mail-archive.com/jump-pilot-devel@lists.sourceforge.net/msg18295.html
  
https://www.mail-archive.com/jump-pilot-devel@lists.sourceforge.net/msg18317.html

On 19.08.2020 15:17, ede wrote:
> Mike,
>
> saving by itself is not the root problem here. this is proven by the second 
> run (when all dates are converted).
>
> it is a combination of opening/editing/saving. if a data reader (eg. JML or 
> GeoJSON now) uses FlexibleFeature's (FF) lazy conversion, time is saved by 
> simply not converting expensive data types in the loader but having FF do 
> that later. in the worst case this is during saving to another format, as you 
> noticed.
>
> if saving data loaded from a database is slow and it is indeed caused by the 
> FlexDateParser, then the loader must be at fault, because it does by design 
> or flaw not convert to java.sql.Date during loading. what database do you 
> load from?
>
> ..ede
>
> On 19.08.2020 14:06, michael michaud wrote:
>> A beanshell test just comparing writing shp from  BasicFeature and from 
>> FlexibleFeature.
>> (to be executed from the Beanshell Editor)
>>
>>
>> Attachments:
>>
>> - 
>> [test_flexible_feature.bsh](https://sourceforge.net/p/jump-pilot/bugs/_discuss/thread/62e6f051ce/6cc8/attachment/test_flexible_feature.bsh)
>>  (1.7 kB; application/octet-stream)
>>
>>
>> ---
>>
>> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>>
>> **Status:** open
>> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
>> **Last Updated:** Wed Aug 19, 2020 10:29 AM UTC
>> **Owner:** michael michaud
>>
>>
>> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
>> with version 1.15 lasted more than 20 minutes.
>> Most of the time is used by FlexibleDateParser.parse().
>>
>>
>> ---
>>
>> Sent from sourceforge.net because you indicated interest in 
>> 
>>
>>
>>
>> To unsubscribe from further messages, please visit 
>> 
>>
>
>
>
> ---
>
> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>
> **Status:** open
> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
> **Last Updated:** Wed Aug 19, 2020 12:06 PM UTC
> **Owner:** michael michaud
>
>
> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
> with version 1.15 lasted more than 20 minutes.
> Most of the time is used by FlexibleDateParser.parse().
>
>
> ---
>
> Sent from sourceforge.net because you indicated interest in 
> 
>
>
>
> To unsubscribe from further messages, please visit 
> 
>



---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 12:06 PM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of FlexibleDateParser

[ede via Jump-pilot-devel]

Mike,

saving by itself is not the root problem here. this is proven by the second run 
(when all dates are converted).

it is a combination of opening/editing/saving. if a data reader (eg. JML or 
GeoJSON now) uses FlexibleFeature's (FF) lazy conversion, time is saved by 
simply not converting expensive data types in the loader but having FF do that 
later. in the worst case this is during saving to another format, as you 
noticed.

if saving data loaded from a database is slow and it is indeed caused by the 
FlexDateParser, then the loader must be at fault, because it does by design or 
flaw not convert to java.sql.Date during loading. what database do you load 
from?

..ede

On 19.08.2020 14:06, michael michaud wrote:
> A beanshell test just comparing writing shp from  BasicFeature and from 
> FlexibleFeature.
> (to be executed from the Beanshell Editor)
>
>
> Attachments:
>
> - 
> [test_flexible_feature.bsh](https://sourceforge.net/p/jump-pilot/bugs/_discuss/thread/62e6f051ce/6cc8/attachment/test_flexible_feature.bsh)
>  (1.7 kB; application/octet-stream)
>
>
> ---
>
> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>
> **Status:** open
> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
> **Last Updated:** Wed Aug 19, 2020 10:29 AM UTC
> **Owner:** michael michaud
>
>
> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
> with version 1.15 lasted more than 20 minutes.
> Most of the time is used by FlexibleDateParser.parse().
>
>
> ---
>
> Sent from sourceforge.net because you indicated interest in 
> 
>
>
>
> To unsubscribe from further messages, please visit 
> 
>



---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 12:06 PM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] Offset in the digitizing tools

[Rahkonen Jukka (MML)]

Hi,

Tested with OpenJUMP-20200818-r6382-PLUS with java versions Corretto 8 and JDK 
11 on Windows. The issue appears also with OpenJUMP-20200415-r6252-PLUS. What 
happens is that the tip of the digitizing tool has some sort of offset compared 
with the map canvas. As a result the digitized point is inserted above the 
tooltip and effectively it is impossible to digitize new features properly. In 
addition, tools like Delete vertex or Move vertex do not seem to find the item 
to edit. Move selected features tool works and it does not that offset issue 
but it works in a bit different way and click does not need to hit any vertex 
or feature.

-Jukka Rahkonen-
___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] #497 Shapefile export slowed down because of FlexibleDateParser

[michael michaud via Jump-pilot-devel]

A beanshell test just comparing writing shp from  BasicFeature and from 
FlexibleFeature.
(to be executed from the Beanshell Editor)


Attachments:

- 
[test_flexible_feature.bsh](https://sourceforge.net/p/jump-pilot/bugs/_discuss/thread/62e6f051ce/6cc8/attachment/test_flexible_feature.bsh)
 (1.7 kB; application/octet-stream)


---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 10:29 AM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] #497 Shapefile export slowed down because of FlexibleDateParser

[Jukka Rahkonen via Jump-pilot-devel]

Hi,

There are mails about slow parsing of JML in July 2017 and I experienced slow 
parsing of dates from Spatialite in February 2019, discussion is also on the 
mailing list. It seems there is a need to test your fixes also with JML and 
SpatiaLite, not only with shapefiles.

-Jukka-

Lähettäjä: ede 
Lähetetty: keskiviikko 19. elokuuta 2020 11.26
Vastaanottaja: [jump-pilot:bugs] <4...@bugs.jump-pilot.p.re.sourceforge.net>
Aihe: [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of 
FlexibleDateParser


On 19.08.2020 09:25, michael michaud wrote:

Indeed, from my tests, things are getting worst :

well. it shouldn't after all i'm trying to improve things :)

I save my dataset to a shapefile after it has been freshly extracted from a 
database (it contains a geometry and a timestamp)
1.15 : 1st save : 55s
2nd save < 1s> r6382 : 1st save : 188s!
2nd save < 1s

let me check. actually r6382 should be faster now. in my tests it is.

what i do is.
load your data as JML (exported earlier), as it is text based and needs to be 
parsed. then save it as SHP and compare times and FlexdateParser behaviour.

what's important is if the loader creates FlexibleFeatures or not. and even 
then, if then lazy conversion is optional, as the loader may of course set the 
attribute as an object of the Date class.

Another thing about your explanation : I remember that you introduced the 
LazyConversion of attributes with the json parser. I think the reason was 
related to the type system of json (which has only strings, numbers and 
boolean).

wrt. GeoJSON that's true. but later on someone noticed that opening JML was 
very slow and we found date parsing to be the root cause, so lazy conversion 
was introduced.

In the case of shapefile or database where data is precisely typed, I think the 
lazy conversion is finally very costly.

that's incorrect ;) brute forcing date formats is what is very_ costly here. 
lazy conversion is merely distributing cost from time A to time B. if we do not 
need to convert at some place we should remove conversion there.

Data should be stored with the right type at the first place. It would avoid 
all these useless conversions of dates. What do you think ? Seems that lazy 
conversion made for the json should stay the exception, not the general way to 
handle data types.

the concept is sound. there is something else goin on.

here check out the commit
https://sourceforge.net/p/jump-pilot/code/5482/

for testing purposes you could simply replace FlexibleFeature with the 
BasicFeature again in your local copy to disable lazy conversion and observe 
the difference.

what i can't find right now where JML/GML parses/parsed the date which made it 
slow. i'll have a look in the bug description again. i think it was Jukka via 
mailing list.

so far.. ede



[bugs:#497] Shapefile export 
slowed down because of FlexibleDateParser

Status: open
Created: Mon Aug 17, 2020 11:47 AM UTC by michael michaud
Last Updated: Tue Aug 18, 2020 09:42 PM UTC
Owner: michael michaud

Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().



Sent from sourceforge.net because you indicated interest in 
https://sourceforge.net/p/jump-pilot/bugs/497/

To unsubscribe from further messages, please visit 
https://sourceforge.net/auth/subscriptions/



[bugs:#497] Shapefile export 
slowed down because of FlexibleDateParser

Status: open
Created: Mon Aug 17, 2020 11:47 AM UTC by michael michaud
Last Updated: Wed Aug 19, 2020 07:39 AM UTC
Owner: michael michaud

Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().



Sent from sourceforge.net because you indicated interest in 
https://sourceforge.net/p/jump-pilot/bugs/497/

To unsubscribe from further messages, please visit 
https://sourceforge.net/auth/subscriptions/



---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 10:29 AM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe 

[JPP-Devel] [jump-pilot:bugs] #497 Shapefile export slowed down because of FlexibleDateParser

[Jukka Rahkonen via Jump-pilot-devel]

There are mails about slow parsing of JML in July 2017 and I experienced slow 
parsing of dates from Spatialite in February 2019, discussion is also on the 
mailing list. I hope that new fixes do not cause regression in other places.



---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 07:39 AM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of FlexibleDateParser

[ede via Jump-pilot-devel]

On 19.08.2020 09:25, michael michaud wrote:
> Indeed, from my tests, things are  getting worst :

well. it shouldn't after all i'm trying to improve things :)

> I save my dataset to a shapefile after it has been freshly extracted from a 
> database (it contains a geometry and a timestamp)
> 1.15 : 1st save : 55s
>  2nd save < 1s>  r6382 : 1st save : 188s!
>  2nd save < 1s

let me check. actually r6382 should be faster now. in my tests it is.

what i do is.
load your data as JML (exported earlier), as it is text based and needs to be 
parsed. then save it as SHP and compare times and FlexdateParser behaviour.

what's important is if the loader creates FlexibleFeatures or not. and even 
then, if then lazy conversion is optional, as the loader may of course set the 
attribute as an object of the Date class.

> Another thing about your explanation : I remember that you introduced the 
> LazyConversion of attributes with the json parser. I think the reason was 
> related to the type system of json (which has only strings, numbers and 
> boolean).

wrt. GeoJSON that's true. but later on someone noticed that opening JML was 
very slow and we found date parsing to be the root cause, so lazy conversion 
was introduced.

>In the case of shapefile or database where data is precisely typed, I think 
>the lazy conversion is finally very costly.

that's incorrect ;) brute forcing date formats is what is very_ costly here. 
lazy conversion is merely distributing cost from time A to time B. if we do not 
need to convert at some place we should remove conversion there.

>Data should be stored with the right type at the first place. It would avoid 
>all these useless conversions of dates. What do you think ? Seems that lazy 
>conversion made for the json should stay the exception, not the general way to 
>handle data types.

the concept is sound. there is something else goin on.

here check out the commit
https://sourceforge.net/p/jump-pilot/code/5482/

for testing purposes you could simply replace FlexibleFeature with the 
BasicFeature again in your local copy to disable lazy conversion and observe 
the difference.

what i can't find right now where JML/GML parses/parsed the date which made it 
slow. i'll have a look in the bug description again. i think it was Jukka via 
mailing list.

so far.. ede

>
>
> ---
>
> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>
> **Status:** open
> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
> **Last Updated:** Tue Aug 18, 2020 09:42 PM UTC
> **Owner:** michael michaud
>
>
> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
> with version 1.15 lasted more than 20 minutes.
> Most of the time is used by FlexibleDateParser.parse().
>
>
> ---
>
> Sent from sourceforge.net because you indicated interest in 
> 
>
>
>
> To unsubscribe from further messages, please visit 
> 
>



---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 07:39 AM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] #497 Shapefile export slowed down because of FlexibleDateParser

[michael michaud via Jump-pilot-devel]

Jukka : you are right that shapefile store only dates, but conversion from 
shapefile dates to java Date (+ time) and from java Date (+ time) to shapefile 
date should be done with a fixed parser ( DbfFile.DATE_PARSER) rather than the 
expensive FlexibleDateParser. Adding/truncating the time part shouldn't be a 
big deal. 


---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 07:26 AM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] #497 Shapefile export slowed down because of FlexibleDateParser

[Jukka Rahkonen via Jump-pilot-devel]

Perhaps parsing is implemented as it is also because dbf and thus shapefile 
supports DATE but not DATETIME and conversion into strings is really needed 
only if time values contain  also hours and so on?


---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 07:25 AM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] #497 Shapefile export slowed down because of FlexibleDateParser

[michael michaud via Jump-pilot-devel]

Indeed, from my tests, things are  getting worst :
I save my dataset to a shapefile after it has been freshly extracted from a 
database (it contains a geometry and a timestamp)
1.15 : 1st save : 55s
 2nd save < 1s
 r6382 : 1st save : 188s!
 2nd save < 1s
 
Another thing about your explanation : I remember that you introduced the 
LazyConversion of attributes with the json parser. I think the reason was 
related to the type system of json (which has only strings, numbers and 
boolean). In the case of shapefile or database where data is precisely typed, I 
think the lazy conversion is finally very costly. Data should be stored with 
the right type at the first place. It would avoid all these useless conversions 
of dates. What do you think ? Seems that lazy conversion made for the json 
should stay the exception, not the general way to handle data types. 


---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Tue Aug 18, 2020 09:42 PM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel