On 18.08.2020 23:42, michael michaud wrote:
> Hi Ede,
> Seems to be a nice tweak but something is still wrong : the first save is
> still very slow.
surely it's faster than before, or? before it tried 125 parser for each entry
before succeeding. now this only happens on the first try and then this one is
reused. it's faster for me.
> Seems that the second save of a dataset benefits your optimization, but the
> save operation on a freshly imported/extracted dataset is still long.
yeah, that's mysterious to me to. oh, wait a second. no. i got it. read on
below.
> Also there is something I don't understand : writing a shapefile (a dbf)
> should not need FlexibleDateParser at all. The date should simply be
> converted to a string by DbfFile.DATE_PARSER. Did you see where and why
> FlexibleDateParser is used in the "writing" process ? The problem also exists
> for JML. I did not check the code, but I also don't understand why it would
> need a flexible "parser"to convert a Date to a String.
that's our code. when speeding up loading datasets we introduced lazy
conversion for date/time fields remember? now the attribute is only
converted/parsed into a date object when the value is actually requested.
that is actually what is happening during saving, the Feature.getAttribute()
need to return a Date Object, hence parsing needs to be done.
on the second save the whole layers features are parsed already and no
conversion is needed anymore, hence lightning speed.
we could introduce a asynchronous thread that converts silently in the
background. problem though would be error handling as the user would be
confused by possible suddenly popping up parsing errors whilst in the middle of
doing something else. solution agn could be, to ignore those and let the user
stumble over them when they need to be converted agn during save or editing.
> Your changed broke two tests, but I removed the tests which were not as
> important as the optimization we try to achieve.
saw it.
yeah, reordering parsers shouldn't break any test. thanks!.. ede
>
>
> ---
>
> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>
> **Status:** open
> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
> **Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
> **Owner:** michael michaud
>
>
> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
> with version 1.15 lasted more than 20 minutes.
> Most of the time is used by FlexibleDateParser.parse().
>
>
> ---
>
> Sent from sourceforge.net because you indicated interest in
> <https://sourceforge.net/p/jump-pilot/bugs/497/>
>
>
>
> To unsubscribe from further messages, please visit
> <https://sourceforge.net/auth/subscriptions/>
>
---
** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Tue Aug 18, 2020 09:42 PM UTC
**Owner:** michael michaud
Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().
---
Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is
subscribed to https://sourceforge.net/p/jump-pilot/bugs/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/jump-pilot/admin/bugs/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
