[JPP-Devel] [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of FlexibleDateParser

[ede via Jump-pilot-devel]

On 18.08.2020 23:42, michael michaud wrote:
> Hi Ede,
> Seems to be a nice tweak but something is still wrong : the first save is 
> still very slow.

surely it's faster than before, or? before it tried 125 parser for each entry 
before succeeding. now this only happens on the first try and then this one is 
reused. it's faster for me.

> Seems that the second save of a dataset benefits your optimization, but the 
> save operation on a freshly imported/extracted dataset is still long.

yeah, that's mysterious to me to. oh, wait a second. no. i got it. read on 
below.

> Also there is something I don't understand : writing a shapefile (a dbf) 
> should not need FlexibleDateParser at all. The date should simply be 
> converted to a string by DbfFile.DATE_PARSER. Did you see where and why 
> FlexibleDateParser is used in the "writing" process ? The problem also exists 
> for JML. I did not check the code, but I also don't understand why it would 
> need a flexible "parser"to convert a Date to a String.

that's our code. when speeding up loading datasets we introduced lazy 
conversion for date/time fields remember? now the attribute is only 
converted/parsed into a date object when the value is actually requested.
that is actually what is happening during saving, the Feature.getAttribute() 
need to return a Date Object, hence parsing needs to be done.

on the second save the whole layers features are parsed already and no 
conversion is needed anymore, hence lightning speed.

we could introduce a asynchronous thread that converts silently in the 
background. problem though would be error handling as the user would be 
confused by possible suddenly popping up parsing errors whilst in the middle of 
doing something else. solution agn could be, to ignore those and let the user 
stumble over them when they need to be converted agn during save or editing.

> Your changed broke two tests, but I removed the tests which were not as 
> important as the optimization we try to achieve.

saw it.
yeah, reordering parsers shouldn't break any test. thanks!.. ede

>
>
> ---
>
> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>
> **Status:** open
> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
> **Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
> **Owner:** michael michaud
>
>
> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
> with version 1.15 lasted more than 20 minutes.
> Most of the time is used by FlexibleDateParser.parse().
>
>
> ---
>
> Sent from sourceforge.net because you indicated interest in 
> 
>
>
>
> To unsubscribe from further messages, please visit 
> 
>



---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Tue Aug 18, 2020 09:42 PM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] #497 Shapefile export slowed down because of FlexibleDateParser

[michael michaud via Jump-pilot-devel]

Hi Ede,
Seems to be a nice tweak but something is still wrong : the first save is still 
very slow.
Seems that the second save of a dataset benefits your optimization, but the 
save operation on a freshly imported/extracted dataset is still long.
Also there is something I don't understand : writing a shapefile (a dbf) should 
not need FlexibleDateParser at all. The date should simply be converted to a 
string by DbfFile.DATE_PARSER. Did you see where and why FlexibleDateParser is 
used in the "writing" process ? The problem also exists for JML. I did not 
check the code, but I also don't understand why it would need a flexible 
"parser"to convert a Date to a String. 
Your changed broke two tests, but I removed the tests which were not as 
important as the optimization we try to achieve.


---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] SVN: [6382] core/trunk/src/jumptest/junit/FlexibleDateParserTestCase.java

[jump-pilot-svn--- via Jump-pilot-devel]

Revision: 6382
  http://sourceforge.net/p/jump-pilot/code/6382
Author:   michaudm
Date: 2020-08-18 21:02:19 + (Tue, 18 Aug 2020)
Log Message:
---
Deactivate 2 tests on FlexibleDateParser

Modified Paths:
--
core/trunk/src/jumptest/junit/FlexibleDateParserTestCase.java

Modified: core/trunk/src/jumptest/junit/FlexibleDateParserTestCase.java
===
--- core/trunk/src/jumptest/junit/FlexibleDateParserTestCase.java   
2020-08-18 19:26:30 UTC (rev 6381)
+++ core/trunk/src/jumptest/junit/FlexibleDateParserTestCase.java   
2020-08-18 21:02:19 UTC (rev 6382)
@@ -80,12 +80,18 @@
 //parser.parse("Jan 06 17:01:02 PST 2003", false));
 assertEquals(simpleFormat1.parse("1970-06-01"),
 parser.parse("Jun 1970", false));
-assertEquals(simpleFormat1.parse(year + "-06-19"),
-parser.parse("Jun 19", false));
+// The following test does not pass anymore because from
+// r6381 the last used parser (here MMM ) is tested first,
+// and in this case, interpret 19 as the year 19.
+//assertEquals(simpleFormat1.parse(year + "-06-19"),
+//parser.parse("Jun 19", false));
 assertEquals(simpleFormat1.parse("1970-06-01"),
 parser.parse("June 1970", false));
-assertEquals(simpleFormat1.parse(year + "-06-19"),
-parser.parse("June 19", false));
+// The following test does not pass anymore because from
+// r6381 the last used parser (here MMM ) is tested first,
+// and in this case, interpret 19 as the year 19.
+//assertEquals(simpleFormat1.parse(year + "-06-19"),
+//parser.parse("June 19", false));
 assertEquals(simpleFormat1.parse("2003-09-19"),
 parser.parse("Sep 19, 2003", false));
 



___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of FlexibleDateParser

[ede via Jump-pilot-devel]

just committed r6381 which speeds up FlexibleDateParser by prefering successful 
DateFormatters (see log below). please test.

other observations.
1. saving the SHP data immediately again is lightning speed fast. not sure why.
2. sometimes writing with the new "Save Dataset as (testing)" dialog fails with 
the stack below

[INFO] 21:23:06.501 Reading 'test_date.jml' took 3.39s.
[INFO] 21:23:07.247 Done. Current committed 
memory:plugin.AbstractPlugIn.executing = Executing271 MB
[INFO] 21:23:07.253 Activating Select Features Tool
[INFO] 21:23:11.837 Executing Save Dataset As (testing)
[INFO] 21:23:19.141 Activating Select Features Tool
[INFO] 21:23:26.271 Writing 'test_date8.shp' took 7.00s.
[INFO] 21:23:26.275 Done. Current committed 
memory:plugin.AbstractPlugIn.executing = Executing321 MB
[INFO] 21:23:35.053 Activating Select Features Tool
[INFO] 21:23:38.203 Executing Save Dataset As (testing)
[INFO] 21:23:42.603 Activating Select Features Tool
[INFO] 21:23:43.037 Writing 'test_date9.shp' took 0.36s.

[ERROR] 21:29:32.398 java.io.EOFException
java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at 
com.vividsolutions.jump.io.EndianDataInputStream.readUnsignedByteLE(EndianDataInputStream.java:86)
at 
org.geotools.dbffile.DbfFile$DbfFileHeader.getDbfFileHeader(DbfFile.java:342)
at org.geotools.dbffile.DbfFile$DbfFileHeader.(DbfFile.java:336)
at org.geotools.dbffile.DbfFile.init(DbfFile.java:172)
at org.geotools.dbffile.DbfFile.(DbfFile.java:80)
at org.geotools.dbffile.DbfFile.(DbfFile.java:66)
at 
com.vividsolutions.jump.io.ShapefileWriter.writeDbf(ShapefileWriter.java:420)
at 
com.vividsolutions.jump.io.ShapefileWriter.write(ShapefileWriter.java:293)
at 
com.vividsolutions.jump.io.datasource.ReaderWriterFileDataSource$1.executeUpdate(ReaderWriterFileDataSource.java:141)
at 
org.openjump.core.ui.io.file.DataSourceFileLayerSaver.write(DataSourceFileLayerSaver.java:62)
at 
org.openjump.core.ui.plugin.file.save.SaveToFileWizard.run(SaveToFileWizard.java:75)
at 
org.openjump.core.ui.plugin.file.SaveWizardPlugIn.run(SaveWizardPlugIn.java:127)
at 
com.vividsolutions.jump.workbench.ui.task.TaskMonitorManager$TaskWrapper.run(TaskMonitorManager.java:151)

On 18.08.2020 20:57, ede wrote:
> ok, got it. SHP reader does not seem to use lazy date parsing via 
> FlexibleDateParser. converting your set to JML and tryinf to save that 
> involves FlexibleDateParser and shows the symptoms you observed.
>
> ..ede
>
> On 18.08.2020 20:24, ede wrote:
>> sorry Mike, can't reproduce it.
>>
>> actually writing is magnitudes faster than reading on my laptop. see
>>
>> [INFO] 20:05:45.910 Reading 'test_date.shp' took 6.64s.
>> [INFO] 20:05:47.011 Done. Current committed 
>> memory:plugin.AbstractPlugIn.executing = Executing199 MB
>> [INFO] 20:05:47.013 Activating Select Features Tool
>> ...
>> [INFO] 20:06:09.648 Executing Save Dataset As (testing)
>> [INFO] 20:06:26.760 Activating Select Features Tool
>> [INFO] 20:06:27.881 Writing 'test_date-out.shp' took 0.93s.
>> [INFO] 20:06:27.891 Done. Current committed 
>> memory:plugin.AbstractPlugIn.executing = Executing227 MB
>>
>> i also added a debug breakpoint at 
>> com.vividsolutions.jump.util.FlexibleDateParser.parse(String s, boolean 
>> lenient) and it isn't even called during saving as SHP.
>>
>> ..ede
>>
>> On 17.08.2020 21:15, michael michaud wrote:
>>> Here is a dataset of 100 000 features including datetimes. It takes nearly 
>>> 1 minute to save as shapefile (less than 2 seconds to read it).
>>>
>>>
>>> Attachments:
>>>
>>> - 
>>> [test_date.zip](https://sourceforge.net/p/jump-pilot/bugs/_discuss/thread/62e6f051ce/89eb/attachment/test_date.zip)
>>>  (1.6 MB; application/x-zip-compressed)
>>>
>>>
>>> ---
>>>
>>> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>>>
>>> **Status:** open
>>> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
>>> **Last Updated:** Mon Aug 17, 2020 04:38 PM UTC
>>> **Owner:** michael michaud
>>>
>>>
>>> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
>>> with version 1.15 lasted more than 20 minutes.
>>> Most of the time is used by FlexibleDateParser.parse().
>>>
>>>
>>> ---
>>>
>>> Sent from sourceforge.net because you indicated interest in 
>>> 
>>>
>>>
>>>
>>> To unsubscribe from further messages, please visit 
>>> 
>>>
>>
>>
>>
>> ---
>>
>> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>>
>> **Status:** open
>> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
>> **Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
>> **Owner:** michael michaud
>>
>>
>> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
>> with version 1.15 lasted more than 20 minutes.
>> Most of the time is 

[JPP-Devel] SVN: [6381] core/trunk/src/com/vividsolutions/jump/util/ FlexibleDateParser.java

[jump-pilot-svn--- via Jump-pilot-devel]

Revision: 6381
  http://sourceforge.net/p/jump-pilot/code/6381
Author:   edso
Date: 2020-08-18 19:26:30 + (Tue, 18 Aug 2020)
Log Message:
---
speedup FlexibleDateParser.parse() by moving successful Parsers to the front of 
the list of Parsers to try

Modified Paths:
--
core/trunk/src/com/vividsolutions/jump/util/FlexibleDateParser.java

Modified: core/trunk/src/com/vividsolutions/jump/util/FlexibleDateParser.java
===
--- core/trunk/src/com/vividsolutions/jump/util/FlexibleDateParser.java 
2020-08-18 19:19:01 UTC (rev 6380)
+++ core/trunk/src/com/vividsolutions/jump/util/FlexibleDateParser.java 
2020-08-18 19:26:30 UTC (rev 6381)
@@ -32,6 +32,7 @@
 package com.vividsolutions.jump.util;
 
 import com.vividsolutions.jts.util.Assert;
+import com.vividsolutions.jump.workbench.Logger;
 
 import java.awt.Color;
 import java.awt.Component;
@@ -61,8 +62,8 @@
 public class FlexibleDateParser {
 private static FlexibleDateParser instance = null;
 
-private static Collection lenientFormatters = null;
-private static Collection unlenientFormatters = null;
+private static List lenientFormatters = null;
+private static List unlenientFormatters = null;
 
 //CellEditor used to be a static field CELL_EDITOR, but I was getting
 //problems calling it from ESETextField (it simply didn't appear).
@@ -180,7 +181,7 @@
 return sortedPatterns;
 }
 
-private Collection lenientFormatters() {
+private List lenientFormatters() {
 if (lenientFormatters == null) {
 load();
 }
@@ -187,7 +188,7 @@
 return lenientFormatters;
 }
 
-private Collection unlenientFormatters() {
+private List unlenientFormatters() {
 if (unlenientFormatters == null) {
 load();
 }
@@ -224,22 +225,36 @@
 }
 }
 
-private Date parse(String s, Collection formatters) 
throws ParseException {
+private Date parse(String s, List formatters) throws 
ParseException {
 ParseException firstParseException = null;
 
+int i = 0;
 for (SimpleDateFormat formatter : formatters) {
-
-if (verbose) {
-System.out.println(
-s
-+ " -- "
+i++;
+if (verbose || Logger.isTraceEnabled()) {
+String msg = s
++ " -- #" + i + ". "
 + formatter.toPattern()
-+ (formatter.isLenient() ? "lenient" : ""));
++ (formatter.isLenient() ? "lenient" : "");
+if (verbose)
+  System.out.println(msg);
+Logger.trace(msg);
 }
 
 try {
   Date d = parse(s, formatter);
-return d;
+  
+  // [ede] 2020-08
+  // moving successful parser to the list's beginning assuming 
+  // that whatever datestring is parsed next will probably be 
+  // in the same format speeding up parsing by magnitudes
+  if (i > 1) {
+int index = formatters.indexOf(formatter);
+formatters.remove(index);
+formatters.add(0, formatter);
+  }
+
+  return d;
 } catch (ParseException e) {
 if (firstParseException == null) {
 firstParseException = e;
@@ -330,7 +345,7 @@
 }
 }
 
-private Collection toFormatters(boolean lenient, 
Collection patterns) {
+private List toFormatters(boolean lenient, 
Collection patterns) {
 List formatters = new ArrayList<>();
 //Sort from least complex to most complex; otherwise, ddMMM 
 //instead of MMMd will match "May 15". [Jon Aquino]



___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] SVN: [6380] core/trunk/src/org/geotools/shapefile/Shapefile.java

[jump-pilot-svn--- via Jump-pilot-devel]

Revision: 6380
  http://sourceforge.net/p/jump-pilot/code/6380
Author:   edso
Date: 2020-08-18 19:19:01 + (Tue, 18 Aug 2020)
Log Message:
---
changing a debug out to trace because of it's level of detail is slowing down 
loading massively

Modified Paths:
--
core/trunk/src/org/geotools/shapefile/Shapefile.java

Modified: core/trunk/src/org/geotools/shapefile/Shapefile.java
===
--- core/trunk/src/org/geotools/shapefile/Shapefile.java2020-08-18 
18:14:31 UTC (rev 6379)
+++ core/trunk/src/org/geotools/shapefile/Shapefile.java2020-08-18 
19:19:01 UTC (rev 6380)
@@ -403,7 +403,7 @@
 raf.getChannel().read(bb, offset*2 + 8);
 shp = new EndianDataInputStream(new 
ByteArrayInputStream(bytes));
 body = handler.read(shp, geometryFactory, length);
-Logger.debug("" + recordNumber + " : from " + offset + " 
for " + length + " (" + body.getNumPoints() + " pts)");
+Logger.trace("" + recordNumber + " : from " + offset + " 
for " + length + " (" + body.getNumPoints() + " pts)");
 list.add(body);
 if (body.getUserData() != null) errors++;
 } catch(Exception e) {



___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of FlexibleDateParser

[ede via Jump-pilot-devel]

ok, got it. SHP reader does not seem to use lazy date parsing via 
FlexibleDateParser. converting your set to JML and tryinf to save that involves 
FlexibleDateParser and shows the symptoms you observed.

..ede

On 18.08.2020 20:24, ede wrote:
> sorry Mike, can't reproduce it.
>
> actually writing is magnitudes faster than reading on my laptop. see
>
> [INFO] 20:05:45.910 Reading 'test_date.shp' took 6.64s.
> [INFO] 20:05:47.011 Done. Current committed 
> memory:plugin.AbstractPlugIn.executing = Executing199 MB
> [INFO] 20:05:47.013 Activating Select Features Tool
> ...
> [INFO] 20:06:09.648 Executing Save Dataset As (testing)
> [INFO] 20:06:26.760 Activating Select Features Tool
> [INFO] 20:06:27.881 Writing 'test_date-out.shp' took 0.93s.
> [INFO] 20:06:27.891 Done. Current committed 
> memory:plugin.AbstractPlugIn.executing = Executing227 MB
>
> i also added a debug breakpoint at 
> com.vividsolutions.jump.util.FlexibleDateParser.parse(String s, boolean 
> lenient) and it isn't even called during saving as SHP.
>
> ..ede
>
> On 17.08.2020 21:15, michael michaud wrote:
>> Here is a dataset of 100 000 features including datetimes. It takes nearly 1 
>> minute to save as shapefile (less than 2 seconds to read it).
>>
>>
>> Attachments:
>>
>> - 
>> [test_date.zip](https://sourceforge.net/p/jump-pilot/bugs/_discuss/thread/62e6f051ce/89eb/attachment/test_date.zip)
>>  (1.6 MB; application/x-zip-compressed)
>>
>>
>> ---
>>
>> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>>
>> **Status:** open
>> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
>> **Last Updated:** Mon Aug 17, 2020 04:38 PM UTC
>> **Owner:** michael michaud
>>
>>
>> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
>> with version 1.15 lasted more than 20 minutes.
>> Most of the time is used by FlexibleDateParser.parse().
>>
>>
>> ---
>>
>> Sent from sourceforge.net because you indicated interest in 
>> 
>>
>>
>>
>> To unsubscribe from further messages, please visit 
>> 
>>
>
>
>
> ---
>
> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>
> **Status:** open
> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
> **Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
> **Owner:** michael michaud
>
>
> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
> with version 1.15 lasted more than 20 minutes.
> Most of the time is used by FlexibleDateParser.parse().
>
>
> ---
>
> Sent from sourceforge.net because you indicated interest in 
> 
>
>
>
> To unsubscribe from further messages, please visit 
> 
>





---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of FlexibleDateParser

[ede via Jump-pilot-devel]

sorry Mike, can't reproduce it.

actually writing is magnitudes faster than reading on my laptop. see

[INFO] 20:05:45.910 Reading 'test_date.shp' took 6.64s.
[INFO] 20:05:47.011 Done. Current committed 
memory:plugin.AbstractPlugIn.executing = Executing199 MB
[INFO] 20:05:47.013 Activating Select Features Tool
...
[INFO] 20:06:09.648 Executing Save Dataset As (testing)
[INFO] 20:06:26.760 Activating Select Features Tool
[INFO] 20:06:27.881 Writing 'test_date-out.shp' took 0.93s.
[INFO] 20:06:27.891 Done. Current committed 
memory:plugin.AbstractPlugIn.executing = Executing227 MB

i also added a debug breakpoint at 
com.vividsolutions.jump.util.FlexibleDateParser.parse(String s, boolean 
lenient) and it isn't even called during saving as SHP.

..ede

On 17.08.2020 21:15, michael michaud wrote:
> Here is a dataset of 100 000 features including datetimes. It takes nearly 1 
> minute to save as shapefile (less than 2 seconds to read it).
>
>
> Attachments:
>
> - 
> [test_date.zip](https://sourceforge.net/p/jump-pilot/bugs/_discuss/thread/62e6f051ce/89eb/attachment/test_date.zip)
>  (1.6 MB; application/x-zip-compressed)
>
>
> ---
>
> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>
> **Status:** open
> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
> **Last Updated:** Mon Aug 17, 2020 04:38 PM UTC
> **Owner:** michael michaud
>
>
> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
> with version 1.15 lasted more than 20 minutes.
> Most of the time is used by FlexibleDateParser.parse().
>
>
> ---
>
> Sent from sourceforge.net because you indicated interest in 
> 
>
>
>
> To unsubscribe from further messages, please visit 
> 
>



---

** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**

**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
**Owner:** michael michaud


Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb) 
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().


---

Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/jump-pilot/bugs/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/jump-pilot/admin/bugs/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

[JPP-Devel] SVN: [6379] core/trunk/src/com/vividsolutions/jump/io/datasource/ ReaderWriterFileDataSource.java

[jump-pilot-svn--- via Jump-pilot-devel]

Revision: 6379
  http://sourceforge.net/p/jump-pilot/code/6379
Author:   edso
Date: 2020-08-18 18:14:31 + (Tue, 18 Aug 2020)
Log Message:
---
add a Log.info() on the time it took to save file
helpful for debugging file writing
using level info so users do not need to switch verbosity to get this oneliner 
their logs

Modified Paths:
--

core/trunk/src/com/vividsolutions/jump/io/datasource/ReaderWriterFileDataSource.java

Modified: 
core/trunk/src/com/vividsolutions/jump/io/datasource/ReaderWriterFileDataSource.java
===
--- 
core/trunk/src/com/vividsolutions/jump/io/datasource/ReaderWriterFileDataSource.java
2020-08-17 16:25:20 UTC (rev 6378)
+++ 
core/trunk/src/com/vividsolutions/jump/io/datasource/ReaderWriterFileDataSource.java
2020-08-18 18:14:31 UTC (rev 6379)
@@ -137,8 +137,9 @@
   
"com.vividsolutions.jump.io.datasource.ReaderWriterFileDataSource.write",
   createDescriptiveName(uri)));
 }
-
+long start = Timer.milliSecondsSince(0);
 writer.write(featureCollection, dp);
+Logger.info("Writing '"+UriUtil.getFileName(uri)+"' took 
"+Timer.secondsSinceString(start)+"s.");
   }
 
   @Override



___
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

Re: [JPP-Devel] MakeValidOp

[Michaud Michael]

OK, I understand,Maybe good to have a separate project to test it. I firstly thought a fix on my local disk would be enough, but sharing it on github is a good idea as this class is also used by OrbisGIS project and hopefully, it will make its way to jts (there is already an implementation in geos (or at least postgis) and I'm quite sure that one day or the other, M. Davis will take up the challenge to implement it properly in jts).Michaelenvoyé : 18 août 2020 à 11:21de : Eric à : jump-pilot-devel@lists.sourceforge.netobjet : Re: [JPP-Devel] MakeValidOp Hi Michaël,  The idea behind openjump-migration is initially to test the SVN to Git migration and the JTS update. If we start updating some specific code or creating a test part, it is going to become something else, and it will be probably difficult to reproduce all these steps when/if you decide to go on with this migration.  What I was simply suggested was the creation of a dedicated repository just to test this class (or more if needed), which would benefit from the JTS update (1.17), not its removal from the OJ core. Simply because it is the only case which doesn't seem to be easy to solve, and which requires JTS 1.15+ to be tested  against (contrary to all the other current changes that can be directly committed into the SVN repository, as usual). Once a solution is found, we can integrate it into the final migration repository.  So just a simple 'divide and conquer' idea, without the need to restructure some parts of openjump-migration, at least not for now. Sorry if I've been unclear.  Eric On 18/08/2020 09:18, Michaud Michael wrote:Eric,Not sure I understand your proposition. There is now a function of OpenJUMP-core using this MakeValidOp. If we move it to another project, it will make the build a bit more difficult isn't it ?About tests, you're completely right. I'm often lazy and just push a few tests in the main() of the class. I think it would be quite easy to put these tests in a dedicated test dierctory. Indeed, we already have one named jumptest, but it is not much used. Benjamin Gudehus, a former contributor tried to improve our habits about testing, but there is so much work to be done in this area that nobody followed him...Let's try to improve it a bit. I'll move MakeValidOp tests to the dedicated test area.   Michaëlenvoyé : 18 août 2020 à 09:55 de : Eric  à : jump-pilot-devel@lists.sourceforge.net objet : Re: [JPP-Devel] MakeValidOp Hi Michaël,  Don't be sorry, it was quite good to understand how it works.  The changes I made seem to fix the part of makeValid which tests if the "geometry" dimension (based on CoordinateSequence, not the geometry dimension itself). If the dimension is 4, it calls restoreDim4 as expected. These shouldn't have an impact on the rest.  The other changes are related to the PackedCoordinateSequenceFactory instances, as there isn't any more a constructor which includes the dimension as a parameter.  As it purely JTS dependent, would you like to create another repository for that in openjump-gis or would you like me to do it? This way we could split the code and the tests, and it would be easier to manage than the full project (especially if the tests are written separately, as it isn't yet clearly done in OJ at the moment). Let me know what you think about it.  Eric On 18/08/2020 07:41, Michaud Michael wrote:Hi Eric,Sorry to have let you with this problem.I also checked on my side. Indeed, when I wrote this class, I gave up the idea of preserving the z/m ordinate during the computation because as you noticed, they are lost by some jts algo I need. Instead, I get z/m back at the end of the process.I think that broken tests are not so important because they test intermediate steps (on CoordinateSequence). I suppose they are broken by the many changes done on jts side about z/m management. Trying to understand why these changes broke my test, I was not completely satisfied about how this is managed in jts and I issued 2 tickets in locationtech/jts repo.-> I have a version of the class which compiles and passed the (modified) tests. I can push it.-> I'll review the code depending on how my issues on jts are answered. I also have to check it more extensively to take into account improvements on z/m management recently done in jts. But it can be done later.Michaël  envoyé : 18 août 2020 à 04:29 de : Eric  à : jump-pilot-devel@lists.sourceforge.net objet : Re: [JPP-Devel] Git migration Hi,  No problem.  I didn't encounter any problems to complete the migration locally (removed WFS parts, updated related WFS classes, JTS 1.17 and related code updated, etc.)... except with the class 'com.vividsolutions.jump.geom.MakeValidOp'.  I managed to modify/update this class and it compiles. But then I tried to test if it was working based on the tests already written in the main. Even if the tests can be run, it seems that there is a problem with the 'nodePolygon(Polygon polygon)' method, during the validation of 3D (but also 

Re: [JPP-Devel] MakeValidOp

[Eric]

The migration, as previously discussed (SVN from Git + JTS 1.17), is now 
complete. See:

https://github.com/openjump-gis/openjump-migration/commit/3c24bce2bc6c69d2c786af5c9c0a4737b07666ad

I'm going to try completing the documentation asap, and adding a 
automated build configuration. I'll let you know when it's ready.


Eric

On 18/08/2020 10:21, Eric wrote:

Hi Michaël,

The idea behind openjump-migration is initially to test the SVN to Git 
migration and the JTS update. If we start updating some specific code 
or creating a test part, it is going to become something else, and it 
will be probably difficult to reproduce all these steps when/if you 
decide to go on with this migration.


What I was simply suggested was the creation of a dedicated repository 
just to test this class (or more if needed), which would benefit from 
the JTS update (1.17), not its removal from the OJ core. Simply 
because it is the only case which doesn't seem to be easy to solve, 
and which requires JTS 1.15+ to be tested against (contrary to all the 
other current changes that can be directly committed into the SVN 
repository, as usual). Once a solution is found, we can integrate it 
into the final migration repository.


So just a simple 'divide and conquer' idea, without the need to 
restructure some parts of openjump-migration, at least not for now. 
Sorry if I've been unclear.


Eric

On 18/08/2020 09:18, Michaud Michael wrote:


Eric,

Not sure I understand your proposition. There is now a function of 
OpenJUMP-core using this MakeValidOp. If we move it to another 
project, it will make the build a bit more difficult isn't it ?


About tests, you're completely right. I'm often lazy and just push a 
few tests in the main() of the class. I think it would be quite easy 
to put these tests in a dedicated test dierctory. Indeed, we already 
have one named jumptest, but it is not much used. Benjamin Gudehus, a 
former contributor tried to improve our habits about testing, but 
there is so much work to be done in this area that nobody followed 
him...Let's try to improve it a bit. I'll move MakeValidOp tests to 
the dedicated test area.


Michaël


*envoyé :* 18 août 2020 à 09:55
*de :* Eric 
*à :* jump-pilot-devel@lists.sourceforge.net
*objet :* Re: [JPP-Devel] MakeValidOp


Hi Michaël,

Don't be sorry, it was quite good to understand how it works.

The changes I made seem to fix the part of makeValid which tests if 
the "geometry" dimension (based on CoordinateSequence, not the 
geometry dimension itself). If the dimension is 4, it calls 
restoreDim4 as expected. These shouldn't have an impact on the rest.


The other changes are related to the PackedCoordinateSequenceFactory 
instances, as there isn't any more a constructor which includes the 
dimension as a parameter.


As it purely JTS dependent, would you like to create another 
repository for that in openjump-gis or would you like me to do it? 
This way we could split the code and the tests, and it would be 
easier to manage than the full project (especially if the tests are 
written separately, as it isn't yet clearly done in OJ at the 
moment). Let me know what you think about it.


Eric

On 18/08/2020 07:41, Michaud Michael wrote:


Hi Eric,

Sorry to have let you with this problem.

I also checked on my side. Indeed, when I wrote this class, I gave 
up the idea of preserving the z/m ordinate during the computation 
because as you noticed, they are lost by some jts algo I need. 
Instead, I get z/m back at the end of the process.


I think that broken tests are not so important because they test 
intermediate steps (on CoordinateSequence). I suppose they are 
broken by the many changes done on jts side about z/m management. 
Trying to understand why these changes broke my test, I was not 
completely satisfied about how this is managed in jts and I issued 
2 tickets in locationtech/jts repo.


-> I have a version of the class which compiles and passed the 
(modified) tests. I can push it.


-> I'll review the code depending on how my issues on jts are 
answered. I also have to check it more extensively to take into 
account improvements on z/m management recently done in jts. But it 
can be done later.


Michaël


*envoyé :* 18 août 2020 à 04:29
*de :* Eric  

*à :* jump-pilot-devel@lists.sourceforge.net 


*objet :* Re: [JPP-Devel] Git migration


Hi,

No problem.

I didn't encounter any problems to complete the migration locally 
(removed WFS parts, updated related WFS classes, JTS 1.17 and 
related code updated, etc.)... except with the class 
'com.vividsolutions.jump.geom.MakeValidOp'.


I managed to modify/update this class and it compiles. But then I 
tried to test if it was working based on the tests already written 
in the main. Even if the tests can be run, it seems that there is 
a problem with the 'nodePolygon(Polygon polygon)' method, during 
the validation of 3D (but also 

Re: [JPP-Devel] MakeValidOp

[Eric]

Hi Michaël,

The idea behind openjump-migration is initially to test the SVN to Git 
migration and the JTS update. If we start updating some specific code or 
creating a test part, it is going to become something else, and it will 
be probably difficult to reproduce all these steps when/if you decide to 
go on with this migration.


What I was simply suggested was the creation of a dedicated repository 
just to test this class (or more if needed), which would benefit from 
the JTS update (1.17), not its removal from the OJ core. Simply because 
it is the only case which doesn't seem to be easy to solve, and which 
requires JTS 1.15+ to be tested  against (contrary to all the other 
current changes that can be directly committed into the SVN repository, 
as usual). Once a solution is found, we can integrate it into the final 
migration repository.


So just a simple 'divide and conquer' idea, without the need to 
restructure some parts of openjump-migration, at least not for now. 
Sorry if I've been unclear.


Eric

On 18/08/2020 09:18, Michaud Michael wrote:


Eric,

Not sure I understand your proposition. There is now a function of 
OpenJUMP-core using this MakeValidOp. If we move it to another 
project, it will make the build a bit more difficult isn't it ?


About tests, you're completely right. I'm often lazy and just push a 
few tests in the main() of the class. I think it would be quite easy 
to put these tests in a dedicated test dierctory. Indeed, we already 
have one named jumptest, but it is not much used. Benjamin Gudehus, a 
former contributor tried to improve our habits about testing, but 
there is so much work to be done in this area that nobody followed 
him...Let's try to improve it a bit. I'll move MakeValidOp tests to 
the dedicated test area.


Michaël


*envoyé :* 18 août 2020 à 09:55
*de :* Eric 
*à :* jump-pilot-devel@lists.sourceforge.net
*objet :* Re: [JPP-Devel] MakeValidOp


Hi Michaël,

Don't be sorry, it was quite good to understand how it works.

The changes I made seem to fix the part of makeValid which tests if 
the "geometry" dimension (based on CoordinateSequence, not the 
geometry dimension itself). If the dimension is 4, it calls 
restoreDim4 as expected. These shouldn't have an impact on the rest.


The other changes are related to the PackedCoordinateSequenceFactory 
instances, as there isn't any more a constructor which includes the 
dimension as a parameter.


As it purely JTS dependent, would you like to create another 
repository for that in openjump-gis or would you like me to do it? 
This way we could split the code and the tests, and it would be 
easier to manage than the full project (especially if the tests are 
written separately, as it isn't yet clearly done in OJ at the 
moment). Let me know what you think about it.


Eric

On 18/08/2020 07:41, Michaud Michael wrote:


Hi Eric,

Sorry to have let you with this problem.

I also checked on my side. Indeed, when I wrote this class, I gave 
up the idea of preserving the z/m ordinate during the computation 
because as you noticed, they are lost by some jts algo I need. 
Instead, I get z/m back at the end of the process.


I think that broken tests are not so important because they test 
intermediate steps (on CoordinateSequence). I suppose they are 
broken by the many changes done on jts side about z/m management. 
Trying to understand why these changes broke my test, I was not 
completely satisfied about how this is managed in jts and I issued 2 
tickets in locationtech/jts repo.


-> I have a version of the class which compiles and passed the 
(modified) tests. I can push it.


-> I'll review the code depending on how my issues on jts are 
answered. I also have to check it more extensively to take into 
account improvements on z/m management recently done in jts. But it 
can be done later.


Michaël


*envoyé :* 18 août 2020 à 04:29
*de :* Eric  

*à :* jump-pilot-devel@lists.sourceforge.net 


*objet :* Re: [JPP-Devel] Git migration


Hi,

No problem.

I didn't encounter any problems to complete the migration locally 
(removed WFS parts, updated related WFS classes, JTS 1.17 and 
related code updated, etc.)... except with the class 
'com.vividsolutions.jump.geom.MakeValidOp'.


I managed to modify/update this class and it compiles. But then I 
tried to test if it was working based on the tests already written 
in the main. Even if the tests can be run, it seems that there is a 
problem with the 'nodePolygon(Polygon polygon)' method, during the 
validation of 3D (but also 4D) geometries (i.e. XYZ and XYZM). It 
replaces the Z values by NaN, thus changing 3D coordinate sequences 
into 2D ones. And because the mapping of the measure (M) is based 
on a XYZ coordinate comparison, this measure is also lost for 4D 
geometries. I didn't test yet this method with the current OJ 
version, to see if it is linked to the JTS update or not.

Re: [JPP-Devel] MakeValidOp

[Michaud Michael]

Eric,Not sure I understand your proposition. There is now a function of OpenJUMP-core using this MakeValidOp. If we move it to another project, it will make the build a bit more difficult isn't it ?About tests, you're completely right. I'm often lazy and just push a few tests in the main() of the class. I think it would be quite easy to put these tests in a dedicated test dierctory. Indeed, we already have one named jumptest, but it is not much used. Benjamin Gudehus, a former contributor tried to improve our habits about testing, but there is so much work to be done in this area that nobody followed him...Let's try to improve it a bit. I'll move MakeValidOp tests to the dedicated test area.   Michaëlenvoyé : 18 août 2020 à 09:55de : Eric à : jump-pilot-devel@lists.sourceforge.netobjet : Re: [JPP-Devel] MakeValidOp Hi Michaël,  Don't be sorry, it was quite good to understand how it works.  The changes I made seem to fix the part of makeValid which tests if the "geometry" dimension (based on CoordinateSequence, not the geometry dimension itself). If the dimension is 4, it calls restoreDim4 as expected. These shouldn't have an impact on the rest.  The other changes are related to the PackedCoordinateSequenceFactory instances, as there isn't any more a constructor which includes the dimension as a parameter.  As it purely JTS dependent, would you like to create another repository for that in openjump-gis or would you like me to do it? This way we could split the code and the tests, and it would be easier to manage than the full project (especially if the tests are written separately, as it isn't yet clearly done in OJ at the moment). Let me know what you think about it.  Eric On 18/08/2020 07:41, Michaud Michael wrote:Hi Eric,Sorry to have let you with this problem.I also checked on my side. Indeed, when I wrote this class, I gave up the idea of preserving the z/m ordinate during the computation because as you noticed, they are lost by some jts algo I need. Instead, I get z/m back at the end of the process.I think that broken tests are not so important because they test intermediate steps (on CoordinateSequence). I suppose they are broken by the many changes done on jts side about z/m management. Trying to understand why these changes broke my test, I was not completely satisfied about how this is managed in jts and I issued 2 tickets in locationtech/jts repo.-> I have a version of the class which compiles and passed the (modified) tests. I can push it.-> I'll review the code depending on how my issues on jts are answered. I also have to check it more extensively to take into account improvements on z/m management recently done in jts. But it can be done later.Michaël  envoyé : 18 août 2020 à 04:29 de : Eric  à : jump-pilot-devel@lists.sourceforge.net objet : Re: [JPP-Devel] Git migration Hi,  No problem.  I didn't encounter any problems to complete the migration locally (removed WFS parts, updated related WFS classes, JTS 1.17 and related code updated, etc.)... except with the class 'com.vividsolutions.jump.geom.MakeValidOp'.  I managed to modify/update this class and it compiles. But then I tried to test if it was working based on the tests already written in the main. Even if the tests can be run, it seems that there is a problem with the 'nodePolygon(Polygon polygon)' method, during the validation of 3D (but also 4D) geometries (i.e. XYZ and XYZM). It replaces the Z values by NaN, thus changing 3D coordinate sequences into 2D ones. And because the mapping of the measure (M) is based on a XYZ coordinate comparison, this measure is also lost for 4D geometries. I didn't test yet this method with the current OJ version, to see if it is linked to the JTS update or not.  I don't think that I can quickly solve it. So what I would suggest for now is to leave this problem aside. Then tomorrow / today, I'll remove the extra code I wrote to locate/fix this possible bug, and I'll push the changes I made locally. This way, you'll be able to access the OJ version (with JTS 1.17 but without WFS). If I have time, I'll also try to add a first script to automate the builds, probably using Travis.  Sorry to not have been able to push my local changes today / yesterday but I really tried to see what was happening with the MakeValidOp app.  For info, I also documented the changes I made during this second phase and how I made them (even if the results / diffs can be visible in the commits / logs). I'll add both documentation (svn to git migration and this one) as soon as it is ready.  Eric On 15/08/2020 17:27, Michaud Michael wrote:Thanks for you effort,I pushed a small modification in the pom to test that I can access and compile the project. Could compile after that. Was just a test, don't hesitate if you have to restart the process from scratch.Michaël envoyé : 15 août 2020 à 12:36 de : Eric  à : jump-pilot-devel@lists.sourceforge.net objet : Re: [JPP-Devel] Git migration   Hi Ede,  On 15/08/2020 11:19, 

Re: [JPP-Devel] MakeValidOp

[Eric]

Hi Michaël,

Don't be sorry, it was quite good to understand how it works.

The changes I made seem to fix the part of makeValid which tests if the 
"geometry" dimension (based on CoordinateSequence, not the geometry 
dimension itself). If the dimension is 4, it calls restoreDim4 as 
expected. These shouldn't have an impact on the rest.


The other changes are related to the PackedCoordinateSequenceFactory 
instances, as there isn't any more a constructor which includes the 
dimension as a parameter.


As it purely JTS dependent, would you like to create another repository 
for that in openjump-gis or would you like me to do it? This way we 
could split the code and the tests, and it would be easier to manage 
than the full project (especially if the tests are written separately, 
as it isn't yet clearly done in OJ at the moment). Let me know what you 
think about it.


Eric

On 18/08/2020 07:41, Michaud Michael wrote:


Hi Eric,

Sorry to have let you with this problem.

I also checked on my side. Indeed, when I wrote this class, I gave up 
the idea of preserving the z/m ordinate during the computation because 
as you noticed, they are lost by some jts algo I need. Instead, I get 
z/m back at the end of the process.


I think that broken tests are not so important because they test 
intermediate steps (on CoordinateSequence). I suppose they are broken 
by the many changes done on jts side about z/m management. Trying to 
understand why these changes broke my test, I was not completely 
satisfied about how this is managed in jts and I issued 2 tickets in 
locationtech/jts repo.


-> I have a version of the class which compiles and passed the 
(modified) tests. I can push it.


-> I'll review the code depending on how my issues on jts are 
answered. I also have to check it more extensively to take into 
account improvements on z/m management recently done in jts. But it 
can be done later.


Michaël


*envoyé :* 18 août 2020 à 04:29
*de :* Eric 
*à :* jump-pilot-devel@lists.sourceforge.net
*objet :* Re: [JPP-Devel] Git migration


Hi,

No problem.

I didn't encounter any problems to complete the migration locally 
(removed WFS parts, updated related WFS classes, JTS 1.17 and related 
code updated, etc.)... except with the class 
'com.vividsolutions.jump.geom.MakeValidOp'.


I managed to modify/update this class and it compiles. But then I 
tried to test if it was working based on the tests already written in 
the main. Even if the tests can be run, it seems that there is a 
problem with the 'nodePolygon(Polygon polygon)' method, during the 
validation of 3D (but also 4D) geometries (i.e. XYZ and XYZM). It 
replaces the Z values by NaN, thus changing 3D coordinate sequences 
into 2D ones. And because the mapping of the measure (M) is based on 
a XYZ coordinate comparison, this measure is also lost for 4D 
geometries. I didn't test yet this method with the current OJ 
version, to see if it is linked to the JTS update or not.


I don't think that I can quickly solve it. So what I would suggest 
for now is to leave this problem aside. Then tomorrow / today, I'll 
remove the extra code I wrote to locate/fix this possible bug, and 
I'll push the changes I made locally. This way, you'll be able to 
access the OJ version (with JTS 1.17 but without WFS). If I have 
time, I'll also try to add a first script to automate the builds, 
probably using Travis.


Sorry to not have been able to push my local changes today / 
yesterday but I really tried to see what was happening with the 
MakeValidOp app.


For info, I also documented the changes I made during this second 
phase and how I made them (even if the results / diffs can be visible 
in the commits / logs). I'll add both documentation (svn to git 
migration and this one) as soon as it is ready.


Eric

On 15/08/2020 17:27, Michaud Michael wrote:


Thanks for you effort,

I pushed a small modification in the pom to test that I can access 
and compile the project. Could compile after that.


Was just a test, don't hesitate if you have to restart the process 
from scratch.


Michaël


envoyé : 15 août 2020 à 12:36
de : Eric  

à : jump-pilot-devel@lists.sourceforge.net 


objet : Re: [JPP-Devel] Git migration


Hi Ede,

On 15/08/2020 11:19, edgar.sol...@web.de 
 wrote:



On 15.08.2020 12:07, Eric wrote:


>> Hi all,>>
>> After 5-6 hours, the result of the migration is finally 
complete: https://github.com/openjump-gis/openjump-migration

>>
>> It includes the commit history from revision 859 to 6242.

just checked, we'll lose some commits to the source this way. did 
you check if you could svn2git from rev.1 ? did it err out?


I'll try it next time from revision 1 as it would be a bit long to 
do it

again just now (it needs to run at night), and unnecessary within the
context of the JTS update.

>> I reinitialised the repository 'openjump-migration' to 

[JPP-Devel] MakeValidOp

[Michaud Michael]

Hi Eric,Sorry to have let you with this problem.I also checked on my side. Indeed, when I wrote this class, I gave up the idea of preserving the z/m ordinate during the computation because as you noticed, they are lost by some jts algo I need. Instead, I get z/m back at the end of the process.I think that broken tests are not so important because they test intermediate steps (on CoordinateSequence). I suppose they are broken by the many changes done on jts side about z/m management. Trying to understand why these changes broke my test, I was not completely satisfied about how this is managed in jts and I issued 2 tickets in locationtech/jts repo.-> I have a version of the class which compiles and passed the (modified) tests. I can push it.-> I'll review the code depending on how my issues on jts are answered. I also have to check it more extensively to take into account improvements on z/m management recently done in jts. But it can be done later.Michaël  envoyé : 18 août 2020 à 04:29de : Eric à : jump-pilot-devel@lists.sourceforge.netobjet : Re: [JPP-Devel] Git migration Hi,  No problem.  I didn't encounter any problems to complete the migration locally (removed WFS parts, updated related WFS classes, JTS 1.17 and related code updated, etc.)... except with the class 'com.vividsolutions.jump.geom.MakeValidOp'.  I managed to modify/update this class and it compiles. But then I tried to test if it was working based on the tests already written in the main. Even if the tests can be run, it seems that there is a problem with the 'nodePolygon(Polygon polygon)' method, during the validation of 3D (but also 4D) geometries (i.e. XYZ and XYZM). It replaces the Z values by NaN, thus changing 3D coordinate sequences into 2D ones. And because the mapping of the measure (M) is based on a XYZ coordinate comparison, this measure is also lost for 4D geometries. I didn't test yet this method with the current OJ version, to see if it is linked to the JTS update or not.  I don't think that I can quickly solve it. So what I would suggest for now is to leave this problem aside. Then tomorrow / today, I'll remove the extra code I wrote to locate/fix this possible bug, and I'll push the changes I made locally. This way, you'll be able to access the OJ version (with JTS 1.17 but without WFS). If I have time, I'll also try to add a first script to automate the builds, probably using Travis.  Sorry to not have been able to push my local changes today / yesterday but I really tried to see what was happening with the MakeValidOp app.  For info, I also documented the changes I made during this second phase and how I made them (even if the results / diffs can be visible in the commits / logs). I'll add both documentation (svn to git migration and this one) as soon as it is ready.  Eric On 15/08/2020 17:27, Michaud Michael wrote:Thanks for you effort,I pushed a small modification in the pom to test that I can access and compile the project. Could compile after that. Was just a test, don't hesitate if you have to restart the process from scratch.Michaël envoyé : 15 août 2020 à 12:36 de : Eric  à : jump-pilot-devel@lists.sourceforge.net objet : Re: [JPP-Devel] Git migration   Hi Ede,  On 15/08/2020 11:19, edgar.sol...@web.de wrote:On 15.08.2020 12:07, Eric wrote:>> Hi all,>> >> After 5-6 hours, the result of the migration is finally complete: https://github.com/openjump-gis/openjump-migration >> >> It includes the commit history from revision 859 to 6242.just checked, we'll lose some commits to the source this way. did you check if you could svn2git from rev.1 ? did it err out?I'll try it next time from revision 1 as it would be a bit long to do it  again just now (it needs to run at night), and unnecessary within the  context of the JTS update.  >> I reinitialised the repository 'openjump-migration' to make easier this initial import, i.e. I deleted it and recreated a new one with the same name, but empty this time. >> >> I'm now going to add a couple of files (gitignore, licence, readme, etc.) then I'll delete the WFS part as discussed, update the Maven configuration, update JTS and all related classes.sounds good. remember to write down the steps, so we can recreate it in case we want/need to.Don't worry, it is well documented. It also explains the reasons behind  some of the choices.  I just added a first gitignore file and the licence. I'm writing a  readme right now, then I'll convert the documentation about the  migration from txt to md. One step at a time.  Ericthanks ..ede   ___ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel  ___ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel ___
Jump-pilot-devel mailing list