Hi Gerd
Since downloading loading britain-and-ireland-latest.osm.pbf I had been
unable to build a gmapsupp because of running out of heap (my hardware
is 32 bit, -Xmx1540M is largest value allowed)
My problem is mainly because I have 1731146 cities (along with 1046096
streets)
Looking at Mdr5 processing, I've changed it in 3 ways to improve memory
usage and garbage collection.
1/ use trimToSize() after all the cities are loaded from the individual
tile .img. I presume that the growth factor gradually increases as it
runs out of allocated array space. I had to change the declaration from
List<Mdr5Record> to ArrayList<Mdr5Record> to allow this, but I can't
see any problem in this.
2/ Move the main part of preWriteImpl into its own method so the first
sortKeys ArrayList and Sort can be freed before calcMdr20/1/2() each
create another massive SortKeys and Sort.
3/ Move the scope of mdr20s to a class variable. This is referenced by
all the Mdr5Records and the scope of where it was declared before
seemed to to cause the garbage collector major problems - it churned
for 5 mins using all the processors before running out of memory. After
moving it, the whole of mdr is built in a couple of mins with cpu usage
mostly < 125%.
Ticker
Index: src/uk/me/parabola/imgfmt/app/mdr/Mdr5.java
===================================================================
--- src/uk/me/parabola/imgfmt/app/mdr/Mdr5.java (revision 4701)
+++ src/uk/me/parabola/imgfmt/app/mdr/Mdr5.java (working copy)
@@ -32,12 +32,16 @@
* @author Steve Ratcliffe
*/
public class Mdr5 extends MdrMapSection {
- private List<Mdr5Record> allCities = new ArrayList<>();
- private List<Mdr5Record> cities = new ArrayList<>();
+ private ArrayList<Mdr5Record> allCities = new ArrayList<>();
+ private ArrayList<Mdr5Record> cities = new ArrayList<>();
private int maxCityIndex;
private int localCitySize;
private int mdr20PointerSize = 0; // bytes for mdr20 pointer, or 0 if no mdr20
+ // We need a common area to save the mdr20 values, since there can be multiple
+ // city records with the same global city index
+ private int[] mdr20s;
+
public Mdr5(MdrConfig config) {
setConfig(config);
}
@@ -54,8 +58,16 @@
*/
@Override
public void preWriteImpl() {
+ allCities.trimToSize();
localCitySize = Utils.numberToPointerSize(maxCityIndex + 1);
+ genCitiesAndMdr20s();
+ // calculate positions for the different road indexes
+ calcMdr20SortPos();
+ calcMdr21SortPos();
+ calcMdr22SortPos();
+ }
+ private void genCitiesAndMdr20s() {
List<SortKey<Mdr5Record>> sortKeys = new ArrayList<>(allCities.size());
Sort sort = getConfig().getSort();
for (Mdr5Record m : allCities) {
@@ -76,9 +88,7 @@
int count = 0;
Mdr5Record lastCity = null;
- // We need a common area to save the mdr20 values, since there can be multiple
- // city records with the same global city index
- int[] mdr20s = new int[sortKeys.size()+1];
+ mdr20s = new int[sortKeys.size()+1];
int mdr20count = 0;
for (SortKey<Mdr5Record> key : sortKeys) {
@@ -99,12 +109,8 @@
lastCity = c;
}
}
- // calculate positions for the different road indexes
- calcMdr20SortPos();
- calcMdr21SortPos();
- calcMdr22SortPos();
}
-
+
/**
* Calculate a position when sorting by name, region, and country- This is used for MDR20.
*/
@@ -287,6 +293,7 @@
protected void releaseMemory() {
allCities = null;
cities = null;
+ mdr20s = null;
}
/**
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
