On 06/09/11 00:17, Felix Hartmann wrote:
could it be that the index starts to be worked one, when the first of
the 4 worker threads (to compile the maps) is finished, with 3 still
working and hence overshooting the memory limit?? I really cannot see
why it would have failed me 4 times in a row on a single call, but
single index creation allways worked...
Maybe - if something else is failing while tiles are being generated.
If you apply the attached patch and run, it will print out how much
memory is available, what threads are running and version information
just before it starts the combining step.
On my machine it looks like this:
About to start combine steps
Thread[Signal Dispatcher,9,system] RUNNABLE
Thread[Reference Handler,10,system] WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread[main,5,main] RUNNABLE
at java.lang.Thread.dumpThreads(Native Method)
at java.lang.Thread.getAllStackTraces(Thread.java:1530)
at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:423)
at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)
at uk.me.parabola.mkgmap.main.Main.main(Main.java:132)
Thread[Finalizer,8,system] WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Memory 2M 105M 881M
Java Version 1.6.0_22 Sun Microsystems Inc., on Linux
2.6.40.3-0.fc15.x86_64
I do have some ideas for some reduction in the memory use by the index
generation.
Well that would be great. It is currently using much more memory than
I looked into this a bit more and found that the simple ideas wouldn't
save much memory unfortunately.
Best would even be if on single index creation, one could specify an mdr
file for existing maps, and just add stuff for maps that you want to add
to the compilation (use case, user has a map of Germany in several .img
with mdr/mdx and wants to add contourlines into the mapset, it would be
great if the index of the Germany map could be given, and the
contourline maps just added to the mdr without needing to regenerate the
full mdr -- don't know the format well enough, but would hope that this
is possible..).
It would be possible, but the existing file would have to be read so
that the new information could be merged in the correct sorted order.
It would be a fair amount of work and would use a similar amount of
memory as doing the whole lot.
..Steve
Index: src/uk/me/parabola/mkgmap/main/Main.java
===================================================================
--- src/uk/me/parabola/mkgmap/main/Main.java (revision 2023)
+++ src/uk/me/parabola/mkgmap/main/Main.java (revision )
@@ -219,6 +219,7 @@
FilenameTask task = new FilenameTask(new Callable<String>() {
public String call() {
log.threadTag(filename);
+ System.out.println("map " + filename);
String output = mp.makeMap(args, filename);
log.debug("adding output name", output);
log.threadTag(null);
@@ -414,6 +415,35 @@
args.setSort(getSort(args));
+ System.gc();
+ try {
+ Thread.sleep(5 * 1000);
+
+ System.out.println("About to start combine steps");
+ Map<Thread, StackTraceElement[]> allStackTraces = Thread.getAllStackTraces();
+ for (Map.Entry<Thread, StackTraceElement[]> ent : allStackTraces.entrySet()) {
+ StackTraceElement[] traceElements = ent.getValue();
+ Thread thread = ent.getKey();
+
+ System.out.println(thread + " " + thread.getState());
+ for (StackTraceElement el : traceElements) {
+ System.out.println(" at " + el);
+ }
+ }
+
+ Runtime runtime = Runtime.getRuntime();
+ int MB = 1024*1024;
+ System.out.printf("Memory %dM %dM %dM\n", (runtime.totalMemory() - runtime.freeMemory()) / MB,
+ runtime.totalMemory() / MB,
+ runtime.maxMemory() / MB
+ );
+ System.out.printf("Java Version %s %s, on %s %s\n", System.getProperty("java.version"),
+ System.getProperty("java.vendor"),
+ System.getProperty("os.name"), System.getProperty("os.version"));
+ } catch (InterruptedException e) {
+ e.printStackTrace();
+ }
+
// Get them all set up.
for (Combiner c : combiners)
c.init(args);
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
