Hi,
yesterday I've got confirmation there are some big bad locks
in the Java rendering subsystem that do prevent a system
to use a multicore architecture effectively:
http://jira.codehaus.org/browse/GEOS-3423
In fact I have been observing CPU utization well below
100% in various mulithreaded benchmarks where the
number of client threads far exceeded the number
of available CPUs... those locks that explains it,
and I had confirmation using Yourkit.
So today I set up to try and have the renderer do something
in the time it cannot render in parallel: try to prefetch
data from the store while rendering is happening.
I've then run again some benchmarks, with mixed results.
StreamingRenderer + shapefile datastore gets quite a benefit
and becomes in some cases 30% faster than the shapefile
renderer itself, in some others 10% slower (but the
datastore is not optimized to work with the streaming
renderer, so I'm confident it can get faster in 100% of
the cases with some more work).
StreamingRendere + postgis datastore is around +10/-10
depending on the workload.
I would have thought to always get better performance
on lighter loads, considering the second core of my
machine could have helped by doing the parallel data
fetching, but that did not always happen (it happened
big time only in one tests, where the speed at light
load almost doubled).
Maybe there is something wrong with the way I've
coded up the prefecthin iterator? Someone more familiar
with the concurrent java utilitis might be able to tell.
More in general, how do people feel about this?
Good for a configuration knob in the StreamingRenderer?
Or just the new default?
Cheers
Andrea
--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.
package org.geotools.renderer.lite;
import java.util.Iterator;
import java.util.NoSuchElementException;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.geotools.feature.FeatureCollection;
public class PrefetchingIterator implements Iterator {
static final ExecutorService pool = Executors.newCachedThreadPool();
BlockingQueue<ValueWrapper> queue;
FeatureCollection collection;
ValueWrapper current;
boolean end = false;
public PrefetchingIterator(FeatureCollection collection) {
this.collection = collection;
queue = new ArrayBlockingQueue<ValueWrapper>(100);
pool.execute(new Prefetcher());
}
public boolean hasNext() {
if(end)
return false;
if(current == null) {
try {
current = queue.take();
} catch(InterruptedException e) {
// nothing to do
}
if(current.end) {
end = true;
return false;
} else if(current.exception != null) {
end = true;
throw new RuntimeException("Error occurred during feature prefetching", current.exception);
} else {
return true;
}
} else {
return true;
}
}
public Object next() {
if(!hasNext())
throw new NoSuchElementException();
Object result = current.value;
current = null;
return result;
}
public void close() {
end = true;
queue = null;
}
public void remove() {
throw new UnsupportedOperationException();
}
final class Prefetcher implements Runnable {
public void run() {
Iterator iterator = null;
try {
iterator = collection.iterator();
while(iterator.hasNext()) {
if(end)
return;
putInQueue(new ValueWrapper(iterator.next()));
}
} catch(Exception e) {
putInQueue(new ValueWrapper(e));
} finally {
collection.close(iterator);
}
putInQueue(new ValueWrapper());
}
void putInQueue(ValueWrapper o) {
try {
queue.put(o);
} catch(InterruptedException e) {
// nothing to do
}
}
}
final class ValueWrapper {
Object value;
Exception exception;
boolean end;
public ValueWrapper(Exception e) {
this.exception = e;
}
public ValueWrapper(Object value) {
this.value = value;
}
public ValueWrapper() {
this.end = true;
}
}
}
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Geotools-devel mailing list
Geotools-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
