Hi there,
I've been following this thread with quite some interest.
I like the idea of one monitoring thread that checks memory
and then calls on the different stores to free some up if
necessary.
As an attachment I included a (very rough, first version, pre-alpha...
you get it;-) StoreJanitor.
To get this to run in your environment do the following:
1) add a new role named "janitor" to org/apache/cocoon/cocoon.roles
2) add an <janitor> element to cocoon.xconf,
similar to the MRUMemoryStore ones
3) Expand the Store interface by a "public void free()" method.
4) Add the Store to the Janitor surveillance, i.e.:
StoreJanitor janitor = (StoreJanitor)manager.lookup(StoreJanitor.ROLE);
janitor.addToSurveillance(this);
As mentioned above, this is far from complete. I'm sending this in
so people that are more knowledgable than I am can take a look at
it.
I'd appreciate any (rocks!, sucks) feedback.
Thanks,
Christian
On Wed, Aug 29, 2001 at 12:18:46AM +0200, Gerhard Froehlich wrote:
> >> >Wow, where have you been before? :)))
> Always there watching and learning much through this project and its
> participants :). But most of the time in the dungeons of commercial
> software development and its deadlines.
> >> >Yes, you are right - that's not necessary and this might be the
> >> >performance issue.
> >> >Reducing checker threads amount down to one would decrease amount of
> >> >runFinalization()/gc()
> >> >calls in two times - giving better performance.
> >> >
> >> >Could we use just one static checker thread?
> We have to test this. When static, we have namely one cleanup thread but
> still two stores to free(). For the moment I don't know if this work.
> But I have the strong feeling that it won't. Too bad that Carsten is
> away for a couple of weeks. He would certainly have some good ideas for
> the pipline configuration. But I will test this with a static thread.
> >> >There is an issue with configuration though... Which
> >configuration to use,
> >> >from what store...
> Maybe we can add a sample configuration to the cocoon.xconf according
> to a reference config for the Xms Xmx parameters. Berin made some
> good suggestions some threads below. I can add it to the cocoon.xconf.
> >> How about a separate component to do the checking?
> >(MRUMemoryStoreJanitor?)
> >> Each MRUMemoryStore would register its requested heapsize &
> >freememory with
> >> the janitor, and it would be responsible for the worker thread. It would
> >
> >Good suggestion and I kind of like this name ;)
> >Gerhard, what do you think?
> Good suggestion. But I would name it StoreJanitor. Because the
> MRUMemoryStore
> is *only* a component. Maybe there will be a LRUMemoryStore or combination
> of
> both in nearer future :). Every store component implements the store
> interface, so
> the StoreJanitor would have access to the free() method of every store
> component.
> Then we would have a seperate cleanup component for every store
> implementation.
> I would implement the stores as singelton classes with one instance for
> better
> control about the memory. Because, how do you decide from which store
> instance
> to free memory. Maybe there is one with many small objects and another with
> less
> but huge objects. You can't determine, like in C, the byte size of each
> object in java easy.
> >> try to keep the JVM totalMemory below the sum of all the registered
> >> heapsizes, and freeMemory below the sum of all registered freememories.
> >Here I think that idea (originally) was that heapsize specifies
> >heapsize of the JVM,
> >but not the size of the store (because it is (almost) impossible
> >to determine), so my
> >understanding is that these parameters (heapsize and freememory)
> >should also
> >go into janitor configuration.
> Yeah, thats right. But I had the opinion that there is only one store
> instance.
> (I tested it too carelessly :( )
> This gc()->jvm.FreeMemory()->gc()->jvm.FreeMemory() is the proposed
> implementation (hack) from javasoft to determine memory consumption
> of objects in java.
> >> If runFinalization/gc isn't enough to create the free memory, a
> >round-robin
> >> approach (or other algorithm) could be used to select a
> >MRUMemoryStore to
> >> free an object from.
> Yes, that would be a possible approach to decide from which store to free().
> But
> I still don't understand why 2 or more stores?
> >> Just my .02 :)
> Mine too :)
>
> Cheers
> Gerhard
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, email: [EMAIL PROTECTED]
>
/*****************************************************************************
* Copyright (C) The Apache Software Foundation. All rights reserved. *
* ------------------------------------------------------------------------- *
* This software is published under the terms of the Apache Software License *
* version 1.1, a copy of which has been included with this distribution in *
* the LICENSE file. *
*****************************************************************************/
package org.apache.cocoon.components.store;
import org.apache.avalon.framework.component.Component;
import org.apache.avalon.framework.configuration.Configurable;
import org.apache.avalon.framework.configuration.Configuration;
import org.apache.avalon.framework.configuration.ConfigurationException;
import org.apache.avalon.framework.parameters.Parameters;
import org.apache.avalon.framework.logger.AbstractLoggable;
import org.apache.avalon.framework.thread.ThreadSafe;
import java.util.List;
import java.util.ArrayList;
import java.util.Collections;
import java.util.ListIterator;
/**
* This class monitors memory usage of thr JVM.
* It calls the <code>free()</code> method in <code>Store</code>s
* when memory gets low to free up some.
*
* @author <a href="mailto:[EMAIL PROTECTED]";>Gerhard Froehlich</a>
* @author <a href="mailto:[EMAIL PROTECTED]";>Davanum Srinivas</a>
* @author <a href="mailto:[EMAIL PROTECTED]";>Christian Schmitt</a>
*
*/
public class StoreJanitor extends AbstractLoggable
implements Component, Configurable, Runnable, ThreadSafe {
/**
* The Avalon role name
*/
final static String ROLE = "org.apache.cocoon.components.store.StoreJanitor";
/**
* Indicates how much memory should be left free in the JVM for
* normal operation.
*/
private int freememory;
/**
* Indicates how big the heap size can grow to before the cleanup thread kicks in.
* The default value is based on the default maximum heap size of 64Mb.
*/
private int heapsize;
/**
* Indicates the time in seconds to sleep between memory checks.
*/
private long cleanupthreadinterval;
/**
* Indicates the daemon thread priority.
*/
private int priority;
/**
* Store of stores
*/
private List stores = Collections.synchronizedList(new ArrayList());
/**
* The Runtime
*/
private Runtime jvm;
public void configure(Configuration config) throws ConfigurationException {
getLogger().debug("About to be configured!");
Parameters params = Parameters.fromConfiguration(config);
this.freememory = params.getParameterAsInteger("freememory", 1000000);
this.heapsize = params.getParameterAsInteger("heapsize", 60000000);
this.cleanupthreadinterval = params.getParameterAsInteger("cleanupthreadinterval", 10);
this.priority = params.getParameterAsInteger("threadpriority",Thread.currentThread().getPriority());
if ((this.priority < 1) || (this.priority > 10)) {
throw new ConfigurationException("StoreJanitor cleanup thread priority must be between 1 and 10!");
}
if ((this.cleanupthreadinterval < 1)) {
throw new ConfigurationException("StoreJanitor cleanup thread interval must be at least 1 second!");
}
Thread janitor = new Thread(this);
janitor.setDaemon(true);
janitor.setPriority(this.priority);
janitor.setName("janitor");
janitor.start();
}
/**
* The background thread that monitors memory.
* If memory is getting low, appropriate actions have to be taken.
*/
public void run() {
while(true) {
if (memoryLow()) {
ListIterator li = stores.listIterator();
while (li.hasNext()) {
Store s = (Store)li.next();
getLogger().debug("Calling " + s.toString() + " to free some memory");
s.free();
}
}
try {
Thread.currentThread().sleep(this.cleanupthreadinterval * 1000);
} catch (InterruptedException ignore) {}
}
}
/**
* Add a Store to be watched.
*/
public void addToSurveillance(Store store) {
stores.add(store);
getLogger().debug(store.toString() + " added to surveillance");
}
/**
* Remove a Store from being watched.
*/
public void removeFromSurveillance(Store store) {
stores.remove(store);
getLogger().debug(store.toString() + " removed from surveillance");
}
/**
* Method to check if memory is running low in the jvm.
*/
private boolean memoryLow() {
return jvm.totalMemory() > heapsize && jvm.freeMemory() < freememory;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]
