Ard Schrijvers wrote:
> Before I say things that are wrong, please consider that the StoreJanitor was
invented long before I looked into the cocoon code, so probably a lot of
discussion and good ideas has been around which I am not aware of. But still, my
ideas about the StoreJanitor (and sorry for the long mail, but perhaps it might
contain something useful):
>
> 1) How it works and its intention (I think :-) ): The StoreJanitor is
originally invented to monitor cocoon's memory useage and does this by checking
some memory values every X (default 10) seconds. Beside the fact that I doubt
users know that it is quite important to configure the store janitor correctly,
I stick to the defaults and use a heapsize of just a little lower then JVM
maxmemory.
>
> Now, every 10 seconds, the StoreJanitor does a check wether
(getJVM().totalMemory() >= getMaxHeapSize() && (getJVM().freeMemory() <
getMinFreeMemory()) is true, and if so, the next store is choosen (compared to
previoud one) and entries are removed from this store (I saw a post that in
trunk not one single store is chosen anymore, but an equal part of all of them
is being removed, right?...probably you can configure which stores to use, i
don't know)
AFAICS there are two freeing algorithms in trunk: round-robin and all-stores.
> 2) My Observations: When running high traffic sites and render them live
(only mod_cache in between which holds pages for 5 to 10 min) like [1] or [2],
then checking every X sec for a JVM to be low on memory doesn't make sense to
me. At the moment of checking, the JVM might be perfectly sound but just needed
some extra memory for a moment, in that case, the Store Janitor is removing
items from cache while not needed. Also, when the JVM is really in trouble, but
the Store Janitor is not checking for 5 more sec....this might be too long for a
JVM in a high traffic site when it is low on memory. Problems that result from
it are:
>
> - Since there is no way to remove cache entries from the used cache impl by
the cache's eviction policy, the cache entries from memory are removed by
starting from entry 0, whatever this might be in the cache. There is a very
likely situation, that at the very next request, the same cache entries are
added again.
>
> - Ones the JVM gets low on memory, and the StoreJanitor is needed, it is
quite likely that from that moment on, the StoreJanitor runs *every* 10 seconds,
and keeps removing cache entries which you perhaps don't want to be removed,
like compiled stylesheets.
yep, that's a problem
> 1) suppose, from one store (or since trunk from multiple stores) 10%
(default) is removed. This 10% is from the number of memory cache entries. I
quite frequently happen to have only 200 entries in memory for each store ( I
have added *many* different stores to enable all we wanted in a high traffic
environment) and the rest is disk store. Now, suppose, the JVM which has 512 Mb
of memory, is low on memory, and removes 10% of 200 entries = 20 entries,
helping me zero!
agreed
> These memory entries are my most important ones, so, on the next request,
they are either added again, or, from diskcache I have a hit, implying that the
cache will put this cache entry in memory again. If I would use 2000 memory
items, I am very sure, the 200 items which are cleaned are put back in memory
before the next StoreJanitor runs.
> 2) I am not sure if in trunk you can configure wether the StoreJanitor
should leave one store alone, like the DefaultTransientStore. In this store,
typically, compiled stylesheets end up, and i18n resource bundles. Since these
files are needed virtually on every request, I had rather not that the
StoreJanitor removes from this store. I think, the StoreJanitor does so, leaving
my "critical app" in an even worse state, and on the next request, the hardly
improved JVM needs to recompile stylesheets and i18n resource bundles.
agreed
> 3) What if the JVM being low is not because of the stores....For example,
you have added some component which has some problems you did not know, and,
that component is the real reason for you OOM. The StoreJanitor, sees your low
memory, and starts removing entries from your perfectly sound cache, leaving you
app in a much worse situation then it already was. Your component with memory
leak has some more memory it now can fill, and hapily does this, making the
StoreJanitor remove more and more entries from cache, untill it ends up with an
empty cache. You could blame the wrong component for this behavior. One of these
wrong components in use is the event registry for event caching, which made our
high traffic sites with 512 Mb crash every two days. Better that I write in
another mail what I did to the event cache registry, why I did not yet post
about it, and if others are interested and how to include it in the trunk.
Bottom line is that there was a major OOM problem if the registry grows,
resulting in a StoreJanitor removing cache entries while this was actually
increasing the problem.
> 4) By default, probably most people are using ehcache. Naturally,
overflow-to-disk is true. In a high traffic site, the number of cache keys can
grow enormously (I have seen mails around people complaing about disk cached
growing to multiple Gbytes). Certainly, when the not very experienced user uses
something like a session attr (or timestamp and many more possibilities) in a
stylesheet parameter which ends up in the cache key (but perhaps, should cocoon
be the target for high traffic sites for the average user, I don't know). Now,
and this is IMO one of the major weakenesses of ehcache (or I missed it
completely), I did not find any way to limit the number of disk store entries.
Actually we don't configure this value. According to
http://ehcache.sourceforge.net/documentation/configuration.html the default
value is 0 meaning unlimited. We should use the 1.2.4 constructor that allows to
set a maxElementsOnDisk parameter.
> This implies, that the disk store can grow indefinitely. For the ones ever
looking at the status page, cache keys in memory of about 2 kb are quite common
in cocoon (actually, the dept of the folder structure of your app is of
influence). The disk store cache keys are kept in *memory*. So, suppose, you run
your app with 128 Mb, and you have overflow-to-disk=true, your app runs into
problem when there are about 50.000 keys in cache. Then your StoreJanitor keep
removing entries from your memory cache, which are refilled with disk store
entries just a few moments later. Now, if you really know how to configure your
stores, you use a time2liveSeconds and time2IdleSeconds to let your store clear
unused cache entries. This is good to do, unless, you depend on something like
an event registry which is currently in cocoon trunk. The problem is, that the
StoreJanitor removes cache entries by calling the free from the correct store,
which, might for example be the eventaware store. This event aware store,
updates (cleans) its registry before removing the cache entry from its delegate.
Now, when you use the internal cleaning of caches by a time2liveSeconds or
time2IdleSeconds, the event registry is not cleaned and will lead to OOM in the
long run.
>
> I have more things about it, but probably nobody will read it anymore, but in
short, my conclusion is that the StoreJanitor never helped me out, but merely
impoverished my app when it ran
I wonder what StoreJanitor is good for at all. EHCache takes care that the
number of items in the memory cache doesn't grow indefinitly and starts its own
cleanup threads for the disc store
(http://ehcache.sourceforge.net/documentation/storage_options.html#DiskStore).
JCS will probably do the same. I guess that original purpose of StoreJanitor was
when Cocoon had its own store implementations (transient, persistent) and we had
to take care of cleaning them up in our code.
Only the persistent store can grow unlimited but since it should only be used
for special usecases, it shouldn't be a real problem.
>
> --------o0o--------
>
> The rules I try to follow to avoid the Store Janitor to run
>
> 1) use readers in noncaching pipelines and use expires on them to avoid
cache/memory polution
> 2) use a different store for repository binary sources which has only a disk
store part and no memory part (cached-binary: protocol added)
> 3) use a different store for repository sources then for pipeline cache
> 4) replaced the abstract double mapping event registry to use weakreferences
and let the JVM clean up my event registry
> 5) (4) gave me undesired behavior by removing weakrefs in combination with
ehcache when overflowing items to disk (i could not reproduce this, but seems
that my references to cachekeys got lost). Testing with JCSCache solved this
problem, gave me faster response times and gave me for free to limit the number
of disk cache entries. Disadvantage of the weakreferences, is that I disabled
persitstent caches for jvm restarts, but, I never wanted this anyway (but this
might be implemented quite easily, but might take long start up times)
> 6) JCSCache has a complex configuration IMO. Therefor, I added default
configurations to choose from, for example:
> <store logger="core.store">
> <parameter name="region-name" value="store"/>
> <parameter name="size" value="small"/>
>
> where size might be small, medium, large or huge.
>
> I think we have created with in this way a setup for cocoon, where it is
harder for unexperienced users to have memory > problems when trying to
implement larger sites.
>
> Hopefully somebody read my mail until here :-) I am curious about what
others think,
> [1] http://www.minfin.nl
> [2] http://www.minbuza.nl
What do we want to do in order to improve the situation? After reading your mail
and from my own experience I'd say
- introduce a maxPersistentObjects parameter and use it in EHDefaultCache to
set maxElementsOnDisk
- make the registration of stores at StoreJanitor configureable
(Though I wonder what the default value should be, true or false?)
- fix EventRegistry
Any further ideas?
P.S. Ard, answering to your mails is very difficult because there are no line
breaks. Is anybody else experiencing the same problem or is it only me?
--
Reinhard Pötz Independent Consultant, Trainer & (IT)-Coach
{Software Engineering, Open Source, Web Applications, Apache Cocoon}
web(log): http://www.poetz.cc
--------------------------------------------------------------------
