Hi Guys, I think I've been leading myself up the garden path a bit here. After some more in-depth analysis I've noticed that this is directly related to mem usage and garbage collection. It would seem that the store janitor and GC is kicking in at about the same time (which makes sense.)
The iowait is caused by a huge amount of memory being swapped out to disk. We're going to try with incremental GC turned on. What is everyone else doing as far as GC is concerned? What is recognised as optimal store janitor / gc config? I suspect that this may be why so many users have reported poor performance when using the persistent store..... Of course, it's all anecdotal. What I'd love to be able to do out of this research is produce a user's guide to configuring these various components. I think a document like this should really form part of a release...... Corin -----Original Message----- From: Corin Moss Sent: Thursday, 11 March 2004 3:55 p.m. To: [EMAIL PROTECTED] Subject: JCS Store - new issues Hi Guys, You may or may not have seen my recent posts about performance with JCS store, but I've gotten a little further into diagnosing what looks like a deadlock issue. After about 30mins of load testing I'm seeing everything lock up (iowait of ~ 100%) the error log is getting this message repeatedly: ERROR (2004-03-11) 15:24.32:587 [core.store.janitor] (Unknown-URI) Unknown-thread/StoreJanitorImpl: Error in freeMemory() java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.RangeCheck(ArrayList.java:507) at java.util.ArrayList.get(ArrayList.java:324) at org.apache.excalibur.store.impl.StoreJanitorImpl.freeMemory(StoreJanitor Impl.java:383) at org.apache.excalibur.store.impl.StoreJanitorImpl.run(StoreJanitorImpl.ja va:202) at java.lang.Thread.run(Thread.java:534) Obviously it's getting it every cycle of the Store Janitor. I'm currently a little confused about it - it seems to me that the only reason this error could be thrown is if the ArrayList which contains the available stores is empty. Is this the case? If it is, then the next question is: what causes the JCS store to fall out of the store list? Would this happen if JCS threw a fatal exception? I suspect it would, but I'm not 100% certain. I have a feeling I know the particular part of JCS that causes the problem. It seems to me that the R-W lock mechanism they use is not quite perfect under load. So - anyone have any more info on the questions above? Thanks, Corin Corin Moss Lead Developer TVNZ Interactive +64 9 916 7367 +64 21 403 054 [EMAIL PROTECTED] ================================================================ CAUTION: This e-mail and any attachment(s) contains information that is intended to be read only by the named recipient(s). It may contain information that is confidential, proprietary or the subject of legal privilege. This information is not to be used by any other person and/or organisation. If you are not the intended recipient, please advise us immediately and delete this e-mail from your system. Do not use any information contained in it. ================================================================ For more information on the Television New Zealand Group, visit us online at http://www.tvnz.co.nz ================================================================ ================================================================ CAUTION: This e-mail and any attachment(s) contains information that is intended to be read only by the named recipient(s). It may contain information that is confidential, proprietary or the subject of legal privilege. This information is not to be used by any other person and/or organisation. If you are not the intended recipient, please advise us immediately and delete this e-mail from your system. Do not use any information contained in it. ================================================================ For more information on the Television New Zealand Group, visit us online at http://www.tvnz.co.nz ================================================================