Aaron,
As far as I know the deadlock was not recoverable and did not occur when iterating the local memory cache but rather on put operations triggered by a master/remotecache update.
The problem once it occurred stopped all updates occurring I think it was blocking the local puts triggered from the mastercache by holding a lock.
I agree automated tests are difficult, I tried, it required high load from a stress tool difficult to setup in an automated test.
On an unrelated note, we tried the threadpool timeout code 1.1.3dev yesterday. It seemed to work in testing but once we staged to production it turned out that deletions were no longer getting propagated from the remote/mastercache resulting in content never updating on the client machines. I tried various config combinations, with threadpool, without threadpool and with/without removeOnPut enabled. I got the same error in every case on the mastercache (failure to enqueue remove events) and had to roll back to the version prior to thread pools. Unfortunately during testing we had only checked the initial items were received, we never tested items were updated - so this was partially my own fault.
Having retested the code now on test servers i definitely can't get the remove events working in the 1.1.3dev build.
If you haven't tested that deletion are propagating with the new threadpool code i suggest you do so, if you have tested this could you send me a copy of your working cache.ccf and remote.cache.ccf in case i missed something.
Many Thanks, Matt.
Aaron Smuts wrote:
I suspect that it was the remove / put problem that I solved in 1.1.2. It was the only known problem of that sort. It may be due to iteration but I thought this was resolved. Iteration is very problematic on an active collection. There is no really good way to do it. We basically copy the keys in fail fast mode. Were you seeing unrecoverable deadlock or just temporary lags?
I'll try some more strenuous testing. The big problem is that it is
very difficult to create automated tests for the remote and lateral
caches. I mainly rely on the tester script. I have a random function
that will beat the hell out of the cache. . . . .
Aaron
-----Original Message----- From: Matthew Cooke [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 11, 2005 8:52 AM To: [email protected] Subject: RE: Thread deadlock in CacheEventQueue class
We had major problems when we used the remotecache configuration in
push
mode rather than delete mode (removeOnPut).
At the time we strongly suspected a deadlock caused by the local puts placing cached items back into the local memory caches when they recieved async put events from the remotecache.
Could the deadlock we were seeing when using remotePuts have been the same thing? We are quite keen to try remotePuts again as this functionality would be very useful to us (would allow iteration over whole caches on individual machines). As the deadlock was only
occuring
under high load we don't have a reliable way to test.
Any advice appreciated!
--------------------------------------------------------------------- To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
