Yes and no. The disk cache will optimize the data file on shutdown. During optimization, basically, the cache reads the keys. For each key it pulls the data from the file and puts it into a new file. After it has read allt he keys, it destroys the old file and puts the new one in its place.
There is no real time optimization or defragmetation. This is pretty tricky and I'm working on a solution now. I'm going to have a scheduled optimization. Also, some timeout background shrinkers that will go through and get rid of expired elements during the optimization process. The first phase of this will take the disk offline temporarily, so you could schedule it for the middle of the night. Later, I'll try to do it live. Either way you run into the problem of removes and puts that have come in during the optimization. Lossing cached entries is not bad, since nothing should be cashed that can't be recreated (or almost nothing), but serving old data is bad. The incoming remove requests are the big problem. Basically, if you remove an item, the key is removed and the element on disk will not be found. It will take up file space until optimization, which only occurs at shutdown, but it will not be available. The system is not wildly inefficient. When you put something into diskcache it first checks to see if the element already exists. If so, if your put is an update, then it checks to see if the serialized value is less than or equal to the old value. If so, it reuses the slot. There is also a purgatory for disk elements. When an entry is spooled from memory it goes to purgatory and then a background process puts it on disk. If you request it while it is in purgatory the background job is cancelled and the element goes back to memory. Aaron > -----Original Message----- > From: Corin Moss [mailto:[EMAIL PROTECTED] > Sent: Wednesday, April 14, 2004 9:39 PM > To: Turbine JCS Developers List > Subject: RE: cvscommit:jakarta-turbine- > jcs/src/java/org/apache/jcs/auxiliary/diskAbstractDiskCache.java > > > Hi Aaron, > > Regarding this round of fixes, I noticed mention of a fix relating to > the removal of expired elements. Does this ensure that things stored > with the on disk store aren't left there after key deletion? > > > I just noticed whilst browsing the code the following: > > /** > * Returns true if the removal was successful; or false if there is > nothing > * to remove. Current implementation always result in a disk orphan. > * > * @return > * @param key > */ > public boolean doRemove( Serializable key ) > > The bit about the disk orphan (if still true) is in my mind incredibly > dangerous. It means that you have no control over the size of your > cache - which can result in a disk blow-out. > > > Am I right in this? I have noticed that my data file is always growing, > never shrinking - this would definitely explain this. > > > Does anyone have any comments on this? If my assumptions are correct, > does anyone have a fix in the works? :) > > Thanks, > > Corin > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > > Sent: Thursday, 15 April 2004 5:06 a.m. > To: [EMAIL PROTECTED] > Subject: cvs > commit:jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/diskAbstrac > tDiskCache.java > > > asmuts 2004/04/14 10:06:12 > > Modified: src/java/org/apache/jcs/utils/locking ReadWriteLock.java > src/java/org/apache/jcs/auxiliary/disk/indexed > IndexedDiskCache.java > src/java/org/apache/jcs/auxiliary/disk > AbstractDiskCache.java > Log: > It should be safe to dispose now. Kill queue, optimize, then play > dead. > > > Fixed a problem in the ReadWriteLock class that let the number of > writelocks get below zero. This seems to have solved many problems. > However, I can't find out why there are more done calls than lock calls. > Added a break into the wait on the readlock in case noone calls notify. > > > This may have fixed the mysterious locking problem that I've never > been able to duplicate. > > > Revision Changes Path > 1.3 +14 -3 > jakarta-turbine-jcs/src/java/org/apache/jcs/utils/locking/ReadWriteLock. > java > > > Index: ReadWriteLock.java > =================================================================== > RCS file: > /home/cvs/jakarta-turbine-jcs/src/java/org/apache/jcs/utils/locking/Read > WriteLock.java,v > retrieving revision 1.2 > retrieving revision 1.3 > diff -u -r1.2 -r1.3 > --- ReadWriteLock.java 22 Aug 2003 11:57:19 -0000 1.2 > +++ ReadWriteLock.java 14 Apr 2004 17:06:12 -0000 1.3 > @@ -117,7 +117,7 @@ > while ( writeLockedThread != null ) > { > log.debug( "readLock wait" ); > - wait(); > + wait(20); > log.debug( "wake up from readLock wait" ); > } > > > @@ -211,9 +211,20 @@ > } > return; > } > + > if ( Thread.currentThread() == writeLockedThread ) > { > - outstandingWriteLocks--; > + > + //log.info( "outstandingWriteLocks= " + > outstandingWriteLocks ); > + if ( outstandingWriteLocks > 0 ) > + { > + outstandingWriteLocks--; > + } > + else > + { > + log.warn( "extra lock release, writelocks are " + > outstandingWriteLocks + "and done was called" ); > + } > + > if ( outstandingWriteLocks > 0 ) > { > log.debug( "writeLock released for a nested writeLock > request." ); > @@ -243,7 +254,7 @@ > if ( waitingForReadLock > 0 ) > { > log.debug( "writeLock released, notified waiting > readers" ); > - > > + > notifyAll(); > } > else > > > > > > > 1.9 +18 -2 > jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/disk/indexed/Index > edDiskCache.java > > > Index: IndexedDiskCache.java > =================================================================== > RCS file: > /home/cvs/jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/disk/ind > exed/IndexedDiskCache.java,v > retrieving revision 1.8 > retrieving revision 1.9 > diff -u -r1.8 -r1.9 > --- IndexedDiskCache.java 14 Apr 2004 06:24:18 -0000 1.8 > +++ IndexedDiskCache.java 14 Apr 2004 17:06:12 -0000 1.9 > @@ -300,6 +300,7 @@ > { > storageLock.done(); > } > + > if ( log.isDebugEnabled() ) > { > log.debug( "Put to file: " + fileName + > @@ -469,8 +470,15 @@ > } > else > { > + > + if ( log.isDebugEnabled() ) > + { > + log.debug( "Disk removal: Removed from key hash, > key " + key ); > + } > + > // remove single item. > return keyHash.remove( key ) != null; > + > } > } > catch ( Exception e ) > @@ -588,7 +596,14 @@ > finally > { > alive = false; > - storageLock.done(); > + > + try > + { > + storageLock.done(); > + } catch ( Exception e ) > + { > + log.error( "Failure releasing lock on shutdown " + e > ); > + } > } > } > > > @@ -673,6 +688,7 @@ > log.debug( fileName + " -- newData.length() = " + > newData.length() ); > } > + > newData.renameTo( newFileName ); > } > keyHash = keyHashTemp; > > > > > > > 1.19 +4 -6 > jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/disk/AbstractDiskC > ache.java > > > Index: AbstractDiskCache.java > =================================================================== > RCS file: > /home/cvs/jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/disk/Abs > tractDiskCache.java,v > retrieving revision 1.18 > retrieving revision 1.19 > diff -u -r1.18 -r1.19 > --- AbstractDiskCache.java 14 Apr 2004 06:24:18 -0000 1.18 > +++ AbstractDiskCache.java 14 Apr 2004 17:06:12 -0000 1.19 > @@ -300,13 +300,11 @@ > public final void dispose() > { > > > - // Invoke any implementation specific disposal code > - > - doDispose(); > - > // FIXME: May lose the end of the queue, need to be more > graceful > - > cacheEventQueue.destroy(); > + > + // Invoke any implementation specific disposal code > + doDispose(); > > > alive = false; > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > ================================================================ > CAUTION: This e-mail and any attachment(s) contains information > that is intended to be read only by the named recipient(s). It > may contain information that is confidential, proprietary or the > subject of legal privilege. This information is not to be used by > any other person and/or organisation. If you are not the intended > recipient, please advise us immediately and delete this e-mail > from your system. Do not use any information contained in it. > > ================================================================ > For more information on the Television New Zealand Group, visit > us online at http://www.tvnz.co.nz > ================================================================ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
