A co-worker had the idea that maybe we need a more sophisticated cache that uses a combination of Strong and Soft References. The idea would be that "newer" things in the cache would have strong and soft references. To be sure that this newer data does not get GC'd. Then at some point when some threshold of time and size are exceeded the oldest strong references would be released.
How does that sound? thanks, Aaron ----- Original Message ----- From: "(Cedric) Qin ZHANG" <[EMAIL PROTECTED]> Date: Tuesday, September 19, 2006 9:54 am Subject: Re: [HACKYSTAT-DEV-L] Weak/Soft/Phantom References To: [email protected] > My bad. It's already SoftReference. I don't why I had the pre- > conception > that WeakHashMap was used in ThreeKeyCache implementation. > > I will probably do some research on the internal implementation of > Sun > GC on my 18-hour trip from here to NY, and report back if I have > any > useful finding. > > By the way, GC in client and server JVM behaves differently. On > client > JVM, it's supposed to do small memory sweeps frequently in order to > hide > the latency from a user. > > Cheers, > > Cedric > > > > Philip Johnson wrote: > > Hmm... did I inadvertantly type 'weak' reference somewhere? > That's a mistake. The only non-strong reference (in the > ThreeKeyCache class, anyway) is a soft reference. And even soft > references were definitely reclaimed too fast, at least in my > trials under JDK 1.4 this summer when I was developing the > FileMetric DPD redesign. > > > > What I've been reading recently in various articles is: > > > > (a) there aren't supposed to be major differences between 1.4 and > 1.5 with respect to this issue; and > > (b) The JDK 1.5 GC does a better job of 'waiting' to reclaim > SoftReferences than prior GCs. > > > > When I eventually get some spare cycles, I will profile Hackystat > and figure out what's currently the case. > > > > Cheers, > > Philip > > > > ----- Original Message ----- > > From: "(Cedric) Qin ZHANG" <[EMAIL PROTECTED]> > > Date: Monday, September 18, 2006 10:57 pm > > Subject: [HACKYSTAT-DEV-L] Weak/Soft/Phantom References > > To: [email protected] > > > > > >> Philip mentioned that Java GC reclaims weakly referenced objects > >> too > >> fast, often times before the cached object could be reused. > >> According to > >> the IBM article listed below, it seems that it's just a matter > of > >> replacing WeakReference with SoftReference in Hackystat code. I > am > >> quite > >> interested in knowing why there need to be more than one type of > >> non-strong reference in Java. Do we have any Java GC expert on > this > >> list > >> that can probably shed light on this topic? > >> > >> Thanks. > >> > >> Cedric > >> > >> > >> > >> Philip Johnson (JIRA) wrote: > >> > >>> [ http://hackydev.ics.hawaii.edu:8080/browse/HACK- > >>> > >> 796?page=comments#action_11837 ] > >> > >>> Philip Johnson commented on HACK-796: > >>> ------------------------------------- > >>> > >>> I found a couple more links that might of use when doing > further > >>> > >> research on this issue: > >> > >>> Shows a soft reference based hashmap: > >>> > >>> > >> > http://www.javaspecialists.co.za/archive/newsletter.do?issue=098&locale=en_US>>> > > >>> An article about soft references: > >>> http://www-128.ibm.com/developerworks/java/library/j-jtp01246.html > >>> > >>> A bug report indicating that maybe soft references don't get > GC'd > >>> > >> so fast in JDK 1.5: > >> > >>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6349930 > >>> > >>> > >>> > >>> > >>>> NullPointerException in DailyProjectFileMetrics > >>>> ----------------------------------------------- > >>>> > >>>> Key: HACK-796 > >>>> URL: http://hackydev.ics.hawaii.edu:8080/browse/HACK-796 > >>>> Project: Hackystat > >>>> Type: Bug > >>>> > >>>> > >>> > >>> > >>>> Reporter: Qin Zhang > >>>> Assignee: Philip Johnson > >>>> Priority: Critical > >>>> Fix For: 7.5 > >>>> > >>>> > >>> > >>> > >>>> To reproduce: use telemetry wall to hit the server hard. It > >>>> > >> seems that DailyProjectFileMetric is not written in a thread- > safe way. > >> > >>>> 09/17 13:13:18 DailyProjectFileMetric failed to initialize! > >>>> Hackystat-7 09-Sep-2006 > >>>> java.lang.NullPointerException > >>>> at java.util.LinkedList.remove(LinkedList.java:647) > >>>> at java.util.LinkedList.removeLast(LinkedList.java:144) > >>>> at > >>>> > >> > org.hackystat.core.kernel.cache.ThreeKeyCache.put(ThreeKeyCache.java:252)>> > at > org.hackystat.core.kernel.sensordata.SensorDataCache.getSensorData(SensorDataCache.java:48) > >> > >>>> at > >>>> > >> > org.hackystat.sdt.filemetric.dailyproject.FileMetricRawDataMap.<init>(FileMetricRawDataMap.java:91)>> > at > org.hackystat.sdt.filemetric.dailyproject.DailyProjectFileMetric.initialize(DailyProjectFileMetric.java:117) > >> > >>>> at > >>>> > >> > org.hackystat.sdt.filemetric.dailyproject.DailyProjectFileMetric.getAllFileTypes(DailyProjectFileMetric.java:152)>> > at > org.hackystat.sdt.filemetric.reducer.LanguageFileMetricReducer.getAllFileTypes(LanguageFileMetricReducer.java:101) > >> > >>>> at > >>>> > >> > org.hackystat.sdt.filemetric.reducer.LanguageFileMetricReducer.compute(LanguageFileMetricReducer.java:72)>> > at > org.hackystat.core.telemetry.reducer.TelemetryReducerManager.compute(TelemetryReducerManager.java:141) > >> > >>>> at > >>>> > >> > org.hackystat.core.telemetry.evaluator.TelemetryEvaluator.resolveReducerCall(TelemetryEvaluator.java:380)>> > at > org.hackystat.core.telemetry.evaluator.TelemetryEvaluator.resolveFunctionCall(TelemetryEvaluator.java:322) > >> > >>>> at > >>>> > >> > org.hackystat.core.telemetry.evaluator.TelemetryEvaluator.resolveExpression(TelemetryEvaluator.java:285)>> > at > org.hackystat.core.telemetry.evaluator.TelemetryEvaluator.evaluate(TelemetryEvaluator.java:56) > >> > >>>> at > >>>> > >> > org.hackystat.core.telemetry.evaluator.TelemetryEvaluator.evaluate(TelemetryEvaluator.java:140)>> > at > org.hackystat.core.telemetry.evaluator.TelemetryEvaluator.evaluate(TelemetryEvaluator.java:259) > >> > >>>> at > >>>> > >> > org.hackystat.core.telemetry.analysis.TelemetryReportAnalysis.prepareReportForJspRendering(TelemetryReportAnalysis.java:133)>> > at > org.hackystat.core.telemetry.analysis.TelemetryExpertAnalysis.process(TelemetryExpertAnalysis.java:146) > >> > >>>> at > >>>> > >> > org.hackystat.core.kernel.mvc.Controller.doPost(Controller.java:271)>> > at org.hackystat.core.kernel.mvc.Controller.doGet(Controller.java:300) > >> > >>>> at > >>>> > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:689)>> > > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > >> > >>>> at > >>>> > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)>> > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > >> > >>>> at > >>>> > >> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)>> > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) > >> > >>>> at > >>>> > >> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)>> > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) > >> > >>>> at > >>>> > >> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)>> > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) > >> > >>>> at > >>>> > >> > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)>> > at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) > >> > >>>> at > >>>> > >> > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)>> > at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) > >> > >>>> at > >>>> > >> > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)>> > at java.lang.Thread.run(Thread.java:595) > >> > >>>> > >>>> > >>> > >>> >
