I committed the "unregister" on close of a clone. This makes it more consistent, but I think it ill not help, because we never close clones. But if we do in future, its already correct :-)
Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Uwe Schindler [mailto:u...@thetaphi.de] > Sent: Wednesday, January 23, 2013 10:45 PM > To: dev@lucene.apache.org > Subject: RE: Win7 64bit, jvm 7 and MMAP OOM > > We have this stress test: TestWeakIdentityMap :-) > > The problem here may be: > The weak references used by those maps are only nuked from the map, > when the map is accessed (get/put/size/... operation, it calls reap() from > there method). So if you have code that allocates millions of IndexInputs and > then sleeps, the IndexInputs are garbage collected, but the WeakReference > instances (not the referees) are only freed when the map is accessed for the > next time, or when it is nuked completely (when you close the master > IndexInput). > Ideally, the cloned IndexInputs would be closed after use, then they could > deregister themselves from their master. But unfortunately we have no > close() any more in TermsEnum or PostingLists... The cloned IndexInput is a > resource that should be closed. > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > -----Original Message----- > > From: Michael McCandless [mailto:luc...@mikemccandless.com] > > Sent: Wednesday, January 23, 2013 2:36 PM > > To: dev@lucene.apache.org > > Subject: Re: Win7 64bit, jvm 7 and MMAP OOM > > > > This seems bad. > > > > Does it mean this JVM on Win7 fails to properly reclaim WeakReferences? > > > > Maybe a simple standalone stress test of WeakIdentityMap would > > reproduce the issue? > > > > Mike McCandless > > > > http://blog.mikemccandless.com > > > > On Tue, Jan 22, 2013 at 2:38 PM, eksdev <eks...@googlemail.com> wrote: > > > Just to share some experience if someone hits the same problem. > > > > > > We had huge problems on Win7 64bit, JVM 64bit 1.7.0_07, (a few days > > > old trunk version, 5.0, default codec) solr under tomcat thread > > > queue limited to 20 . NRTCaching and MMAP have the same problems > (no > > updates > > > , just search). Unmap hack was on > > > > > > Problem: > > > After hitting it for 15 minutes with 10 search threads, gc() did not > > > manage > > to catch-up and free enough memory and the server repeatably spiralled > > to OOM (giving more max heap did not help as well). Server is running > > on 8 cores, and 10 client threads are normally not an issue. > > > > > > Observations: > > > - Tweaking jvm memory and gc() options did not help at all. > > > - exactly the same configuration and tests on 3 linux flavours > > > had > > absolutely no problems. > > > - Win using FSDirectory works slower, but stable > > > - When "OOM" spiralling happens, major culprits are, by occupied > > > memory > > and Noo of instances: > > > o.a.l.util.WeakIdentityMap$IdentityWeakReference > > > java.util.concurrent.ConcurrentHashMap$HashEntry > > > java.util.concurrent.ConcurrentHashMap$HashEntry[] > > > - if pausing search requests for really long time (5 to 10 minutes!) > > > these > > references get eventually released. > > > ---------- > > > > > > > > > I know java+MMAP on win platforms has problems (slowly releasing > > mapped regions), but I did not expect it is that bad, to the point of > > being useless. > > > It is not an itch currently, all our production is on linux, but if > > > someone has > > an idea how to work around it, we would be glad to try it. > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For > > > additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For > > additional commands, e-mail: dev-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional > commands, e-mail: dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
