+1 for having soft reference an available option by configuration, keeping the current behavior as default.
Bill 2009/10/20 Noble Paul നോബിള് नोब्ळ् <[email protected]> > On Tue, Oct 20, 2009 at 6:07 PM, Mark Miller <[email protected]> > wrote: > > > I'm +1 obviously ;) No one is talking about making it the default. And I > > think its well known that soft value caches can be a valid choice - > > thats why google has one in their collections here ;) Its a nice way to > > let your cache grow and shrink based on the available RAM. Its not > > always the right choice, but sure is a nice option. And it doesn't have > > much to do with Lucene's FieldCaches. The main reason for a soft value > > cache is not to avoid OOM. Set your cache sizes correctly for that. And > > even if it was to avoid OOM, who cares if something else causes more of > > them? Thats like not fixing a bug in a piece of code because another > > piece of code has more bugs. Anyway, their purpose is to allow the cache > > to size depending on the available free RAM IMO. > > > +1 > > > > > Noble Paul നോബിള് नोब्ळ् wrote: > > > So , is everyone now in favor of this feature? Who has a -1 on this? > and > > > what is the concern? > > > > > > On Tue, Oct 20, 2009 at 3:56 PM, Mark Miller <[email protected]> > > wrote: > > > > > > > > >> On Oct 20, 2009, at 12:12 AM, Shalin Shekhar Mangar < > > >> [email protected]> wrote: > > >> > > >> I don't think the debate is about weak reference vs. soft references. > > >> > > >> There appears to be confusion between the two here no matter what the > > >> debate - soft references are for cachinh, weak references are not so > > much. > > >> Getting it right is important. > > >> > > >> I > > >> > > >>> guess the point that Lance is making is that using such a technique > > will > > >>> make application performance less predictable. There's also a good > > chance > > >>> that a soft reference based cache will cause cache thrashing and will > > hide > > >>> OOMs caused by inadequate cache sizes. So basically we trade an OOM > for > > >>> more > > >>> CPU usage (due to re-computation of results). > > >>> > > >>> > > >> That's the whole point. Your not hiding anything. I don't follow you. > > >> > > >> > > >> > > >> > > >>> Personally, I think giving an option is fine. What if the user does > not > > >>> have > > >>> enough RAM and he is willing to pay the price? Right now, there is no > > way > > >>> he > > >>> can do that at all. However, the most frequent reason behind OOMs is > > not > > >>> having enough RAM to create the field caches and not Solr caches, so > > I'm > > >>> not > > >>> sure how important this is. > > >>> > > >>> > > >> How important is any feature? You don't have a use for it, so it's not > > >> important to you - someone else does so it is important to them. Soft > > value > > >> caches can be useful. > > >> > > >> > > >> > > >> > > >>> On Tue, Oct 20, 2009 at 8:41 AM, Mark Miller <[email protected]> > > >>> wrote: > > >>> > > >>> There is a difference - weak references are not for very good for > > caches > > >>> > > >>>> - > > >>>> soft references (soft values here) are good for caches in most jvms. > > They > > >>>> can be very nice. Weak refs are eagerly reclaimed - it's suggested > > that > > >>>> impls should not eagerly reclaim soft refs. > > >>>> > > >>>> - Mark > > >>>> > > >>>> http://www.lucidimagination.com (mobile) > > >>>> > > >>>> > > >>>> On Oct 19, 2009, at 8:22 PM, Lance Norskog <[email protected]> > wrote: > > >>>> > > >>>> "Soft references" then. "Weak pointers" is an older term. (They're > > >>>> > > >>>> > > >>>>> "weak" because some bully can steal their candy.) > > >>>>> > > >>>>> On Sun, Oct 18, 2009 at 8:37 PM, Jason Rutherglen > > >>>>> <[email protected]> wrote: > > >>>>> > > >>>>> Lance, > > >>>>> > > >>>>>> Do you mean soft references? > > >>>>>> > > >>>>>> On Sun, Oct 18, 2009 at 3:59 PM, Lance Norskog <[email protected] > > > > >>>>>> wrote: > > >>>>>> > > >>>>>> -1 for weak references in caching. > > >>>>>> > > >>>>>>> This makes memory management less deterministic (predictable) and > > at > > >>>>>>> peak can cause cache-thrashing. In other words, the worst case > gets > > >>>>>>> even more worse. When designing a system I want predictability > and > > I > > >>>>>>> want to control the worst case, because system meltdowns are > caused > > by > > >>>>>>> the worst case. Having thousands of small weak references does > the > > >>>>>>> opposite. > > >>>>>>> > > >>>>>>> On Sat, Oct 17, 2009 at 2:00 AM, Noble Paul (JIRA) < > > [email protected]> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>> [ > > >>>>>>>> > > >>>>>>>> > > > https://issues.apache.org/jira/browse/SOLR-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766864#action_12766864 > > >>>>>>>> ] > > >>>>>>>> > > >>>>>>>> Noble Paul commented on SOLR-1513: > > >>>>>>>> ---------------------------------- > > >>>>>>>> > > >>>>>>>> bq.Google Collections is already checked in as a dependency of > > Carrot > > >>>>>>>> clustering. > > >>>>>>>> > > >>>>>>>> in that e need to move it to core. > > >>>>>>>> > > >>>>>>>> Jason . We do not need to remove the original option. We can > > probably > > >>>>>>>> add an extra parameter say softRef="true" or something. That way > , > > we > > >>>>>>>> are > > >>>>>>>> not screwing up anything and perf benefits can be studied > > separately. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Use Google Collections in ConcurrentLRUCache > > >>>>>>>> > > >>>>>>>> > > >>>>>>>>> -------------------------------------------- > > >>>>>>>>> > > >>>>>>>>> Key: SOLR-1513 > > >>>>>>>>> URL: > > https://issues.apache.org/jira/browse/SOLR-1513 > > >>>>>>>>> Project: Solr > > >>>>>>>>> Issue Type: Improvement > > >>>>>>>>> Components: search > > >>>>>>>>> Affects Versions: 1.4 > > >>>>>>>>> Reporter: Jason Rutherglen > > >>>>>>>>> Priority: Minor > > >>>>>>>>> Fix For: 1.5 > > >>>>>>>>> > > >>>>>>>>> Attachments: google-collect-snapshot.jar, SOLR-1513.patch > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> ConcurrentHashMap is used in ConcurrentLRUCache. The Google > > >>>>>>>>> Colletions concurrent map implementation allows for soft values > > that > > >>>>>>>>> are > > >>>>>>>>> great for caches that potentially exceed the allocated heap. > > Though > > >>>>>>>>> I > > >>>>>>>>> suppose Solr caches usually don't use too much RAM? > > >>>>>>>>> http://code.google.com/p/google-collections/ > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> -- > > >>>>>>>> This message is automatically generated by JIRA. > > >>>>>>>> - > > >>>>>>>> You can reply to this email to add a comment to the issue > online. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>> -- > > >>>>>>> Lance Norskog > > >>>>>>> [email protected] > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>> -- > > >>>>> Lance Norskog > > >>>>> [email protected] > > >>>>> > > >>>>> > > >>>>> > > >>> -- > > >>> Regards, > > >>> Shalin Shekhar Mangar. > > >>> > > >>> > > > > > > > > > > > > > > > -- > > - Mark > > > > http://www.lucidimagination.com > > > > > > > > > > > -- > ----------------------------------------------------- > Noble Paul | Principal Engineer| AOL | http://aol.com >
