On Tue, Oct 20, 2009 at 6:07 PM, Mark Miller <[email protected]> wrote:
> I'm +1 obviously ;) No one is talking about making it the default. And I > think its well known that soft value caches can be a valid choice - > thats why google has one in their collections here ;) Its a nice way to > let your cache grow and shrink based on the available RAM. Its not > always the right choice, but sure is a nice option. And it doesn't have > much to do with Lucene's FieldCaches. The main reason for a soft value > cache is not to avoid OOM. Set your cache sizes correctly for that. And > even if it was to avoid OOM, who cares if something else causes more of > them? Thats like not fixing a bug in a piece of code because another > piece of code has more bugs. Anyway, their purpose is to allow the cache > to size depending on the available free RAM IMO. > +1 > > Noble Paul നോബിള് नोब्ळ् wrote: > > So , is everyone now in favor of this feature? Who has a -1 on this? and > > what is the concern? > > > > On Tue, Oct 20, 2009 at 3:56 PM, Mark Miller <[email protected]> > wrote: > > > > > >> On Oct 20, 2009, at 12:12 AM, Shalin Shekhar Mangar < > >> [email protected]> wrote: > >> > >> I don't think the debate is about weak reference vs. soft references. > >> > >> There appears to be confusion between the two here no matter what the > >> debate - soft references are for cachinh, weak references are not so > much. > >> Getting it right is important. > >> > >> I > >> > >>> guess the point that Lance is making is that using such a technique > will > >>> make application performance less predictable. There's also a good > chance > >>> that a soft reference based cache will cause cache thrashing and will > hide > >>> OOMs caused by inadequate cache sizes. So basically we trade an OOM for > >>> more > >>> CPU usage (due to re-computation of results). > >>> > >>> > >> That's the whole point. Your not hiding anything. I don't follow you. > >> > >> > >> > >> > >>> Personally, I think giving an option is fine. What if the user does not > >>> have > >>> enough RAM and he is willing to pay the price? Right now, there is no > way > >>> he > >>> can do that at all. However, the most frequent reason behind OOMs is > not > >>> having enough RAM to create the field caches and not Solr caches, so > I'm > >>> not > >>> sure how important this is. > >>> > >>> > >> How important is any feature? You don't have a use for it, so it's not > >> important to you - someone else does so it is important to them. Soft > value > >> caches can be useful. > >> > >> > >> > >> > >>> On Tue, Oct 20, 2009 at 8:41 AM, Mark Miller <[email protected]> > >>> wrote: > >>> > >>> There is a difference - weak references are not for very good for > caches > >>> > >>>> - > >>>> soft references (soft values here) are good for caches in most jvms. > They > >>>> can be very nice. Weak refs are eagerly reclaimed - it's suggested > that > >>>> impls should not eagerly reclaim soft refs. > >>>> > >>>> - Mark > >>>> > >>>> http://www.lucidimagination.com (mobile) > >>>> > >>>> > >>>> On Oct 19, 2009, at 8:22 PM, Lance Norskog <[email protected]> wrote: > >>>> > >>>> "Soft references" then. "Weak pointers" is an older term. (They're > >>>> > >>>> > >>>>> "weak" because some bully can steal their candy.) > >>>>> > >>>>> On Sun, Oct 18, 2009 at 8:37 PM, Jason Rutherglen > >>>>> <[email protected]> wrote: > >>>>> > >>>>> Lance, > >>>>> > >>>>>> Do you mean soft references? > >>>>>> > >>>>>> On Sun, Oct 18, 2009 at 3:59 PM, Lance Norskog <[email protected]> > >>>>>> wrote: > >>>>>> > >>>>>> -1 for weak references in caching. > >>>>>> > >>>>>>> This makes memory management less deterministic (predictable) and > at > >>>>>>> peak can cause cache-thrashing. In other words, the worst case gets > >>>>>>> even more worse. When designing a system I want predictability and > I > >>>>>>> want to control the worst case, because system meltdowns are caused > by > >>>>>>> the worst case. Having thousands of small weak references does the > >>>>>>> opposite. > >>>>>>> > >>>>>>> On Sat, Oct 17, 2009 at 2:00 AM, Noble Paul (JIRA) < > [email protected]> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> [ > >>>>>>>> > >>>>>>>> > https://issues.apache.org/jira/browse/SOLR-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766864#action_12766864 > >>>>>>>> ] > >>>>>>>> > >>>>>>>> Noble Paul commented on SOLR-1513: > >>>>>>>> ---------------------------------- > >>>>>>>> > >>>>>>>> bq.Google Collections is already checked in as a dependency of > Carrot > >>>>>>>> clustering. > >>>>>>>> > >>>>>>>> in that e need to move it to core. > >>>>>>>> > >>>>>>>> Jason . We do not need to remove the original option. We can > probably > >>>>>>>> add an extra parameter say softRef="true" or something. That way , > we > >>>>>>>> are > >>>>>>>> not screwing up anything and perf benefits can be studied > separately. > >>>>>>>> > >>>>>>>> > >>>>>>>> Use Google Collections in ConcurrentLRUCache > >>>>>>>> > >>>>>>>> > >>>>>>>>> -------------------------------------------- > >>>>>>>>> > >>>>>>>>> Key: SOLR-1513 > >>>>>>>>> URL: > https://issues.apache.org/jira/browse/SOLR-1513 > >>>>>>>>> Project: Solr > >>>>>>>>> Issue Type: Improvement > >>>>>>>>> Components: search > >>>>>>>>> Affects Versions: 1.4 > >>>>>>>>> Reporter: Jason Rutherglen > >>>>>>>>> Priority: Minor > >>>>>>>>> Fix For: 1.5 > >>>>>>>>> > >>>>>>>>> Attachments: google-collect-snapshot.jar, SOLR-1513.patch > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ConcurrentHashMap is used in ConcurrentLRUCache. The Google > >>>>>>>>> Colletions concurrent map implementation allows for soft values > that > >>>>>>>>> are > >>>>>>>>> great for caches that potentially exceed the allocated heap. > Though > >>>>>>>>> I > >>>>>>>>> suppose Solr caches usually don't use too much RAM? > >>>>>>>>> http://code.google.com/p/google-collections/ > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> -- > >>>>>>>> This message is automatically generated by JIRA. > >>>>>>>> - > >>>>>>>> You can reply to this email to add a comment to the issue online. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> -- > >>>>>>> Lance Norskog > >>>>>>> [email protected] > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> -- > >>>>> Lance Norskog > >>>>> [email protected] > >>>>> > >>>>> > >>>>> > >>> -- > >>> Regards, > >>> Shalin Shekhar Mangar. > >>> > >>> > > > > > > > > > -- > - Mark > > http://www.lucidimagination.com > > > > -- ----------------------------------------------------- Noble Paul | Principal Engineer| AOL | http://aol.com
