hmmmmm, relevance is before filtering, probably during indexing? Dennis Gearon
Signature Warning ---------------- It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' EARTH has a Right To Life, otherwise we all die. ----- Original Message ---- From: Lance Norskog <goks...@gmail.com> To: solr-user@lucene.apache.org Sent: Tue, November 9, 2010 7:07:45 PM Subject: Re: Using Multiple Cores for Multiple Users There is a standard problem with this: relevance is determined from all of the words in a field of all documents, not just the documents that match the query. That is, when user A searches for 'monkeys' and one of his feeds has a document with this word, but someone else is a zoophile, 'monkeys' will be a common word in the index. This will skew the relevance computation for user A. You could have a separate text field for each user. This might work better- but you can't use field norms (they take up space for all documents). Lance On Tue, Nov 9, 2010 at 6:00 PM, Adam Estrada <estrada.adam.gro...@gmail.com> wrote: > Thanks a lot for all the tips, guys! I think that we may explore both > options just to see what happens. I'm sure that scalability will be a huge > mess with the core-per-user scenario. I like the idea of creating a user ID > field and agree that it's probably the best approach. We'll see...I will be > sure to let the list know what I find! Please don't stop posting your > comments everyone ;-) My inquiring mind wants to know... > > Adam > > On Tue, Nov 9, 2010 at 7:34 PM, Jonathan Rochkind <rochk...@jhu.edu> wrote: > >> If storing in a single index (possibly sharded if you need it), you can >> simply include a solr field that specifies the user ID of the saved thing. >> On the client side, in your application, simply ensure that there is an fq >> parameter limiting to the current user, if you want to limit to the current >> user's stuff. Relevancy ranking should work just as if you had 'seperate >> cores', there is no relevancy issue. >> >> It IS true that when your index gets very large, commits will start taking >> longer, which can be a problem. I don't mean commits will take longer just >> because there is more stuff to commit -- the larger the index, the longer an >> update to a single document will take to commit. >> >> In general, i suspect that having dozens or hundreds (or thousands!) of >> cores is not going to scale well, it is not going to make good use of your >> cpu/ram/hd resources. Not really the intended use case of multiple cores. >> >> However, you are probably going to run into some issues with the single >> index approach too. In general, how to deal with "multi-tenancy" in Solr is >> an oft-asked question that there doesn't seem to be any "just works and does >> everything for you without needing to think about it" solution for in solr. >> Judging from past thread. I am not a Solr developer or expert. >> >> ________________________________________ >> From: Markus Jelsma [markus.jel...@openindex.io] >> Sent: Tuesday, November 09, 2010 6:57 PM >> To: solr-user@lucene.apache.org >> Cc: Adam Estrada >> Subject: Re: Using Multiple Cores for Multiple Users >> >> Hi, >> >> > All, >> > >> > I have a web application that requires the user to register and then >> login >> > to gain access to the site. Pretty standard stuff...Now I would like to >> > know what the best approach would be to implement a "customized" search >> > experience for each user. Would this mean creating a separate core per >> > user? I think that this is not possible without restarting Solr after >> each >> > core is added to the multi-core xml file, right? >> >> No, you can dynamically manage cores and parts of their configuration. >> Sometimes you must reindex after a change, the same is true for reloading >> cores. Check the wiki on this one [1]. >> >> > >> > My use case is this...User A would like to index 5 RSS feeds and User B >> > would like to index 5 completely different RSS feeds and he is not >> > interested at all in what User A is interested in. This means that they >> > would have to be separate index cores, right? >> >> If you view documents within an rss feed as a separate documents, you can >> assign an user ID to those documents, creating a multi user index with rss >> documents per user, or group or whatever. >> >> Having a core per user isn't a good idea if you have many users. It takes >> up >> additional memory and disk space, doesn't share caches etc. There is also >> more maintenance and your need some support scripts to dynamically create >> new >> cores - Solr currently doesn't create a new core directory structure. >> >> But, reindexing a very large index takes up a lot more time and resources >> and >> relevancy might be an issue depending on the rss feeds' contents. >> >> > >> > What is the best approach for this kind of thing? >> >> I'd usually store the feeds in a single index and shard if it's too many >> for a >> single server with your specifications. Unless the demands are too >> specific. >> >> > >> > Thanks in advance, >> > Adam >> >> [1]: http://wiki.apache.org/solr/CoreAdmin >> >> Cheers >> > -- Lance Norskog goks...@gmail.com
