custom scoring

Hi, I would like to customize the lucene scoring to remove the effect of the "coord()" parameter in Similarity for the number of fields in which the whole query is found. First, here's what I'm trying to achieve: For example, if I have the query "New York" and that the documents have 2 fields: n

custom scoring

the following message comes from Explanation explain 0.09375 = (MATCH) fieldWeight(name:85 in 8687), product of 1.0 = tf(termFreq(name:85)=1) 1.0 = idf(docFreq=1406, maxDocs=14129) 0.09375 = fieldNorm(field=name, doc=8687) fieldNorm 0.09375 the value 0.09375 is what determines an

Custom scoring

Hello, I am trying to implement my own Jaccard similarity for Lucene. So far i have the following code public class JaccardSimilarity extends DefaultSimilarity { int numberOfDocumentTerms; //String field="contents"; // Should the Jaccard similarity be only based in the contents field

custom scoring

Hi all,What would be the best approach for a custom scoring that requires a "global" view of the result set. For example, I have a field call "color" and I would like to have constraints that there are at most 3 docs with color:red, 4 docs with color:blue in the first 16 hits.

RE: custom scoring

Hi Sébastien, Have you looked into the DisjunctionMaxQuery ? From that page: A query that generates the union of documents produced by its subqueries, and that scores each document with the maximum

Custom scoring example ...

I am looking for an example if anyone has done any custom scoring with Lucene. I need to implement a Query similar to DisjunctionMaxQuery, the only difference would be it should score based on sum of score of sub queries' scores instead of max. Any custom scoring example will help. (On one

Custom scoring algorithm

Hi. I am developing an application and I would like to add searching capabilities. I have a database with items. Each item has a number of "features" with a numeric value. Example: feature_x=100, feature_y=200. Items can have common or different "features". And they can have a variable number of "

custom scoring help

Hello, I'm having a hard time implementing / understanding a very simple custom scoring situation. I have created my Similarity class for testing which overrides all the relevant (I think) methods below, returning 1 for all but coord(int, int) which returns q / maxOverlap so scores are s

Re: custom scoring

Some values in the norm/boost area are stored encoded with some loss of precision. Details in the javadocs somewhere. What values do you get when you change the boost? -- Ian. 2012/1/14 ltomuno : > the following message comes from  Explanation explain >  0.09375  = (MATCH) fieldWeight(name:85

Re: Custom scoring

> The problem is that coord() method is not used (or at least > so that i understand) neither in searching nor in indexing > What do i do wrong? If you want to see coord() values, use a multi-word query (two or more query terms) and go to last page of result set. --

Re: custom scoring

through the TopDocs and apply the constraints I need toI think this will work, but have some concern about performance. What would you think?Thanks,Tri.On Apr 06, 2012, at 10:06 AM, Tri Cao wrote:Hi all,What would be the best approach for a custom scoring that requires a "global" view of

Lucene custom scoring / analyzer

Hello! I`d like to do something like that: When I add a document and some text is going with (e.g.) quotes it should mean that this text has to be exactly in the query. Better with an examples - text: green "blue apple" juice query : blue apple - result: hit. query : blue apple juice - result: h

Re: Custom scoring algorithm

Hi again. I've made a proof of concept using the boost factor. I have done the following: add a field for each feature and put the field boost factor as the feature value. private static void addDocument(String id, Map features, IndexWriter writer) throws IOException { Doc

Re: custom scoring help

This code is in fact working. I had an error in my test case. Things seem to work as advertised. sorry / thanks - C>T> On Fri, Apr 2, 2010 at 10:20 AM, Christopher Tignor wrote: > Hello, > > I'm having a hard time implementing / understanding a very simple custom > s

Question on custom scoring

it difficult for me to weight the different scores properly. Is there an easy way to get the normalized "text index" score in customScore() API so I can easaily weigh in the other factors relative to it? Thanks a lot in advance, Srinivas -- View this message in context: htt

Re: Lucene custom scoring / analyzer

I think you'll need a SpanQuery with the inOrder flag set: https://lucene.apache.org/core/8_8_1/core/org/apache/lucene/search/spans/SpanNearQuery.html Charlie On 17/03/2021 10:30, Vlad Smirnovskiy wrote: Hello! I`d like to do something like that: When I add a document and some text is going wi

Re: Question on custom scoring

) still appreciated. I'm also starting to look at the code to understand the weight normalization process. If I find something useful, I'll post it here. I hope this thread becomes useful to others that are figuring out how to best weigh in different factors while doing custom scorin

Re: Question on custom scoring

: [1] I need to rank matches by some combination of keyword match, popularity : and recency of the doc. I read the docs about CustomScoreQuery and seems to : be a resonable fit. An alternate way of achieving my goals is to use a : custom sort. What are the trade-offs between these two approaches?

Re: Question on custom scoring

easaily weigh in the other factors > : relative to it? > > normalized relative to what? > > > > -Hoss > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > -- View this m

Re: Question on custom scoring

: document in the scoring formula, and I thought the CustomScoreQuery would be : useful, but I am realizing that it may not be easy because the "relevance" : score from Lucene has no absolute meaning. The relevance score could be 5 or : 500 and there is no way for me gauge what that number means an

Custom scoring algorithm and Explanation extending.

Hello 1. I would like to use my custom scoring algorithm. Is it make sense to use Lucene with other scoring algorithm? What is the best way for that - implement Similarity and own Queries? 2. I'm reasearching Elasticsearch/Lucene capabilities. Elastichsearch contains request parameter &qu

Custom scoring for searhing geographic objects

Hi, Please give me advise how to create custom scoring. I need to result that documents were in order, depending on how popular each term in the document (popular = how many times it appears in the index) and length of the document (less terms - higher in search results). For example, index

Re: Custom scoring algorithm and Explanation extending.

Hi Vadim, Le jeu. 16 nov. 2017 à 18:09, Vadim Gindin a écrit : > 1. I would like to use my custom scoring algorithm. Is it make sense to use > Lucene with other scoring algorithm? What is the best way for that - > implement Similarity and own Queries? > It really depends what y

Re: Custom scoring algorithm and Explanation extending.

Thank's a lot! On Mon, Nov 20, 2017 at 11:22 PM, Adrien Grand wrote: > Hi Vadim, > > Le jeu. 16 nov. 2017 à 18:09, Vadim Gindin a écrit > : > > > 1. I would like to use my custom scoring algorithm. Is it make sense to > use > > Lucene with other scoring algor

Re: Custom scoring for searhing geographic objects

that is much more complex. A web search for "lucene scoring" should find you lots of info. -- Ian. On Wed, Dec 15, 2010 at 3:28 PM, Pavel Minchenkov wrote: > Hi, > Please give me advise how to create custom scoring. I need to result that > documents were in order, depen

Re: Custom scoring for searhing geographic objects

gt; > There is also the org.apache.lucene.search.function package but that > is much more complex. > > > A web search for "lucene scoring" should find you lots of info. > > > -- > Ian. > > > On Wed, Dec 15, 2010 at 3:28 PM, Pavel Minchenkov > wrote: > >

Re: Custom scoring for searhing geographic objects

Have a look at http://lucene.apache.org/java/3_0_2/scoring.html on how Lucene's scoring works. You can override the Similarity class in Solr as well via the schema.xml file. On Dec 15, 2010, at 10:28 AM, Pavel Minchenkov wrote: > Hi, > Please give me advise how to create custom

Re: Custom scoring for searhing geographic objects

que identifying the match. If it's then return matching street. If there's no any match proceed using the same algorithm with towns, regions, countries. HTH, Alexey On Wed, Dec 15, 2010 at 6:28 PM, Pavel Minchenkov wrote: > Hi, > Please give me advise how to create custom scorin

Get values in custom scoring during document retrieval

directly get that value after scoring as its already calculated. -- View this message in context: http://lucene.472066.n3.nabble.com/Get-values-in-custom-scoring-during-document-retrieval-tp4313853.html Sent from the Lucene - Java Users mailing list archive at Nabble.com

Question about how to speed up custom scoring

I am trying to come up with a performant query that will allow me to use a custom score where the custom score is a sum-product over a set of query time weights where each weight gets applied only if the query time term exists in the document . So for example if I have a doc with three fields: comp

Re: Get values in custom scoring during document retrieval

mputing it again cant i > directly > get that value after scoring as its already calculated. > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Get-values-in-custom-scoring-during-document-retrieval-tp4313853.html > Sent from the Lucene

Re: Get values in custom scoring during document retrieval

Sorry I just saw your other message that has a bit more information. Actually you do not need the distance for displaying purposes but both for filtering and custom scoring. That said, I think recomputing the distances is still the way to go. Geo-distance filters have optimizations that allow them

Re: Get values in custom scoring during document retrieval

rather than storing the distances somewhere when doing the custom scoring? -- View this message in context: http://lucene.472066.n3.nabble.com/Get-values-in-custom-scoring-during-document-retrieval-tp4313853p4314417.html Sent from the Lucene - Java Users mailing list archive at Nabble.com

Re: Get values in custom scoring during document retrieval

y >rather than storing the distances somewhere when doing the custom >scoring? > > > >-- >View this message in context: >http://lucene.472066.n3.nabble.com/Get-values-in-custom-scoring-during-document-retrieval-tp4313853p4314417.html >Sent from the L

Re: Question about how to speed up custom scoring

Oops, forgot to include the class I mentioned. Here it is: public class QueryTermBoostingQuery extends CustomScoreQuery { private Map queryTermWeights; private float bias; private IndexReader indexReader; public QueryTermBoostingQuery( Query q, Map termWeights, IndexReader indexReader, fl

Re: Question about how to speed up custom scoring

I suspect your problem here is the line: document = indexReader.document( doc ); See the caution in the docs You could try using lazy loading (so you don't load all the terms of the document, just those you're interested in). And I *think* (but it's been a while) that if the terms you load are in

Re: Question about how to speed up custom scoring

Erick Erickson wrote: I suspect your problem here is the line: document = indexReader.document( doc ); See the caution in the docs You could try using lazy loading (so you don't load all the terms of the document, just those you're interested in). And I *think* (but it's been a while) that if t

Re: Question about how to speed up custom scoring

Thanks for the suggestions Erick. I am using Lucene 2.3. Terms are stored and given Andrzej's comments in the follow up email sounds like it's not the stored field issue. I'll keep investigating... thanks, Scott On Thu, Oct 8, 2009 at 8:06 AM, Erick Erickson wrote: > I suspect your problem here

Re: Question about how to speed up custom scoring

Scott, To reiterate what Erick and Andrzej's said: calling IndexReader.document(docId) in your inner scoring loop is the source of your performance problem - iterating over all these stored fields is what is killing you. To do this a better way, can you try to explain exactly what this Scorer

Re: Question about how to speed up custom scoring

If you are trying to add specific term weights to terms in the index and then incorporate them into scoring, you might benefit from payloads and the PayloadTermQuery option. See http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/ -Grant On Oct 8, 2009, at 11:56 AM

Re: Question about how to speed up custom scoring

Oops, just reread and realized you wanted query time weights. Payloads are an index time thing. On Oct 9, 2009, at 5:49 PM, Grant Ingersoll wrote: If you are trying to add specific term weights to terms in the index and then incorporate them into scoring, you might benefit from payloads a

Re: Question about how to speed up custom scoring

Right exactly. I looked into payload initially and realized it wouldn't work for my use case. On Fri, Oct 9, 2009 at 2:00 PM, Grant Ingersoll wrote: > Oops, just reread and realized you wanted query time weights. Payloads are > an index time thing. > > > On Oct 9, 2009, at 5:49 PM, Grant Ingers

Re: Question about how to speed up custom scoring

(Apologies if this message gets sent more than once. I received an error sending it the first two times so sent directly to Jake but reposting to group.) Hi Jake -- Thanks for the feedback. What I am trying to implement is a way to custom score documents using a scoring function that takes as inp

Re: Question about how to speed up custom scoring

Hey Scott, I'm still not sure I understand what your dynamic boosts are for: they are the names of fields, right, not terms in the fields? So in terms of your example { company = microsoft, city = redmond, size = big }, the three possible choices for keys in your map are company, city, or size,

Re: Question about how to speed up custom scoring

Hi Jake -- Sorry for the confusion. I have two similar but slightly different use cases in mind and the example I gave you corresponds to one use case while the code corresponds to the other slightly more complicated one. Ignore the original example, and let me restate the one I have in mind so it

Re: Question about how to speed up custom scoring

On Fri, Oct 9, 2009 at 3:07 PM, scott w wrote: > Example Document: > model_1_score = 0.9 > model_2_score = 0.3 > model_3_score = 0.7 > > I want to be able to pass in the following map at query time: > {model_1_score=0.4, model_2_score=0.7} and have that map get used as input > to a custom score f

Re: Question about how to speed up custom scoring

Thanks Jake! I will test this out and report back soon in case it's helpful to others. Definitely appreciate the help. Scott On Fri, Oct 9, 2009 at 3:33 PM, Jake Mannix wrote: > On Fri, Oct 9, 2009 at 3:07 PM, scott w wrote: > > > Example Document: > > model_1_score = 0.9 > > model_2_score = 0

Re: Question about how to speed up custom scoring

Great Scott (hah!) - please do report back, even if it just works fine and you have no more questions, I'd like to know whether this really is what you were after and actually works for you. Note that the FieldCache is kinda "magic" - it's lazy (so the first query will be slow and you should fire

Re: Question about how to speed up custom scoring

Haven't tried it yet but looking at it closer it looks like it's not something I can plug in on top of my original query. I am definitely happy using an approximation for the sake of performance but I do need to be able to have the original results stay the same. On Fri, Oct 9, 2009 at 5:32 PM, Ja

Re: Question about how to speed up custom scoring

What do you mean "not something I can plug in on top of my original query"? Do you mean that you can't do it like the more complex example in the class you posted earlier in the thread, where you take a linear combination of the Map -based score, and the regular text score? Another option is to j

Re: Question about how to speed up custom scoring

On Sun, Oct 11, 2009 at 9:10 AM, Jake Mannix wrote: > What do you mean "not something I can plug in on top of my original query"? > > Do you mean that you can't do it like the more complex example in the class > you posted earlier in the thread, where you take a linear combination of > the > Map