Re: get term frequency, just only keywords search
Hi, jack i have a same problem as danielitos85 i want to search like research development but termfreq function not work as per your messages and you said that use phraseFreq but we can get it from debug query. my problem is i want to sort on research development count, higher count document will display first in list. so how can i sort on that. can you please help me asap. Thanks. -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4133260.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
You need to use a shingle filter at index time so that pairs of adjacent words get indexed as single terms, then you can do a term frequency for the shingled pair of terms (Research Development as a single term). Be sure to manually apply any other filters, such as lower case or stemming. See: http://lucene.apache.org/core/4_7_0/analyzers-common/org/apache/lucene/analysis/shingle/ShingleFilterFactory.html http://lucene.apache.org/core/4_7_0/analyzers-common/org/apache/lucene/analysis/shingle/ShingleFilter.html But, note that you don't need to do any of this if you simply want to boost documents containing a phrase - just use the pf, pf2, and pf3 parameters of edsimax or explicitly boost the phrase, such as research development^20. -- Jack Krupansky -Original Message- From: ksmith Sent: Saturday, April 26, 2014 5:38 AM To: solr-user@lucene.apache.org Subject: Re: get term frequency, just only keywords search Hi, jack i have a same problem as danielitos85 i want to search like research development but termfreq function not work as per your messages and you said that use phraseFreq but we can get it from debug query. my problem is i want to sort on research development count, higher count document will display first in list. so how can i sort on that. can you please help me asap. Thanks. -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4133260.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
Thanks a lot guys, @Jack in my search I use dismax (how defType) and I search either term or phrase, but I need to get the number that show me how many time that term or phrase is in the document. I could get it from debugQuery but I would like get it directly from the results. What do you suggest? Thanks a lot for support. -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4085831.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
Probably your best bet is to use the debug.explain.structured parameter, set to true, to get the XML version of the debug explain section and then you can traverse looking for the desired phrase and then the phraseFreq. But, be aware that the terms in a Lucene query have been analyzed, so they won't necessarily be exactly the same as your source query characters. My advise would be to abandon phrase frequency since it is probably more effort than it is worth. But if your management insists on having the feature, roll up your sleeves and write the code needed to ferret it out. -- Jack Krupansky -Original Message- From: danielitos85 Sent: Wednesday, August 21, 2013 4:41 AM To: solr-user@lucene.apache.org Subject: Re: get term frequency, just only keywords search Thanks a lot guys, @Jack in my search I use dismax (how defType) and I search either term or phrase, but I need to get the number that show me how many time that term or phrase is in the document. I could get it from debugQuery but I would like get it directly from the results. What do you suggest? Thanks a lot for support. -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4085831.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
Thanks Jack, but if my keyword search are two words? for example french fries ? how is the right syntax? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4085399.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
french fries is a phrase, not a term or a keyword. It consists of two terms or keywords, french and fries. They have to be treated separately. -- Jack Krupansky -Original Message- From: danielitos85 Sent: Monday, August 19, 2013 4:30 AM To: solr-user@lucene.apache.org Subject: Re: get term frequency, just only keywords search Thanks Jack, but if my keyword search are two words? for example french fries ? how is the right syntax? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4085399.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
there isn't a way to get termFreq about a search like french fries (sentence)? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4085454.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
Term frequency is about terms, nothing else. So, by definition, a phrase or any other collection of terms does not have a termfreq - in Lucene. -- Jack Krupansky -Original Message- From: danielitos85 Sent: Monday, August 19, 2013 9:59 AM To: solr-user@lucene.apache.org Subject: Re: get term frequency, just only keywords search there isn't a way to get termFreq about a search like french fries (sentence)? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4085454.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
ok I undestand it (thanks) but if I search a sentence and type debugQuery=on, in the explain I obtain termFreq=2.0 and it right. Is it possible to obtain that parameter? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4085464.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
There are a series of functions that can deal with _some_ relevance data, see: http://wiki.apache.org/solr/FunctionQuery#Relevance_Functions Best Erick On Mon, Aug 19, 2013 at 10:25 AM, danielitos85 danydany@gmail.comwrote: ok I undestand it (thanks) but if I search a sentence and type debugQuery=on, in the explain I obtain termFreq=2.0 and it right. Is it possible to obtain that parameter? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4085464.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
The Lucene PhraseQuery goes through a lot of effort to calculate phrase frequency (phraseFreq) - but that is not the same as term frequency (don't confuse terms and phrases). Feel free to pick that number out of the debugQuery output, or from the XML variant of the explain output. For reference, an example: 0.2169777 = (MATCH) weight(text_t:quick brown in 0) [DefaultSimilarity], result of: 0.2169777 = fieldWeight in 0, product of: 1.4142135 = tf(freq=2.0), with freq of: 2.0 = phraseFreq=2.0 0.61370564 = idf(), sum of: 0.30685282 = idf(docFreq=1, maxDocs=1) 0.30685282 = idf(docFreq=1, maxDocs=1) 0.25 = fieldNorm(doc=0) So, it is really phraseFreq and then Lucene treats phraseFreq as if it were term frequency (tf) in the similarity calculation. -- Jack Krupansky -Original Message- From: danielitos85 Sent: Monday, August 19, 2013 10:25 AM To: solr-user@lucene.apache.org Subject: Re: get term frequency, just only keywords search ok I undestand it (thanks) but if I search a sentence and type debugQuery=on, in the explain I obtain termFreq=2.0 and it right. Is it possible to obtain that parameter? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4085464.html Sent from the Solr - User mailing list archive at Nabble.com.
get term frequency, just only keywords search
I need to get the TermFrequency in Solr4 but just about my keywords search and not for all the keywords that a field cointains. I try to explain with an example: I have this fiels: field name=idtype=string indexed=true stored=truerequired=true / field name=foods type=text indexed=true stored=truerequired=truetermVectors=true termPositions=truetermOffsets=true/ when I try to searh into field foods the keyword pizza like this: http://localhost:8983/solr/mycore/select?q=pizzafl=idtv=truedf=foodstv.tf_idf=truetv.tf=truetv.df=truetv.fl=foodsomitHeader=truedefType=dismax this is my results: response result name=response numFound=1 start=0 doc str name=id1/str /doc /result lst name=termVectors str name=uniqueKeyFieldNameid/str lst name=1 str name=uniqueKey1/str lst name=foods lst name=pizza int name=tf5/int int name=df3/int double name=tf-idf1.6667/double /lst lst name=ice-cream int name=tf2/int int name=df3/int double name=tf-idf0./double /lst lst name=tomato int name=tf2/int int name=df1/int double name=tf-idf2.0/double /lst /lst /lst /lst Now, my question is: why I get the tf for all the indexed keywords into field foods and not only for my keyword search (pizza)? Please, have you any suggests? Thanks in advance -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: get term frequency, just only keywords search
Try the TermsComponent. It will return one or more terms and their counts for a given field only. -Original message- From:danielitos85 danydany@gmail.com Sent: Wednesday 14th August 2013 11:30 To: solr-user@lucene.apache.org Subject: get term frequency, just only keywords search I need to get the TermFrequency in Solr4 but just about my keywords search and not for all the keywords that a field cointains. I try to explain with an example: I have this fiels: field name=idtype=string indexed=true stored=truerequired=true / field name=foods type=text indexed=true stored=truerequired=truetermVectors=true termPositions=truetermOffsets=true/ when I try to searh into field foods the keyword pizza like this: http://localhost:8983/solr/mycore/select?q=pizzafl=idtv=truedf=foodstv.tf_idf=truetv.tf=truetv.df=truetv.fl=foodsomitHeader=truedefType=dismax this is my results: response result name=response numFound=1 start=0 doc str name=id1/str /doc /result lst name=termVectors str name=uniqueKeyFieldNameid/str lst name=1 str name=uniqueKey1/str lst name=foods lst name=pizza int name=tf5/int int name=df3/int double name=tf-idf1.6667/double /lst lst name=ice-cream int name=tf2/int int name=df3/int double name=tf-idf0./double /lst lst name=tomato int name=tf2/int int name=df1/int double name=tf-idf2.0/double /lst /lst /lst /lst Now, my question is: why I get the tf for all the indexed keywords into field foods and not only for my keyword search (pizza)? Please, have you any suggests? Thanks in advance -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: get term frequency, just only keywords search
Thanks for your answer, but I'm tring to use TermsComponent but the output is similar. TermsComponent returns all the terms (*and not just for the term that I have search*) and their counts for a given field. something wrong? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4084518.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: get term frequency, just only keywords search
Why? Using terms.limit or a ^term$ regex should limit the response to the exact term right? -Original message- From:danielitos85 danydany@gmail.com Sent: Wednesday 14th August 2013 12:20 To: solr-user@lucene.apache.org Subject: RE: get term frequency, just only keywords search Thanks for your answer, but I'm tring to use TermsComponent but the output is similar. TermsComponent returns all the terms (*and not just for the term that I have search*) and their counts for a given field. something wrong? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4084518.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: get term frequency, just only keywords search
thanks a lot Markus ;) If I use regex parameter it works -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4084525.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: get term frequency, just only keywords search
sorry, but now I give more attention at the results and it don't return that I needed. If I have two documents indexed: the text of my first document: ice-cream pizza pizza pizza the text of my second document: pizza tomato it returns the followed code: response result name=response numFound=2 start=0 doc str name=id1/str /doc doc str name=id3/str /doc /result lst name=terms lst name=mytext int name=pizza2/int *I want that it returns 3 for the first document and 1 for the second document and not 2* /lst /lst /response Thanks a lot -- View this message in context: http://lucene.472066.n3.nabble.com/SOLVED-get-term-frequency-just-only-keywords-search-tp4084510p4084530.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
You can use the termfreq or tf function query in your field list to return the term frequency for a term, like: fl=id,tf(foods,'pizza') -- Jack Krupansky -Original Message- From: danielitos85 Sent: Wednesday, August 14, 2013 5:29 AM To: solr-user@lucene.apache.org Subject: get term frequency, just only keywords search I need to get the TermFrequency in Solr4 but just about my keywords search and not for all the keywords that a field cointains. I try to explain with an example: I have this fiels: field name=idtype=string indexed=true stored=truerequired=true / field name=foods type=text indexed=true stored=truerequired=truetermVectors=true termPositions=truetermOffsets=true/ when I try to searh into field foods the keyword pizza like this: http://localhost:8983/solr/mycore/select?q=pizzafl=idtv=truedf=foodstv.tf_idf=truetv.tf=truetv.df=truetv.fl=foodsomitHeader=truedefType=dismax this is my results: response result name=response numFound=1 start=0 doc str name=id1/str /doc /result lst name=termVectors str name=uniqueKeyFieldNameid/str lst name=1 str name=uniqueKey1/str lst name=foods lst name=pizza int name=tf5/int int name=df3/int double name=tf-idf1.6667/double /lst lst name=ice-cream int name=tf2/int int name=df3/int double name=tf-idf0./double /lst lst name=tomato int name=tf2/int int name=df1/int double name=tf-idf2.0/double /lst /lst /lst /lst Now, my question is: why I get the tf for all the indexed keywords into field foods and not only for my keyword search (pizza)? Please, have you any suggests? Thanks in advance -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
Thanks Jack, I'm tring to use tf function but I don't understand: why he returns a float value and not integer? At the start of this topic I explained an example where I used term Frequency but it don't works how I need because he returns the term frequency about all the terms of my field. Thanks in advance. -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4084629.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
The rationale for float vs. integer is probably that function queries are primarily intended for computing a boost for scoring which is a float. The term vectors search component is designed to provide all term vectors for each selected document. -- Jack Krupansky -Original Message- From: danielitos85 Sent: Wednesday, August 14, 2013 12:50 PM To: solr-user@lucene.apache.org Subject: Re: get term frequency, just only keywords search Thanks Jack, I'm tring to use tf function but I don't understand: why he returns a float value and not integer? At the start of this topic I explained an example where I used term Frequency but it don't works how I need because he returns the term frequency about all the terms of my field. Thanks in advance. -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4084629.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
Thanks. I tried to use termfreq function and it is ok for me but, last my question is: if I have a query like this: q=pizza+tomatofl=id,termfreq(myfield,**) how set my query (pizza+tomato) in second param in termfreq function? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4084643.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: get term frequency, just only keywords search
q=pizza+tomatofl=id,termfreq(myfield,'pizza'),termfreq(myfield,'tomato') -- Jack Krupansky -Original Message- From: danielitos85 Sent: Wednesday, August 14, 2013 1:22 PM To: solr-user@lucene.apache.org Subject: Re: get term frequency, just only keywords search Thanks. I tried to use termfreq function and it is ok for me but, last my question is: if I have a query like this: q=pizza+tomatofl=id,termfreq(myfield,**) how set my query (pizza+tomato) in second param in termfreq function? -- View this message in context: http://lucene.472066.n3.nabble.com/get-term-frequency-just-only-keywords-search-tp4084510p4084643.html Sent from the Solr - User mailing list archive at Nabble.com.
