Re: payload similarity
Erick,
That's a neat example. Thanks! Btw, noticed, that luke can decode the
payloads quite well. Extra goody under debugger's belt.
Regards,
Dmitry
On Fri, Apr 24, 2015 at 9:22 PM, Erick Erickson
wrote:
> I put up a complete example not too long ago that may help, see:
>
> http://lucidworks.com/blog/end-to-end-payload-example-in-solr/
>
> Best,
> Erick
>
> On Fri, Apr 24, 2015 at 6:33 AM, Dmitry Kan wrote:
> > Ahmet, exactly. As I have just illustrated with code, simultaneously with
> > your reply. Thanks!
> >
> > On Fri, Apr 24, 2015 at 4:30 PM, Ahmet Arslan >
> > wrote:
> >
> >> Hi Dmitry,
> >>
> >> I think, it is activated by PayloadTermQuery.
> >>
> >> Ahmet
> >>
> >>
> >>
> >> On Friday, April 24, 2015 2:51 PM, Dmitry Kan
> >> wrote:
> >> Hi,
> >>
> >>
> >> Using the approach here
> >> http://lucidworks.com/blog/getting-started-with-payloads/ I have
> >> implemented my own PayloadSimilarity class. When debugging the code I
> have
> >> noticed, that the scorePayload method is never called. What could be
> wrong?
> >>
> >>
> >> [code]
> >>
> >> class PayloadSimilarity extends DefaultSimilarity {
> >> @Override
> >> public float scorePayload(int doc, int start, int end, BytesRef
> >> payload) {
> >> float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
> >> System.out.println("payloadValue = " + payloadValue);
> >> return payloadValue;
> >> }
> >> }
> >>
> >> [/code]
> >>
> >>
> >> Here is how the similarity is injected during indexing:
> >>
> >> [code]
> >>
> >> PayloadEncoder encoder = new FloatEncoder();
> >> IndexWriterConfig indexWriterConfig = new
> >> IndexWriterConfig(Version.LUCENE_4_10_4, new
> >> PayloadAnalyzer(encoder));
> >> payloadSimilarity = new PayloadSimilarity();
> >> indexWriterConfig.setSimilarity(payloadSimilarity);
> >> IndexWriter writer = new IndexWriter(dir, indexWriterConfig);
> >>
> >> [/code]
> >>
> >>
> >> and during searching:
> >>
> >> [code]
> >>
> >> IndexReader indexReader = DirectoryReader.open(dir);
> >> IndexSearcher searcher = new IndexSearcher(indexReader);
> >> searcher.setSimilarity(payloadSimilarity);
> >>
> >> TermQuery termQuery = new TermQuery(new Term("body", "dogs"));
> >> termQuery.setBoost(1.1f);
> >> TopDocs topDocs = searcher.search(termQuery, 10);
> >> printResults(searcher, termQuery, topDocs);
> >>
> >>
> >> [/code]
> >>
> >> --
> >> Dmitry Kan
> >> Luke Toolbox: http://github.com/DmitryKey/luke
> >> Blog: http://dmitrykan.blogspot.com
> >> Twitter: http://twitter.com/dmitrykan
> >> SemanticAnalyzer: www.semanticanalyzer.info
> >>
> >
> >
> >
> > --
> > Dmitry Kan
> > Luke Toolbox: http://github.com/DmitryKey/luke
> > Blog: http://dmitrykan.blogspot.com
> > Twitter: http://twitter.com/dmitrykan
> > SemanticAnalyzer: www.semanticanalyzer.info
>
--
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info
Re: payload similarity
I put up a complete example not too long ago that may help, see:
http://lucidworks.com/blog/end-to-end-payload-example-in-solr/
Best,
Erick
On Fri, Apr 24, 2015 at 6:33 AM, Dmitry Kan wrote:
> Ahmet, exactly. As I have just illustrated with code, simultaneously with
> your reply. Thanks!
>
> On Fri, Apr 24, 2015 at 4:30 PM, Ahmet Arslan
> wrote:
>
>> Hi Dmitry,
>>
>> I think, it is activated by PayloadTermQuery.
>>
>> Ahmet
>>
>>
>>
>> On Friday, April 24, 2015 2:51 PM, Dmitry Kan
>> wrote:
>> Hi,
>>
>>
>> Using the approach here
>> http://lucidworks.com/blog/getting-started-with-payloads/ I have
>> implemented my own PayloadSimilarity class. When debugging the code I have
>> noticed, that the scorePayload method is never called. What could be wrong?
>>
>>
>> [code]
>>
>> class PayloadSimilarity extends DefaultSimilarity {
>> @Override
>> public float scorePayload(int doc, int start, int end, BytesRef
>> payload) {
>> float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
>> System.out.println("payloadValue = " + payloadValue);
>> return payloadValue;
>> }
>> }
>>
>> [/code]
>>
>>
>> Here is how the similarity is injected during indexing:
>>
>> [code]
>>
>> PayloadEncoder encoder = new FloatEncoder();
>> IndexWriterConfig indexWriterConfig = new
>> IndexWriterConfig(Version.LUCENE_4_10_4, new
>> PayloadAnalyzer(encoder));
>> payloadSimilarity = new PayloadSimilarity();
>> indexWriterConfig.setSimilarity(payloadSimilarity);
>> IndexWriter writer = new IndexWriter(dir, indexWriterConfig);
>>
>> [/code]
>>
>>
>> and during searching:
>>
>> [code]
>>
>> IndexReader indexReader = DirectoryReader.open(dir);
>> IndexSearcher searcher = new IndexSearcher(indexReader);
>> searcher.setSimilarity(payloadSimilarity);
>>
>> TermQuery termQuery = new TermQuery(new Term("body", "dogs"));
>> termQuery.setBoost(1.1f);
>> TopDocs topDocs = searcher.search(termQuery, 10);
>> printResults(searcher, termQuery, topDocs);
>>
>>
>> [/code]
>>
>> --
>> Dmitry Kan
>> Luke Toolbox: http://github.com/DmitryKey/luke
>> Blog: http://dmitrykan.blogspot.com
>> Twitter: http://twitter.com/dmitrykan
>> SemanticAnalyzer: www.semanticanalyzer.info
>>
>
>
>
> --
> Dmitry Kan
> Luke Toolbox: http://github.com/DmitryKey/luke
> Blog: http://dmitrykan.blogspot.com
> Twitter: http://twitter.com/dmitrykan
> SemanticAnalyzer: www.semanticanalyzer.info
Re: payload similarity
Ahmet, exactly. As I have just illustrated with code, simultaneously with
your reply. Thanks!
On Fri, Apr 24, 2015 at 4:30 PM, Ahmet Arslan
wrote:
> Hi Dmitry,
>
> I think, it is activated by PayloadTermQuery.
>
> Ahmet
>
>
>
> On Friday, April 24, 2015 2:51 PM, Dmitry Kan
> wrote:
> Hi,
>
>
> Using the approach here
> http://lucidworks.com/blog/getting-started-with-payloads/ I have
> implemented my own PayloadSimilarity class. When debugging the code I have
> noticed, that the scorePayload method is never called. What could be wrong?
>
>
> [code]
>
> class PayloadSimilarity extends DefaultSimilarity {
> @Override
> public float scorePayload(int doc, int start, int end, BytesRef
> payload) {
> float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
> System.out.println("payloadValue = " + payloadValue);
> return payloadValue;
> }
> }
>
> [/code]
>
>
> Here is how the similarity is injected during indexing:
>
> [code]
>
> PayloadEncoder encoder = new FloatEncoder();
> IndexWriterConfig indexWriterConfig = new
> IndexWriterConfig(Version.LUCENE_4_10_4, new
> PayloadAnalyzer(encoder));
> payloadSimilarity = new PayloadSimilarity();
> indexWriterConfig.setSimilarity(payloadSimilarity);
> IndexWriter writer = new IndexWriter(dir, indexWriterConfig);
>
> [/code]
>
>
> and during searching:
>
> [code]
>
> IndexReader indexReader = DirectoryReader.open(dir);
> IndexSearcher searcher = new IndexSearcher(indexReader);
> searcher.setSimilarity(payloadSimilarity);
>
> TermQuery termQuery = new TermQuery(new Term("body", "dogs"));
> termQuery.setBoost(1.1f);
> TopDocs topDocs = searcher.search(termQuery, 10);
> printResults(searcher, termQuery, topDocs);
>
>
> [/code]
>
> --
> Dmitry Kan
> Luke Toolbox: http://github.com/DmitryKey/luke
> Blog: http://dmitrykan.blogspot.com
> Twitter: http://twitter.com/dmitrykan
> SemanticAnalyzer: www.semanticanalyzer.info
>
--
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info
Re: payload similarity
Answering my own question:
in order to account for payloads, PayloadTermQuery should be used instead
of TermQuery:
PayloadTermQuery payloadTermQuery = new PayloadTermQuery(new Term("body",
"dogs"), new MaxPayloadFunction());
Then in the query explanation we get:
---
Results for body:dogs of type:
org.apache.lucene.search.payloads.PayloadTermQuery
Doc: doc=0 score=3.125 shardIndex=-1
payloadValue = 10.0
Explain: 3.125 = (MATCH) btq, product of:
0.3125 = weight(body:dogs in 0) [PayloadSimilarity], result of:
0.3125 = fieldWeight in 0, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = phraseFreq=1.0
1.0 = idf(docFreq=3, maxDocs=10)
0.3125 = fieldNorm(doc=0)
10.0 = MaxPayloadFunction.docScore()
Doc: doc=9 score=3.125 shardIndex=-1
payloadValue = 10.0
Explain: 3.125 = (MATCH) btq, product of:
0.3125 = weight(body:dogs in 9) [PayloadSimilarity], result of:
0.3125 = fieldWeight in 9, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = phraseFreq=1.0
1.0 = idf(docFreq=3, maxDocs=10)
0.3125 = fieldNorm(doc=9)
10.0 = MaxPayloadFunction.docScore()
Doc: doc=1 score=0.3125 shardIndex=-1
Explain: 0.3125 = (MATCH) btq, product of:
0.3125 = weight(body:dogs in 1) [PayloadSimilarity], result of:
0.3125 = fieldWeight in 1, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = phraseFreq=1.0
1.0 = idf(docFreq=3, maxDocs=10)
0.3125 = fieldNorm(doc=1)
1.0 = MaxPayloadFunction.docScore()
On Fri, Apr 24, 2015 at 2:50 PM, Dmitry Kan wrote:
> Hi,
>
>
> Using the approach here
> http://lucidworks.com/blog/getting-started-with-payloads/ I have
> implemented my own PayloadSimilarity class. When debugging the code I have
> noticed, that the scorePayload method is never called. What could be wrong?
>
>
> [code]
>
> class PayloadSimilarity extends DefaultSimilarity {
> @Override
> public float scorePayload(int doc, int start, int end, BytesRef payload) {
> float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
> System.out.println("payloadValue = " + payloadValue);
> return payloadValue;
> }
> }
>
> [/code]
>
>
> Here is how the similarity is injected during indexing:
>
> [code]
>
> PayloadEncoder encoder = new FloatEncoder();
> IndexWriterConfig indexWriterConfig = new
> IndexWriterConfig(Version.LUCENE_4_10_4, new PayloadAnalyzer(encoder));
> payloadSimilarity = new PayloadSimilarity();
> indexWriterConfig.setSimilarity(payloadSimilarity);
> IndexWriter writer = new IndexWriter(dir, indexWriterConfig);
>
> [/code]
>
>
> and during searching:
>
> [code]
>
> IndexReader indexReader = DirectoryReader.open(dir);
> IndexSearcher searcher = new IndexSearcher(indexReader);
> searcher.setSimilarity(payloadSimilarity);
>
> TermQuery termQuery = new TermQuery(new Term("body", "dogs"));
> termQuery.setBoost(1.1f);
> TopDocs topDocs = searcher.search(termQuery, 10);
> printResults(searcher, termQuery, topDocs);
>
>
> [/code]
>
> --
> Dmitry Kan
> Luke Toolbox: http://github.com/DmitryKey/luke
> Blog: http://dmitrykan.blogspot.com
> Twitter: http://twitter.com/dmitrykan
> SemanticAnalyzer: www.semanticanalyzer.info
>
>
--
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info
Re: payload similarity
Hi Dmitry,
I think, it is activated by PayloadTermQuery.
Ahmet
On Friday, April 24, 2015 2:51 PM, Dmitry Kan wrote:
Hi,
Using the approach here
http://lucidworks.com/blog/getting-started-with-payloads/ I have
implemented my own PayloadSimilarity class. When debugging the code I have
noticed, that the scorePayload method is never called. What could be wrong?
[code]
class PayloadSimilarity extends DefaultSimilarity {
@Override
public float scorePayload(int doc, int start, int end, BytesRef payload) {
float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
System.out.println("payloadValue = " + payloadValue);
return payloadValue;
}
}
[/code]
Here is how the similarity is injected during indexing:
[code]
PayloadEncoder encoder = new FloatEncoder();
IndexWriterConfig indexWriterConfig = new
IndexWriterConfig(Version.LUCENE_4_10_4, new
PayloadAnalyzer(encoder));
payloadSimilarity = new PayloadSimilarity();
indexWriterConfig.setSimilarity(payloadSimilarity);
IndexWriter writer = new IndexWriter(dir, indexWriterConfig);
[/code]
and during searching:
[code]
IndexReader indexReader = DirectoryReader.open(dir);
IndexSearcher searcher = new IndexSearcher(indexReader);
searcher.setSimilarity(payloadSimilarity);
TermQuery termQuery = new TermQuery(new Term("body", "dogs"));
termQuery.setBoost(1.1f);
TopDocs topDocs = searcher.search(termQuery, 10);
printResults(searcher, termQuery, topDocs);
[/code]
--
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info
payload similarity
Hi,
Using the approach here
http://lucidworks.com/blog/getting-started-with-payloads/ I have
implemented my own PayloadSimilarity class. When debugging the code I have
noticed, that the scorePayload method is never called. What could be wrong?
[code]
class PayloadSimilarity extends DefaultSimilarity {
@Override
public float scorePayload(int doc, int start, int end, BytesRef payload) {
float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
System.out.println("payloadValue = " + payloadValue);
return payloadValue;
}
}
[/code]
Here is how the similarity is injected during indexing:
[code]
PayloadEncoder encoder = new FloatEncoder();
IndexWriterConfig indexWriterConfig = new
IndexWriterConfig(Version.LUCENE_4_10_4, new
PayloadAnalyzer(encoder));
payloadSimilarity = new PayloadSimilarity();
indexWriterConfig.setSimilarity(payloadSimilarity);
IndexWriter writer = new IndexWriter(dir, indexWriterConfig);
[/code]
and during searching:
[code]
IndexReader indexReader = DirectoryReader.open(dir);
IndexSearcher searcher = new IndexSearcher(indexReader);
searcher.setSimilarity(payloadSimilarity);
TermQuery termQuery = new TermQuery(new Term("body", "dogs"));
termQuery.setBoost(1.1f);
TopDocs topDocs = searcher.search(termQuery, 10);
printResults(searcher, termQuery, topDocs);
[/code]
--
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info
