Re: payload similarity

2015-04-25 Thread Dmitry Kan 
Erick,

That's a neat example. Thanks! Btw, noticed, that luke can decode the
payloads quite well. Extra goody under debugger's belt.

Regards,
Dmitry

On Fri, Apr 24, 2015 at 9:22 PM, Erick Erickson erickerick...@gmail.com
wrote:

 I put up a complete example not too long ago that may help, see:

 http://lucidworks.com/blog/end-to-end-payload-example-in-solr/

 Best,
 Erick

 On Fri, Apr 24, 2015 at 6:33 AM, Dmitry Kan solrexp...@gmail.com wrote:
  Ahmet, exactly. As I have just illustrated with code, simultaneously with
  your reply. Thanks!
 
  On Fri, Apr 24, 2015 at 4:30 PM, Ahmet Arslan iori...@yahoo.com.invalid
 
  wrote:
 
  Hi Dmitry,
 
  I think, it is activated by PayloadTermQuery.
 
  Ahmet
 
 
 
  On Friday, April 24, 2015 2:51 PM, Dmitry Kan solrexp...@gmail.com
  wrote:
  Hi,
 
 
  Using the approach here
  http://lucidworks.com/blog/getting-started-with-payloads/ I have
  implemented my own PayloadSimilarity class. When debugging the code I
 have
  noticed, that the scorePayload method is never called. What could be
 wrong?
 
 
  [code]
 
  class PayloadSimilarity extends DefaultSimilarity {
  @Override
  public float scorePayload(int doc, int start, int end, BytesRef
  payload) {
  float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
  System.out.println(payloadValue =  + payloadValue);
  return payloadValue;
  }
  }
 
  [/code]
 
 
  Here is how the similarity is injected during indexing:
 
  [code]
 
  PayloadEncoder encoder = new FloatEncoder();
  IndexWriterConfig indexWriterConfig = new
  IndexWriterConfig(Version.LUCENE_4_10_4, new
  PayloadAnalyzer(encoder));
  payloadSimilarity = new PayloadSimilarity();
  indexWriterConfig.setSimilarity(payloadSimilarity);
  IndexWriter writer = new IndexWriter(dir, indexWriterConfig);
 
  [/code]
 
 
  and during searching:
 
  [code]
 
  IndexReader indexReader = DirectoryReader.open(dir);
  IndexSearcher searcher = new IndexSearcher(indexReader);
  searcher.setSimilarity(payloadSimilarity);
 
  TermQuery termQuery = new TermQuery(new Term(body, dogs));
  termQuery.setBoost(1.1f);
  TopDocs topDocs = searcher.search(termQuery, 10);
  printResults(searcher, termQuery, topDocs);
 
 
  [/code]
 
  --
  Dmitry Kan
  Luke Toolbox: http://github.com/DmitryKey/luke
  Blog: http://dmitrykan.blogspot.com
  Twitter: http://twitter.com/dmitrykan
  SemanticAnalyzer: www.semanticanalyzer.info
 
 
 
 
  --
  Dmitry Kan
  Luke Toolbox: http://github.com/DmitryKey/luke
  Blog: http://dmitrykan.blogspot.com
  Twitter: http://twitter.com/dmitrykan
  SemanticAnalyzer: www.semanticanalyzer.info




-- 
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info

Re: payload similarity

2015-04-24 Thread Dmitry Kan 
Answering my own question:

in order to account for payloads, PayloadTermQuery should be used instead
of TermQuery:

PayloadTermQuery payloadTermQuery = new PayloadTermQuery(new Term(body,
dogs), new MaxPayloadFunction());

Then in the query explanation we get:

---
Results for body:dogs of type:
org.apache.lucene.search.payloads.PayloadTermQuery
Doc: doc=0 score=3.125 shardIndex=-1
payloadValue = 10.0
Explain: 3.125 = (MATCH) btq, product of:
  0.3125 = weight(body:dogs in 0) [PayloadSimilarity], result of:
0.3125 = fieldWeight in 0, product of:
  1.0 = tf(freq=1.0), with freq of:
1.0 = phraseFreq=1.0
  1.0 = idf(docFreq=3, maxDocs=10)
  0.3125 = fieldNorm(doc=0)
  10.0 = MaxPayloadFunction.docScore()

Doc: doc=9 score=3.125 shardIndex=-1
payloadValue = 10.0
Explain: 3.125 = (MATCH) btq, product of:
  0.3125 = weight(body:dogs in 9) [PayloadSimilarity], result of:
0.3125 = fieldWeight in 9, product of:
  1.0 = tf(freq=1.0), with freq of:
1.0 = phraseFreq=1.0
  1.0 = idf(docFreq=3, maxDocs=10)
  0.3125 = fieldNorm(doc=9)
  10.0 = MaxPayloadFunction.docScore()

Doc: doc=1 score=0.3125 shardIndex=-1
Explain: 0.3125 = (MATCH) btq, product of:
  0.3125 = weight(body:dogs in 1) [PayloadSimilarity], result of:
0.3125 = fieldWeight in 1, product of:
  1.0 = tf(freq=1.0), with freq of:
1.0 = phraseFreq=1.0
  1.0 = idf(docFreq=3, maxDocs=10)
  0.3125 = fieldNorm(doc=1)
  1.0 = MaxPayloadFunction.docScore()

On Fri, Apr 24, 2015 at 2:50 PM, Dmitry Kan solrexp...@gmail.com wrote:

 Hi,


 Using the approach here
 http://lucidworks.com/blog/getting-started-with-payloads/ I have
 implemented my own PayloadSimilarity class. When debugging the code I have
 noticed, that the scorePayload method is never called. What could be wrong?


 [code]

 class PayloadSimilarity extends DefaultSimilarity {
 @Override
 public float scorePayload(int doc, int start, int end, BytesRef payload) {
 float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
 System.out.println(payloadValue =  + payloadValue);
 return payloadValue;
 }
 }

 [/code]


 Here is how the similarity is injected during indexing:

 [code]

 PayloadEncoder encoder = new FloatEncoder();
 IndexWriterConfig indexWriterConfig = new 
 IndexWriterConfig(Version.LUCENE_4_10_4, new PayloadAnalyzer(encoder));
 payloadSimilarity = new PayloadSimilarity();
 indexWriterConfig.setSimilarity(payloadSimilarity);
 IndexWriter writer = new IndexWriter(dir, indexWriterConfig);

 [/code]


 and during searching:

 [code]

 IndexReader indexReader = DirectoryReader.open(dir);
 IndexSearcher searcher = new IndexSearcher(indexReader);
 searcher.setSimilarity(payloadSimilarity);

 TermQuery termQuery = new TermQuery(new Term(body, dogs));
 termQuery.setBoost(1.1f);
 TopDocs topDocs = searcher.search(termQuery, 10);
 printResults(searcher, termQuery, topDocs);


 [/code]

 --
 Dmitry Kan
 Luke Toolbox: http://github.com/DmitryKey/luke
 Blog: http://dmitrykan.blogspot.com
 Twitter: http://twitter.com/dmitrykan
 SemanticAnalyzer: www.semanticanalyzer.info




-- 
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info

Re: payload similarity

2015-04-24 Thread Ahmet Arslan 
Hi Dmitry,

I think, it is activated by PayloadTermQuery.

Ahmet



On Friday, April 24, 2015 2:51 PM, Dmitry Kan solrexp...@gmail.com wrote:
Hi,


Using the approach here
http://lucidworks.com/blog/getting-started-with-payloads/ I have
implemented my own PayloadSimilarity class. When debugging the code I have
noticed, that the scorePayload method is never called. What could be wrong?


[code]

class PayloadSimilarity extends DefaultSimilarity {
@Override
public float scorePayload(int doc, int start, int end, BytesRef payload) {
float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
System.out.println(payloadValue =  + payloadValue);
return payloadValue;
}
}

[/code]


Here is how the similarity is injected during indexing:

[code]

PayloadEncoder encoder = new FloatEncoder();
IndexWriterConfig indexWriterConfig = new
IndexWriterConfig(Version.LUCENE_4_10_4, new
PayloadAnalyzer(encoder));
payloadSimilarity = new PayloadSimilarity();
indexWriterConfig.setSimilarity(payloadSimilarity);
IndexWriter writer = new IndexWriter(dir, indexWriterConfig);

[/code]


and during searching:

[code]

IndexReader indexReader = DirectoryReader.open(dir);
IndexSearcher searcher = new IndexSearcher(indexReader);
searcher.setSimilarity(payloadSimilarity);

TermQuery termQuery = new TermQuery(new Term(body, dogs));
termQuery.setBoost(1.1f);
TopDocs topDocs = searcher.search(termQuery, 10);
printResults(searcher, termQuery, topDocs);


[/code]

-- 
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info

Re: payload similarity

2015-04-24 Thread Dmitry Kan 
Ahmet, exactly. As I have just illustrated with code, simultaneously with
your reply. Thanks!

On Fri, Apr 24, 2015 at 4:30 PM, Ahmet Arslan iori...@yahoo.com.invalid
wrote:

 Hi Dmitry,

 I think, it is activated by PayloadTermQuery.

 Ahmet



 On Friday, April 24, 2015 2:51 PM, Dmitry Kan solrexp...@gmail.com
 wrote:
 Hi,


 Using the approach here
 http://lucidworks.com/blog/getting-started-with-payloads/ I have
 implemented my own PayloadSimilarity class. When debugging the code I have
 noticed, that the scorePayload method is never called. What could be wrong?


 [code]

 class PayloadSimilarity extends DefaultSimilarity {
 @Override
 public float scorePayload(int doc, int start, int end, BytesRef
 payload) {
 float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
 System.out.println(payloadValue =  + payloadValue);
 return payloadValue;
 }
 }

 [/code]


 Here is how the similarity is injected during indexing:

 [code]

 PayloadEncoder encoder = new FloatEncoder();
 IndexWriterConfig indexWriterConfig = new
 IndexWriterConfig(Version.LUCENE_4_10_4, new
 PayloadAnalyzer(encoder));
 payloadSimilarity = new PayloadSimilarity();
 indexWriterConfig.setSimilarity(payloadSimilarity);
 IndexWriter writer = new IndexWriter(dir, indexWriterConfig);

 [/code]


 and during searching:

 [code]

 IndexReader indexReader = DirectoryReader.open(dir);
 IndexSearcher searcher = new IndexSearcher(indexReader);
 searcher.setSimilarity(payloadSimilarity);

 TermQuery termQuery = new TermQuery(new Term(body, dogs));
 termQuery.setBoost(1.1f);
 TopDocs topDocs = searcher.search(termQuery, 10);
 printResults(searcher, termQuery, topDocs);


 [/code]

 --
 Dmitry Kan
 Luke Toolbox: http://github.com/DmitryKey/luke
 Blog: http://dmitrykan.blogspot.com
 Twitter: http://twitter.com/dmitrykan
 SemanticAnalyzer: www.semanticanalyzer.info




-- 
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info

Re: payload similarity

2015-04-24 Thread Erick Erickson 
I put up a complete example not too long ago that may help, see:

http://lucidworks.com/blog/end-to-end-payload-example-in-solr/

Best,
Erick

On Fri, Apr 24, 2015 at 6:33 AM, Dmitry Kan solrexp...@gmail.com wrote:
 Ahmet, exactly. As I have just illustrated with code, simultaneously with
 your reply. Thanks!

 On Fri, Apr 24, 2015 at 4:30 PM, Ahmet Arslan iori...@yahoo.com.invalid
 wrote:

 Hi Dmitry,

 I think, it is activated by PayloadTermQuery.

 Ahmet



 On Friday, April 24, 2015 2:51 PM, Dmitry Kan solrexp...@gmail.com
 wrote:
 Hi,


 Using the approach here
 http://lucidworks.com/blog/getting-started-with-payloads/ I have
 implemented my own PayloadSimilarity class. When debugging the code I have
 noticed, that the scorePayload method is never called. What could be wrong?


 [code]

 class PayloadSimilarity extends DefaultSimilarity {
 @Override
 public float scorePayload(int doc, int start, int end, BytesRef
 payload) {
 float payloadValue = PayloadHelper.decodeFloat(payload.bytes);
 System.out.println(payloadValue =  + payloadValue);
 return payloadValue;
 }
 }

 [/code]


 Here is how the similarity is injected during indexing:

 [code]

 PayloadEncoder encoder = new FloatEncoder();
 IndexWriterConfig indexWriterConfig = new
 IndexWriterConfig(Version.LUCENE_4_10_4, new
 PayloadAnalyzer(encoder));
 payloadSimilarity = new PayloadSimilarity();
 indexWriterConfig.setSimilarity(payloadSimilarity);
 IndexWriter writer = new IndexWriter(dir, indexWriterConfig);

 [/code]


 and during searching:

 [code]

 IndexReader indexReader = DirectoryReader.open(dir);
 IndexSearcher searcher = new IndexSearcher(indexReader);
 searcher.setSimilarity(payloadSimilarity);

 TermQuery termQuery = new TermQuery(new Term(body, dogs));
 termQuery.setBoost(1.1f);
 TopDocs topDocs = searcher.search(termQuery, 10);
 printResults(searcher, termQuery, topDocs);


 [/code]

 --
 Dmitry Kan
 Luke Toolbox: http://github.com/DmitryKey/luke
 Blog: http://dmitrykan.blogspot.com
 Twitter: http://twitter.com/dmitrykan
 SemanticAnalyzer: www.semanticanalyzer.info




 --
 Dmitry Kan
 Luke Toolbox: http://github.com/DmitryKey/luke
 Blog: http://dmitrykan.blogspot.com
 Twitter: http://twitter.com/dmitrykan
 SemanticAnalyzer: www.semanticanalyzer.info