Pass Analyzed Field to SignatureUpdateProcessorFactory
Hi all,
We have successfully integrated Solr in our application, and now we are
facing a requirement where the application should be able to search for
duplicate records in Solr core based on equality in 3 distinct fields.
Tried using SignatureUpdateProcessorFactory as described in
https://cwiki.apache.org/confluence/display/solr/De-Duplication and
Lookup3Signature and everything seems to work fine, signature field is
being filled with unique hash values.
One issue we have, is that we need to pass to
SignatureUpdateProcessorFactory the stemmed value of 1 of 3 fields.
Currenty, the following documents produce different hash values, and we
need them to produce unique.
Analysis for field1 and values "value1_a" and "value1_b" produce stemmed
value "value1"
documentA: {
field1: value1_a,
field2: value2,
field3: value3,
signature: hash_value1
}
documentB: {
field1: value1_b,
field2: value2,
field3: value3,
signature: hash_value2
}
I would like to ask whether it is possible to have required behavior, and
some tips about how to accomplish this task.
Thank you in advance,
Leonidas
RE: Pass Analyzed Field to SignatureUpdateProcessorFactory
Hello,
This is not possible out of the box, you would need to manually pass the input
through an analyzer with a tokenizer and your steming token filter, and put the
output together again.
Markus
-Original message-
> From:Leonidas Zagkaretos
> Sent: Wednesday 25th January 2017 17:51
> To: solr-user@lucene.apache.org
> Subject: Pass Analyzed Field to SignatureUpdateProcessorFactory
>
> Hi all,
>
> We have successfully integrated Solr in our application, and now we are
> facing a requirement where the application should be able to search for
> duplicate records in Solr core based on equality in 3 distinct fields.
>
> Tried using SignatureUpdateProcessorFactory as described in
> https://cwiki.apache.org/confluence/display/solr/De-Duplication and
> Lookup3Signature and everything seems to work fine, signature field is
> being filled with unique hash values.
>
> One issue we have, is that we need to pass to
> SignatureUpdateProcessorFactory the stemmed value of 1 of 3 fields.
> Currenty, the following documents produce different hash values, and we
> need them to produce unique.
> Analysis for field1 and values "value1_a" and "value1_b" produce stemmed
> value "value1"
>
> documentA: {
> field1: value1_a,
> field2: value2,
> field3: value3,
> signature: hash_value1
> }
>
> documentB: {
> field1: value1_b,
> field2: value2,
> field3: value3,
> signature: hash_value2
> }
>
> I would like to ask whether it is possible to have required behavior, and
> some tips about how to accomplish this task.
>
> Thank you in advance,
>
> Leonidas
>
Re: Pass Analyzed Field to SignatureUpdateProcessorFactory
It might be possible by sticking additional update request processors
before the signature one. For example clone field, regex instead of
tokenizing on the clone, then signature. If a clone is too much of a
burden, it may even be possible to then add IgnoreField URP to remove
it or map it in the schema to index/store/docValues=false field.
Regards,
Alex.
P.s. The full all-in-one list of URPs is available at:
http://www.solr-start.com/info/update-request-processors/
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 25 January 2017 at 12:00, Markus Jelsma wrote:
> Hello,
>
> This is not possible out of the box, you would need to manually pass the
> input through an analyzer with a tokenizer and your steming token filter, and
> put the output together again.
>
> Markus
>
>
>
> -Original message-
>> From:Leonidas Zagkaretos
>> Sent: Wednesday 25th January 2017 17:51
>> To: solr-user@lucene.apache.org
>> Subject: Pass Analyzed Field to SignatureUpdateProcessorFactory
>>
>> Hi all,
>>
>> We have successfully integrated Solr in our application, and now we are
>> facing a requirement where the application should be able to search for
>> duplicate records in Solr core based on equality in 3 distinct fields.
>>
>> Tried using SignatureUpdateProcessorFactory as described in
>> https://cwiki.apache.org/confluence/display/solr/De-Duplication and
>> Lookup3Signature and everything seems to work fine, signature field is
>> being filled with unique hash values.
>>
>> One issue we have, is that we need to pass to
>> SignatureUpdateProcessorFactory the stemmed value of 1 of 3 fields.
>> Currenty, the following documents produce different hash values, and we
>> need them to produce unique.
>> Analysis for field1 and values "value1_a" and "value1_b" produce stemmed
>> value "value1"
>>
>> documentA: {
>> field1: value1_a,
>> field2: value2,
>> field3: value3,
>> signature: hash_value1
>> }
>>
>> documentB: {
>> field1: value1_b,
>> field2: value2,
>> field3: value3,
>> signature: hash_value2
>> }
>>
>> I would like to ask whether it is possible to have required behavior, and
>> some tips about how to accomplish this task.
>>
>> Thank you in advance,
>>
>> Leonidas
>>
Re: Pass Analyzed Field to SignatureUpdateProcessorFactory
Finally, I was able to implement desirable behavior using your suggestions
as follows:
- Added StatelessScriptUpdateProcessorFactory before
SignatureUpdateProcessorFactory in order to analyze "field1" and set
analyzed value to "field1_tmp_ss"
- Passed "field1_tmp_ss" to SignatureUpdateProcessorFactory
- Used IgnoreFieldUpdateProcessorFactory to ignore "field1_tmp_ss" from
document stored
Everything seems to work fine and as expected.
Thank you very much,
Have a nice day,
Leonidas
2017-01-25 19:19 GMT+02:00 Alexandre Rafalovitch :
> It might be possible by sticking additional update request processors
> before the signature one. For example clone field, regex instead of
> tokenizing on the clone, then signature. If a clone is too much of a
> burden, it may even be possible to then add IgnoreField URP to remove
> it or map it in the schema to index/store/docValues=false field.
>
> Regards,
>Alex.
> P.s. The full all-in-one list of URPs is available at:
> http://www.solr-start.com/info/update-request-processors/
>
>
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 25 January 2017 at 12:00, Markus Jelsma
> wrote:
> > Hello,
> >
> > This is not possible out of the box, you would need to manually pass the
> input through an analyzer with a tokenizer and your steming token filter,
> and put the output together again.
> >
> > Markus
> >
> >
> >
> > -----Original message-----
> >> From:Leonidas Zagkaretos
> >> Sent: Wednesday 25th January 2017 17:51
> >> To: solr-user@lucene.apache.org
> >> Subject: Pass Analyzed Field to SignatureUpdateProcessorFactory
> >>
> >> Hi all,
> >>
> >> We have successfully integrated Solr in our application, and now we are
> >> facing a requirement where the application should be able to search for
> >> duplicate records in Solr core based on equality in 3 distinct fields.
> >>
> >> Tried using SignatureUpdateProcessorFactory as described in
> >> https://cwiki.apache.org/confluence/display/solr/De-Duplication and
> >> Lookup3Signature and everything seems to work fine, signature field is
> >> being filled with unique hash values.
> >>
> >> One issue we have, is that we need to pass to
> >> SignatureUpdateProcessorFactory the stemmed value of 1 of 3 fields.
> >> Currenty, the following documents produce different hash values, and we
> >> need them to produce unique.
> >> Analysis for field1 and values "value1_a" and "value1_b" produce stemmed
> >> value "value1"
> >>
> >> documentA: {
> >> field1: value1_a,
> >> field2: value2,
> >> field3: value3,
> >> signature: hash_value1
> >> }
> >>
> >> documentB: {
> >> field1: value1_b,
> >> field2: value2,
> >> field3: value3,
> >> signature: hash_value2
> >> }
> >>
> >> I would like to ask whether it is possible to have required behavior,
> and
> >> some tips about how to accomplish this task.
> >>
> >> Thank you in advance,
> >>
> >> Leonidas
> >>
>
