Re: Solr SearchComponent-like functionality?
Thanks for your answer ! I have some followup questions
1. I wrote a custom script to return a constant score of 2.5. It works for
the following query
{
"query" : {
"function_score": {
"query":{
"match_all":{}
},
"script_score": {
"lang":"native",
"script": "constant-score"
}
}
}
}
in the result i see the value of _score is "2.5" for all docs. But when i
include a query string it doesnt work
{
"query" : {
"function_score": {
"query":{
"query_string":{
"query": "dog"
}
},
"script_score": {
"lang":"native",
"script": "constant-score"
}
}
}
}
2. I would like to access the search keyword (if possible analyzed version)
to some custom processing depending on the keyword. is that possible ?
3. You mentioned I can consider rescore api, I am interested in trying that
out. I have some external data stored outside the index (in-memory) , how
do i make the rescore function pick up data from that external data
structure instead of index ?
Thanks in advance for any help !
Thanks
Srini
On Friday, April 18, 2014 11:48:25 AM UTC-7, Matt Weber wrote:
>
> Well, the scripts runs against all matching documents of the query so you
> can do a match_all query [1] to have the logic applied to all your
> documents. This is going to be expensive though, so try to filter out as
> many documents as possible before applying the custom scoring. Maybe even
> perform a rescore [2] on the top X docs. It really all depends on your
> requirements though. Run some tests and tune based on those results.
>
> When I said to be careful. I mean don't do a lot of blocking IO or long
> running calculations as the script is ran against each matching document.
> Cache results and make the script return as quick as possible.
>
> [1]
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-match-all-query.html
> [2]
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-rescore.html
>
> Thanks,
> Matt Weber
>
>
>
> On Fri, Apr 18, 2014 at 9:46 AM, Srinivasan Ramaswamy
>
> > wrote:
>
>> Thats great, thanks for your reply. This looks like a good solution for
>> my requirement ! Is this script applied in each shard ? I want to apply
>> this function to all the documents so that the Top N picked from each shard
>> is picked by my custom score.
>>
>> Also, can you elaborate a little bit on "be careful you can significantly
>> impact your query performance if you are not careful". I would like to
>> understand the best practices there.
>>
>> On Friday, April 18, 2014 8:14:54 AM UTC-7, Matt Weber wrote:
>>>
>>> Yes, you can use the Function Score Query [1] in combination with a
>>> native script written in java [2]. With the native script you can
>>> basically do whatever you want, but be careful you can significantly impact
>>> your query performance if you are not careful.
>>>
>>> [1] http://www.elasticsearch.org/guide/en/elasticsearch/
>>> reference/current/query-dsl-function-score-query.html
>>> [2] http://www.elasticsearch.org/guide/en/elasticsearch/
>>> reference/current/modules-scripting.html#_native_java_scripts
>>>
>>> Thanks,
>>> Matt Weber
>>>
>>>
>>> On Thu, Apr 17, 2014 at 11:54 PM, Srinivasan Ramaswamy <
>>> ursv...@gmail.com> wrote:
>>>
I would like to influence the ranking with few fields that are not
stored in the index (eg click data for keyword-documents). I have used
custom SearchComponent in Solr to implement similar functionality in the
past. I am wondering how can i achieve the same in ElasticSearch.
I know this thread is a very old thread, but i didnt find much
information on how to do custom scoring (in elasticsearch) with data thats
not stored in the index. This thread looked very relevant to my
requirement, so trying to see whether you guys have solved similar
requirements with elasticsearch.
Thanks
Srini
On Wednesday, September 7, 2011 12:18:09 PM UTC-7, Lukáš Vlček wrote:
>
> Hi Otis,
>
> So if I understand it correctly (providing my knowledge is quite
> limited here) you are asking if
> 1) it is possible to hook into query processing flow and inject or
> extend custom handlers for individual flow phases and
> 2) if we can find in ES the same functionality which is currently
> provided by components listed here: http://wiki.apache.org/s
> olr/SearchComponent (or here: http://lucene.apache.org/solr/
> api/org/apache/solr/handler/component/SearchComponent.html).
>
> As for #1, frankly, I do not know. I have been playing with plugins a
> bit but did not have a chance to explore full potential of it yet. I
> remember that Shay mentioned that not every aspect of ES is pluggable now
> but that is all I know about it (personally, I did not hit the limits by
> myself yet, may be I would if I wanted to employ Carrot2 clus
Re: Solr SearchComponent-like functionality?
Well, the scripts runs against all matching documents of the query so you can do a match_all query [1] to have the logic applied to all your documents. This is going to be expensive though, so try to filter out as many documents as possible before applying the custom scoring. Maybe even perform a rescore [2] on the top X docs. It really all depends on your requirements though. Run some tests and tune based on those results. When I said to be careful. I mean don't do a lot of blocking IO or long running calculations as the script is ran against each matching document. Cache results and make the script return as quick as possible. [1] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-match-all-query.html [2] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-rescore.html Thanks, Matt Weber On Fri, Apr 18, 2014 at 9:46 AM, Srinivasan Ramaswamy wrote: > Thats great, thanks for your reply. This looks like a good solution for my > requirement ! Is this script applied in each shard ? I want to apply this > function to all the documents so that the Top N picked from each shard is > picked by my custom score. > > Also, can you elaborate a little bit on "be careful you can significantly > impact your query performance if you are not careful". I would like to > understand the best practices there. > > On Friday, April 18, 2014 8:14:54 AM UTC-7, Matt Weber wrote: >> >> Yes, you can use the Function Score Query [1] in combination with a >> native script written in java [2]. With the native script you can >> basically do whatever you want, but be careful you can significantly impact >> your query performance if you are not careful. >> >> [1] http://www.elasticsearch.org/guide/en/elasticsearch/ >> reference/current/query-dsl-function-score-query.html >> [2] http://www.elasticsearch.org/guide/en/elasticsearch/ >> reference/current/modules-scripting.html#_native_java_scripts >> >> Thanks, >> Matt Weber >> >> >> On Thu, Apr 17, 2014 at 11:54 PM, Srinivasan Ramaswamy > > wrote: >> >>> I would like to influence the ranking with few fields that are not >>> stored in the index (eg click data for keyword-documents). I have used >>> custom SearchComponent in Solr to implement similar functionality in the >>> past. I am wondering how can i achieve the same in ElasticSearch. >>> >>> I know this thread is a very old thread, but i didnt find much >>> information on how to do custom scoring (in elasticsearch) with data thats >>> not stored in the index. This thread looked very relevant to my >>> requirement, so trying to see whether you guys have solved similar >>> requirements with elasticsearch. >>> >>> Thanks >>> Srini >>> >>> On Wednesday, September 7, 2011 12:18:09 PM UTC-7, Lukáš Vlček wrote: Hi Otis, So if I understand it correctly (providing my knowledge is quite limited here) you are asking if 1) it is possible to hook into query processing flow and inject or extend custom handlers for individual flow phases and 2) if we can find in ES the same functionality which is currently provided by components listed here: http://wiki.apache.org/s olr/SearchComponent (or here: http://lucene.apache.org/solr/ api/org/apache/solr/handler/component/SearchComponent.html). As for #1, frankly, I do not know. I have been playing with plugins a bit but did not have a chance to explore full potential of it yet. I remember that Shay mentioned that not every aspect of ES is pluggable now but that is all I know about it (personally, I did not hit the limits by myself yet, may be I would if I wanted to employ Carrot2 clustering or something like that) As for #2, if you are after one-to-one comparison of Solr SearchComponents and ES then I think we would find some matches and also some misses. Still it could be an interesting exercise to do (although we should be careful to include only those features that do work well in distributed environment). We could probably end up identifying new feature requests, so this can be useful. Regards, Lukas On Wed, Sep 7, 2011 at 6:17 PM, Otis Gospodnetic >>> > wrote: > Hi Lukas, > > Yes, SearchComponents are about extensibility, but specifically about > extending how queries are handled within Solr once Solr gets them. I > know ES has other types of plugins, and you've listed several of them, > but I'm wondering about which of them is SearchComponent-like. > I've looked at http://www.elasticsearch.org/guide/reference/modules/ > plugins.html > , but couldn't find the answer to my Q there. Maybe I'm looking at > the wrong place? > > Thanks, > Otis > -- > Sematext is hiring Search Engineers -- http://sematext.com/about/jobs > .html > > On Sep 6, 2:57 pm, Lukáš Vlček wrote: > > Hi, > > > > I am not Solr ex
Re: Solr SearchComponent-like functionality?
Thats great, thanks for your reply. This looks like a good solution for my requirement ! Is this script applied in each shard ? I want to apply this function to all the documents so that the Top N picked from each shard is picked by my custom score. Also, can you elaborate a little bit on "be careful you can significantly impact your query performance if you are not careful". I would like to understand the best practices there. On Friday, April 18, 2014 8:14:54 AM UTC-7, Matt Weber wrote: > > Yes, you can use the Function Score Query [1] in combination with a native > script written in java [2]. With the native script you can basically do > whatever you want, but be careful you can significantly impact your query > performance if you are not careful. > > [1] > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html > [2] > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_native_java_scripts > > Thanks, > Matt Weber > > > On Thu, Apr 17, 2014 at 11:54 PM, Srinivasan Ramaswamy > > > wrote: > >> I would like to influence the ranking with few fields that are not stored >> in the index (eg click data for keyword-documents). I have used custom >> SearchComponent in Solr to implement similar functionality in the past. I >> am wondering how can i achieve the same in ElasticSearch. >> >> I know this thread is a very old thread, but i didnt find much >> information on how to do custom scoring (in elasticsearch) with data thats >> not stored in the index. This thread looked very relevant to my >> requirement, so trying to see whether you guys have solved similar >> requirements with elasticsearch. >> >> Thanks >> Srini >> >> On Wednesday, September 7, 2011 12:18:09 PM UTC-7, Lukáš Vlček wrote: >>> >>> Hi Otis, >>> >>> So if I understand it correctly (providing my knowledge is quite limited >>> here) you are asking if >>> 1) it is possible to hook into query processing flow and inject or >>> extend custom handlers for individual flow phases and >>> 2) if we can find in ES the same functionality which is currently >>> provided by components listed here: http://wiki.apache.org/ >>> solr/SearchComponent (or here: http://lucene.apache.org/solr/ >>> api/org/apache/solr/handler/component/SearchComponent.html). >>> >>> As for #1, frankly, I do not know. I have been playing with plugins a >>> bit but did not have a chance to explore full potential of it yet. I >>> remember that Shay mentioned that not every aspect of ES is pluggable now >>> but that is all I know about it (personally, I did not hit the limits by >>> myself yet, may be I would if I wanted to employ Carrot2 clustering or >>> something like that) >>> >>> As for #2, if you are after one-to-one comparison of Solr >>> SearchComponents and ES then I think we would find some matches and also >>> some misses. Still it could be an interesting exercise to do (although we >>> should be careful to include only those features that do work well in >>> distributed environment). We could probably end up identifying new feature >>> requests, so this can be useful. >>> >>> Regards, >>> Lukas >>> >>> On Wed, Sep 7, 2011 at 6:17 PM, Otis Gospodnetic >>> wrote: >>> Hi Lukas, Yes, SearchComponents are about extensibility, but specifically about extending how queries are handled within Solr once Solr gets them. I know ES has other types of plugins, and you've listed several of them, but I'm wondering about which of them is SearchComponent-like. I've looked at http://www.elasticsearch.org/guide/reference/modules/ plugins.html , but couldn't find the answer to my Q there. Maybe I'm looking at the wrong place? Thanks, Otis -- Sematext is hiring Search Engineers -- http://sematext.com/about/ jobs.html On Sep 6, 2:57 pm, Lukáš Vlček wrote: > Hi, > > I am not Solr expert but to me it seems that SearchComponents in Solr are > about extensibility of out of the box functionality. If that is the case > then I would say that we can talk about plugins in ES world. Although there > is no official doc about how to implement custom plugins yet it is really > not difficult. Apart from that there are several plugins that are part of > distribution (river plugins, attachments mapper, ICU analysis, scripting > languages ... to name a few) and they can be used as an inspiration if a new > plugin implementation is needed. > > My 2 cents. > > Lukas > > On Tue, Sep 6, 2011 at 5:35 PM, Otis Gospodnetic < otis.gospodne...@gmail.com > > > > > > > > > wrote: > > Hello, > > > A long time Solr user posted a good question about ES over on Sematext > > Blog, about an equivalent of Solr's SearchComponen
Re: Solr SearchComponent-like functionality?
Yes, you can use the Function Score Query [1] in combination with a native script written in java [2]. With the native script you can basically do whatever you want, but be careful you can significantly impact your query performance if you are not careful. [1] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html [2] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_native_java_scripts Thanks, Matt Weber On Thu, Apr 17, 2014 at 11:54 PM, Srinivasan Ramaswamy wrote: > I would like to influence the ranking with few fields that are not stored > in the index (eg click data for keyword-documents). I have used custom > SearchComponent in Solr to implement similar functionality in the past. I > am wondering how can i achieve the same in ElasticSearch. > > I know this thread is a very old thread, but i didnt find much information > on how to do custom scoring (in elasticsearch) with data thats not stored > in the index. This thread looked very relevant to my requirement, so trying > to see whether you guys have solved similar requirements with elasticsearch. > > Thanks > Srini > > On Wednesday, September 7, 2011 12:18:09 PM UTC-7, Lukáš Vlček wrote: >> >> Hi Otis, >> >> So if I understand it correctly (providing my knowledge is quite limited >> here) you are asking if >> 1) it is possible to hook into query processing flow and inject or extend >> custom handlers for individual flow phases and >> 2) if we can find in ES the same functionality which is currently >> provided by components listed here: http://wiki.apache.org/ >> solr/SearchComponent (or here: http://lucene.apache.org/solr/ >> api/org/apache/solr/handler/component/SearchComponent.html). >> >> As for #1, frankly, I do not know. I have been playing with plugins a bit >> but did not have a chance to explore full potential of it yet. I remember >> that Shay mentioned that not every aspect of ES is pluggable now but that >> is all I know about it (personally, I did not hit the limits by myself yet, >> may be I would if I wanted to employ Carrot2 clustering or something like >> that) >> >> As for #2, if you are after one-to-one comparison of Solr >> SearchComponents and ES then I think we would find some matches and also >> some misses. Still it could be an interesting exercise to do (although we >> should be careful to include only those features that do work well in >> distributed environment). We could probably end up identifying new feature >> requests, so this can be useful. >> >> Regards, >> Lukas >> >> On Wed, Sep 7, 2011 at 6:17 PM, Otis Gospodnetic >> wrote: >> >>> Hi Lukas, >>> >>> Yes, SearchComponents are about extensibility, but specifically about >>> extending how queries are handled within Solr once Solr gets them. I >>> know ES has other types of plugins, and you've listed several of them, >>> but I'm wondering about which of them is SearchComponent-like. >>> I've looked at http://www.elasticsearch.org/guide/reference/modules/ >>> plugins.html >>> , but couldn't find the answer to my Q there. Maybe I'm looking at >>> the wrong place? >>> >>> Thanks, >>> Otis >>> -- >>> Sematext is hiring Search Engineers -- http://sematext.com/about/ >>> jobs.html >>> >>> On Sep 6, 2:57 pm, Lukáš Vlček wrote: >>> > Hi, >>> > >>> > I am not Solr expert but to me it seems that SearchComponents in Solr >>> are >>> > about extensibility of out of the box functionality. If that is the >>> case >>> > then I would say that we can talk about plugins in ES world. Although >>> there >>> > is no official doc about how to implement custom plugins yet it is >>> really >>> > not difficult. Apart from that there are several plugins that are part >>> of >>> > distribution (river plugins, attachments mapper, ICU analysis, >>> scripting >>> > languages ... to name a few) and they can be used as an inspiration if >>> a new >>> > plugin implementation is needed. >>> > >>> > My 2 cents. >>> > >>> > Lukas >>> > >>> > On Tue, Sep 6, 2011 at 5:35 PM, Otis Gospodnetic < >>> otis.gospodne...@gmail.com >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > > wrote: >>> > > Hello, >>> > >>> > > A long time Solr user posted a good question about ES over on >>> Sematext >>> > > Blog, about an equivalent of Solr's SearchComponents in ES: >>> > >>> > >http://blog.sematext.com/2010/05/03/elastic-search- >>> distributed-lucene... >>> > >>> > > I'm curious, too. Thanks. >>> > >>> > > Otis >>> > > -- >>> > > Sematext is hiring Search Engineers --http://sematext.com/about/ >>> jobs.html >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/16add2bc-c629-4613-934f-004c8cc749df%40googlegroups.com
Re: Solr SearchComponent-like functionality?
I would like to influence the ranking with few fields that are not stored in the index (eg click data for keyword-documents). I have used custom SearchComponent in Solr to implement similar functionality in the past. I am wondering how can i achieve the same in ElasticSearch. I know this thread is a very old thread, but i didnt find much information on how to do custom scoring (in elasticsearch) with data thats not stored in the index. This thread looked very relevant to my requirement, so trying to see whether you guys have solved similar requirements with elasticsearch. Thanks Srini On Wednesday, September 7, 2011 12:18:09 PM UTC-7, Lukáš Vlček wrote: > > Hi Otis, > > So if I understand it correctly (providing my knowledge is quite limited > here) you are asking if > 1) it is possible to hook into query processing flow and inject or extend > custom handlers for individual flow phases and > 2) if we can find in ES the same functionality which is currently provided > by components listed here: http://wiki.apache.org/solr/SearchComponent (or > here: > http://lucene.apache.org/solr/api/org/apache/solr/handler/component/SearchComponent.html > ). > > As for #1, frankly, I do not know. I have been playing with plugins a bit > but did not have a chance to explore full potential of it yet. I remember > that Shay mentioned that not every aspect of ES is pluggable now but that > is all I know about it (personally, I did not hit the limits by myself yet, > may be I would if I wanted to employ Carrot2 clustering or something like > that) > > As for #2, if you are after one-to-one comparison of Solr SearchComponents > and ES then I think we would find some matches and also some misses. Still > it could be an interesting exercise to do (although we should be careful to > include only those features that do work well in distributed environment). > We could probably end up identifying new feature requests, so this can be > useful. > > Regards, > Lukas > > On Wed, Sep 7, 2011 at 6:17 PM, Otis Gospodnetic > > > wrote: > >> Hi Lukas, >> >> Yes, SearchComponents are about extensibility, but specifically about >> extending how queries are handled within Solr once Solr gets them. I >> know ES has other types of plugins, and you've listed several of them, >> but I'm wondering about which of them is SearchComponent-like. >> I've looked at >> http://www.elasticsearch.org/guide/reference/modules/plugins.html >> , but couldn't find the answer to my Q there. Maybe I'm looking at >> the wrong place? >> >> Thanks, >> Otis >> -- >> Sematext is hiring Search Engineers -- >> http://sematext.com/about/jobs.html >> >> On Sep 6, 2:57 pm, Lukáš Vlček wrote: >> > Hi, >> > >> > I am not Solr expert but to me it seems that SearchComponents in Solr >> are >> > about extensibility of out of the box functionality. If that is the case >> > then I would say that we can talk about plugins in ES world. Although >> there >> > is no official doc about how to implement custom plugins yet it is >> really >> > not difficult. Apart from that there are several plugins that are part >> of >> > distribution (river plugins, attachments mapper, ICU analysis, scripting >> > languages ... to name a few) and they can be used as an inspiration if >> a new >> > plugin implementation is needed. >> > >> > My 2 cents. >> > >> > Lukas >> > >> > On Tue, Sep 6, 2011 at 5:35 PM, Otis Gospodnetic < >> otis.gospodne...@gmail.com >> > >> > >> > >> > >> > >> > >> > >> > > wrote: >> > > Hello, >> > >> > > A long time Solr user posted a good question about ES over on Sematext >> > > Blog, about an equivalent of Solr's SearchComponents in ES: >> > >> > >http://blog.sematext.com/2010/05/03/elastic-search-distributed-lucene. >> .. >> > >> > > I'm curious, too. Thanks. >> > >> > > Otis >> > > -- >> > > Sematext is hiring Search Engineers -- >> http://sematext.com/about/jobs.html >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/16add2bc-c629-4613-934f-004c8cc749df%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
