Re: custom sorting of search result

[Alexandre Rafalovitch]

Latest versions of Solr have collapsing and expanding plugins,
reranking plugins and post-filters. Some combinations of these seem
like it might be relevant.

And, of course, there is always carrot2 clustering.

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 3 November 2014 18:07,   wrote:
> Hello,
>
>
> We need to order solr search results according to specific rules.
>
>
> I will explain with an example. Let say solr returns 1000 results for query 
> "sport".
> These results must be divided into three buckets according to rules that come 
> from database.
> Then one doc must be chosen from each bucket and put in the results 
> subsequently until all buckets are empty.
>
>
> One approach was to modify/override solr code where it gets results, sorts 
> them and return #rows of elements.
> However, from the code in Weight.java scoreAll function we see that docs have 
> only internal document id and nothing else.
>
>
> We expect unique solr document id in order to match documents with the custom 
> scoring.
> We also  see that Lucene code handles those doc ids to scoreAll function, and 
> for now We do not want to modify Lucene code
>  and prefer to solve this issue as a Solr  plugin .
>
>
> Any ideas are welcome.
>
>
>
>
> Thanks.
> Alex.
>
>
>
>

Re: Custom sorting based on external (database) data

[Sujit Pal]

Thank you Ahmet, looks like we could use this. Basically we would do
periodic dumps of the (unique_id|computed_score) sorted by score and
write it out to this file followed by a commit.

Found some more info here, for the benefit of others looking for
something similar:
http://dev.tailsweep.com/solr-external-scoring/ 

On Thu, 2011-05-05 at 13:12 -0700, Ahmet Arslan wrote:
> 
> --- On Thu, 5/5/11, Sujit Pal  wrote:
> 
> > From: Sujit Pal 
> > Subject: Custom sorting based on external (database) data
> > To: "solr-user" 
> > Date: Thursday, May 5, 2011, 11:03 PM
> > Hi,
> > 
> > Sorry for the possible double post, I wrote this up but had
> > the
> > incorrect sender address, so I am guessing that my previous
> > one is going
> > to be rejected by the list moderation daemon.
> > 
> > I am trying to figure out options for the following
> > problem. I am on
> > Solr 1.4.1 (Lucene 2.9.1).
> > 
> > I have search results which are going to be ranked by the
> > user (using a
> > thumbs up/down) and would translate to a score between -1
> > and +1. 
> > 
> > This data is stored in a database table (
> > unique_id
> > thumbs_up
> > thumbs_down
> > num_calls
> > 
> > as the thumbs up/down component is clicked.
> > 
> > We want to be able to sort the results by the following
> > score =
> > (thumbs_up - thumbs_down) / (num_calls). The unique_id
> > field refers to
> > the one referenced as  in the schema.xml.
> > 
> > Based on the following conversation:
> > http://www.mail-archive.com/solr-user@lucene.apache.org/msg06322.html
> > 
> > 
> > ...my understanding is that I need to:
> > 
> > 1) subclass FieldType to create my own RankFieldType. 
> > 2) In this class I override the getSortField() method to
> > return my
> > custom FieldSortComparatorSource object.
> > 3) Build the custom FieldSortComparatorSource object which
> > returns a
> > custom FieldSortComparator object in newComparator().
> > 4) Configure the field type of class RankFieldType
> > (rank_t), and a field
> > (called rank) of field type rank_t in schema.xml of type
> > RankFieldType.
> > 5) use sort=rank+desc to do the sort.
> > 
> > My question is: is there a simpler/more performant way? The
> > number of
> > database lookups seems like its going to be pretty high
> > with this
> > approach. And its hard to believe that my problem is new,
> > so I am
> > guessing this is either part of some Solr configuration I
> > am missing, or
> > there is some other (possibly simpler) approach I am
> > overlooking.
> > 
> > Pointers to documentation or code (or even keywords I could
> > google)
> > would be much appreciated.
> 
> Looks like it can be done with 
> http://lucene.apache.org/solr/api/org/apache/solr/schema/ExternalFileField.html
>  
> and 
> http://wiki.apache.org/solr/FunctionQuery
> 
> You can dump your table into three text files. Issue a commit to load these 
> changes.
> 
> Sort by function query is available in Solr3.1 though.

Re: Custom sorting based on external (database) data

[Ahmet Arslan]

--- On Thu, 5/5/11, Sujit Pal  wrote:

> From: Sujit Pal 
> Subject: Custom sorting based on external (database) data
> To: "solr-user" 
> Date: Thursday, May 5, 2011, 11:03 PM
> Hi,
> 
> Sorry for the possible double post, I wrote this up but had
> the
> incorrect sender address, so I am guessing that my previous
> one is going
> to be rejected by the list moderation daemon.
> 
> I am trying to figure out options for the following
> problem. I am on
> Solr 1.4.1 (Lucene 2.9.1).
> 
> I have search results which are going to be ranked by the
> user (using a
> thumbs up/down) and would translate to a score between -1
> and +1. 
> 
> This data is stored in a database table (
> unique_id
> thumbs_up
> thumbs_down
> num_calls
> 
> as the thumbs up/down component is clicked.
> 
> We want to be able to sort the results by the following
> score =
> (thumbs_up - thumbs_down) / (num_calls). The unique_id
> field refers to
> the one referenced as  in the schema.xml.
> 
> Based on the following conversation:
> http://www.mail-archive.com/solr-user@lucene.apache.org/msg06322.html
> 
> 
> ...my understanding is that I need to:
> 
> 1) subclass FieldType to create my own RankFieldType. 
> 2) In this class I override the getSortField() method to
> return my
> custom FieldSortComparatorSource object.
> 3) Build the custom FieldSortComparatorSource object which
> returns a
> custom FieldSortComparator object in newComparator().
> 4) Configure the field type of class RankFieldType
> (rank_t), and a field
> (called rank) of field type rank_t in schema.xml of type
> RankFieldType.
> 5) use sort=rank+desc to do the sort.
> 
> My question is: is there a simpler/more performant way? The
> number of
> database lookups seems like its going to be pretty high
> with this
> approach. And its hard to believe that my problem is new,
> so I am
> guessing this is either part of some Solr configuration I
> am missing, or
> there is some other (possibly simpler) approach I am
> overlooking.
> 
> Pointers to documentation or code (or even keywords I could
> google)
> would be much appreciated.

Looks like it can be done with 
http://lucene.apache.org/solr/api/org/apache/solr/schema/ExternalFileField.html 
and 
http://wiki.apache.org/solr/FunctionQuery

You can dump your table into three text files. Issue a commit to load these 
changes.

Sort by function query is available in Solr3.1 though.

RE: Custom Sorting

[Michael Owen]

Ok thank you for the discussion. As I thought regard to not possible within 
performance limits.
I think the way to go is to document some more stats at index time, and use 
them in boost queries. :)
Thanks
Mike

> Date: Tue, 19 Apr 2011 15:12:00 -0400
> Subject: Re: Custom Sorting
> From: erickerick...@gmail.com
> To: solr-user@lucene.apache.org
> 
> As I understand it, sorting by field is what caches are all
> about. You have a big list in memory of all of the terms for
> a field, indexed by Lucene doc ID so fetching the term to
> compare by doc ID is fast, and also why the caches need
> to be warmed, and why sort fields should be single-valued.
> 
> If you try to do this yourself and fetch data from each document,
> you can incur a huge performance hit, since you'll be seeking
> all over your disk...
> 
> Score is special though since it's transient. Internally, all Lucene
> has to do is keep track of the top N scores encountered where
> N is something like "start + queryResultWindowSize", this
> latter from solrconfig.xml, with no seeks to disk at all...
> 
> Best
> Erick
> 
> On Tue, Apr 19, 2011 at 2:50 PM, Jonathan Rochkind  wrote:
> > On 4/19/2011 1:43 PM, Jan Høydahl wrote:
> >>
> >> Hi,
> >>
> >> Not possible :)
> >> Lucene compares each matching document against the query and produces a
> >> score for each.
> >> Documents are not compared to eachother like normal sort, that would be
> >> way too costly.
> >
> > That might be true for sort by 'score' (although even if you have all the
> > scores, it still seems like some kind of sort must be neccesary to see which
> > comes first), but when you sort by a field value, which is also possible,
> > Lucene must be doing some kind of 'normal sort' algorithm, no?  Ah, I guess
> > it could just be using each term's position in the index, which is available
> > in constant time, always kept track of in an index? Maybe, I don't know?
> >
> >
> >
  

Re: Custom Sorting

[Erick Erickson]

As I understand it, sorting by field is what caches are all
about. You have a big list in memory of all of the terms for
a field, indexed by Lucene doc ID so fetching the term to
compare by doc ID is fast, and also why the caches need
to be warmed, and why sort fields should be single-valued.

If you try to do this yourself and fetch data from each document,
you can incur a huge performance hit, since you'll be seeking
all over your disk...

Score is special though since it's transient. Internally, all Lucene
has to do is keep track of the top N scores encountered where
N is something like "start + queryResultWindowSize", this
latter from solrconfig.xml, with no seeks to disk at all...

Best
Erick

On Tue, Apr 19, 2011 at 2:50 PM, Jonathan Rochkind  wrote:
> On 4/19/2011 1:43 PM, Jan Høydahl wrote:
>>
>> Hi,
>>
>> Not possible :)
>> Lucene compares each matching document against the query and produces a
>> score for each.
>> Documents are not compared to eachother like normal sort, that would be
>> way too costly.
>
> That might be true for sort by 'score' (although even if you have all the
> scores, it still seems like some kind of sort must be neccesary to see which
> comes first), but when you sort by a field value, which is also possible,
> Lucene must be doing some kind of 'normal sort' algorithm, no?  Ah, I guess
> it could just be using each term's position in the index, which is available
> in constant time, always kept track of in an index? Maybe, I don't know?
>
>
>

Re: Custom Sorting

[Jonathan Rochkind]

On 4/19/2011 1:43 PM, Jan Høydahl wrote:

Hi,

Not possible :)
Lucene compares each matching document against the query and produces a score 
for each.
Documents are not compared to eachother like normal sort, that would be way too 
costly.


That might be true for sort by 'score' (although even if you have all 
the scores, it still seems like some kind of sort must be neccesary to 
see which comes first), but when you sort by a field value, which is 
also possible, Lucene must be doing some kind of 'normal sort' 
algorithm, no?  Ah, I guess it could just be using each term's position 
in the index, which is available in constant time, always kept track of 
in an index? Maybe, I don't know?

Re: Custom Sorting

[lboutros]

You could create a new Similarity class  plugin that take in account every
parameters you need. :

http://wiki.apache.org/solr/SolrPlugins?highlight=%28similarity%29#Similarity

but, as Jan said, be carefull with the cost of the the similarity function.

Ludovic.

2011/4/19 Jan Høydahl / Cominvent [via Lucene] <
ml-node+2839526-2100261518-383...@n3.nabble.com>

> Hi,
>
> Not possible :)
> Lucene compares each matching document against the query and produces a
> score for each.
> Documents are not compared to eachother like normal sort, that would be way
> too costly.
>
> But if you explain your use case, I'm sure we can find ways to express your
> needs in other ways
>
> Perhaps it is possible for you to use Sort by Function?
> http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function
> Then you can decide exactly what goes into your sort score.
> If you want to do conditional stuff, you may need to pre-process your
> documents a bit and create new fields which can be used in a FunctionQuery.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> On 19. apr. 2011, at 19.02, Michael Owen wrote:
>
> >
> > Hi,
> > I want to able to have a custom sorting algorithm such that for each
> comparison of document results (A v B) I can rank them. i.e. writing a
> comparator like I would normally do in Java (Compares its two arguments for
> order. Returns a negative integer, zero, or a positive integer as the first
> argument is less than, equal to, or greater than the second).
> > In the comparator I want to be able to take into account the score of the
> results, as well as other fields in the documents.
> > I've looked at using things such as the score/boost/bf parameters etc,
> however, want the flexibility of being able to code the comparator, so I can
> do if conditions and such.
> > Is this possible? And if so what's the best way of doing this? I've
> upgraded to use the latest version of Solr 3.1, and of course for this use
> case would expect to have to build from source, in order to add custom
> source.
> > Or/and, when using the score/boost/bf parameters etc - is it possible to
> use the score parameter in functions, to say scale it between 0 and 1?
> > Thanks
> > Mike
> >
> >
> >
> >
>
>
>
> --
>  If you reply to this email, your message will be added to the discussion
> below:
> http://lucene.472066.n3.nabble.com/Custom-Sorting-tp2839375p2839526.html
>  To start a new topic under Solr - User, email
> ml-node+472068-1765922688-383...@n3.nabble.com
> To unsubscribe from Solr - User, click 
> here.
>
>


-
Jouve
France.
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Custom-Sorting-tp2839375p2839593.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Custom Sorting

[Jan Høydahl]

Hi,

Not possible :)
Lucene compares each matching document against the query and produces a score 
for each.
Documents are not compared to eachother like normal sort, that would be way too 
costly.

But if you explain your use case, I'm sure we can find ways to express your 
needs in other ways

Perhaps it is possible for you to use Sort by Function? 
http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function
Then you can decide exactly what goes into your sort score.
If you want to do conditional stuff, you may need to pre-process your documents 
a bit and create new fields which can be used in a FunctionQuery.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

On 19. apr. 2011, at 19.02, Michael Owen wrote:

> 
> Hi,
> I want to able to have a custom sorting algorithm such that for each 
> comparison of document results (A v B) I can rank them. i.e. writing a 
> comparator like I would normally do in Java (Compares its two arguments for 
> order. Returns a negative integer, zero, or a positive integer as the first 
> argument is less than, equal to, or greater than the second).
> In the comparator I want to be able to take into account the score of the 
> results, as well as other fields in the documents.
> I've looked at using things such as the score/boost/bf parameters etc, 
> however, want the flexibility of being able to code the comparator, so I can 
> do if conditions and such.
> Is this possible? And if so what's the best way of doing this? I've upgraded 
> to use the latest version of Solr 3.1, and of course for this use case would 
> expect to have to build from source, in order to add custom source.
> Or/and, when using the score/boost/bf parameters etc - is it possible to use 
> the score parameter in functions, to say scale it between 0 and 1?
> Thanks
> Mike
> 
> 
> 
> 

Re: Custom Sorting in Solr

[Ezequiel Calderara]

Ok i imagined that the double linked list would be far too complicated for
solr.

Now, how can i achieve that solr connects to a webservice and do the import?

I'm sorry if i'm not clear, sometimes my english gets fuzzy :P

On Fri, Oct 29, 2010 at 4:51 PM, Yonik Seeley wrote:

> On Fri, Oct 29, 2010 at 3:39 PM, Ezequiel Calderara 
> wrote:
> > Hi all guys!
> > I'm in a weird situation here.
> > We have index a set of documents which are ordered using a linked list
> (each
> > documents has the reference of the previous and the next).
> >
> > Is there a way when sorting in the solr search, Use the linked list to
> sort?
>
> It seems like you should be able to encode this linked list as an
> integer instead, and sort by that?
> If there are multiple linked lists in the index, it seems like you
> could even use the high bits of the int to designate which list the
> doc belongs to, and the low order bits as the order in that list.
>
> -Yonik
> http://www.lucidimagination.com
>



-- 
__
Ezequiel.

Http://www.ironicnet.com

Re: Custom Sorting in Solr

[Yonik Seeley]

On Fri, Oct 29, 2010 at 3:39 PM, Ezequiel Calderara  wrote:
> Hi all guys!
> I'm in a weird situation here.
> We have index a set of documents which are ordered using a linked list (each
> documents has the reference of the previous and the next).
>
> Is there a way when sorting in the solr search, Use the linked list to sort?

It seems like you should be able to encode this linked list as an
integer instead, and sort by that?
If there are multiple linked lists in the index, it seems like you
could even use the high bits of the int to designate which list the
doc belongs to, and the low order bits as the order in that list.

-Yonik
http://www.lucidimagination.com

RE: Custom Sorting in Solr

[Jonathan Rochkind]

There's no way I know of to make Solr use that kind of data to create the sort 
order you want. 

Generally for 'custom' sorts, you want to create a field in your Solr index 
with possibly artificially constructed values that will 'naturally' sort the 
way you want. 

How to do that with a linked list seems kind of tricky, before you index you 
may have to write code to analyze your whole graph order and then just supply 
sort order keys.  And then if you sometimes update just a few documents, but 
not your whole thing.. Geez, i'm not really sure. It's kind of a tricky 
problem.  That kind of data is not really the expected use case for Solr 
sorting. 

Sorry, I'm not sure what this means or how it would help: "use the DIH to 
access a Service in WCF or a Webservice?"  Maybe someone else will know exactly 
what you mean. Or maybe if you rephrase with more specificity as to how you 
think this will help you solve your problem, it will be more clear. 

Recall that you don't need to use DIH to index at all, it's just one of several 
methods, it simplifies things for common patterns, it's possible you fall out 
of the common pattern nad it would be simpler not to use DIH.   Although even 
without DIH, I can't think of a particularly simple way to solve your problem. 

Just curious, but is your _entire_ corpus, your entire document set, part of a 
_single_ linked list?  Or do you have several different linked lists in there? 
If several, what do you want to happen with sort if two documents in the result 
set aren't even part of the same linked list?   This kind of thing is one 
reason translating the sort of data you have to a solr sort order starts to 
seem kind of confusing to me. 


From: Ezequiel Calderara [ezech...@gmail.com]
Sent: Friday, October 29, 2010 3:39 PM
To: Solr Mailing List
Subject: Custom Sorting in Solr

Hi all guys!
I'm in a weird situation here.
We have index a set of documents which are ordered using a linked list (each
documents has the reference of the previous and the next).

Is there a way when sorting in the solr search, Use the linked list to sort?


If that is not possible, how can i use the DIH to access a Service in WCF or
a Webservice? Should i develop my own DIH?


--
__
Ezequiel.

Http://www.ironicnet.com

Re: custom sorting / help overriding FieldComparator

[Chris Hostetter]

Brad:

1) if you haven't already figured this out, i would suggest emailin the 
java-user mailing list.  It's got a bigger collection of users who are 
familiar with the internals of the Lucnee-Java API (that's the level it 
seems like you are having difficulty at)

2) Maybe you mentioned your sorting algorithm in a previous thread, but 
i'm not remembering it -- it's possibly this is an XY problem, if you 
describe the algorithm you need (or show us the code for your Comparable 
impl) we might be able to suggest an efficient way to do this with out any 
custom code in Solr...
http://people.apache.org/~hossman/#xyproblem


: I'm trying to get my (overly complex and strange) product IDs sorting 
properly in Solr.
: 
: Approaches I've tried so far, that I've given up on for various reasons:
: --Normalizing/padding the IDs so they naturally sort 
alphabetically/alphanumerically.
: --Splitting the ID into multiple Solr fields and sending a longer, 
multi-field "sort" argument in the GET request.
: --(both of those approaches do work "most of the time", but aren't quite 
perfect)
: 
: However, in another project, I already have a Comparble class 
defined in Java that represents a ProductID and does sort them correctly every 
time.  It's not yet in lucene/solr, though.  So I'm trying to make a FieldType 
plugin for Solr that uses the existing ProductID class/datatype.
: 
: I need some help extending the lucene FieldComparator class.  I don't know 
much about the rest of the solr / lucene codebase, so I'm fumbling around a 
bit, especially with the required setNextReader() method.  setNextReader() 
looks like it checks the FieldCache to see if this value is there already, 
otherwise grabs a bunch of documents from the index.  I think I should call 
some form of FieldCache.getCustom() for this, but FieldCache.getCustom() itself 
accepts a comparator as an argument, and is marked as "@deprecated Please 
implement FieldComparatorSource directly, instead" ... but isn't that what I'm 
doing?
: 
: So, I'm just a bit confused.  Any help?  Specifically, any help implementing 
a setNextReader() method in a customComparator?
: 
: (solr 1.4.1 / lucene 2.9.3)
: 
: Thanks,
: Brad
: 
: 
: 
: 

-Hoss

--
http://lucenerevolution.org/  ...  October 7-8, Boston
http://bit.ly/stump-hoss  ...  Stump The Chump!

Re: Custom sorting

[Daniel Cassiano]

Hi Dan,

It seems that you want a SearchComponent[1], something like the
QueryElevationComponent[2].
Take a look how at him and I think you can build your custom solution.

[1]-
http://lucene.apache.org/solr/api/org/apache/solr/handler/component/SearchComponent.html
[2]- http://wiki.apache.org/solr/QueryElevationComponent


Cheers,

-- Daniel Cassiano

http://dcassiano.wordpress.com


On Wed, May 19, 2010 at 6:46 AM, dan sutton  wrote:

> Hi,
>
> I have a requirement to do the following:
>
> For up to the first 10 results (i.e. only on the first page) show
> sponsored category ads, in order of bid, but no more than 2 / category,
> and only if all sponsored cat' ads are more that min% of the highest
> score. e.g. If I had the following:
>
> min% =1
>
>
> doc score bid  cat_id sponsored
>  1   100   x   x 0
>  255x   x 0
>
>  3502   2 1
>  4202   2 1
>  5052   2 1
>
>  6801   1 1
>  7701   1 1
>  8601   1 1
>
> x = dont care
>
> sorted order would be:
>
> 3
> 4
>
> 6
> 7
>
> 1
> 8
> 2
> 5
>
> I'm not sure if this can be implemented with a custom comparator as I
> need access to the final score to enforce min%, I'm thinking I'm
> probably going to have to implement a subclass of QParserPlugin with a
> custom sort. but was wondering if there were alternatives ?
>
> Many thanks in advance.
> Dan
>

Re: Custom Sorting Based on Relevancy

[Otis Gospodnetic]

Or you could collapse search results with SOLR-236.

 Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: David Giffin 
> To: solr-user@lucene.apache.org
> Sent: Monday, May 4, 2009 12:37:29 PM
> Subject: Custom Sorting Based on Relevancy
> 
> Hi There,
> 
> I'm working on a sorting issue. Our site currently sorts by creation
> date descending, so users list similar products multiple times to show
> up at the top of the results. When sorting based on score, we want to
> move items by the "same user" with the "same title" down search
> results. It would be best if the first item stayed in place based on
> score, and each additional item is moved out (rows * repeated
> user/title).
> 
> Is custom sorting the best way? or is there something else I'm not
> thinking about. At the moment I'm looking at doing roughly the
> opposite of the Query Elevate Search component.
> 
> Thanks,
> David

Re: Custom Sorting

[psyron]

I was sucessful with your hint and just need to solve another problem:

The problem I have is that I have implemented a custome sorting by following
your advice to code a
QParserPlugin and to create a custom comparator as described in your book,
and it really works
But now I also would like to return those computed sort values by adding
them to the SolrQueryResponse.
I am calculating distances and would like to return the distance from the
origin for each search result.

In your book you describe that it is possible by using this lucene search
function:
TopFieldDocs docs = searcher.search(query, null, 3, sort);

and then to read the sort values:
FieldDoc fieldDoc = (FieldDoc) docs.scoreDocs[0];
return -> fieldDoc.fields[0]

But how can I do this inside Solr?
I am using the default QueryComponent and of course I don’t want to make too
many changes, because
I don’t understand the inside of solr so much – it’s quite big and
complicated and I didn’t find many
documents explaining Solr.

Is there maybe a workaround? Can I just store all my sort values and add
them to the SolrQueryResponse
at the end?

Thanks,
Markus


Erik Hatcher wrote:
> 
> Markus,
> 
> A couple of code pointers for you:
> 
>* QueryComponent - this is where results are generated, it uses a  
> SortSpec from the QParser.
> 
>* QParser#getSort - creating a custom QParser you'll be able to  
> wire in your own custom sort
> 
> You can write your own QParserPlugin and QParser, and configure it  
> into solrconfig.xml and should be good to go.  Subclassing existing  
> classes, this should only be a handful of lines of code to do.
> 
>   Erik
> 
> 
> On Dec 16, 2008, at 3:54 AM, psyron wrote:
> 
>>
>> I have the same problem, also need to plugin my "customComparator",  
>> but as
>> there is no explanation of the framework, how a RequestHandler is  
>> working,
>> what comes in, what comes out ... just impossible!
>>
>> Can someone explain where i have to add which code, to just have the  
>> same
>> functionality as the StandardRequestHandler, but also adding a custom
>> sorting?
>>
>> Thanks,
>> Markus
>>
>>
>> hossman wrote:
>>>
>>>
>>> : Sort sort = new Sort(new SortField[]
>>> : { SortField.FIELD_SCORE, new SortField(customValue,
>>> SortField.FLOAT,
>>> : true) });
>>> : indexSearcher.search(q, sort)
>>>
>>> that appears to just be a sort on score withe a secondary reversed
>>> float sort on whatever field name is in the variable  
>>> "customValue" ...
>>> assuming hte field name is "FIELD" that's hte same thing as...
>>>   sort=score+asc,+FIELD+desc
>>>
>>> : Sort sort = new Sort(new SortField(customValue, customComparator))
>>> : indexSearcher.search(q, sort)
>>>
>>> this is using a custom SortComparatorSource -- code you (or someone  
>>> else)
>>> has written which is not part of Lucene and which tells lucene how to
>>> order the documents using whatever crazy logic it wants ... for  
>>> obvious
>>> reasons Solr can't do that same logic (since it doesn't know what  
>>> it is)
>>>
>>> although many things in Solr are easily customizable, just by  
>>> writting a
>>> little factory and configuring it by class name, i'm afraind
>>> SortComparatorSources aren't once of them.  You could write a custom
>>> RequestHandler which used your SortComparatorSource, or you could  
>>> write a
>>> custom FieldType that used it anything someone sorted on that  
>>> field ...
>>> but those are the best options i cna think of.
>>>
>>>
>>>
>>> -Hoss
>>>
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://www.nabble.com/Custom-Sorting-tp1659p21029370.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
> 
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Custom-Sorting-tp1659p22248512.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Custom Sorting Algorithm

[Yonik Seeley]

On Wed, Feb 4, 2009 at 4:45 PM, wojtekpia  wrote:
> Ok, so maybe a better question is: should I bother trying to change the
> "sorting" algorithm? I'm concerned that with large data sets, sorting
> becomes a severe bottleneck (this is an assumption, I haven't profiled
> anything to verify).

No... Lucene/Solr never sorts the complete result set.
If you ask for the top 10 results, a priority queue (heap) of the
current top 10 results is maintained... far more efficient and
scalable than sorting all the hits at the end.

-Yonik

Re: Custom Sorting Algorithm

[wojtekpia]

Ok, so maybe a better question is: should I bother trying to change the
"sorting" algorithm? I'm concerned that with large data sets, sorting
becomes a severe bottleneck (this is an assumption, I haven't profiled
anything to verify). Does it become a severe bottleneck? Do you know if
alternate sort algorithms have been tried during Lucene development? 



markrmiller wrote:
> 
> It would not be simple to use a new algorithm. The current 
> implementation takes place at the Lucene level and uses a priority 
> queue. When you ask for the top n results, a priority queue of size n is 
> filled with all of the matching documents. The ordering in the priority 
> queue is the sort. The on Sort method orders by relevance score - the 
> Sort method orders by field, relevance, or doc id.
> 

-- 
View this message in context: 
http://www.nabble.com/Custom-Sorting-Algorithm-tp21837721p21840299.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Custom Sorting Algorithm

[Mark Miller]

It would not be simple to use a new algorithm. The current 
implementation takes place at the Lucene level and uses a priority 
queue. When you ask for the top n results, a priority queue of size n is 
filled with all of the matching documents. The ordering in the priority 
queue is the sort. The on Sort method orders by relevance score - the 
Sort method orders by field, relevance, or doc id.


- Mark

wojtekpia wrote:

That's not quite what I meant. I'm not looking for a custom comparator, I'm
looking for a custom sorting algorithm. Is there a way to use quick sort or
merge sort or... rather than the current algorithm? Also, what is the
current algorithm?


Otis Gospodnetic wrote:
  

You can use one of the exiting function queries (if they fit your need) or
write a custom function query to reorder the results of a query.





  

Re: Custom Sorting Algorithm

[wojtekpia]

That's not quite what I meant. I'm not looking for a custom comparator, I'm
looking for a custom sorting algorithm. Is there a way to use quick sort or
merge sort or... rather than the current algorithm? Also, what is the
current algorithm?


Otis Gospodnetic wrote:
> 
> 
> You can use one of the exiting function queries (if they fit your need) or
> write a custom function query to reorder the results of a query.
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Custom-Sorting-Algorithm-tp21837721p21838804.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Custom Sorting Algorithm

[Otis Gospodnetic]

Hi,

You can use one of the exiting function queries (if they fit your need) or 
write a custom function query to reorder the results of a query.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: wojtekpia 
> To: solr-user@lucene.apache.org
> Sent: Wednesday, February 4, 2009 2:28:56 PM
> Subject: Custom Sorting Algorithm
> 
> 
> Is an easy way to choose/create an alternate sorting algorithm? I'm
> frequently dealing with large result sets (a few million results) and I
> might be able to benefit domain knowledge in my sort.
> -- 
> View this message in context: 
> http://www.nabble.com/Custom-Sorting-Algorithm-tp21837721p21837721.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Custom Sorting

[psyron]

Thanks Erik, that helped me a lot ...

but still have somthing, i am not sure about:

If i am using a custom sort - like the DistanceComparator example described
in "oh" your book - and i debugged the code and seem to understand that
the the distances-array is created for all indexed documents - not only
for the search result. The compare-function is then called only for the
docs of the search result, right?
My problem is now, that i wonder, if it is not possible to compute only the
distances from the documents of the search result (that should help the
performance, if there are a lot of documents, but the search result is
mostly very small, right?)

Another point:
Of course it also could be interesting to compute all distances for all
documents the first time a new start location is given, in the case, that
you want do a lot of queries from the same location. But this would then
only make sense, if all distances are cached together with the location
value.
I am not sure how things are actually handled in lucene/solr. What and
at which timer things are cached?

To compute distances only for the search result, i could
- store the reader instance in a variable
- for every doc-id called in the compare function the first time, i could
  compute the distance at this moment
- and then compare
Would this work? Or is there a better way to compute the distances
only on the search result?

A lot of questions, i know,

Thanks for the good book,
Markus


Erik Hatcher wrote:
> 
>* QueryComponent - this is where results are generated, it uses a  
> SortSpec from the QParser.
> 
>* QParser#getSort - creating a custom QParser you'll be able to  
> wire in your own custom sort
> 
> You can write your own QParserPlugin and QParser, and configure it  
> into solrconfig.xml and should be good to go.  Subclassing existing  
> classes, this should only be a handful of lines of code to do.
> 

-- 
View this message in context: 
http://www.nabble.com/Custom-Sorting-tp1659p21825900.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Custom Sorting

[Erik Hatcher]

Markus,

A couple of code pointers for you:

  * QueryComponent - this is where results are generated, it uses a  
SortSpec from the QParser.


  * QParser#getSort - creating a custom QParser you'll be able to  
wire in your own custom sort


You can write your own QParserPlugin and QParser, and configure it  
into solrconfig.xml and should be good to go.  Subclassing existing  
classes, this should only be a handful of lines of code to do.


Erik


On Dec 16, 2008, at 3:54 AM, psyron wrote:



I have the same problem, also need to plugin my "customComparator",  
but as
there is no explanation of the framework, how a RequestHandler is  
working,

what comes in, what comes out ... just impossible!

Can someone explain where i have to add which code, to just have the  
same

functionality as the StandardRequestHandler, but also adding a custom
sorting?

Thanks,
Markus


hossman wrote:



: Sort sort = new Sort(new SortField[]
: { SortField.FIELD_SCORE, new SortField(customValue,
SortField.FLOAT,
: true) });
: indexSearcher.search(q, sort)

that appears to just be a sort on score withe a secondary reversed
float sort on whatever field name is in the variable  
"customValue" ...

assuming hte field name is "FIELD" that's hte same thing as...
  sort=score+asc,+FIELD+desc

: Sort sort = new Sort(new SortField(customValue, customComparator))
: indexSearcher.search(q, sort)

this is using a custom SortComparatorSource -- code you (or someone  
else)

has written which is not part of Lucene and which tells lucene how to
order the documents using whatever crazy logic it wants ... for  
obvious
reasons Solr can't do that same logic (since it doesn't know what  
it is)


although many things in Solr are easily customizable, just by  
writting a

little factory and configuring it by class name, i'm afraind
SortComparatorSources aren't once of them.  You could write a custom
RequestHandler which used your SortComparatorSource, or you could  
write a
custom FieldType that used it anything someone sorted on that  
field ...

but those are the best options i cna think of.



-Hoss





--
View this message in context: 
http://www.nabble.com/Custom-Sorting-tp1659p21029370.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Custom Sorting

[Shalin Shekhar Mangar]

On Tue, Dec 16, 2008 at 2:24 PM, psyron  wrote:

>
> I have the same problem, also need to plugin my "customComparator", but as
> there is no explanation of the framework, how a RequestHandler is working,
> what comes in, what comes out ... just impossible!
>
> Can someone explain where i have to add which code, to just have the same
> functionality as the StandardRequestHandler, but also adding a custom
> sorting?


I do not know much about this but I can see that the
org.apache.solr.handler.component.QueryElevationComponent inside Solr uses a
custom sort to push documents at the top of the results.

HTH.

-- 
Regards,
Shalin Shekhar Mangar.

Re: Custom Sorting

[psyron]

I have the same problem, also need to plugin my "customComparator", but as
there is no explanation of the framework, how a RequestHandler is working,
what comes in, what comes out ... just impossible!

Can someone explain where i have to add which code, to just have the same
functionality as the StandardRequestHandler, but also adding a custom
sorting?

Thanks,
Markus


hossman wrote:
> 
> 
> : Sort sort = new Sort(new SortField[]
> : { SortField.FIELD_SCORE, new SortField(customValue,
> SortField.FLOAT,
> : true) });
> : indexSearcher.search(q, sort)
> 
> that appears to just be a sort on score withe a secondary reversed
> float sort on whatever field name is in the variable "customValue" ...
> assuming hte field name is "FIELD" that's hte same thing as...
>sort=score+asc,+FIELD+desc
> 
> : Sort sort = new Sort(new SortField(customValue, customComparator))
> : indexSearcher.search(q, sort)
> 
> this is using a custom SortComparatorSource -- code you (or someone else)
> has written which is not part of Lucene and which tells lucene how to
> order the documents using whatever crazy logic it wants ... for obvious
> reasons Solr can't do that same logic (since it doesn't know what it is)
> 
> although many things in Solr are easily customizable, just by writting a
> little factory and configuring it by class name, i'm afraind
> SortComparatorSources aren't once of them.  You could write a custom
> RequestHandler which used your SortComparatorSource, or you could write a
> custom FieldType that used it anything someone sorted on that field ...
> but those are the best options i cna think of.
> 
> 
> 
> -Hoss
> 
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Custom-Sorting-tp1659p21029370.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: custom sorting

[Chris Hostetter]

: If you went with the FunctionQuery approach for sorting by distance, would
: there be any way to use the output of the FunctionQuery to limit the
: documents to those within a certain radius?  Or is it just for boosting
: documents, not for filtering?

FunctionQueries don't restrict the set of documents at all, so you would 
need to combine it with a seperate query that limits the documents ... the 
simplest way would be as you say: by combining it with two range queries 
that would define a lat/lon "bounding box"

: Also, even if you're just using it for boosting, is there a way to avoid
: running the expensive function on all docs in the index?  Could you somehow

that's the bueaty of "skipTo" in query scoring ... BooleanQueries keep 
track of the "next" document each clause can match (in order by docid), 
and tell all of the other queries to "skipTo" that doc and not bother 
trying to score any doc ids below that.



-Hoss

Re: custom sorting

[Doug Daniels]

If you went with the FunctionQuery approach for sorting by distance, would
there be any way to use the output of the FunctionQuery to limit the
documents to those within a certain radius?  Or is it just for boosting
documents, not for filtering?

Also, even if you're just using it for boosting, is there a way to avoid
running the expensive function on all docs in the index?  Could you somehow
nest bounding-box RangeQuery for latitude and longitude inside as
ValueSources?

Thanks,
Doug


hossman wrote:
> 
> 
> : leaks, etc.).  (Speaking of which, could anyone with more Lucene/Solr
> : experience than I comment on the performance characteristics of the
> : locallucene implementation mentioned on the list recently?  I've taken
> : a first look and it seems reasonable to me.)
> 
> i cna't speak for anyone else, but i haven't had a chacne to drill into it 
> yet.
> 
> : Using a function query, as Yonik suggests above, is another approach.
> : But to get a true sort, you have to boost the original query to zero?
> 
> or a very close approximation there of (0.01 perhaps)
> 
> keep in mind: a "true" distance sort while easy to explain may not be as 
> useful as a sort by score where the distance is factored into the score 
> ... there have been some threads about this on the java-user list in the 
> past and it's been discussed that a really relevant result 2 miles away is 
> probably better then a mildly relevent result 1.5 miles away ... that's 
> where a function query with well choosen boosts might serve you better.
> 
> : How does this impact the results returned by the original query?  Will
> : the requirements (and boosts) of the original (now nested) query
> : remain intact, only sorted by the function?  Also, is there any way to
> 
> it should ... but i won't swear to that.
> 
> : do this with the dismax handler?
> 
> a strict sort on the value of a a function?  put the function in the bf
> param, don't bother with bq or pf params and change your qf params to all 
> have really small boosts.
> 
> 
> 
> -Hoss
> 
> 
> 

-- 
View this message in context: 
http://www.nabble.com/custom-sorting-tf4521989.html#a13436617
Sent from the Solr - User mailing list archive at Nabble.com.

Re: custom sorting

[Chris Hostetter]

: leaks, etc.).  (Speaking of which, could anyone with more Lucene/Solr
: experience than I comment on the performance characteristics of the
: locallucene implementation mentioned on the list recently?  I've taken
: a first look and it seems reasonable to me.)

i cna't speak for anyone else, but i haven't had a chacne to drill into it 
yet.

: Using a function query, as Yonik suggests above, is another approach.
: But to get a true sort, you have to boost the original query to zero?

or a very close approximation there of (0.01 perhaps)

keep in mind: a "true" distance sort while easy to explain may not be as 
useful as a sort by score where the distance is factored into the score 
... there have been some threads about this on the java-user list in the 
past and it's been discussed that a really relevant result 2 miles away is 
probably better then a mildly relevent result 1.5 miles away ... that's 
where a function query with well choosen boosts might serve you better.

: How does this impact the results returned by the original query?  Will
: the requirements (and boosts) of the original (now nested) query
: remain intact, only sorted by the function?  Also, is there any way to

it should ... but i won't swear to that.

: do this with the dismax handler?

a strict sort on the value of a a function?  put the function in the bf
param, don't bother with bq or pf params and change your qf params to all 
have really small boosts.



-Hoss

Re: custom sorting

[Chris Hostetter]

: > Using something like this, how would the custom SortComparatorSource
: > get a parameter from the request to use in sorting calculations?

in general: you wouldn't you would have to specify all options as init 
params for the FieldType -- which makes it pretty horrible for distance 
calculations, and isn't something i considered when i posted that.

the only way i can think of that you can really solve the problem with a 
plugin at the moment (without some serious internal changes that yonik 
describes below) would be to use a dynamicField when you want geodistance 
sort, and encode the center lat/lon point in the field name, ala:

   sort=geodist_-124.75_93.45

: or extend solr's sorting mechanisms to allow specifying a function to sort by.
: 
: sort="dist(10.4,20.2,geoloc) asc"

thta would in fact, kick ass.  even if there is a better solution for the 
distance stuff the idea of being able to specify a raw function as a sort 
would be pretty sick. (NOTE: that's "sick" as in "so good it's amazing" 
... since the last person i used that idiom with didn't understand and 
thought i ment "bad")



-Hoss

RE: custom sorting

[Sandeep Shetty]

i have been testing locallucene with our data for the last couple of days.
one issue i faced with it is during when using geo sorting is that it seems
to eat up all the memory, however big and become progressively slower,
finally after several requests (10 or so in my case) it throws up a
java.lang.OutOfMemoryError: Java heap space error.

is there a way to get around this?

-Original Message-
From: Jon Pierce [mailto:[EMAIL PROTECTED]
Sent: 28 September 2007 15:48
To: solr-user@lucene.apache.org
Subject: Re: custom sorting


Is the machinery in place to do this now (hook up a function query to
be used in sorting)?

I'm trying to figure out what's the best way to do a distance sort:
custom comparator or function query.

Using a custom comparator seems straightforward and reusable across
both the standard and dismax handlers.  But it also seems most likely
to impact performance (or at least require the most work/knowledge to
get right by minimizing calculations, caching, watching out for memory
leaks, etc.).  (Speaking of which, could anyone with more Lucene/Solr
experience than I comment on the performance characteristics of the
locallucene implementation mentioned on the list recently?  I've taken
a first look and it seems reasonable to me.)

Using a function query, as Yonik suggests above, is another approach.
But to get a true sort, you have to boost the original query to zero?
How does this impact the results returned by the original query?  Will
the requirements (and boosts) of the original (now nested) query
remain intact, only sorted by the function?  Also, is there any way to
do this with the dismax handler?

Thanks,
- Jon

On 9/27/07, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> On 9/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
> > Using something like this, how would the custom SortComparatorSource
> > get a parameter from the request to use in sorting calculations?
>
> perhaps hook in via function query:
>   dist(10.4,20.2,geoloc)
>
> And either manipulate the score with that and sort by score,
>
> q=+(foo bar)^0 dist(10.4,20.2,geoloc)
> sort=score asc
>
> or extend solr's sorting mechanisms to allow specifying a function to sort
by.
>
> sort="dist(10.4,20.2,geoloc) asc"
>
> -Yonik
>

This email is confidential and may also be privileged. If you are not the 
intended recipient please notify us immediately by telephoning +44 (0)20 7452 
5300 or email [EMAIL PROTECTED] You should not copy it or use it for any 
purpose nor disclose its contents to any other person. Touch Local cannot 
accept liability for statements made which are clearly the sender's own and are 
not made on behalf of the firm.

Touch Local Limited
Registered Number: 2885607
VAT Number: GB896112114
Cardinal Tower, 12 Farringdon Road, London EC1M 3NN
+44 (0)20 7452 5300

Re: custom sorting

[Jon Pierce]

Is the machinery in place to do this now (hook up a function query to
be used in sorting)?

I'm trying to figure out what's the best way to do a distance sort:
custom comparator or function query.

Using a custom comparator seems straightforward and reusable across
both the standard and dismax handlers.  But it also seems most likely
to impact performance (or at least require the most work/knowledge to
get right by minimizing calculations, caching, watching out for memory
leaks, etc.).  (Speaking of which, could anyone with more Lucene/Solr
experience than I comment on the performance characteristics of the
locallucene implementation mentioned on the list recently?  I've taken
a first look and it seems reasonable to me.)

Using a function query, as Yonik suggests above, is another approach.
But to get a true sort, you have to boost the original query to zero?
How does this impact the results returned by the original query?  Will
the requirements (and boosts) of the original (now nested) query
remain intact, only sorted by the function?  Also, is there any way to
do this with the dismax handler?

Thanks,
- Jon

On 9/27/07, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> On 9/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
> > Using something like this, how would the custom SortComparatorSource
> > get a parameter from the request to use in sorting calculations?
>
> perhaps hook in via function query:
>   dist(10.4,20.2,geoloc)
>
> And either manipulate the score with that and sort by score,
>
> q=+(foo bar)^0 dist(10.4,20.2,geoloc)
> sort=score asc
>
> or extend solr's sorting mechanisms to allow specifying a function to sort by.
>
> sort="dist(10.4,20.2,geoloc) asc"
>
> -Yonik
>

Re: custom sorting

[Narayanan Palasseri]

Hi all,
Regarding this issue, we tried using a custom request handler which inturn
uses the CustomCompartor. But this has a memory leak and we are almost got
stuck up at that point. As somebody mentioned, we are thinking of moving
towards function query to achieve the same. Please let me know whether
anybody has faced similar issue or is it that we are doing something wrong.
The additional code that we have return from the default handler is as given
below.

*

if* ("*myappRequestHandler*".equalsIgnoreCase(requestHandler))

{

sort = getSortCriteria(*new* SimpleSortComparatorSourceImpl());

}

Thanks and Regards
Narayanan


On 9/28/07, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>
> On 9/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
> > Using something like this, how would the custom SortComparatorSource
> > get a parameter from the request to use in sorting calculations?
>
> perhaps hook in via function query:
> dist(10.4,20.2,geoloc)
>
> And either manipulate the score with that and sort by score,
>
> q=+(foo bar)^0 dist(10.4,20.2,geoloc)
> sort=score asc
>
> or extend solr's sorting mechanisms to allow specifying a function to sort
> by.
>
> sort="dist(10.4,20.2,geoloc) asc"
>
> -Yonik
>

Re: custom sorting

[Yonik Seeley]

On 9/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
> Using something like this, how would the custom SortComparatorSource
> get a parameter from the request to use in sorting calculations?

perhaps hook in via function query:
  dist(10.4,20.2,geoloc)

And either manipulate the score with that and sort by score,

q=+(foo bar)^0 dist(10.4,20.2,geoloc)
sort=score asc

or extend solr's sorting mechanisms to allow specifying a function to sort by.

sort="dist(10.4,20.2,geoloc) asc"

-Yonik

Re: custom sorting

[Erik Hatcher]

On Sep 27, 2007, at 2:50 PM, Chris Hostetter wrote:

to answer the broader question of using customized
LUcene SortComparatorSource objects in solr -- it is in fact possible.

In Solr, all decisisons about how to sort are driven by  
FieldTypes.  You
can subclass any of the FieldTypes that come with Solr and override  
just
the getSortField method to use whatever sort logic you want and  
then use

your new FieldType as you would any other plugin...

http://wiki.apache.org/solr/SolrPlugins

In the case where you have a custom SortComparatorSource that is not
"field" specific (or uses data from morethen one field) you would  
need to
make your field type smart enough to let you cofigure (via the  

declaration in the schema) which fields (if any) to get it's data  
from,
and then create a marker field of that type, which you don't use to  
index

or store any data, but you use to indicate when to trigger your custom
sort logic, ie...



   
   

...and then use "sort=distance+asc" in your query


Using something like this, how would the custom SortComparatorSource  
get a parameter from the request to use in sorting calculations?


I haven't looked under the covers of the local-solr stuff that flew  
by earlier, but looks quite well done.  I think I can speak for many  
that would love to have geo field types / sorting capability built  
into Solr.


Erik

Re: custom sorting

[Chris Hostetter]

: > Previously we were using lucene to do this. by using the
: > SortComparatorSource we could sort the documents returned by distance
: > nicely. we are now switching over to lucene because of the features it
: > provides, however i am not able to see a way to do this in Solr. 

Someone started another thread where they specificly discuss the 
"Geographical distance searching" aspect of your question.

to answer the broader question of using customized 
LUcene SortComparatorSource objects in solr -- it is in fact possible.

In Solr, all decisisons about how to sort are driven by FieldTypes.  You 
can subclass any of the FieldTypes that come with Solr and override just 
the getSortField method to use whatever sort logic you want and then use 
your new FieldType as you would any other plugin...

http://wiki.apache.org/solr/SolrPlugins

In the case where you have a custom SortComparatorSource that is not 
"field" specific (or uses data from morethen one field) you would need to 
make your field type smart enough to let you cofigure (via the  
declaration in the schema) which fields (if any) to get it's data from, 
and then create a marker field of that type, which you don't use to index 
or store any data, but you use to indicate when to trigger your custom 
sort logic, ie...


 

   

...and then use "sort=distance+asc" in your query



-Hoss

Re: custom sorting

[Mike Klaas]

On 26-Sep-07, at 5:14 AM, Sandeep Shetty wrote:


Hi Guys,

this question as been asked before but i was unable to find an answer
thats good for me, so hope you guys can help again
i am working on a website where we need to sort the results by  
distance
from the location entered by the user. I have indexed the lat and  
long
info for each record in solr and also i can get the lat and long  
of the

location input by the user.
Previously we were using lucene to do this. by using the
SortComparatorSource we could sort the documents returned by distance
nicely. we are now switching over to lucene because of the  
features it

provides, however i am not able to see a way to do this in Solr.

If someone can point me in the right direction i would be very  
grateful!


Thanks in advance,
Sandeep


This email is confidential and may also be privileged. If you are  
not the intended recipient please notify us immediately by  
telephoning +44 (0)20 7452 5300 or email [EMAIL PROTECTED]  
You should not copy it or use it for any purpose nor disclose its  
contents to any other person. Touch Local cannot accept liability  
for statements made which are clearly the sender's own and are not  
made on behalf of the firm.


Sorry, I'm afraid the above email is already irrevokably publicly  
archived.


-Mike

Re: Custom Sorting

[Chris Hostetter]

: Sort sort = new Sort(new SortField[]
: { SortField.FIELD_SCORE, new SortField(customValue, SortField.FLOAT,
: true) });
: indexSearcher.search(q, sort)

that appears to just be a sort on score withe a secondary reversed
float sort on whatever field name is in the variable "customValue" ...
assuming hte field name is "FIELD" that's hte same thing as...
   sort=score+asc,+FIELD+desc

: Sort sort = new Sort(new SortField(customValue, customComparator))
: indexSearcher.search(q, sort)

this is using a custom SortComparatorSource -- code you (or someone else)
has written which is not part of Lucene and which tells lucene how to
order the documents using whatever crazy logic it wants ... for obvious
reasons Solr can't do that same logic (since it doesn't know what it is)

although many things in Solr are easily customizable, just by writting a
little factory and configuring it by class name, i'm afraind
SortComparatorSources aren't once of them.  You could write a custom
RequestHandler which used your SortComparatorSource, or you could write a
custom FieldType that used it anything someone sorted on that field ...
but those are the best options i cna think of.



-Hoss

Re: custom sorting for multivalued field

[Chris Hostetter]

: Is it possible to assign a custom sorting value for
: each of the values in the multivalued field?  So that
: the document gets sorted differently, depending on the
: matched value in the multivalued field.

Sorting happens extremely independently from matching ... there is no
mechanism available in the underlying Lucene code to allow the Sorting
logic to know why a particular document is a match.

: The other approach would be to store each
: document/keyword pair as a separate document with the
: sorting value as an explicit field.  Is it possible to
: filter the results on the Solr end (based on the
: relevancy of the matched keyword), so that the same
: original document doesn't appear in the result set
: twice?

can you elaborate a bit more on what exactly it is you are trying to
achieve? ...i'm having a hard time understanding hte motivation for
sorting on a keyword field where the sort order is on the keyword that
matches .. for simple single word queries all document will sort
identically, for multi-word queries you might as well just search on each
word seperately and concatenate the result sets in order -- except in the
case where a single document matches on more then one of your query terms,
but you've already said you just want it to appear once ... but why would
you want documents in this order in the first place?

my first assumption would be thaty ou just want docs which match on very
rare keyword to come first and you are only doing searches on this keyword
field, then regualr sort by score should do what you want ... but you
might wnat to omitNOrms and maybe change the coordFactor in your
similarity.




-Hoss

Re: custom sorting for multivalued field

[Otis Gospodnetic]

I'll try answering:
- sorting by the term that happened to match a multi-term field is, as far as I 
know, not possible
- "de-duping" search results is, I believe, possible with a patch that is still 
in JIRA:
https://issues.apache.org/jira/browse/SOLR-236

Otis
--
Lucene Consulting -- http://lucene-consulting.com/



- Original Message 
From: Del Raco <[EMAIL PROTECTED]>
To: solr-user@lucene.apache.org
Sent: Wednesday, July 11, 2007 9:24:10 PM
Subject: custom sorting for multivalued field

Hi,

We are trying to set up Solr to search documents with
multiple keywords, which we have implemented as a
multivalued field.

Is it possible to assign a custom sorting value for
each of the values in the multivalued field?  So that
the document gets sorted differently, depending on the
matched value in the multivalued field.

The other approach would be to store each
document/keyword pair as a separate document with the
sorting value as an explicit field.  Is it possible to
filter the results on the Solr end, so that the same
original document doesn't appear in the result set
twice?

Any ideas or suggestions would be greatly appreciated.
 

Thanks!


   

Building a website is a piece of cake. Yahoo! Small Business gives you all the 
tools to get online.
http://smallbusiness.yahoo.com/webhosting