Group by in Lucene ?

2007-11-04 Thread Marcus Herou
Hi. I have a situation where I'm searching amongst some 100K feeds and only want one result per site in return. I have developed a really simple method of grouping which just scrolls through the resultset(hitset) until a maxNum docs of feeds from a set of unique sites is populated. Since I don't w

GROUP BY in Lucene

2015-08-09 Thread Gimantha Bandara
Hi all, Is there a way to achieve $subject? For example, consider the following SQL query. SELECT A, B, C SUM(D) as E FROM `table` WHERE time BETWEEN fromDate AND toDate *GROUP BY X,Y,Z* In the above query we can group the records by, X,Y,Z. Is there a way to achieve the same in Lucene? (I gues

Re: GROUP BY in Lucene

2016-03-18 Thread Gimantha Bandara
Hi Rob, Thanks a lot for above very descriptive answer. I will give it a try. On Friday, March 18, 2016, Rob Audenaerde wrote: > Hi Gimantha, > > You don't need to store the aggregates and don't need to retrieve > Documents. The aggregates are calculated during collection using the > BinaryDocV

Re: GROUP BY in Lucene

2016-03-19 Thread Gimantha Bandara
Hi Rob, Thank you for explaining your approach. Still I have a few questions. Do I need to store the values being aggregated as STORED at indexing time? and how does the collector handle a large number of documents when aggregating? I mean lets say I have several millions documents in an index and

Re: GROUP BY in Lucene

2016-03-19 Thread Rob Audenaerde
Hi Gimantha, You don't need to store the aggregates and don't need to retrieve Documents. The aggregates are calculated during collection using the BinaryDocValues from the facet-module. What I do, is that I need to store values in the facets using AssociationFacetFields. (for example FloatAssocia

Re: Group by in Lucene ?

2009-01-27 Thread Marcus Herou
checon.com > >> > >> Lucene Helpful Hints: > >> http://wiki.apache.org/lucene-java/BasicsOfPerformance > >> http://wiki.apache.org/lucene-java/LuceneFAQ > >> > >> > >> > >> ---

Re: Group by in Lucene ?

2009-01-28 Thread ninaS
: Iterate through the result documents and take one of each group. Each document has a groupingKey. I remember which groupingKey is already used and don't take another document of this group into the result list. Regards, Nina -- View this message in context: http://www.nabble.com/Gro

Re: Group by in Lucene ?

2009-01-28 Thread ninaS
gt; document has a groupingKey. I remember which groupingKey is already used > and don't take another document of this group into the result list. > > Regards, > Nina > -- View this message in context: http://www.nabble.com/Group-by-in-Lucene---tp13581760p21702742.html

Re: Group by in Lucene ?

2009-01-28 Thread Marcus Herou
; > It is easier to first do a normal search und "group by" afterwards: > > > > Iterate through the result documents and take one of each group. Each > > document has a groupingKey. I remember which groupingKey is already used > > and don't take another doc

Re: Group by in Lucene ?

2009-01-28 Thread Marcus Herou
thing. >> >> >> ninaS wrote: >> > >> > Hello, >> > >> > yes I tried HitCollector but I am not satisfied with it because you can >> > not use sorting with HitCollector unless you implement a way to use >> > TopFieldTocCollector. I did

Re: Group by in Lucene ?

2009-01-28 Thread Erick Erickson
> is a good choice. If you only count you don't need to sort anything. > > > > > > ninaS wrote: > > > > > > Hello, > > > > > > yes I tried HitCollector but I am not satisfied with it because you can > > > not use sorting wi

Re: Group by in Lucene ?

2009-01-28 Thread Mark Miller
Group-by in Lucene/Solr has not been solved in a great general way yet to my knowledge. Ideally, we would want a solution that does not need to fit into memory. However, you need the value of the field for each document. to do the grouping As you are finding, this is not cheap to get

Re: Group by in Lucene ?

2009-02-01 Thread Marcus Herou
> > > >public int getDistinctCount() > >{ > >distinctSize = set.size(); > >return distinctSize; > > } > > } > > > > > > On Wed, Jan 2

Re: Group by in Lucene ?

2009-02-01 Thread Marcus Herou
Marcus On Wed, Jan 28, 2009 at 5:02 PM, Mark Miller wrote: > Group-by in Lucene/Solr has not been solved in a great general way yet to > my knowledge. > > Ideally, we would want a solution that does not need to fit into memory. > However, you need the value

Re: Group by in Lucene ?

2009-08-01 Thread mschipperheyn
http://code.google.com/p/bobo-browse looks like it may be the ticket. Marc -- View this message in context: http://www.nabble.com/Group-by-in-Lucene---tp13581760p24767693.html Sent from the Lucene - Java Users mailing list archive at Nabble.com

Re: Group by in Lucene ?

2009-08-02 Thread Erik Hatcher
Don't overlook Solr: http://lucene.apache.org/solr Erik On Aug 1, 2009, at 5:43 AM, mschipperheyn wrote: http://code.google.com/p/bobo-browse looks like it may be the ticket. Marc -- View this message in context: http://www.nabble.com/Group-by-in-Lucene---tp13581760p24767693

Re: Group by in Lucene ?

2007-11-05 Thread Grant Ingersoll
Solr has an issue outstanding right now that implements something that may be close to what you want. They are calling it Field Collapsing. See https://issues.apache.org/jira/browse/SOLR-236 -Grant On Nov 5, 2007, at 12:57 AM, Marcus Herou wrote: Hi. I have a situation where I'm searchi

Re: Group by in Lucene ?

2007-11-05 Thread Marcus Herou
Thanks. They seem to have got real far in the dev cycle on this. Seems like it will hit the road in Solr 1.3. However I would really like this feature to be developed for Core Lucene, how do I start that process? Develop it yourself you would say :) I'm serious isn't it a really cool and useful fe

Re: Group by in Lucene ?

2007-11-05 Thread Grant Ingersoll
On Nov 5, 2007, at 7:49 AM, Marcus Herou wrote: Thanks. They seem to have got real far in the dev cycle on this. Seems like it will hit the road in Solr 1.3. However I would really like this feature to be developed for Core Lucene, how do I start that process? Develop it yourself you woul

Re: Group by in Lucene ?

2007-11-06 Thread Marcus Herou
Cool. I'll do since this is a field which I can spend time in. Kindly //Marcus On 11/5/07, Grant Ingersoll <[EMAIL PROTECTED]> wrote: > > > On Nov 5, 2007, at 7:49 AM, Marcus Herou wrote: > > > Thanks. They seem to have got real far in the dev cycle on this. > > Seems like > > it will hit the ro

Re: Group by in Lucene ?

2007-12-05 Thread ninaS
l Hints: >> http://wiki.apache.org/lucene-java/BasicsOfPerformance >> http://wiki.apache.org/lucene-java/LuceneFAQ >> >> >> >> - >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For add

Re: GROUP BY in Lucene

2015-08-10 Thread Michael McCandless
Lucene has a grouping module that has several approaches for grouping search hits, though it's only by a single field I believe. Mike McCandless http://blog.mikemccandless.com On Sun, Aug 9, 2015 at 2:55 PM, Gimantha Bandara wrote: > Hi all, > > Is there a way to achieve $subject? For example,

Re: GROUP BY in Lucene

2015-08-10 Thread Rob Audenaerde
You can write a custom (facet) collector to do this. I have done something similar, I'll describe my approach: For all the values that need grouping or aggregating, I have added a FacetField ( an AssociatedFacetField, so I can store the value alongside the ordinal) . The main search stays the same