RE: solr 4.0 - pagination
Well, right now, I'm using SOLR in a LOT of my projects. I'm VERY fond of it, proud of it and VERY happy that such a team exists to make it work. Of course the pagination issue is a bit frustrating on the field collapsing... But... heck... I'm currently de-normalizing my postgresql database and... I'm just counting the total unique rows using SQL :D And let SOLR do the rest of the job So... As a generic idea for the SOLR JAVA expert-people, I think the counting of groups per each field should be something to be done at indexing time rather than at query-time. So basically when you index each document, you compute the grouping thingie IF the user would so choose to have his field as eligible for grouping inside his schema.xml... If you guys geather my meaning... So I would have a field like this: field GROUPING=TRUE name=unit_id type=tint indexed=true stored=true required=true / Or something like that... and the indexer will know to compute the number of groups that exist for this field, so that number would be available at query-time without too much stress on the memory or CPU. Hope this helps In the meanwhile... just count the total from SQL... And keep your index in-sync often :D Cip. -- View this message in context: http://lucene.472066.n3.nabble.com/solr-4-0-pagination-tp1812384p2123255.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: solr 4.0 - pagination
Then what will be when we filter out only some result and want to group ,how your index time group count will help. - Grijesh -- View this message in context: http://lucene.472066.n3.nabble.com/solr-4-0-pagination-tp1812384p2124747.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: solr 4.0 - pagination
Dear Yonik, this is fantastic, but can you tell any time it will be ready ? I would need this feature in two weeks. Is it possible to finish and make an update in this time or should I look for another solution cocerning the pgaination (like implement just more results link instead of pagination) ? best regards, Rich -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Saturday, October 30, 2010 19:29 To: solr-user@lucene.apache.org Subject: Re: solr 4.0 - pagination On Sat, Oct 30, 2010 at 12:22 PM, Papp Richard ccode...@gmail.com wrote: I'm using Solr 4.0 with grouping (field collapsing), but unfortunately I can't solve the pagination. It's not implemented yet, but I'm working on that right now. -Yonik http://www.lucidimagination.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5576 (20101029) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5598 (20101107) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com
Re: solr 4.0 - pagination
On Sun, Nov 7, 2010 at 10:55 AM, Papp Richard ccode...@gmail.com wrote: this is fantastic, but can you tell any time it will be ready ? It already is ;-) Grab the latest trunk or the latest nightly build. -Yonik http://www.lucidimagination.com
RE: solr 4.0 - pagination
thank you very much Yonik! you are a magician! regards, Rich -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Sunday, November 07, 2010 18:04 To: Papp Richard Cc: solr-user@lucene.apache.org Subject: Re: solr 4.0 - pagination On Sun, Nov 7, 2010 at 10:55 AM, Papp Richard ccode...@gmail.com wrote: this is fantastic, but can you tell any time it will be ready ? It already is ;-) Grab the latest trunk or the latest nightly build. -Yonik http://www.lucidimagination.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5598 (20101107) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5598 (20101107) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com
RE: solr 4.0 - pagination
Hi Yonik, I've just tried the latest stable version from nightly build: apache-solr-4.0-2010-11-05_08-06-28.war I have some concerns however: I have 3 documents; 2 in the first group, 1 in the 2nd group. 1. I got for matches 3 - which is good, but I still don't know how many groups I have. (using start = 0, rows = 10) 2. as far as I see the start / rows is working now, but the matches is returned incorrectly = it said matches = 3 instead of = 1, when I used start = 1, rows = 1 so can you help me, how to compute how many pages I'll have, because the matches can't use for this. regards, Rich -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Sunday, November 07, 2010 18:04 To: Papp Richard Cc: solr-user@lucene.apache.org Subject: Re: solr 4.0 - pagination On Sun, Nov 7, 2010 at 10:55 AM, Papp Richard ccode...@gmail.com wrote: this is fantastic, but can you tell any time it will be ready ? It already is ;-) Grab the latest trunk or the latest nightly build. -Yonik http://www.lucidimagination.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5598 (20101107) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5599 (20101107) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com
Re: solr 4.0 - pagination
On Sun, Nov 7, 2010 at 2:45 PM, Papp Richard ccode...@gmail.com wrote: Hi Yonik, I've just tried the latest stable version from nightly build: apache-solr-4.0-2010-11-05_08-06-28.war I have some concerns however: I have 3 documents; 2 in the first group, 1 in the 2nd group. 1. I got for matches 3 - which is good, but I still don't know how many groups I have. (using start = 0, rows = 10) 2. as far as I see the start / rows is working now, but the matches is returned incorrectly = it said matches = 3 instead of = 1, when I used start = 1, rows = 1 matches is the number of documents before grouping, so start/rows or group.offset/group.limit will not affect this number. so can you help me, how to compute how many pages I'll have, because the matches can't use for this. Solr doesn't even know given the current algorithm, hence it can't return that info. The issue is that to calculate the total number of groups, we would need to keep each group in memory (which could cause a big blowup if there are tons of groups). The current algorithm only keeps the top 10 groups (assuming rows=10) in memory at any one time, hence it has no idea what the total number of groups is. -Yonik http://www.lucidimagination.com
RE: solr 4.0 - pagination
Hey Yonik, Sorry, I think the matches is ok - because it probably returns always the total document number - however I don't know how to compute the number of pages. thanks, Rich -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Sunday, November 07, 2010 18:04 To: Papp Richard Cc: solr-user@lucene.apache.org Subject: Re: solr 4.0 - pagination On Sun, Nov 7, 2010 at 10:55 AM, Papp Richard ccode...@gmail.com wrote: this is fantastic, but can you tell any time it will be ready ? It already is ;-) Grab the latest trunk or the latest nightly build. -Yonik http://www.lucidimagination.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5598 (20101107) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5599 (20101107) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com
RE: solr 4.0 - pagination
I see. Let's assume that there are 1000 groups. Can I use safely (with no negative impact on memory usage or slowness) the start = 990, rows = 10 to get the latest page? Or this will not work, due you will need to compute all the groups till 1000, in order to return the last 10, and because of this the whole will be slow / memory usage will increase considerably. regards, Rich -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Sunday, November 07, 2010 21:54 To: Papp Richard Cc: solr-user@lucene.apache.org Subject: Re: solr 4.0 - pagination On Sun, Nov 7, 2010 at 2:45 PM, Papp Richard ccode...@gmail.com wrote: Hi Yonik, I've just tried the latest stable version from nightly build: apache-solr-4.0-2010-11-05_08-06-28.war I have some concerns however: I have 3 documents; 2 in the first group, 1 in the 2nd group. 1. I got for matches 3 - which is good, but I still don't know how many groups I have. (using start = 0, rows = 10) 2. as far as I see the start / rows is working now, but the matches is returned incorrectly = it said matches = 3 instead of = 1, when I used start = 1, rows = 1 matches is the number of documents before grouping, so start/rows or group.offset/group.limit will not affect this number. so can you help me, how to compute how many pages I'll have, because the matches can't use for this. Solr doesn't even know given the current algorithm, hence it can't return that info. The issue is that to calculate the total number of groups, we would need to keep each group in memory (which could cause a big blowup if there are tons of groups). The current algorithm only keeps the top 10 groups (assuming rows=10) in memory at any one time, hence it has no idea what the total number of groups is. -Yonik http://www.lucidimagination.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5599 (20101107) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5599 (20101107) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com
solr 4.0 - pagination
Hi all, I'm using Solr 4.0 with grouping (field collapsing), but unfortunately I can't solve the pagination. Mainly there are two problems: - the query fields start rows doesn't work anymore - beside of the values, it always returns the data as the start would be 0 (start = 0) - the result contains just the total document number and not the total groups number Can anyone help me, how to solve this? regards, Rich
Re: solr 4.0 - pagination
On Sat, Oct 30, 2010 at 12:22 PM, Papp Richard ccode...@gmail.com wrote: I'm using Solr 4.0 with grouping (field collapsing), but unfortunately I can't solve the pagination. It's not implemented yet, but I'm working on that right now. -Yonik http://www.lucidimagination.com
RE: solr 4.0 - pagination
Can you estimate please when it will be done? thanks, Rich -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Saturday, October 30, 2010 19:29 To: solr-user@lucene.apache.org Subject: Re: solr 4.0 - pagination On Sat, Oct 30, 2010 at 12:22 PM, Papp Richard ccode...@gmail.com wrote: I'm using Solr 4.0 with grouping (field collapsing), but unfortunately I can't solve the pagination. It's not implemented yet, but I'm working on that right now. -Yonik http://www.lucidimagination.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5576 (20101029) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5576 (20101029) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com
Re: solr 4.0 - pagination
See: https://issues.apache.org/jira/browse/SOLR-2207 https://issues.apache.org/jira/browse/SOLR-2207You can track the progress here, but it hasn't been committed to trunk yet. Your options are: 1 get a recent trunk source tree and apply the patch and compile. 2 wait until it gets put in the trunk and get a nightly build. 3 volunteer to test it a lot G... Best Erick On Sat, Oct 30, 2010 at 1:27 PM, Papp Richard ccode...@gmail.com wrote: Can you estimate please when it will be done? thanks, Rich -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Saturday, October 30, 2010 19:29 To: solr-user@lucene.apache.org Subject: Re: solr 4.0 - pagination On Sat, Oct 30, 2010 at 12:22 PM, Papp Richard ccode...@gmail.com wrote: I'm using Solr 4.0 with grouping (field collapsing), but unfortunately I can't solve the pagination. It's not implemented yet, but I'm working on that right now. -Yonik http://www.lucidimagination.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5576 (20101029) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com __ Information from ESET NOD32 Antivirus, version of virus signature database 5576 (20101029) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com