UNOFFICIAL Hi Shai, Thank you! My application is now working fine with your changes. I really appreciate the time you spent on this.
I'm pleased to hear that you can store more than one NumericDocValueField. I'm not sure where I read that you can't, looks like I got it wrong. Anyway, I'll change the app to use this rather than a stored field. Thanks again for your help, Steve From: Shai Erera [mailto:[email protected]] Sent: Monday, 21 October 2013 7:06 PM To: [email protected] Subject: Re: Creating a SumFacetRequest class [SEC=UNOFFICIAL] I found the problem. You pass IndexSearcher to SumFacetRequest, but you don't implement setNextReader correctly. So what happens in aggreate is that you are given the relative docID in that segment (which in this test it's always 0 since you commit after every addDocument()) and therefore you read the stored value of the same absolute docID=0. If I made these changes to the code, the prints look ok (except for the category which you add value 0 to it, and it's not printed): @Override public boolean setNextReader(AtomicReaderContext context) throws IOException { this.context = context; return true; } and @Override public void aggregate(int docID, float score, IntsRef ordinals) throws IOException { Document doc = context.reader().document(docID); int value = doc.getField(fieldName).numericValue().intValue(); for (int i = 0; i < ordinals.length; i++) { System.out.println("docID=" + docID + ", ord=" + ordinals.ints[i] + ", value=" + value); sumArray[ordinals.ints[i]] += value; } } You shouldn't call commit() after every addDocument. I don't know if that's what you do in your program, or it's just for this test, but it's unnecessary. And you can store more than one NumericDocValueField in a document (where did you read that you can't?), you just have to give them different names. Shai On Mon, Oct 21, 2013 at 8:21 AM, Stephen GRAY <[email protected]<mailto:[email protected]>> wrote: UNOFFICIAL Hi Shai, Thanks very much for the helpful response. No I'm not using the latest Lucene. I'm using 4.3.0 (which happens to be the latest in our Maven repository). I've attached a test case which shows the problem. I've tried to make this as simple as possible, but it's still 240 lines (sorry). I did consider NumericDocValuesField, but I need to get sums for more than one field and my understanding was that you can only store one NumericDocValues field per document. There's currently not many stored fields in each document, but I might look at using FieldSelector to ensure I only read the field I'm looking for. Thanks for looking at this, Steve From: Shai Erera [mailto:[email protected]<mailto:[email protected]>] Sent: Friday, 18 October 2013 8:28 PM To: [email protected]<mailto:[email protected]> Subject: Re: Creating a SumFacetRequest class [SEC=UNOFFICIAL] Hi Stephen, The code seems correct in general (I have some comments below). The ordinals that you get are those that are associated with that document (docID). I assume this is not the newest Lucene though, right? Can you boil this down to a simple testcase adding a couple of documents with the value which you would like to aggregate and print the actual values each facet gets? About the code, I see that you read the value from a stored field. I recommend that you store the value in a NumericDocValuesField as it's loaded much faster and more efficiently than what you do. Your code currently reads all stored fields for the document, which is both expensive and inefficient. Also, if you move up to the latest Lucene (4.5.0), the API is more segment-oriented, so you're given all matching documents up front, and then you can ask for their NumericDocValues once while you iterate over them. These comments are related to efficiency though. As for your original question, a simple testcase demonstrating the problem will help me spot the issue. Shai On Fri, Oct 18, 2013 at 8:57 AM, Stephen GRAY <[email protected]<mailto:[email protected]>> wrote: UNOFFICIAL Hi everyone, I need to get a sum of the values in an int field in all the documents in a facet. Because there is only a CountFacetRequest in Lucene I am trying to write a SumFacetRequest with associated Aggregator which does this. However the results I am getting when I use my SumFacetRequest are not correct. Here is the aggregate method from the Aggregator I have written (based on CountingAggregator): @Override public void aggregate(int docID, float score, IntsRef ordinals) throws IOException { Document doc = searcher.doc(docID); int value = doc.getField(fieldName).numericValue().intValue(); for (int i = 0; i < ordinals.length; i++) { sumArray[ordinals.ints[i]] += value; } } Would someone be able to tell me if this is correct? I have been assuming that ordinals.ints[i] returns an id for a facet that contains the document but maybe this is not correct. Any help would be greatly appreciated. Apologies if this is not the correct forum to post this. Thanks, Steve UNOFFICIAL -------------------------------------------------------------------- Important Notice: If you have received this email by mistake, please advise the sender and delete the message and attachments immediately. This email, including attachments, may contain confidential, sensitive, legally privileged and/or copyright information. Any review, retransmission, dissemination or other use of this information by persons or entities other than the intended recipient is prohibited. DIAC respects your privacy and has obligations under the Privacy Act 1988. The official departmental privacy policy can be viewed on the department's website at www.immi.gov.au<http://www.immi.gov.au>. See: http://www.immi.gov.au/functional/privacy.htm --------------------------------------------------------------------- UNOFFICIAL --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected]<mailto:[email protected]> For additional commands, e-mail: [email protected]<mailto:[email protected]> UNOFFICIAL
