In case anyone was checking this out for me, I have this matter resolved. It appears that the culprit was that I wasn't resetting my Boolean flag in preparation of processing the next TokenStream.
Thanks! Josh Edwards -----Original Message----- From: Edwards, Joshua [mailto:joshua.edwa...@capitalone.com] Sent: Tuesday, August 19, 2014 9:08 AM To: dev@lucene.apache.org Subject: RE: Inconsistent Results Issue with Filter It is a custom filter. Essentially, what I'm trying to do is combine words that are associated with each other into a single token. Here is the incrementToken() method, where most of the work is being performed: private String consumedToken; private boolean alreadyReachedEOF = false; public final boolean incrementToken() throws IOException { //If there is a token sitting in the buffer, then return it without consuming the next token from the buffer. if (consumedToken != null) { input.clearAttributes(); termAtt.setEmpty().append(consumedToken); consumedToken = null; return true; } StringBuffer str = new StringBuffer(); while (true) { State prePop = this.captureState(); boolean reachedEnd = alreadyReachedEOF || !input.incrementToken(); if (reachedEnd && str.length() == 0) return false; else if (reachedEnd) { termAtt.setEmpty(); termAtt.append(str.toString().trim()); alreadyReachedEOF = true; return true; } consumedToken = termAtt.toString(); //tokenMatchesCombineList does a String comparison, and if it is any of the items in the list it returns true, otherwise false. if (tokenMatchesCombineList(consumedToken)) { //Normalizeby removing dashes. consumedToken = consumedToken.replaceAll("-", " "); str.append(consumedToken + " "); termAtt.setEmpty(); termAtt.append(str.toString().trim()); consumedToken = null; } else if (str.length() == 0){ //Nothing in buffer, and a word that shouldn't be combined. Leave it alone. consumedToken = null; return true; } else { //A word that shouldn't be combined after some that should be. Put it back on the stack and return. this.restoreState(prePop); return true; } Thanks! Josh Edwards -----Original Message----- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Monday, August 18, 2014 5:47 PM To: dev@lucene.apache.org Subject: Re: Inconsistent Results Issue with Filter Are you saying you have a custom filter in here? Or this is just OOB code? If the former, we'd need to see the salient parts of the code. If the latter, some example docs would be helpful. Best, Erick On Mon, Aug 18, 2014 at 12:17 PM, Edwards, Joshua <joshua.edwa...@capitalone.com> wrote: > So, as I continue to try to make things work in different ways with > numbers, I have run into a very strange situation. When I perform a > search (for example my_num:”thirty seven”), it returns the correct > results. However, if I run this exact same search again, it returns > no results, and continues giving no results. If I restart my instance, it > gives results again (once). > Any ideas on what might be happening? I’ve been using the Lucene Solr > admin page, and I see that the tokens are there how I expect them to > be – yet things aren’t matching. Or, more specifically, seem to stop > matching after the first search. I’m pretty sure that my Filter is > somehow the culprit, I just don’t know why a filter would cause the > results to change from one query to the next, since the filter seems > to be returning the same tokens each time (according to the analysis > page), and also works in all of my unit tests. > > > > Thanks, > > Josh Edwards > > > ________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates. The information > transmitted herewith is intended only for use by the individual or > entity to which it is addressed. If the reader of this message is not > the intended recipient, you are hereby notified that any review, > retransmission, dissemination, distribution, copying or other use of, > or taking of any action in reliance upon this information is strictly > prohibited. If you have received this communication in error, please > contact the sender and delete the material from your computer. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
