Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Benson Margulies
I've posted a self-contained test case to github of a mystery. git://github.com/bimargulies/lucene-4-update-case.git The code can be seen at https://github.com/bimargulies/lucene-4-update-case/blob/master/src/test/java/org/apache/lucene/BadFieldTokenizedFlagTest.java. I write a doc to an index,

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Robert Muir
I think the issue is that your analyzer is standardanalyzer, yet field text value is value-1 So standardanalyzer will tokenize this into two terms: value and 1 But later, you proceed to do TermQueries on value-1. This term won't exist... TermQuery etc that take Term don't analyze any text.

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Benson Margulies
On Tue, Mar 6, 2012 at 9:20 AM, Robert Muir rcm...@gmail.com wrote: I think the issue is that your analyzer is standardanalyzer, yet field text value is value-1 Robert, Why is this field analyzed at all? It's built with StringField.TYPE_STORED. I'll push another copy that shows that it works

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Benson Margulies
On Tue, Mar 6, 2012 at 9:23 AM, Benson Margulies bimargul...@gmail.com wrote: On Tue, Mar 6, 2012 at 9:20 AM, Robert Muir rcm...@gmail.com wrote: I think the issue is that your analyzer is standardanalyzer, yet field text value is value-1 Robert, Why is this field analyzed at all? It's

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Robert Muir
On Tue, Mar 6, 2012 at 9:23 AM, Benson Margulies bimargul...@gmail.com wrote: On Tue, Mar 6, 2012 at 9:20 AM, Robert Muir rcm...@gmail.com wrote: I think the issue is that your analyzer is standardanalyzer, yet field text value is value-1 Robert, Why is this field analyzed at all? It's

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Michael McCandless
Hmm something is up here... I'll dig. Seems like we are somehow analyzing StringField when we shouldn't... Mike McCandless http://blog.mikemccandless.com On Tue, Mar 6, 2012 at 9:33 AM, Robert Muir rcm...@gmail.com wrote: On Tue, Mar 6, 2012 at 9:23 AM, Benson Margulies bimargul...@gmail.com

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Benson Margulies
On Tue, Mar 6, 2012 at 9:33 AM, Robert Muir rcm...@gmail.com wrote: On Tue, Mar 6, 2012 at 9:23 AM, Benson Margulies bimargul...@gmail.com wrote: On Tue, Mar 6, 2012 at 9:20 AM, Robert Muir rcm...@gmail.com wrote: I think the issue is that your analyzer is standardanalyzer, yet field text

RE: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Uwe Schindler
String field is analyzed, but with KeywordTokenizer, so all should be fine. - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de -Original Message- From: Michael McCandless [mailto:luc...@mikemccandless.com] Sent: Tuesday, March 06,

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Benson Margulies
On Tue, Mar 6, 2012 at 9:47 AM, Uwe Schindler u...@thetaphi.de wrote: String field is analyzed, but with KeywordTokenizer, so all should be fine. I filed LUCENE-3854. - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de -Original

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Robert Muir
Thanks Benson: look like the problem revolves around indexing Document/Fields you get back from IR.document... this has always been 'lossy', but I think this is a real API trap. Please keep testing :) On Tue, Mar 6, 2012 at 9:58 AM, Benson Margulies bimargul...@gmail.com wrote: On Tue, Mar 6,

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Benson Margulies
On Tue, Mar 6, 2012 at 10:04 AM, Robert Muir rcm...@gmail.com wrote: Thanks Benson: look like the problem revolves around indexing Document/Fields you get back from IR.document... this has always been 'lossy', but I think this is a real API trap. Please keep testing :) Got a suggestion for

Re: Problem with updating a document or TermQuery with current trunk

2012-03-06 Thread Michael McCandless
On Tue, Mar 6, 2012 at 10:06 AM, Benson Margulies bimargul...@gmail.com wrote: On Tue, Mar 6, 2012 at 10:04 AM, Robert Muir rcm...@gmail.com wrote: Thanks Benson: look like the problem revolves around indexing Document/Fields you get back from IR.document... this has always been 'lossy', but I