RE: Confusion with Analyzer.tokenStream() re-use in 4.1

2013-02-27 Thread Uwe Schindler
e- > From: Konstantyn Smirnov [mailto:inject...@yahoo.com] > Sent: Thursday, February 28, 2013 12:18 AM > To: java-user@lucene.apache.org > Subject: RE: Confusion with Analyzer.tokenStream() re-use in 4.1 > > Thanks for the answer Uwe! > > so the behavior has changed since t

RE: Confusion with Analyzer.tokenStream() re-use in 4.1

2013-02-27 Thread Konstantyn Smirnov
Thanks for the answer Uwe! so the behavior has changed since the 3.6, hasn't it? Now I need to instantiate the analyzer each time I feed the field with the tokenStream, or it happens behind the scenes if I use new (String name, String value, Field.Store store). Another question then... Now I tr

RE: Confusion with Analyzer.tokenStream() re-use in 4.1

2013-02-27 Thread Uwe Schindler
age- > From: Uwe Schindler [mailto:u...@thetaphi.de] > Sent: Wednesday, February 27, 2013 8:03 PM > To: 'java-user@lucene.apache.org' > Subject: RE: Confusion with Analyzer.tokenStream() re-use in 4.1 > > The problem here is that the tokenstream is instantiated in the same

RE: Confusion with Analyzer.tokenStream() re-use in 4.1

2013-02-27 Thread Uwe Schindler
The problem here is that the tokenstream is instantiated in the same thread from 2 different code paths and consumed later. If you add fields, the indexer will fetch a new reused TokenStream one after each other and consume them directly after getting. It will not interleave this. In your case,