RE: Lucene 3.4 : shift bug in possibly invalid use of NumericTokenStream

2011-12-16 Thread Uwe Schindler
Hi, Thanks, this *may* cause the exception, but it is impossible that the exception stack trace you are posting occurs in Lucene's code with a default precision step on a numeric field, as you use here. I assume it's a 32bit integer (NumericField.setIntValue or setFloatValue)? Please provide us y

Re: Lucene 3.4 : shift bug in possibly invalid use of NumericTokenStream

2011-12-16 Thread Thushara Wijeratna
Yes, there is one. This is how the field is being created: new NumericField("timestamp", Field.Store.NO, true); Thus, the field is not stored, but indexed. thx, thushara On Fri, Dec 16, 2011 at 3:28 PM, Uwe Schindler wrote: > Do you have NumericFields? If yes, how are they configured? > > -

RE: Lucene 3.4 : shift bug in possibly invalid use of NumericTokenStream

2011-12-16 Thread Uwe Schindler
Do you have NumericFields? If yes, how are they configured? - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Thushara Wijeratna [mailto:thu...@gmail.com] > Sent: Saturday, December 17, 2011 12:25 AM > To:

Lucene 3.4 : shift bug in possibly invalid use of NumericTokenStream

2011-12-16 Thread Thushara Wijeratna
I got this exception while indexing with Lucene 3.4: Exception in thread "Thread-0" java.lang.IllegalArgumentException: Illegal shift value, must be 0..31 at org.apache.lucene.util.NumericUtils.intToPrefixCoded(NumericUtils.java:157) at org.apache.lucene.analysis.NumericTokenStream.incrementToke

Re: Why is the old value still in the index

2011-12-16 Thread Paul Taylor
On 16/12/2011 22:51, Rene Hackl-Sommer wrote: Maybe you could just use MatchAllDocsQuery? http://lucene.apache.org/java/3_5_0/api/core/org/apache/lucene/search/MatchAllDocsQuery.html Rene Ah thanks Rene, thats what I wanted Paul

Re: Why is the old value still in the index

2011-12-16 Thread Rene Hackl-Sommer
Maybe you could just use MatchAllDocsQuery? http://lucene.apache.org/java/3_5_0/api/core/org/apache/lucene/search/MatchAllDocsQuery.html Rene Am 16.12.2011 22:58, schrieb Paul Taylor: On 16/12/2011 20:54, Paul Taylor wrote: Thanks I think you might have it, but tell me if forceMergeDelete

Re: Why is the old value still in the index

2011-12-16 Thread Paul Taylor
On 16/12/2011 20:54, Paul Taylor wrote: On 16/12/2011 17:43, Uwe Schindler wrote: Hi, I'm adding documents to an index, at a later date I modify a document and update the index, close the writer and open a new IndexReader. My indexreader iterates over terms for that field and docFreq() returns

Re: Why is the old value still in the index

2011-12-16 Thread Paul Taylor
On 16/12/2011 17:43, Uwe Schindler wrote: Hi, I'm adding documents to an index, at a later date I modify a document and update the index, close the writer and open a new IndexReader. My indexreader iterates over terms for that field and docFreq() returns one as I would expect, however the iter

Table Defn and/or ER Diagram of Segment files

2011-12-16 Thread Dr. Ray Hoare
Is there an entity-relationship of the segment files and/or Berkeley DB tables (with table definitions)? I'm trying understand the segment files of Lucene and know that a Berkeley DB can be used to store the directory but can't locate any ER diagram or table definitions for the DB. Thanks Ray --

Re: Using Lucene to match document sets to each other

2011-12-16 Thread Erick Erickson
Have you looked at Lucene's "MoreLikeThis"? I confess I haven't worked with this enough to recommend *how* to use it, but it seems like it's in the general area you're talking about. http://lucene.apache.org/java/3_5_0/api/contrib-queries/org/apache/lucene/search/similar/MoreLikeThis.html Best Er

RE: Why is the old value still in the index

2011-12-16 Thread Austin, Carl
The .docFreq() call returns the number of documents that the current term in the enum is in, not all terms in the term enum. Also be aware of, from the lucene wiki : "Once a document is deleted it will not appear in TermDocs nor TermPositions enumerations, nor any search results. Attempts to load

Re: Using Lucene to match document sets to each other

2011-12-16 Thread Josh Stone
Thanks for the response Donna. That would make more sense, but the items I'm pulling in from the web contain large bodies of text (descriptions) whereas the products in my catalog consist of shorter fields such as product name, manufacturer, product code, etc. So using the smaller fields from my ca

RE: Why is the old value still in the index

2011-12-16 Thread Uwe Schindler
Hi, > I'm adding documents to an index, at a later date I modify a document and > update the index, close the writer and open a new IndexReader. My > indexreader iterates over terms for that field and docFreq() returns one as I > would expect, however the iterator returns both the old value of the

Re: Why is the old value still in the index

2011-12-16 Thread Paul Taylor
On 16/12/2011 17:10, Ian Lea wrote: Shouldn't iw.updateDocument(new Term(FIELD1,"term1"),document); be iw.updateDocument(new Term(FIELD1,"test"),document); if you want to replace the first doc? Hmm, you are right if I change it I then get TermDocsFreq1 test TermDocsFreq1 test2 (but doesn

Re: Why is the old value still in the index

2011-12-16 Thread Ian Lea
Shouldn't iw.updateDocument(new Term(FIELD1,"term1"),document); be iw.updateDocument(new Term(FIELD1,"test"),document); if you want to replace the first doc? -- Ian. On Fri, Dec 16, 2011 at 4:54 PM, Paul Taylor wrote: > I'm adding documents to an index, at a later date I modify a document a

Why is the old value still in the index

2011-12-16 Thread Paul Taylor
I'm adding documents to an index, at a later date I modify a document and update the index, close the writer and open a new IndexReader. My indexreader iterates over terms for that field and docFreq() returns one as I would expect, however the iterator returns both the old value of the documen

Re: Using Lucene to match document sets to each other

2011-12-16 Thread Donna L Gresh
Maybe I'm misunderstanding what you're trying to do, but why not do it the other way around; that is, index the items in your catalog, and use the items on the web as the query into the catalog. I have an analogous process (though completely different application area) and I index the stuff th

Have Anyone Needed a Method, getAnalyzer(String fieldName) of PerFieldAnalyzerWrapper Class, So Far?

2011-12-16 Thread 안재희 [Jade]
Hello~ Have anyone needed a method, getAnalyzer(String fieldName) of PerFieldAnalyzerWrapper class, so far? A method getDefaultAnalyzer() as well. It is useful in my project. The needs would be unusual, though. Implementing these methods is very simple. Does Apache Lucene have any plan to add