Re: LUCENE-1282

2008-07-01 Thread Michael McCandless
As long as you are really sure it's not corrupt (by running CheckIndex) then you do not need to rebuild it. Mike dan at gmail wrote: Mike, I have one more question. I have another set of index data which were built with Lucene 2.3.2 and JRE 1.6.0_06 and this set index has not been co

Re: LUCENE-1282

2008-07-01 Thread dan at gmail
Mike, I have one more question. I have another set of index data which were built with Lucene 2.3.2 and JRE 1.6.0_06 and this set index has not been corrupted yet. If I downgrade to JRE 1.6.0_03, do I need to rebuild this index with JRE 1.6.0_03 or I can continue to use this index with 1.6.0_0

Re: LUCENE-1282

2008-07-01 Thread dan at gmail
Thanks Mike. Michael McCandless-2 wrote: > > > Yes, downgrading is really the only option now, unfortunately. > > That, and voting for this bug at Sun (note you can vote for the same > bug 3 times), which seems to be the root cause of the corruption: > > http://bugs.sun.com/bugdatabas

Re: LUCENE-1282

2008-07-01 Thread Michael McCandless
Yes, downgrading is really the only option now, unfortunately. That, and voting for this bug at Sun (note you can vote for the same bug 3 times), which seems to be the root cause of the corruption: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6707044 It's been open for 6 weeks now

LUCENE-1282

2008-07-01 Thread dan at gmail
Hello, I don't have a good understanding of what options for avoid this corrupted index problem described in LUCENE-1282. It seems to me that I either downgrade JRE from 1.6.0_06 to 1.6.0_03, or wait for an official release of Lucene 2.4.0 which includes a workaround for this JRE bug. http://is

Re: FileNotFoundException in ConcurrentMergeScheduler

2008-07-01 Thread Mark Miller
Mark Miller wrote: Paul J. Lucas wrote: Sorry for the radio silence. I changed my code around so that a single IndexReader and IndexSearcher are shared. Since doing that, I've not seen the problem. That being the case, I didn't pursue the issue. I still think there's a bug because the cod

Re: FileNotFoundException in ConcurrentMergeScheduler

2008-07-01 Thread Mark Miller
Paul J. Lucas wrote: Sorry for the radio silence. I changed my code around so that a single IndexReader and IndexSearcher are shared. Since doing that, I've not seen the problem. That being the case, I didn't pursue the issue. I still think there's a bug because the code I had previously,

Re: FileNotFoundException in ConcurrentMergeScheduler

2008-07-01 Thread Paul J. Lucas
That really can't be it. I have *one* client connecting to my server. And there isn't a descriptor leak. My mergeFactor is 10. - Paul On Jul 1, 2008, at 1:37 AM, Michael McCandless wrote: Hmmm then it sounds possible you were in fact running out of file descriptors. What was your merg

Re: Problems with reopening IndexReader while pushing documents to the index

2008-07-01 Thread Michael McCandless
OK I've opened: https://issues.apache.org/jira/browse/LUCENE-1323 I'll commit the fix (to trunk, to be included in 2.4) soon. Mike Michael McCandless wrote: Aha! OK now I see how that led to your exception. When you create a MultiReader, passing in the array of IndexReaders, MultiR

Re: Problems with reopening IndexReader while pushing documents to the index

2008-07-01 Thread Michael McCandless
Aha! OK now I see how that led to your exception. When you create a MultiReader, passing in the array of IndexReaders, MultiReader simply holds onto your array. It also computes & caches norms() the first time its called, based on the total # docs it sees in all the readers in that array

Re: IndexDeletionPolicy and optimized indices

2008-07-01 Thread Michael McCandless
You're right IndexCommit doesn't know that it represents an optimized index. Likewise, IndexCommit doesn't know other "semantic" things about the index, eg, you've just called expungeDeletes, or, you just finished adding batch X of documents to the index, etc. Also, realize that with au

IndexDeletionPolicy and optimized indices

2008-07-01 Thread Shalin Shekhar Mangar
Hi, I'm implementing a custom IndexDeletionPolicy. An IndexCommit object does not have any information whether it's index is optimized or not. How can a IndexDeletionPolicy know which IndexCommit instances corresponded to optimized indices? -- Regards, Shalin Shekhar Mangar. ---

Re: Problems with reopening IndexReader while pushing documents to the index

2008-07-01 Thread Sascha Fahl
Yes I am using IndexReader.reopen(). Here is my code doing this: public void refreshIndeces() throws CorruptIndexException, IOException { if ((System.currentTimeMillis() - this.lastRefresh) > this.REFRESH_PERIOD) { this.lastRefresh = System.currentTimeMillis();

Re: Problems with reopening IndexReader while pushing documents to the index

2008-07-01 Thread Michael McCandless
That's interesting. So you are using IndexReader.reopen() to get a new reader? Are you closing the previous reader? The exception goes away if you create a new IndexSearcher on the reopened IndexReader? I don't yet see how that could explain the exception, though. If you reopen() the

Re: Sorting case-insensitively

2008-07-01 Thread Erik Hatcher
On Jun 30, 2008, at 11:08 PM, Paul J. Lucas wrote: On Jun 30, 2008, at 7:00 PM, Erik Hatcher wrote: On Jun 30, 2008, at 8:55 PM, Paul J. Lucas wrote: If I have a SortField with a type of STRING, is there any way to sort in a case-insensitive manner? Only if you unify the case (lower case

Re: Nested Proximity searches

2008-07-01 Thread John Byrne
The QueryParser syntax does not support what you're trying to do. However, it is possible to do with the API. If you can construct your Query programatically, you can use the SpanQuery API to do what you need. Proximity searching is achieved using the SpanNearQuery; the constructor of this obj

Re: Problems with reopening IndexReader while pushing documents to the index

2008-07-01 Thread Michael McCandless
By "does not help" do you mean CheckIndex never detects this corruption, yet you then hit that exception when searching? By "reopening fails" what do you mean? I thought reopen works fine, but then it's only the search that fails? Mike Sascha Fahl wrote: Checking the index after adding

Re: Problems with reopening IndexReader while pushing documents to the index

2008-07-01 Thread Michael McCandless
OK thanks for the answers below. One thing to realize is, with this specific corruption, you will only hit the exception if the one term that has the corruption is queried on. Ie, only a certain term in a query will hit the corruption. That's great news that it's easily reproduced -- can

Re: FileNotFoundException in ConcurrentMergeScheduler

2008-07-01 Thread Michael McCandless
Hmmm then it sounds possible you were in fact running out of file descriptors. What was your mergeFactor set to? Mike Paul J. Lucas wrote: Sorry for the radio silence. I changed my code around so that a single IndexReader and IndexSearcher are shared. Since doing that, I've not seen