Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-29 Thread Chris Hostetter
: I get these NullPointException records every once and a while, always : from SolrCore and SolrDispatchFilter. Don't get a stack trace, and no : nearby errors seem to clarify what might have happened. that's really strange ... i'm not sure what's causing those, or why you aren't seeing a stack

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-28 Thread Chris Harris
Well I've done two or three more runs with Sun java build 1.6.0_10-rc-b28, with and without the infoStream change, and I can't reproduce the problem. So maybe everything was indeed the fault of the release version of Sun hotspot. Not too confident about that. Maybe instead I let the disk get too fu

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-21 Thread Michael McCandless
OK indeed that revision of Lucene is before the workaround for that nasty JRE bug was committed. Can you test one of those JRE versions (known not to have this particular JRE bug) and see if you can get the original "massive deletion" problem to happen. I guess first try it without the

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-21 Thread Chris Harris
I'll see about using a newer/older JVM. In the meantime, according to the Solr admin page, which seems to get its info like so LucenePackage.class.getPackage().getImplementationVersion() what I've been testing is Lucene r652650. The Solr version is r654965, now modified of course to do some

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-21 Thread Michael McCandless
Urgh, I was hoping we could repro the "massive deletion" with infoStream turned on. Uh-oh: that "off by 1" corruption is very likely due to the Sun JRE bug described here: https://issues.apache.org/jira/browse/LUCENE-1282 Can you downgrade to 1.6.0_03, or, upgrade to the latest beta

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-21 Thread Chris Harris
Well shoot, upon further examination it appears that this time around there weren't actually any segment deletion problems. The "only" problem was a "doc counts differ" error. Interestingly, the count is only off by one. >From the CheckIndex tool: Opening index @ /ssd/solr-/solr/exhibit

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-20 Thread Michael McCandless
Did the same FileNotFoundException / massive deletion of files occur? Actually if you could zip them all up and post them, I'll dig through them to see if they give any clues... Mike Chris Harris wrote: Ok, I did what you suggested, giving each SolrIndexWriter its own "infoStream" log fil

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-20 Thread Chris Harris
Ok, I did what you suggested, giving each SolrIndexWriter its own "infoStream" log file, created in the init() method. The thing is, I now have like 3400 infostream log files, I guess reflecting how solr created like 3400 SolrIndexWriters over the course of the run. (Hopefully this is plausible.) C

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-18 Thread Yonik Seeley
On Mon, Aug 18, 2008 at 6:05 AM, Michael McCandless <[EMAIL PROTECTED]> wrote: > The output from CheckIndex shows quite a few missing files! Is there any > possibility that two instances of Solr were somehow sharing the same index > directory? To eliminate that possibility, the lock factory shoul

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-18 Thread Yonik Seeley
On Mon, Aug 18, 2008 at 1:12 PM, Michael McCandless <[EMAIL PROTECTED]> wrote: > > Alas, I think this won't actually turn on IndexWriter's infoStream. > > I think you may need to modify the SolrIndexWriter.java sources, in the init > method, to add a call to setInfoStream(...). > > Can any Solr dev

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-18 Thread Fuad Efendi
Lucene v.2.1 has a bug with autocommit...

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-18 Thread Michael McCandless
Alas, I think this won't actually turn on IndexWriter's infoStream. I think you may need to modify the SolrIndexWriter.java sources, in the init method, to add a call to setInfoStream(...). Can any Solr developers confirm this? Mike Chris Harris wrote: I'm assuming that one way to do thi

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-18 Thread Chris Harris
I'm assuming that one way to do this would be to set the logging level to "FINEST" in the "logging" page in the solr admin tool, and then to make sure my logging.properties file is also set to record the FINEST logging level. Let me know if that won't enable to sort of debugging info you are talkin

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-18 Thread Michael McCandless
The output from CheckIndex shows quite a few missing files! Is there any possibility that two instances of Solr were somehow sharing the same index directory? It looks like you are using the 2.3 version of the Lucene jar (not the trunk version). Which version of Solr are you using? Si

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-16 Thread Walter Underwood
I hate to blame the JDK, but we tried 1.6 for our production webapp and it was crashing too often. Unless you need 1.6, you might try 1.5. --wunder On 8/16/08 1:54 PM, "Chris Harris" <[EMAIL PROTECTED]> wrote: > On Sat, Aug 16, 2008 at 4:33 AM, Grant Ingersoll <[EMAIL PROTECTED]> wrote: >> What v

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-16 Thread Otis Gospodnetic
- Original Message > From: Chris Harris <[EMAIL PROTECTED]> > To: solr-user@lucene.apache.org > Sent: Friday, August 15, 2008 1:35:20 PM > Subject: "Auto commit error" and java.io.FileNotFoundException > > I have an index (different from the ones mentioned yeste

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-16 Thread Otis Gospodnetic
TED]> > To: solr-user@lucene.apache.org > Sent: Friday, August 15, 2008 4:30:07 PM > Subject: Re: "Auto commit error" and java.io.FileNotFoundException > > I've done some more sniffing on the Lucene list, and noticed that Otis > made the following comment about a FileNotFoun

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-16 Thread Chris Harris
On Sat, Aug 16, 2008 at 4:33 AM, Grant Ingersoll <[EMAIL PROTECTED]> wrote: > Can you try Lucene's CheckIndex tool on it and report what it says? > > On Aug 15, 2008, at 1:35 PM, Chris Harris wrote: > >> I have an index (different from the ones mentioned yesterday) that was >> working fine with 3M

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-16 Thread Chris Harris
On Sat, Aug 16, 2008 at 4:33 AM, Grant Ingersoll <[EMAIL PROTECTED]> wrote: > What version of Java do you have on Linux? The Java version on *Linux* (where I'm seeing the trouble): java version "1.6.0" OpenJDK Runtime Environment (build 1.6.0-b09) OpenJDK 64-Bit Server VM (build 1.6.0

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-16 Thread Grant Ingersoll
What version of Java do you have on Linux? Also, is this easily reproducible? How many threads are you adding documents with? What is your Auto Commit setting? Can you try Lucene's CheckIndex tool on it and report what it says? On Aug 15, 2008, at 1:35 PM, Chris Harris wrote: I have an i

Re: "Auto commit error" and java.io.FileNotFoundException

2008-08-15 Thread Chris Harris
I've done some more sniffing on the Lucene list, and noticed that Otis made the following comment about a FileNotFoundException problem in late 2005: Are you using Windows and a compound index format (look at your index dir - does it have .cfs file(s))? This may be a bad combination,

"Auto commit error" and java.io.FileNotFoundException

2008-08-15 Thread Chris Harris
I have an index (different from the ones mentioned yesterday) that was working fine with 3M docs or so, but when I added a bunch more docs, bringing it closer to 4M docs, the index seemed to get corrupted. In particular, now when I start Solr up, or when when my indexing process tries add a documen