Looking at the timestamps on the tlog files they seem to have all been created around the same time (04:55).. starting around this time I start seeing the exception below (there were 1628).. in fact its getting tons of these (200k+) but most of the time inside regular commits...
2013-15-05 04:55:06.634 ERROR UpdateLog [recoveryExecutor-6-thread-7922] - java.lang.ArrayIndexOutOfBoundsException: 2603 at org.apache.lucene.codecs.lucene40.BitVector.get(BitVector.java:146) at org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:492) at org.apache.lucene.index.BufferedDeletesStream.applyTermDeletes(BufferedDeletesStream.java:407) at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:273) at org.apache.lucene.index.IndexWriter.applyAllDeletes(IndexWriter.java:2973) at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:2964) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2704) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2839) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536) at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1339) at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1163) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) On Thu, May 16, 2013 at 9:35 AM, Yonik Seeley <yo...@lucidworks.com> wrote: > See https://issues.apache.org/jira/browse/SOLR-3939 > > Do you see these log messages from this in your logs? > log.info("I may be the new leader - try and sync"); > > How reproducible is this bug for you? It would be great to know if > the patch in the issue fixes things. > > -Yonik > http://lucidworks.com > > > On Wed, May 15, 2013 at 6:04 PM, Steven Bower <sbo...@alcyon.net> wrote: > > They are visible to ls... > > > > > > On Wed, May 15, 2013 at 5:49 PM, Yonik Seeley <yo...@lucidworks.com> > wrote: > > > >> On Wed, May 15, 2013 at 5:20 PM, Steven Bower <sbo...@alcyon.net> > wrote: > >> > when the TransactionLog objects are dereferenced > >> > their RandomAccessFile object is not closed.. > >> > >> Have the files been deleted (unlinked from the directory), or are they > >> still visible via "ls"? > >> > >> -Yonik > >> http://lucidworks.com > >> >