thanks this was really helpful to understand whats going on.. i got these for 2 of my indexes -
WARNING: 29 broken segments (containing 385395 documents) detected WARNING: would write new segments file, and 385395 documents would be lost, if -fix were specified WARNING: 29 broken segments (containing 385395 documents) detected WARNING: would write new segments file, and 385395 documents would be lost, if -fix were specified so my only option is to run with -fix and live with the data loss right? no other option right? will running fsck help? sometimes my ls or less commands also hang for a good 10 sec.. this somehow indicates that there is some corruption. thanks On Mon, Jul 9, 2012 at 6:27 AM, Erick Erickson <erickerick...@gmail.com>wrote: > no, you can't delete those files, and you can't regenerate just those > files, > all the various segment files are necessary and intertwined... > > Consider using the CheckIndex facility, see: > http://solr.pl/en/2011/01/17/checkindex-for-the-rescue/ > > note, the CheckIndex class is contained in the lucene core jar.... > > You can run it with the -fix option to repair (at, perhaps, the expense > of loss of some documents) if you choose, but running it without > that option first is probably a good idea.. > > Best > Erick > > On Mon, Jul 9, 2012 at 7:43 AM, T Vinod Gupta <tvi...@readypulse.com> > wrote: > > this is on local file system on amazon ec2 host. the file system was fine > > until a week ago when the outage happened and there were probably some > > system glitches. i have seen this issue since then.. sometimes regular > > commands like less or ls hang for many seconds even though there is no > > cpu/memory pressure on the machine. > > > > in my case, there are only 2 unique entries for which i see this error. > one > > for a .fdt file and one for a .tis file. is it possible to regenerate > those > > files somehow? if i delete those 2 files, will the entire index get > > corrupted? im ok to live with some data loss if it makes it more stable > and > > performant. > > > > thanks > > > > On Mon, Jul 9, 2012 at 2:28 AM, Ian Lea <ian....@gmail.com> wrote: > > > >> Is this on a local or remote file system? Is the file system itself > >> OK? Is something else messing with your lucene index at the same > >> time? > >> > >> > >> -- > >> Ian. > >> > >> > >> On Sun, Jul 8, 2012 at 8:58 PM, T Vinod Gupta <tvi...@readypulse.com> > >> wrote: > >> > Hi, > >> > My log files are showing the below exceptions almost at twice a minute > >> > frequency. what is causing it and how can i fix it? I am not using > lucene > >> > directly but instead using elasticsearch (0.18.7 version). but since > the > >> > stack trace is all lucene, i am sending it to this mailing list. > >> > > >> > also, my queries are taking a long time to execute (sometimes take a > >> > minute). could this be contributing to it somehow? > >> > > >> > [2012-07-08 19:44:19,887][WARN ][index.merge.scheduler ] [<my node > >> > name>] [twitter][4] failed to merge > >> > java.io.IOException: Input/output error: > >> > NIOFSIndexInput(path="/media/ephemeral0 > >> > /ES_data/elasticsearch/nodes/0/indices/twitter/4/index/_2h29k.tis") > >> > at > >> > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(N > >> > IOFSDirectory.java:180) > >> > at > >> > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput. > >> > java:229) > >> > at > >> > > >> > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39) > >> > at > org.apache.lucene.store.DataInput.readVInt(DataInput.java:105) > >> > at > >> > > >> > org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:197) > >> > at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:86) > >> > at > >> > org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:133) > >> > at > >> > > org.apache.lucene.index.SegmentMergeInfo.next(SegmentMergeInfo.java:72) > >> > at > >> > > >> > org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:546) > >> > at > >> > > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:473) > >> > at > >> > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:109) > >> > at > >> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4295) > >> > at > >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3940) > >> > at > >> > > >> > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) > >> > at > >> > > >> > org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:88) > >> > at > >> > > >> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) > >> > Caused by: java.io.IOException: Input/output error > >> > at sun.nio.ch.FileDispatcher.pread0(Native Method) > >> > at sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:49) > >> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:248) > >> > at sun.nio.ch.IOUtil.read(IOUtil.java:224) > >> > at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:663) > >> > at > >> > > >> > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:162) > >> > ... 15 more > >> > > >> > thanks > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
