Getting exception while initializing FSDirectory

2016-06-15 Thread Mukul Ranjan
Hi, I'm getting below exception while initializing FSDirectory- Caused by: java.lang.IllegalAccessError: tried to access method org.apache.lucene.store.MMapDirectory.unmapHackImpl()Ljava/lang/Object; from class org.apache.lucene.store.MMapDirectory$$dtt &

Re: Lucene 5.2.1: FSDirectory, is it possible to open existing output for append?

2015-09-10 Thread Vlad K
Thanks Uwe, this will help us to clarify details and make a decision. If you write your own data files into a given index directory, it is very > likely that you may corrupt your index. In later Lucene versions (5.x) we > are very strict with not allowing files in the index directory, which were >

RE: Lucene 5.2.1: FSDirectory, is it possible to open existing output for append?

2015-09-10 Thread Uwe Schindler
k. > > > > Uwe > > > > - > > Uwe Schindler > > H.-H.-Meier-Allee 63, D-28213 Bremen > > http://www.thetaphi.de > > eMail: u...@thetaphi.de > > > > > -Original Message- > > > From: Vlad K [mailto:kuzmi...@gmail.com] > >

Re: Lucene 5.2.1: FSDirectory, is it possible to open existing output for append?

2015-09-09 Thread Vlad K
> Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > -Original Message- > > From: Vlad K [mailto:kuzmi...@gmail.com] > > Sent: Wednesday, September 02, 2015 8:07 AM > > To: java-us

RE: Lucene 5.2.1: FSDirectory, is it possible to open existing output for append?

2015-09-02 Thread Uwe Schindler
rom: Vlad K [mailto:kuzmi...@gmail.com] > Sent: Wednesday, September 02, 2015 8:07 AM > To: java-user@lucene.apache.org > Subject: Lucene 5.2.1: FSDirectory, is it possible to open existing output for > append? > > FSDirectory createOutput re-creates file because it opens stream w

Lucene 5.2.1: FSDirectory, is it possible to open existing output for append?

2015-09-01 Thread Vlad K
FSDirectory createOutput re-creates file because it opens stream with TRUNCATE_EXISTING. What is the way to open existing file and append data? I used it at Lucene 4.1 to create store with raw messages. I could use Files.newOutputStream directly to do that but I just want to understand what is the

Re: MMapDirectory or FSDirectory

2015-02-05 Thread sreedevi s
t; H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > -Original Message- > > From: sreedevi s [mailto:sreedevi.payik...@gmail.com] > > Sent: Thursday, February 05, 2015 10:13 AM > > To: java-user@lucene.apache.org >

RE: MMapDirectory or FSDirectory

2015-02-05 Thread Uwe Schindler
evi.payik...@gmail.com] > Sent: Thursday, February 05, 2015 10:13 AM > To: java-user@lucene.apache.org > Subject: MMapDirectory or FSDirectory > > Hi, > I am doing some performance analysis with lucene. I have 1 million resources > with 1000 attributes. > According to how I index

MMapDirectory or FSDirectory

2015-02-05 Thread sreedevi s
Hi, I am doing some performance analysis with lucene. I have 1 million resources with 1000 attributes. According to how I index, I will have 1 million documents with 1000 fields. For me the total data was about 100 GB and while using FSDirectory to store my indices, index size was almost 6 GB. I

Re: RAMDirectory with FSDirectory merging Versus large mergeFactor and RAMBufferSizeMB

2012-06-05 Thread Lance Norskog
ng RAMDirectory as a buffer (perhaps even several in parallel) and > later merging it using addIndexes to FSDirectory. > > So my question is the following: > In case I have only 1 thread with RAMDirectory - is that pretty much the same > as method 1? Since it's in memory

RAMDirectory with FSDirectory merging Versus large mergeFactor and RAMBufferSizeMB

2012-06-04 Thread Maxim Terletsky
Hi guys, There are two approaches I see in Lucene In Action about speeding up the indexing process. 1) Simply increase the mergeFactor and RAMBufferSizeMB. 2) Using RAMDirectory as a buffer (perhaps even several in parallel) and later merging it using addIndexes to FSDirectory. So my question

Re: Can I use multiple writers of different applications on a same FSDirectory?

2012-02-14 Thread Cheng
thanks On Wed, Feb 15, 2012 at 1:14 AM, Mihai Caraman wrote: > you can use a fsdirectory for each writer and then, search on all of them > at once. This is the recomended way if you have different apps. > > În data de 14 februarie 2012, 19:06, Ian Lea a scris: > > > You can

Re: Can I use multiple writers of different applications on a same FSDirectory?

2012-02-14 Thread Mihai Caraman
you can use a fsdirectory for each writer and then, search on all of them at once. This is the recomended way if you have different apps. În data de 14 februarie 2012, 19:06, Ian Lea a scris: > You can only have one writer against one index at a time. Lucene's > locking will preve

Re: Can I use multiple writers of different applications on a same FSDirectory?

2012-02-14 Thread Ian Lea
You can only have one writer against one index at a time. Lucene's locking will prevent anything else. -- Ian. On Tue, Feb 14, 2012 at 4:49 PM, Cheng wrote: > Hi, > > I need to manage multiple applications, each having its own writer yet on a > same FSdirectory. How to make

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Cheng
ike RAMDirectory for > >> small flushed segments. > >> > >> Mike McCandless > >> > >> http://blog.mikemccandless.com > >> > >> On Mon, Feb 6, 2012 at 10:45 AM, Cheng wrote: > >> > Uwe, when I meant speed is slow, I didn'

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Michael McCandless
>> >> Also, you can use NRTCachingDirectory which acts like RAMDirectory for >> small flushed segments. >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> On Mon, Feb 6, 2012 at 10:45 AM, Cheng wrote: >> > Uwe, when I

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Cheng
all flushed segments. > > Mike McCandless > > http://blog.mikemccandless.com > > On Mon, Feb 6, 2012 at 10:45 AM, Cheng wrote: > > Uwe, when I meant speed is slow, I didn't refer to instant visibility of > > changes, but that the changes may be synchronized with FSDirecto

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Michael McCandless
MDirectory for small flushed segments. Mike McCandless http://blog.mikemccandless.com On Mon, Feb 6, 2012 at 10:45 AM, Cheng wrote: > Uwe, when I meant speed is slow, I didn't refer to instant visibility of > changes, but that the changes may be synchronized with FSDirectory when

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Cheng
; > To: java-user@lucene.apache.org > > Subject: Re: Configure writer to write to FSDirectory? > > > > Uwe, when I meant speed is slow, I didn't refer to instant visibility of > changes, > > but that the changes may be synchronized with FSDirectory when I us

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Cheng
My original question is if there exists a way to configure writer when to writer to FSDirectory. I think there may be something in the IndexWriterConfig that can helps. On Mon, Feb 6, 2012 at 11:50 PM, Ian Lea wrote: > Well, yes. What would you expect? From the javadocs

RE: Configure writer to write to FSDirectory?

2012-02-06 Thread Uwe Schindler
...@thetaphi.de > -Original Message- > From: Cheng [mailto:zhoucheng2...@gmail.com] > Sent: Monday, February 06, 2012 4:45 PM > To: java-user@lucene.apache.org > Subject: Re: Configure writer to write to FSDirectory? > > Uwe, when I meant speed is slow, I didn't refe

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Ian Lea
t; changes, but that the changes may be synchronized with FSDirectory when I > use writer.commit(). > > When I use RAMDirectory, the writer.commit() seems much faster than using > NRTManager built upon FSDirectory. So, I am guessing the difference is the > index synchronization. > &

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Cheng
h the FSDirectory. The slower speed of using NRTManager built upon FSDirectory may be caused by the frequent updates or modification of indexes. That is my guess. On Mon, Feb 6, 2012 at 11:41 PM, Ian Lea wrote: > What exactly do you mean by the "speed is slower"? Time taken to > up

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Cheng
Uwe, when I meant speed is slow, I didn't refer to instant visibility of changes, but that the changes may be synchronized with FSDirectory when I use writer.commit(). When I use RAMDirectory, the writer.commit() seems much faster than using NRTManager built upon FSDirectory. So, I am gue

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Ian Lea
>> >>> >> All thread safe so you don't have to worry about any complications >>> >> there.  And I bet it'll be blindingly fast. >>> >> >>> >> Don't forget to close() things down at the end. >>> >> >>

RE: Configure writer to write to FSDirectory?

2012-02-06 Thread Uwe Schindler
://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Cheng [mailto:zhoucheng2...@gmail.com] > Sent: Monday, February 06, 2012 4:27 PM > To: java-user@lucene.apache.org > Subject: Re: Configure writer to write to FSDirectory? > > Ian, > > I encounte

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Cheng
rry about any complications >> >> there. And I bet it'll be blindingly fast. >> >> >> >> Don't forget to close() things down at the end. >> >> >> >> >> >> -- >> >> Ian. >> >> >> &g

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Cheng
t;> > >> On Mon, Feb 6, 2012 at 12:15 AM, Cheng wrote: > >> > I was trying to, but don't know how to even I read some of your blogs. > >> > > >> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless < > >> > luc...@mikemcc

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Ian Lea
t;> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless < >> > luc...@mikemccandless.com> wrote: >> > >> >> Are you using near-real-time readers? >> >> >> >> (IndexReader.open(IndexWriter)) >> >> >> >> Mike McCand

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Cheng
p://blog.mikemccandless.com > >> > >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng wrote: > >> > Hi Uwe, > >> > > >> > My challenge is that I need to update/modify the indexes frequently > while > >> > providing the sear

Re: Configure writer to write to FSDirectory?

2012-02-06 Thread Ian Lea
>> Mike McCandless >> >> http://blog.mikemccandless.com >> >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng wrote: >> > Hi Uwe, >> > >> > My challenge is that I need to update/modify the indexes frequently while >> > providing the searc

Re: Configure writer to write to FSDirectory?

2012-02-05 Thread Cheng
//blog.mikemccandless.com > > On Sun, Feb 5, 2012 at 9:03 AM, Cheng wrote: > > Hi Uwe, > > > > My challenge is that I need to update/modify the indexes frequently while > > providing the search capability. I was trying to use FSDirectory, but > found > >

Re: Configure writer to write to FSDirectory?

2012-02-05 Thread Michael McCandless
ility. I was trying to use FSDirectory, but found > out that the reading and writing from/to FSDirectory is unbearably slow. So > I now am trying the RAMDirectory, which is fast. > > I don't know of  MMapDirectory, and wonder if it is as fast as RAMDirectory. > > > On Sun,

Re: Configure writer to write to FSDirectory?

2012-02-05 Thread Cheng
Hi Uwe, My challenge is that I need to update/modify the indexes frequently while providing the search capability. I was trying to use FSDirectory, but found out that the reading and writing from/to FSDirectory is unbearably slow. So I now am trying the RAMDirectory, which is fast. I don't

RE: Configure writer to write to FSDirectory?

2012-02-05 Thread Uwe Schindler
.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Cheng [mailto:zhoucheng2...@gmail.com] > Sent: Sunday, February 05, 2012 7:56 AM > To: java-user@lucene.apache.org > Subject: Configure writer to write to FSD

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two

2012-01-12 Thread Cheng
Seem not related to the OpenMode.CREATE > option. > > > > > > -- Original -- > > From: "Ian Lea"; > > Date: Wed, Jan 11, 2012 05:20 PM > > To: "java-user"; > > > > Subject: Re: Build RAMDirectory on FSDirectory, and then synchr

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two

2012-01-12 Thread Sanne Grinovero
eem not related to the OpenMode.CREATE option. > > > -- Original -- > From: "Ian Lea"; > Date: Wed, Jan 11, 2012 05:20 PM > To: "java-user"; > > Subject: Re: Build RAMDirectory on FSDirectory, and then synchronzing the

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two

2012-01-11 Thread dyzc
Subject: Re: Build RAMDirectory on FSDirectory, and then synchronzing the two > I tried IndexWriterConfig.OpenMode CREATE, and the size is doubled. Prove it. -- Ian. > The only way that is effective is the writer's deleteAll() methods. > > On Mon, Jan 9, 2012 at 5:23 AM, Ian Lea

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two

2012-01-11 Thread Ian Lea
I'd better provide a snapshot of my code for people to understand my >> issues: >> > >> > >> > File file=new File("c:/index_files"); >> > FSDirectory fsDir=new FSDirectory(file); >> > RAMDirectory ramDir=new RAMDirectory(fsDir, new &

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two

2012-01-10 Thread Cheng
t; index or overwrite the original. IndexWriterConfig.OpenMode CREATE. > > > -- > Ian. > > > On Mon, Jan 9, 2012 at 4:29 AM, dyzc <1393975...@qq.com> wrote: > > I'd better provide a snapshot of my code for people to understand my > issues: > > > &g

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two

2012-01-09 Thread Ian Lea
> > File file=new File("c:/index_files"); > FSDirectory fsDir=new FSDirectory(file); > RAMDirectory ramDir=new RAMDirectory(fsDir, new > IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer()); > > > IndexWriter iw = new IndexWriter(ramDir, iwc); >

Re:Build RAMDirectory on FSDirectory, and then synchronzing the two

2012-01-08 Thread dyzc
I'd better provide a snapshot of my code for people to understand my issues: File file=new File("c:/index_files"); FSDirectory fsDir=new FSDirectory(file); RAMDirectory ramDir=new RAMDirectory(fsDir, new IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer()); Index

Build RAMDirectory on FSDirectory, and then synchronzing the two

2012-01-08 Thread Cheng
Hi, I new a RAMDirectory based upon a FSDirectory. After a few modifications, I would like to synchronize the two. Some on the mailing list provided a solution that uses addIndex() function. However, the FSDirectory simply combines with the RAMDirectory, and the size doubled. How can I do a

Re: i'm having some trouble with class FSDirectory

2011-08-24 Thread Simon Willnauer
just use the static factory method FSDirectory#open(File) to obtain a FSDirectory instance simon On Wed, Aug 24, 2011 at 3:55 PM, Mostafa Hadian wrote: > hello. > there is this piece of code in the book "lucene in action" : > Directory dir = new FSDirectory(new File(inde

RE: i'm having some trouble with class FSDirectory

2011-08-24 Thread Sendros, Jason
Hi Mostafa, Try looking through the API for help with these types of questions: http://lucene.apache.org/java/3_3_0/api/all/org/apache/lucene/store/FSDi rectory.html You can use a number of FSDirectory subclasses depending on your circumstances. Hope this helps! Jason -Original Message

i'm having some trouble with class FSDirectory

2011-08-24 Thread Mostafa Hadian
hello. there is this piece of code in the book "lucene in action" : Directory dir = new FSDirectory(new File(indexDir), null); but class FSDirectory is an abstract class and cannot be instantiated like this. thank you very much for your helping.

Re: FSDirectory source code mistake!

2011-07-10 Thread Simon Willnauer
Hey, you are right FSDirectory is abstract, yet the fact that it doesn't declare an abstract method doesn't make it wrong. Please use the static factory methods FSDirectory.open(File) to create an instance. This factory will create a reasonable default for your platform. see the j

FSDirectory source code mistake!

2011-07-10 Thread Parsa Moshrefi
Greeting, There's a mistake in writing the type of the class FSDirectory in the core of Lucene 3.3. This class is defined as an abstract one while No abstract method is found inside. Also it yields some other problems like not being able to create an instance of FSDirectory in the code. P

Re: RAMDirectory doesn't win over FSDirectory all the time, why?

2011-06-17 Thread Sanne Grinovero
there's enough memory, my conclusion is that when you have memory, you should limit the JVM heap and leave that to the OS to make better use of FSDirectory, as this implementation is really well optimized, at least for local disks. When you don't have enough available memory, I would s

Re: RAMDirectory doesn't win over FSDirectory all the time, why?

2011-06-16 Thread Lance Norskog
The RAMDirectory uses Java memory, an FSDirectory does not. Holding Java memory makes garbage collection work harder. The operating system is very very good at managing disk buffers, and does a better job using spare memory than Java does. For real-world sites, RAMDirectory is almost always

RE: RAMDirectory doesn't win over FSDirectory all the time, why?

2011-06-07 Thread zhoucheng2008
Makes sense. Thanks -Original Message- From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk] Sent: Tuesday, June 07, 2011 4:28 PM To: java-user@lucene.apache.org Subject: Re: RAMDirectory doesn't win over FSDirectory all the time, why? On Mon, 2011-06-06 at 15:29 +0200, zhouchen

Re: RAMDirectory doesn't win over FSDirectory all the time, why?

2011-06-07 Thread Toke Eskildsen
On Mon, 2011-06-06 at 15:29 +0200, zhoucheng2008 wrote: > I read the lucene in action book and just tested the > FSversusRAMDirectoryTest.java with the following uncommented: > [...]Here is the output: > > RAMDirectory Time: 805 ms > > FSDirectory Time : 728 ms This is

RE: RAMDirectory doesn't win over FSDirectory all the time, why?

2011-06-06 Thread zhoucheng2008
- From: Uwe Schindler [mailto:u...@thetaphi.de] Sent: Tuesday, June 07, 2011 12:04 AM To: java-user@lucene.apache.org Subject: RE: RAMDirectory doesn't win over FSDirectory all the time, why? Hi, It depends on the Lucene version, so if the test uses latest Lucene on a 64bit OS, it ma

RE: RAMDirectory doesn't win over FSDirectory all the time, why?

2011-06-06 Thread Uwe Schindler
.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Michael McCandless [mailto:luc...@mikemccandless.com] > Sent: Monday, June 06, 2011 5:58 PM > To: java-user@lucene.apache.org > Subject: Re: RAMDirectory doesn&#

Re: RAMDirectory doesn't win over FSDirectory all the time, why?

2011-06-06 Thread Michael McCandless
with the following uncommented: > > > >    //    /** > >    //    // change to adjust performance of indexing with FSDirectory > >        writer.mergeFactor = 100; > >        writer.maxMergeDocs = 99; > >        writer.minMergeDocs = 1000; > > //    */ > > >

RAMDirectory doesn't win over FSDirectory all the time, why?

2011-06-06 Thread zhoucheng2008
I read the lucene in action book and just tested the FSversusRAMDirectoryTest.java with the following uncommented: ///** //// change to adjust performance of indexing with FSDirectory writer.mergeFactor = 100; writer.maxMergeDocs = 99

Re: OutOfMemoryError with FSDirectory

2011-04-05 Thread Michael McCandless
to work with, but, for only 80K docs it should be possible as long as the docs are not large. Mike http://blog.mikemccandless.com On Mon, Apr 4, 2011 at 6:39 AM, Claudio R wrote: > Hi, > > I am using Lucene 2.9.4 with FSDirectory. > My index has 80 thousand documents (each document h

OutOfMemoryError with FSDirectory

2011-04-05 Thread Claudio R
Hi, I am using Lucene 2.9.4 with FSDirectory. My index has 80 thousand documents (each document has 12 fields). My jvm has 70Mb of RAM memory (limited by my hosting). I am getting various OutOfMemoryError. I ran jmap and I got: num   #instances    #bytes    Class description

OutOfMemoryError with FSDirectory

2011-04-05 Thread Claudio R
Hi, I am using Lucene 2.9.4 with FSDirectory. My index has 80 thousand documents (each document has 12 fields). My jvm has 70Mb of RAM memory (limited by my hosting). I am getting various OutOfMemoryError. I ran jmap and I got: num   #instances    #bytes    Class description

OutOfMemoryError with FSDirectory

2011-04-05 Thread Claudio R
Hi, I am using Lucene 2.9.4 with FSDirectory. My index has 80 thousand documents (each document has 12 fields). My jvm has 70Mb of RAM memory (limited by my hosting). I am getting various OutOfMemoryError. I ran jmap and I got: num   #instances    #bytes    Class description

Re: OutOfMemoryError with FSDirectory

2011-04-04 Thread Claudio
Ok Erick, Thanks for your quick answer. FSDirectory will, indeed, store the index on disk. However, when *using* that index, lots of stuff happens. Specifically: When indexing, there is a buffer that accumulates documents until it's flushed to disk. Are you indexing? When searching (and

Re: OutOfMemoryError with FSDirectory

2011-04-04 Thread Erick Erickson
FSDirectory will, indeed, store the index on disk. However, when *using* that index, lots of stuff happens. Specifically: When indexing, there is a buffer that accumulates documents until it's flushed to disk. Are you indexing? When searching (and this is the more important part), various c

OutOfMemoryError with FSDirectory

2011-04-04 Thread Claudio
Hi, I am using Lucene 2.9.4 with FSDirectory. My index has 80 thousand documents (each document has 12 fields). My jvm has 70Mb of RAM memory (limited by my hosting). I am getting various OutOfMemoryError. I ran jmap and I got: num #instances#bytesClass description

Re: About FSDirectory, File

2011-02-13 Thread narayan bhati
Hi, you can export index with jar itself and do not provide absolute path relative to file system instead create folder named index in your project space then as below you can get index directory directory=FSDirectory.open(new File("index")); here path is relative to you workspace which will not

About FSDirectory, File

2011-02-13 Thread Gong Li
Hi, I need to generate executable JAR. In my code, it has some lines as following: String path = "d:\\project\\"; File f = new File(path); Directory dir = FSDirectory.open(f); In the path, there is a wordnet index which is used to search by synonyms. When I run the JAR, it needs local directory

Re: integrating RAMDirectory in FSDirectory

2010-10-16 Thread Yakob
? what's the best solution to copy directory from RAMDirectory to FSDirectory besides using copy? thanks. http://stackoverflow.com/questions/3913180/how-to-integrate-ramdirectory-into-fsdirectory-in-lucene/3923914#3923914 -- http://jacobian.web.id

Re: integrating RAMDirectory in FSDirectory or other way round

2010-10-12 Thread Erick Erickson
you can't try. It's trivial to instantiate a RAMdirectory by calling the constructor with a Directory. See Ian's comment. Best Erick On Tue, Oct 12, 2010 at 1:39 AM, wrote: > > I am also having a similar requirement .But its other way round. > > Basically ,I have ind

Re: integrating RAMDirectory in FSDirectory

2010-10-12 Thread Erick Erickson
wrote: > well actually I am doing a kind of a thesis regarding information > retrieval.and my tutor wanted me to be able to create a program that > firstly index a document in memory using RAMDirectory and then > flushing it to disk using FSDirectory periodically. I was having a > hard tim

Re: integrating RAMDirectory in FSDirectory or other way round

2010-10-11 Thread Ian Lea
Start by reading the javadocs for RAMDirectory. -- Ian. On Tue, Oct 12, 2010 at 6:39 AM, wrote: > > I am also having a similar requirement .But its other way round. > > Basically ,I have indexes in FSDirectory and which is transferred to > another machine on regular basis.

Re: integrating RAMDirectory in FSDirectory or other way round

2010-10-11 Thread suman.holani
I am also having a similar requirement .But its other way round. Basically ,I have indexes in FSDirectory and which is transferred to another machine on regular basis. But now for the reason of faster searches, it would be better to copy the indexes onto RAM. (RAMDirectory). Not sure of how it

integrating RAMDirectory in FSDirectory

2010-10-11 Thread Yakob
well actually I am doing a kind of a thesis regarding information retrieval.and my tutor wanted me to be able to create a program that firstly index a document in memory using RAMDirectory and then flushing it to disk using FSDirectory periodically. I was having a hard time implementing it in my

Re: Deadlock in using FSDirectory

2009-03-06 Thread Michael McCandless
MakMak wrote: Hey Mike, thanks for the quick response, I tried passing Directory to IndexReader.open() and there were no deadlocks!! I will get rid of synchronizing on FSDirectory too. Great! However do you think it will be better to modify the docs for FSDirectory and remove the sync

Re: Deadlock in using FSDirectory

2009-03-06 Thread MakMak
Hey Mike, thanks for the quick response, I tried passing Directory to IndexReader.open() and there were no deadlocks!! I will get rid of synchronizing on FSDirectory too. However do you think it will be better to modify the docs for FSDirectory and remove the sync part of "Directorie

Re: Deadlock in using FSDirectory

2009-03-06 Thread Michael McCandless
mentReader lock. Thread2 - 1. Issues prevIndexReader.close() which locks a MultiSegmentReader and then goes ahead and waits on a lock for FSDirectory Actual code: Thread 1-> at org .apache .lucene.index.DirectoryIndexReader.reopen(DirectoryIndexReader.java: 85) - wa

Deadlock in using FSDirectory

2009-03-06 Thread MakMak
prevIndexReader.close() which locks a MultiSegmentReader and then goes ahead and waits on a lock for FSDirectory Actual code: Thread 1-> at org.apache.lucene.index.DirectoryIndexReader.reopen(DirectoryIndexReader.java:85) - waiting to lock <0xfffecaba16

Re: Pattern for maintaining FSDirectory copy of RAMDirectory

2009-02-16 Thread Erick Erickson
is create some kind of marker for whether your app was gracefully shut down. Then, when gracefully shutting down, use Directory.copy to copy FROM your RAMDir TO your FSDir. Upon startup, you can check your marker and, if all is well, read your FSDirectory into your RAMDir, and away you go. If yo

Pattern for maintaining FSDirectory copy of RAMDirectory

2009-02-16 Thread Joel Halbert
Hi, I have a RAMDirectory based index. The document source for the index is a database table, where content to be indexed is stored alongside a status (pending_index, indexed, pending_delete, deleted). Each time the application is started, and periodically thereafter, all documents from the databa

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
OK I opened: https://issues.apache.org/jira/browse/LUCENE-1331 Mike Paul Taylor wrote: Michael McCandless wrote: Hmmm, you should not close the directory if you are then going to use it to instantiate a searcher. how come it works ? Your code below never closes the searcher? I th

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
It works because Lucene doesn't currently check for it, and, because closing an FSDirectory does not actually make it unusable. In fact it also doesn't catch a double-close call. But it may cause subtle problems, because FSDirectory has this invariant: only a single i

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Paul Taylor
Michael McCandless wrote: Hmmm, you should not close the directory if you are then going to use it to instantiate a searcher. how come it works ? Your code below never closes the searcher? I think that is most likely the source of your file descriptor leaks. Ok fixed paul --

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
documentation is not very clear on this point. I see your poibnt about the try/finally I'll make that chnage. There are many other parts of the code that use filedescriptors, but the problem has never occurred before moving to a FSDirectory thanks paul heres an example of my sea

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
I'll make that chnage. There are many other parts of the code that use filedescriptors, but the problem has never occurred before moving to a FSDirectory thanks paul heres an example of my search code, is this ok ? public boolean recNoColumnMatchesSearch(Integer columnId, Integer

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Paul Taylor
see your poibnt about the try/finally I'll make that chnage. There are many other parts of the code that use filedescriptors, but the problem has never occurred before moving to a FSDirectory thanks paul heres an example of my search code, is this ok ? public boolean recNoColumnMatchesSea

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
Technically you should call directory.close() as well, but missing that will not lead to too many open files. How often is that RuntimeException being thrown? EG if a single document is frequently hitting an exception during analysis, your code doesn't close the IndexWriter in that situa

Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Paul Taylor
Hi, I have been using a RAMDirectory for indexing without any problem, but I then moved to a file based directory to reduce memory usage. this has been working fine on Windows and OSX and my version of linux (redhat) but is failing on a version of linux (archlinux) with 'Too many files opened'

Re: Regarding RAMDirectory, FSDirectory, tmpFS...

2008-06-30 Thread Anshum
Hi Devashish, The difference between these modes of operation would be that firsly, RAMDirectory and FSDirectory on one hand are implementations of the directory class of lucene. http://lucene.apache.org/java/2_2_0/api/org/apache/lucene/store/Directory.html Lucene's directory is just a flat

Regarding RAMDirectory, FSDirectory, tmpFS...

2008-06-30 Thread Devashish
What is the difference between these three modes of operating with lucene... And are there any other modes/ways of operation also, using which we can more effectively run applications with lucene. I have heard there is also something called 'Memory' in contrib directory of lucene source package wh

RE: Max size of index (FSDirectory )

2008-01-14 Thread spring
> OG: again, it depends. If the index you'd get by merging is > of manageable size, then merge your indices. OK, this is what I tought. A single index should be faster than multiple indexes with a MultiSearcher, right? But what about the ParallelMultiSearcher? As I understand the docs it searc

Re: Max size of index (FSDirectory )

2008-01-13 Thread Otis Gospodnetic
Hi, - Original Message From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> To: java-user@lucene.apache.org Sent: Sunday, January 13, 2008 12:41:18 PM Subject: Max size of index (FSDirectory ) Hi, is there any maximum size for an index? OG: There is: doc IDs are currently

Max size of index (FSDirectory )

2008-01-13 Thread spring
Hi, is there any maximum size for an index? Are there any recommendations for a useful max size? I want to index in parallel. So I have to create multiple indexes. Shall I merge them together or shall I let them as they are using (Parallel)MultiSearcher? Thank you. ---

Re: FSDirectory Again

2007-12-02 Thread Doron Cohen
This is from Lucene's CHANGES.txt: LUCENE-773: Deprecate the FSDirectory.getDirectory(*) methods that take a boolean "create" argument. Instead you should use IndexWriter's "create" argument to create a new index. (Mike McCandless) So you should create the FSDir with FSDirect

Re: FSDirectory Again

2007-11-30 Thread Donna L Gresh
In general it is much nicer to say "I did not make myself clear" than "you are not getting me" If you look on the java doc page for FSDirectory it tells you what do do instead of the deprecated method: getDirectory(File file, boolean create) Deprecated. Use

FSDirectory Again

2007-11-30 Thread Liaqat Ali
No you are not getting me. I have this original code. What i should use instead of this code to create a directory, because the dir =FSDirectory.getDirectory(indexDir, true) is deprecated. import org.apache.lucene.store.Directory; import org.apache.lucene.store.FSDirectory; protected Directo

Re: FSDirectory

2007-11-30 Thread Erick Erickson
x27;s create flag, instead, to create a new index. *But I'd expect the same behavior from IndexWriter with the create flag... Best Erick On Nov 30, 2007 8:34 AM, Liaqat Ali <[EMAIL PROTECTED]> wrote: > I m facing problem with this code.. > > dir = new FSDirectory(); >

FSDirectory

2007-11-30 Thread Liaqat Ali
I m facing problem with this code.. dir = new FSDirectory(); dir.getDirectory(indexDir, true); i get error that FSDirectory has protected access. So what i should use instead of it... Liaqat - To unsubscribe, e-mail

Re: RAMDirectory vs FSDirectory

2007-11-27 Thread German Kondolf
change any classes of Lucece: Add the "implements Serialzable" in > any classes. > > > > > On Nov 27, 2007 4:28 AM, Chhabra, Kapil <[EMAIL PROTECTED]> wrote: > > > > one can improve search performance by using a RAMDirectory created > > from an

Re: RAMDirectory vs FSDirectory

2007-11-27 Thread Grant Ingersoll
RAMDirectory has a constructor that takes in another Directory and loads it into memory. No Serialization necessary. Just index to a FSDirectory using Lucene's normal indexing methods (it takes care of buffering them internally) and then load the FSDirectory into a RAMDirectory. H

Re: RAMDirectory vs FSDirectory

2007-11-27 Thread Haroldo Nascimento
07 4:28 AM, Chhabra, Kapil <[EMAIL PROTECTED]> wrote: > > one can improve search performance by using a RAMDirectory created > from an underlying FSDirectory using one of the parameterised > constructors. Is this correct? > Absolutly > > > Will a FSDirectory not automatical

RE: RAMDirectory vs FSDirectory

2007-11-26 Thread Chhabra, Kapil
> one can improve search performance by using a RAMDirectory created from an underlying FSDirectory using one of the parameterised constructors. Is this correct? Absolutly > Will a FSDirectory not automatically load the index into memory provided enough RAM is available? Not all index fil

RAMDirectory vs FSDirectory

2007-11-26 Thread Hardy Ferentschik
Hi there, I am using currently a FSDirectory to build my index. The reason for using a file system based index is that a full index rebuild takes around 30 minutes and I want to keep a persistent index. In 'Lucene in Action' I've read that one can improve search performan

  1   2   >