HI testn,
I used to search now 3 folders of total size 1.5.GB.still it
consumes lot of memory for every search.i used to close all the IndexReader
once i finish the search.i optimize the FIles using Luke.when i set the
IndexSearcher object as application level object,its not possible to see
the updated records.
could you guide me how to resolve this memory problem.
testn wrote:
>
> As I mentioned, IndexReader is the one that holds the memory. You should
> explicitly close the underlying IndexReader to make sure that the reader
> releases the memory.
>
>
>
> Sebastin wrote:
>>
>> Hi testn,
>> Every IndexFolder is of size 1.5 GB of size,eventhough when
>> i used to Open and close the IndexSearcher it wont release the memory for
>> all the searches.
>> When i set the IndexSearcher object as the Application
>> Scope object its not possile for me to see current day records.
>>
>> Could you give me an Idea how to trigger out this
>> problem.
>>
>>
>> testn wrote:
>>>
>>> If you know that there are only 15 days of indexes you need to search
>>> on, you just need to open only the latest 15 indexes at a time right?
>>> You can simply create a wrapper that return MultiReader which you can
>>> cache for a while and close the oldest index once the date rolls.
>>>
>>>
>>> Sebastin wrote:
>>>>
>>>> HI testn,
>>>>
>>>> it gives performance improvement while optimizing the Index.
>>>>
>>>> Now i seprate the IndexStore on a daily basis.(ie)
>>>> For Every Day it create a new Index store ,sep- 08-2007,sep-09-2007
>>>> like wise it will minimize the size of the IndexStore.could you give me
>>>> an idea on how to open every day folders for every search.
>>>>
>>>> Query I use here is,
>>>>
>>>> 9840836588 AND dateSc:[070901 TO 070910]
>>>>
>>>> 07---->year (2007)
>>>> 09---->month(september)
>>>> 01----->day
>>>>
>>>> i restrict for 15 days that it is possible to search 15 days record in
>>>> my application.at a time 10 users aare going to search every store.is
>>>> there any other better way to improve the search performance to avoid
>>>> memory problem as well as speed of the search.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> testn wrote:
>>>>>
>>>>> So did you see any improvement in performance?
>>>>>
>>>>> Sebastin wrote:
>>>>>>
>>>>>> It works finally .i use Lucene 2.2 in my application.thanks testn
>>>>>> and Mike
>>>>>>
>>>>>> Michael McCandless-2 wrote:
>>>>>>>
>>>>>>>
>>>>>>> It sounds like there may be a Lucene version mismatch? When Luke
>>>>>>> was used
>>>>>>> it was likely based on Lucene 2.2, but it sounds like an older
>>>>>>> version of
>>>>>>> Lucene is now being used to open the index?
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> "testn" <[EMAIL PROTECTED]> wrote:
>>>>>>>>
>>>>>>>> Should the file be "segments_8" and "segments.gen"? Why is it
>>>>>>>> "Segment"?
>>>>>>>> The
>>>>>>>> case is different.
>>>>>>>>
>>>>>>>>
>>>>>>>> Sebastin wrote:
>>>>>>>> >
>>>>>>>> > java.io.IoException:File Not Found- Segments is the error
>>>>>>>> message
>>>>>>>> >
>>>>>>>> > testn wrote:
>>>>>>>> >>
>>>>>>>> >> What is the error message? Probably Mike, Erick or Yonik can
>>>>>>>> help you
>>>>>>>> >> better on this since I'm no one in index area.
>>>>>>>> >>
>>>>>>>> >> Sebastin wrote:
>>>>>>>> >>>
>>>>>>>> >>> HI testn,
>>>>>>>> >>> 1.I optimize the Large Indexes of size 10 GB using
>>>>>>>> Luke.it
>>>>>>>> >>> optimize all the content into a single CFS file and it
>>>>>>>> generates
>>>>>>>> >>> segments.gen and segments_8 file when i search the item it
>>>>>>>> shows an
>>>>>>>> >>> error that segments file is not there.could you help me in this
>>>>>>>> >>>
>>>>>>>> >>> testn wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> 1. You can close the searcher once you're done. If you want to
>>>>>>>> reopen
>>>>>>>> >>>> the index, you can close and reopen only the updated 3 readers
>>>>>>>> and keep
>>>>>>>> >>>> the 2 old indexreaders and reuse it. It should reduce the time
>>>>>>>> to
>>>>>>>> >>>> reopen it.
>>>>>>>> >>>> 2. Make sure that you optimize it every once in a while
>>>>>>>> >>>> 3. You might consider separating indices in separated storage
>>>>>>>> and use
>>>>>>>> >>>> ParallelReader
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>> Sebastin wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> The problem in my pplication are as follows:
>>>>>>>> >>>>> 1.I am not able to see the updated records
>>>>>>>> in my
>>>>>>>> >>>>> index store because i instantiate
>>>>>>>> >>>>> IndexReader and IndexSearcher class once that is in the first
>>>>>>>> >>>>> search.further searches use the same IndexReaders(5
>>>>>>>> Directories) and
>>>>>>>> >>>>> IndexSearcher with different queries.
>>>>>>>> >>>>>
>>>>>>>> >>>>> 2.My search is very very slow First 2
>>>>>>>> Directories of
>>>>>>>> >>>>> size 10 GB each which are having old index records and no
>>>>>>>> update in
>>>>>>>> >>>>> that remaining 3 Diretories are updated every second.
>>>>>>>> >>>>>
>>>>>>>> >>>>> 3.i am Indexing 20 million records per day so
>>>>>>>> the
>>>>>>>> >>>>> Index store gets growing and it makes search very very
>>>>>>>> slower.
>>>>>>>> >>>>>
>>>>>>>> >>>>> 4.I am using searcherOne class as the global
>>>>>>>> >>>>> application helper class ,with the scope as APPLICATION it
>>>>>>>> consists of
>>>>>>>> >>>>> one IndexReader and IndexSearcher get set method which will
>>>>>>>> hold the
>>>>>>>> >>>>> IndexReader and IndexSearcher object after the First
>>>>>>>> Search.it is used
>>>>>>>> >>>>> for all other searches.
>>>>>>>> >>>>>
>>>>>>>> >>>>> 5.I am using Lucene 2.2.0 version, in a WEB
>>>>>>>> Application
>>>>>>>> >>>>> which index 15 fields per document and Index 5 Fieds,store 10
>>>>>>>> Fields.i
>>>>>>>> >>>>> am not using any sort in my query.for a single query upto the
>>>>>>>> maximum
>>>>>>>> >>>>> it fetches 600 records from the index store(5 direcories)
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>> hossman wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> : I set IndexSearcher as the application Object after the
>>>>>>>> first
>>>>>>>> >>>>>> search.
>>>>>>>> >>>>>> ...
>>>>>>>> >>>>>> : how can i reconstruct the new IndexSearcher for every hour
>>>>>>>> to see
>>>>>>>> >>>>>> the
>>>>>>>> >>>>>> : updated records .
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> i'm confused ... my understanding based on the comments you
>>>>>>>> made
>>>>>>>> >>>>>> below
>>>>>>>> >>>>>> (in an earlier message) was that you already *were*
>>>>>>>> constructing a
>>>>>>>> >>>>>> new
>>>>>>>> >>>>>> IndexSearcher once an hour -- but every time you do that,
>>>>>>>> your memory
>>>>>>>> >>>>>> usage grows, and and that sometimes you got OOM Errors.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> if that's not what you said, then i think you need to
>>>>>>>> explain, in
>>>>>>>> >>>>>> detail,
>>>>>>>> >>>>>> in one message, exactly what your problem is. And don't
>>>>>>>> assume we
>>>>>>>> >>>>>> understand anything -- tell us *EVERYTHING* (like, for
>>>>>>>> example, what
>>>>>>>> >>>>>> the
>>>>>>>> >>>>>> word "crore" means, how "searcherOne" is implemented, and
>>>>>>>> the answer
>>>>>>>> >>>>>> to
>>>>>>>> >>>>>> the specfic question i asked in my last message: does your
>>>>>>>> >>>>>> application,
>>>>>>>> >>>>>> contain anywhere in it, any code that will close anything
>>>>>>>> >>>>>> (IndexSearchers
>>>>>>>> >>>>>> or IndexReaders) ?
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> : > : I use StandardAnalyzer.the records daily ranges from 5
>>>>>>>> crore to
>>>>>>>> >>>>>> 6 crore.
>>>>>>>> >>>>>> : > for
>>>>>>>> >>>>>> : > : every second i am updating my Index. i instantiate
>>>>>>>> >>>>>> IndexSearcher object
>>>>>>>> >>>>>> : > one
>>>>>>>> >>>>>> : > : time for all the searches. for an hour can i see the
>>>>>>>> updated
>>>>>>>> >>>>>> records in
>>>>>>>> >>>>>> : > the
>>>>>>>> >>>>>> : > : indexstore by reinstantiating IndexSearcher object.but
>>>>>>>> the
>>>>>>>> >>>>>> problem when
>>>>>>>> >>>>>> : > i
>>>>>>>> >>>>>> : > : reinstantiate IndexSearcher ,my RAM memory gets
>>>>>>>> appended.is
>>>>>>>> >>>>>> there any
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> : > IndexSearcher are you explicitly closing both the old
>>>>>>>> >>>>>> IndexSearcher as
>>>>>>>> >>>>>> : > well as all of 4 of those old IndexReaders and the
>>>>>>>> MultiReader?
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> -Hoss
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> >>>>>> To unsubscribe, e-mail:
>>>>>>>> [EMAIL PROTECTED]
>>>>>>>> >>>>>> For additional commands, e-mail:
>>>>>>>> [EMAIL PROTECTED]
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12655880
>>>>>>>> Sent from the Lucene - Java Users mailing list archive at
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>>>>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>>>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
--
View this message in context:
http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12775620
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
