I've never paid attention to post/commit ration. I usually do a commit
after maybe 100 posts. Is there a guideline about this? Thanks.
On Wed, May 13, 2009 at 1:10 PM, Otis Gospodnetic
wrote:
> 2) ramBufferSizeMB dictates, more or less, how much Lucene/Solr will consume
> during indexing. Ther
I think that if you have in your index any documents with norms, you
will still use norms for those fields even if the schema is changed
later. Did you wipe and re-index after all your schema changes?
-Peter
On Fri, May 15, 2009 at 9:14 PM, vivek sar wrote:
> Some more info,
>
> Profiling the
Some more info,
Profiling the heap dump shows
"org.apache.lucene.index.ReadOnlySegmentReader" as the biggest object
- taking up almost 80% of total memory (6G) - see the attached screen
shot for a smaller dump. There is some norms object - not sure where
are they coming from as I've omitnorms=tr
Thanks Mark.
I checked all the items you mentioned,
1) I've omitnorms=true for all my indexed fields (stored only fields I
guess doesn't matter)
2) I've tried commenting out all caches in the solrconfig.xml, but
that doesn't help much
3) I've tried commenting out the first and new searcher listen
800 million docs is on the high side for modern hardware.
If even one field has norms on, your talking almost 800 MB right there.
And then if another Searcher is brought up well the old one is serving
(which happens when you update)? Doubled.
Your best bet is to distribute across a couple mac
Some update on this issue,
1) I attached jconsole to my app and monitored the memory usage.
During indexing the memory usage goes up and down, which I think is
normal. The memory remains around the min heap size (4 G) for
indexing, but as soon as I run a search the tenured heap usage jumps
up to 6
I don't know if field type has any impact on the memory usage - does it?
Our use cases require complete matches, thus there is no need of any
analysis in most cases - does it matter in terms of memory usage?
Also, is there any default caching used by Solr if I comment out all
the caches under que
only 20K and you said this is a large index? That doesn't smell
> right...
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> - Original Message
>> From: vivek sar
>> To: solr-user@lucene.apache.org
>> Sent:
On May 13, 2009, at 6:53 PM, vivek sar wrote:
Disabling first/new searchers did help for the initial load time, but
after 10-15 min the heap memory start climbing up again and reached
max within 20 min. Now the GC is coming up all the time, which is
slowing down the commit and search cycles.
T
t.com/ -- Lucene - Solr - Nutch
- Original Message
> From: vivek sar
> To: solr-user@lucene.apache.org
> Sent: Wednesday, May 13, 2009 5:12:00 PM
> Subject: Re: Solr memory requirements?
>
> Otis,
>
> In that case, I'm not sure why Solr is taking up so mu
May 13, 2009 5:53:45 PM
> Subject: Re: Solr memory requirements?
>
> Just an update on the memory issue - might be useful for others. I
> read the following,
>
> http://wiki.apache.org/solr/SolrCaching?highlight=(SolrCaching)
>
> and looks like the first and new
t.
> >>
> >> Thanks,
> >> -vivek
> >>
> >>
> >>
> >>
> >> On Wed, May 13, 2009 at 1:50 PM, Otis Gospodnetic
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Sorting is trigger
Warning: I'm wy out of my competency range when I comment
on SOLR, but I've seen the statement that string fields are NOT
tokenized while text fields are, and I notice that almost all of your fields
are string type.
Would someone more knowledgeable than me care to comment on whether
this is at
> >>
>> >> I'm trying to find out if there is a way to tell how much memory Solr
>> >> would consume and way to cap it.
>> >>
>> >> Thanks,
>> >> -vivek
>> >>
>> >>
>> >>
>> >>
&g
;
> >> I'm trying to find out if there is a way to tell how much memory Solr
> >> would consume and way to cap it.
> >>
> >> Thanks,
> >> -vivek
> >>
> >>
> >>
> >>
> >> On Wed, May 13, 2009 at 1:50 PM, Otis Gospo
; -vivek
>>
>>
>>
>>
>> On Wed, May 13, 2009 at 1:50 PM, Otis Gospodnetic
>> wrote:
>>>
>>> Hi,
>>>
>>> Sorting is triggered by the sort parameter in the URL, not a characteristic
>>> of a field. :)
>>>
>>>
Otis Gospodnetic
> wrote:
>>
>> Hi,
>>
>> Sorting is triggered by the sort parameter in the URL, not a characteristic
>> of a field. :)
>>
>> Otis
>> --
>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>
>>
>>
f a field. :)
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
From: vivek sar
To: solr-user@lucene.apache.org
Sent: Wednesday, May 13, 2009 4:42:16 PM
Subject: Re: Solr memory requirements?
Thanks Otis.
Our use case doesn't require any
RL, not a characteristic
> of a field. :)
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> - Original Message
>> From: vivek sar
>> To: solr-user@lucene.apache.org
>> Sent: Wednesday, May 13, 2009 4:42:16 PM
&g
:42:16 PM
> Subject: Re: Solr memory requirements?
>
> Thanks Otis.
>
> Our use case doesn't require any sorting or faceting. I'm wondering if
> I've configured anything wrong.
>
> I got total of 25 fields (15 are indexed and stored, other 10 are just
> s
Thanks Otis.
Our use case doesn't require any sorting or faceting. I'm wondering if
I've configured anything wrong.
I got total of 25 fields (15 are indexed and stored, other 10 are just
stored). All my fields are basic data type - which I thought are not
sorted. My id field is unique key.
Is th
Hi,
Some answers:
1) .tii files in the Lucene index. When you sort, all distinct values for the
field(s) used for sorting. Similarly for facet fields. Solr caches.
2) ramBufferSizeMB dictates, more or less, how much Lucene/Solr will consume
during indexing. There is no need to commit every 5
22 matches
Mail list logo
