From: [EMAIL PROTECTED] To: solr-user@lucene.apache.org Subject: RE: Out of
memory on Solr sorting Date: Tue, 29 Jul 2008 10:43:05 -0700 A sneaky
source of OutOfMemory errors is the permanent generation. If you add this:
-XX:PermSize=64m -XX:MaxPermSize=96m You will increase the size
Hi Sundar,
If increasing LRU cache helps you:
- you are probably using 'tokenized' field for sorting (could you
confirm please?)...
...you should use 'non-tokenized single-valued non-boolean' for better
performance of
sorting...
Fuad Efendi
==
http://www.tokenizer.org
The field is of type text_ws. Is this not recomended. Should I use text
instead?
Date: Tue, 5 Aug 2008 10:58:35 -0700 From: [EMAIL PROTECTED] To: [EMAIL
PROTECTED] Subject: RE: Out of memory on Solr sorting Hi Sundar, If
increasing LRU cache helps you: - you are probably using
My understanding of Lucene Sorting is that it will sort by 'tokens'
and not by 'full fields'... so that for sorting you need 'full-string'
(non-tokenized) field, and to search you need another one tokenized.
For instance, use 'string' for sorting, and 'text_ws' for search; and
use
Best choice for sorting field:
!-- This is an example of using the KeywordTokenizer along
With various TokenFilterFactories to produce a sortable field
that does not include some properties of the source text
--
fieldType name=alphaOnlySort class=solr.TextField
On Tue, Aug 5, 2008 at 1:59 PM, Fuad Efendi [EMAIL PROTECTED] wrote:
If increasing LRU cache helps you:
- you are probably using 'tokenized' field for sorting (could you confirm
please?)...
Sorting does not utilize any Solr caches.
-Yonik
I know, and this is strange... I was guessing filterCache is used
implicitly to get DocSet for token; as Sundar wrote, increase of
LRUCache helped him (he is sorting on 'text-ws' field)
-Fuad
If increasing LRU cache helps you:
- you are probably using 'tokenized' field for sorting (could you
of memory on Solr sorting I
know, and this is strange... I was guessing filterCache is used implicitly
to get DocSet for token; as Sundar wrote, increase of LRUCache helped him
(he is sorting on 'text-ws' field) -Fuad If increasing LRU cache helps
you: - you are probably using 'tokenized
Sundar, very strange that increase of size/initialSize of LRUCache
helps with OutOfMemoryError...
2048 is number of entries in cache and _not_ 2Gb of memory...
Making size==initialSize of HashMap-based LRUCache would help with
performance anyway; may be with OOMs (probably no need to resize
Oh Wow, I didnt know that was the case. I am completely left baffled now. BAck
to square one I guess. :)
Date: Tue, 5 Aug 2008 14:31:28 -0700 From: [EMAIL PROTECTED] To:
solr-user@lucene.apache.org Subject: RE: Out of memory on Solr sorting
Sundar, very strange that increase of size
that is not reclaimed, and so each undeploy/redeploy cycle
eats up the permanent generation pool.
-Original Message-
From: david w [mailto:[EMAIL PROTECTED]
Sent: Tuesday, July 29, 2008 7:20 AM
To: solr-user@lucene.apache.org
Subject: Re: Out of memory on Solr sorting
Hi, Daniel
I got
@lucene.apache.org
Subject: RE: Out of memory on Solr sorting
Yes, it is a cache, it stores sorted by sorted field array of
Document IDs together with sorted fields; query results can intersect
with it and reorder accordingly.
But memory requirements should be well documented.
It uses internally WeakHashMap
-Xmx2048m -XX:MinHeapFreeRatio=50
-XX:NewSize=1024m -XX:NewRatio=2 -Dsun.rmi.dgc.client.gcInterval=360
-Dsun.rmi.dgc.server.gcInterval=360
Jboss 4.05
Subject: RE: Out of memory on Solr sorting
Date: Wed, 23 Jul 2008 10:49:06 +0100
From: [EMAIL PROTECTED]
To: solr-user
From: [EMAIL PROTECTED]
To: solr-user@lucene.apache.org
Subject: Out of memory on Solr sorting
Date: Tue, 22 Jul 2008 19:11:02 +
Hi,
Sorry again fellos. I am not sure whats happening. The day with solr is bad
for me I guess. EZMLM didnt let me send any mails this morning. Asked
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403)
- this piece of code do not request Array[100M] (as I seen with
Lucene), it asks only few bytes / Kb for a field...
Probably 128 - 512 is not enough; it is also advisable to use equal sizes
-Xms1024M -Xmx1024M
sort of that too just 10 rows??
-Sundar
Date: Tue, 22 Jul 2008 12:24:35 -0700 From: [EMAIL PROTECTED] To:
solr-user@lucene.apache.org Subject: RE: Out of memory on Solr sorting
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403)
- this piece of code do
Date: Tue, 22 Jul 2008 12:24:35 -0700 From: [EMAIL PROTECTED] To: solr-user@lucene.apache.org Subject: RE: Out of memory on Solr sorting org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403) - this piece of code do not request Array[100M] (as I seen with Lucene
fine too. But why just sort of that too just 10 rows??
-Sundar
Date: Tue, 22 Jul 2008 12:24:35 -0700 From: [EMAIL PROTECTED] To:
solr-user@lucene.apache.org Subject: RE: Out of memory on Solr
sorting
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403
] To:
solr-user@lucene.apache.org Subject: RE: Out of memory on Solr
sorting
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403) - this piece of code do not request Array[100M] (as I seen with Lucene), it asks only few bytes / Kb for a field... Probably 128
Subject: Re: Out of memory on Solr sorting
Because to sort efficiently, Solr loads the term to sort on for each doc
in the index into an array. For ints,longs, etc its just an array the
size of the number of docs in your index (i believe deleted or not). For
a String its an array to hold
posted on the turn
arounds.Thanks-Sundar Date: Tue, 22 Jul 2008 15:46:04 -0400 From: [EMAIL
PROTECTED] To: solr-user@lucene.apache.org Subject: Re: Out of memory on Solr
sorting Because to sort efficiently, Solr loads the term to sort on for each
doc in the index into an array. For ints,longs
:35 -0700 From: [EMAIL PROTECTED] To:
solr-user@lucene.apache.org Subject: RE: Out of memory on Solr
sorting
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403)
- this piece of code do not request Array[100M] (as I seen with
Lucene), it asks only few bytes
Mark,
Question: how much memory I need for 25,000,000 docs if I do a sort by
string field, 256 bytes. 6.4Gb?
Quoting Mark Miller [EMAIL PROTECTED]:
Because to sort efficiently, Solr loads the term to sort on for each
doc in the index into an array. For ints,longs, etc its just an array
. But why just sort of that too just 10
rows??
-Sundar
Date: Tue, 22 Jul 2008 12:24:35 -0700 From: [EMAIL PROTECTED]
To: solr-user@lucene.apache.org Subject: RE: Out of memory
on Solr sorting
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403
]
To: solr-user@lucene.apache.org Subject: RE: Out of memory
on Solr sorting
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403)
- this piece of code do not request Array[100M] (as I seen
with Lucene), it asks only few bytes / Kb for a field...
Probably 128
with just about a gig of memory and the
whole 1024 megs have been allocated to heap. The dev is a linux with over 2
Gigs of memory and 1024 allocated to heap now. :S
-Sundar
Date: Tue, 22 Jul 2008 13:17:40 -0700 From: [EMAIL PROTECTED] To:
solr-user@lucene.apache.org Subject: Re: Out of memory
Thanks for your help Mark. Lemme explore a little more and see if some one else
can help me out too. :)
Date: Tue, 22 Jul 2008 16:53:47 -0400 From: [EMAIL PROTECTED] To:
solr-user@lucene.apache.org Subject: Re: Out of memory on Solr sorting
Someone else is going to have to take over Sundar
From: [EMAIL PROTECTED]
To: solr-user@lucene.apache.org Subject: RE: Out of
memory on Solr sorting
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403) - this piece of code do not request Array[100M] (as I seen with Lucene), it asks only few bytes / Kb
??
-Sundar
Date: Tue, 22 Jul 2008 12:24:35 -0700 From: [EMAIL PROTECTED]
To: solr-user@lucene.apache.org Subject: RE: Out of
memory on Solr sorting
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403) - this piece of code do not request Array[100M
had to be loaded.
Date: Tue, 22 Jul 2008 14:26:26 -0700 From: [EMAIL PROTECTED] To:
solr-user@lucene.apache.org Subject: Re: Out of memory on Solr sorting
Ok, after some analysis of FieldCacheImpl: - it is supposed that (sorted)
Enumeration of terms is less than total number
. But don't
know why all of the fields have had to be loaded.
Date: Tue, 22 Jul 2008 14:26:26 -0700 From: [EMAIL PROTECTED] To:
solr-user@lucene.apache.org Subject: Re: Out of memory on Solr
sorting Ok, after some analysis of FieldCacheImpl: - it is
supposed that (sorted
probably be better. But
don't know why all of the fields have had to be loaded.
Date: Tue, 22 Jul 2008 14:26:26 -0700 From: [EMAIL PROTECTED] To:
solr-user@lucene.apache.org Subject: Re: Out of memory on Solr
sorting Ok, after some analysis of FieldCacheImpl: - it is
supposed
32 matches
Mail list logo
