Yes, it is better to use ints for ids than strings. Also, the Trie int
fields have a compressed format that may cut the storage needs even
more. 8m * 4 = 32mb, times "a few hundred", we'll say 300, is 900mb of
IDs. I don't know how these fields are stored, but if they are
separate objects we've blown up to several gigs (per-object overheads
are surprising).
4G is probably not enough for what you want. If you watch the total
memory with 'top' and hit it with different queries, you will get a
stronger sense of how much memory your use cases need.
On Tue, Jun 29, 2010 at 4:32 PM, Robert Petersen <rober...@buy.com> wrote:
> Hello I am trying to find the right max and min settings for Java 1.6 on 20GB
> index with 8 million docs, running 1.6_018 JVM with solr 1.4, and am
> currently have java set to an even 4GB (export JAVA_OPTS="-Xmx4096m
> -Xms4096m") for both min and max which is doing pretty well but occasionally
> still getting the below OOM errors. We're running on dual quad core xeons
> with 16GB memory installed. I've been getting the below OOM exceptions still
> though.
>
> Is the memsize mentioned in the INFO for the uninvert in bytes? Ie is
> memSize=29604020 mean 29MB? We have a few hundred of these fields and they
> contain ints used as IDs, and so I guess could they eat all the memory to
> uninvert them all after we apply load and enough queries are performed. Does
> the field type matter, would int be better than string if these are lookup
> ids sparsely populated across the index? BTW these are used for faceting and
> filtering only.
>
> <dynamicField name="*_contentAttributeToken" type="string"
> indexed="true" multiValued="true" stored="true" required="false"/>
>
> Jun 29, 2010 3:54:50 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field
> {field=768_contentAttributeToken,memSize=29604014,tindexSize=50,time=1841,phase1=1824,nTerms=1,bigTerms=0,termInstances=18,uses=0}
> Jun 29, 2010 3:54:52 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field
> {field=749_contentAttributeToken,memSize=29604020,tindexSize=56,time=1847,phase1=1829,nTerms=143,bigTerms=0,termInstances=951,uses=0}
> Jun 29, 2010 3:54:59 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.request.UnInvertedField.uninvert(UnInvertedField.java:191)
> at
> org.apache.solr.request.UnInvertedField.<init>(UnInvertedField.java:178)
> at
> org.apache.solr.request.UnInvertedField.getUnInvertedField(UnInvertedField.java:839)
> at
> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:250)
> at
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:283)
> at
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:166)
>
--
Lance Norskog
goks...@gmail.com
