Hi Mike,
Looks like I need lower memory limits for JVMs...
I have 12Gb RAM, 2 x Opteron 252, SuSE 10 ES, Java 5_09, Tomcat 5.5.20, SOLR
1.1
Of course, Java needs at least 20-40Mb per-thread, but SOLR-Client is
executed in separate JVM.
4Gb for Tomcat
2Gb-4Gb for Client (144 threads).
6Gb for database.
Too much memory for a database and JVMs...
About amount of threads: I run about 1024-2048 threads simultaneously a
while ago, with 'The Grinder' load-stress simulator, Windows 2000, no any
problem (if RAM is enough)...
>I'd suggest leaving a healthy chunk free for the OS to cache the index.
I thought that this cache is inside JVM memory, inside 4Gb allocated to
Tomcat... RAM is not enough, I have 9Gb of User Memory, and 11Gb of Used
Swap.
Currently I have about 100 threads (rest of threads already finished crawl
and are sleeping an hour); I have setting '7 seconds' before subsequent
execution of each transaction. In average, it gives about 15 transactins per
second, and only few of them are SOLR "add"...
I am recalling... Problems began when I disabled programmatic "commit" and
started to rely on SOLR
<autocommit>
<maxDocs>1000<maxDocs>
</autocommit>
Before that, each thread executed "add", "delete", and single static method
executed "commit" after each 100s of total "adds". Now, I disabled real-time
"delete" (going to make it batch job), and I am relying to SOLR autocommit
feature.
So far so good... Already 2 hours, same environment, no any problem... May
be tomorrow... I set 2 hours of HTTP expiration headers, so we can't see
'real-time' SOLR updates (everything is cached by front-end)...
I need to check Tomcat logs, 1.4Gb per-day is too much... I can't even open
it...
Thanks,
Fuad
P.S.
I'll set lower memory limits, and rerun an application (tomorrow); looks
like I have huge swap file.
Now:
10.6 of 11.7 Gb - user memory
9.6 of 16Gb used swap
2xCPU: 90-100%
P.P.S.
right before sending Email, I tried to check solr admin screen... something
happened... SOLR stopped itself! I can't even see process Id, and admin
screen does not work, but CPU is still 95%...
