Chuck Williams wrote:
I followed this same logic in ParallelWriter and got burned. My first
implementation (still the version submitted as a patch in jira) used
dynamic threads to add the subdocuments to the parallel subindexes
simultaneously. This hit a problem with abnormal native heap OOM's in
the jvm. At first I thought it was simply a thread stack size / java
heap size configuration issue, but adjusting these did not resolve the
issue. This was on linux. ps -L showed large numbers of defunct
threads. jconsole showed enormous growing total-ever-allocated thread
counts. I switched to a thread pool and the issue went away with the
same config settings.
Can you demonstrate the problem with a standalone program?
Way back in the 90's I implemented a system at Excite that spawned one
or more Java threads per request, and it ran for days on end, handling
20 or more requests per second. The thread spawning overhead was
insignificant. That was JDK 1.2 on Solaris. Have things gotten that
much worse in the interim? Today Hadoop's RPC allocates a thread per
connection, and we see good performance. So I certainly have
counterexamples.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
