maybe someone interested. I just remembered, we tested pure Hadop RPC a few (5+) months ago in simple setup, kind of balancing server getting and distributing requests to 3 "search units"...
we went that far as java RMI proved to have ugly latency problems (or we did not get it right, don't know for sure) in well-under-second scenario. All in all, spawning new thread (hadoop RPC) on java 1.5, roundtrip over 2 net nodes (100MB ) there and back was slightly under 3k / second on a few kB message. ----- Original Message ---- From: Doug Cutting <[EMAIL PROTECTED]> To: java-dev@lucene.apache.org Sent: Monday, 13 November, 2006 9:50:28 PM Subject: Re: ParallelMultiSearcher reimplementation Chuck Williams wrote: > I followed this same logic in ParallelWriter and got burned. My first > implementation (still the version submitted as a patch in jira) used > dynamic threads to add the subdocuments to the parallel subindexes > simultaneously. This hit a problem with abnormal native heap OOM's in > the jvm. At first I thought it was simply a thread stack size / java > heap size configuration issue, but adjusting these did not resolve the > issue. This was on linux. ps -L showed large numbers of defunct > threads. jconsole showed enormous growing total-ever-allocated thread > counts. I switched to a thread pool and the issue went away with the > same config settings. Can you demonstrate the problem with a standalone program? Way back in the 90's I implemented a system at Excite that spawned one or more Java threads per request, and it ran for days on end, handling 20 or more requests per second. The thread spawning overhead was insignificant. That was JDK 1.2 on Solaris. Have things gotten that much worse in the interim? Today Hadoop's RPC allocates a thread per connection, and we see good performance. So I certainly have counterexamples. Doug --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] Send instant messages to your online friends http://uk.messenger.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
