Thanks Bryan. Indeed, after creating a new connection for every client thread, the problem went away.
-- http://kpowerinfinity.wordpress.com http://www.linkedin.com/in/kpowerinfinity On Mon, Nov 23, 2009 at 9:45 AM, Bryan Duxbury <[email protected]> wrote: > The TThreadPoolServer components are thread-safe. However, any given client > for a service is NOT thread-safe. This means that if you are using multiple > threads on the client side, you either need multiple connections or > application-level thread safety. > > -Bryan > > On Nov 22, 2009, at 11:09 AM, kpowerinfinity wrote: > >> Hello, >> >> We've been using Thrift for over a year now and have been very happy >> with how it makes it easy to implement RPC across languages. However, >> we were facing a perf bottleneck since each call was taking about >> 50-75ms which means we can't make more than 50k calls in an hour. We >> are using this for a message broker and want to be able to send out >> millions of messages per hour. This was on a single thread. On trying >> out the most obvious multi-threaded implementation where we pump out >> the calls on multiple-threads, I get the exceptions below. >> >> What I am not sure about is why thrift has an out of memory problem on >> the server (thrift-handler) end when I am pumping in messages on >> multiple-threads from the client. I have a TThreadPoolServer on the >> server end. >> >> Is there something I am missing? >> >> Regards >> Krishna. >> >> Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space >> at >> com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown Source) >> at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown >> Source) >> at >> com.facebook.thrift.protocol.TBinaryProtocol.readMessageBegin(Unknown >> Source) >> at >> in.capillary.ifaces.nsadmin.NSAdminService$Client.recv_processMessageFromClient(NSAdminService.java:154) >> at >> in.capillary.ifaces.nsadmin.NSAdminService$Client.processMessageFromClient(NSAdminService.java:139) >> at >> in.capillary.msging.system.NSAdminClientAdapter.send(NSAdminClientAdapter.java:108) >> at >> in.capillary.msging.system.NotificationDeliveryClient.send(NotificationDeliveryClient.java:42) >> at >> in.capillary.msging.system.ThreadWorker.retrieveAndSendSingleMessage(ThreadWorker.java:64) >> at >> in.capillary.msging.system.ThreadWorker.makeDeliveriesTillStop(ThreadWorker.java:82) >> at >> in.capillary.msging.test.TestShopbookPerf$3.run(TestShopbookPerf.java:108) >> at java.lang.Thread.run(Thread.java:619) >> 00:24:21,092 INFO [Thread-0] NSAdminClientAdapter:80 - Registering with >> NSAdmin >> 00:24:21,094 ERROR [Thread-0] NSAdminClientAdapter:24 - Exception >> thrown: com.facebook.thrift.TApplicationException: >> processMessageFromClient failed: unknown result >> At: >> >> NSAdminService.java:in.capillary.ifaces.nsadmin.NSAdminService$Client:recv_processMessageFromClient:169 >> >> NSAdminService.java:in.capillary.ifaces.nsadmin.NSAdminService$Client:processMessageFromClient:139 >> >> NSAdminClientAdapter.java:in.capillary.msging.system.NSAdminClientAdapter:send:108 >> >> NotificationDeliveryClient.java:in.capillary.msging.system.NotificationDeliveryClient:send:42 >> >> ThreadWorker.java:in.capillary.msging.system.ThreadWorker:retrieveAndSendSingleMessage:64 >> >> ThreadWorker.java:in.capillary.msging.system.ThreadWorker:makeDeliveriesTillStop:82 >> TestShopbookPerf.java:in.capillary.msging.test.TestShopbookPerf$1:run:86 >> Thread.java:java.lang.Thread:run:619 >> com.facebook.thrift.TApplicationException: processMessageFromClient >> failed: unknown result >> at >> in.capillary.msging.system.NSAdminClientAdapter.send(NSAdminClientAdapter.java:111) >> at >> in.capillary.msging.system.NotificationDeliveryClient.send(NotificationDeliveryClient.java:42) >> at >> in.capillary.msging.system.ThreadWorker.retrieveAndSendSingleMessage(ThreadWorker.java:64) >> at >> in.capillary.msging.system.ThreadWorker.makeDeliveriesTillStop(ThreadWorker.java:82) >> at >> in.capillary.msging.test.TestShopbookPerf$1.run(TestShopbookPerf.java:86) >> at java.lang.Thread.run(Thread.java:619) >> >> -- >> http://www.linkedin.com/in/kpowerinfinity >> >> >> Note: this is the original facebook implementation of thrift. > >
