Re: Hadoop RPC call response post processing
Hi Ted, I don't think the problem is allocation but garbage collection. When the gc kicks in everything freezes. Of course changing the gc algorithm helps a little. Stefan On Dec 27, 2010, at 11:21 PM, Ted Dunning wrote: I would be very surprised if allocation itself is the problem as opposed to good old fashioned excess copying. It is very hard to write an allocator faster than the java generational gc, especially if you are talking about objects that are ephemeral. Have you looked at the tenuring distribution? On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf s...@101tec.com wrote: Hi All, I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor slot that allows me to handle a RPC call response writable after it was send over the wire. Does anybody has an idea how break into the RPC code from outside. All the interesting methods are private. :( Background: Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems that the garbage collect can get so busy that the jvm almost freezes for seconds. Things like zookeeper sessions time out in that cases. My idea is to create an object pool for writables. Borrowing an object from the pool is simple since this happen in our custom code, though we do know when the writable return was send over the wire and can be returned into the pool. A dirty hack would be to overwrite the write(out) method in the writable, assuming that is the last thing done with the writable, though turns out that this method is called in other cases too, e.g. to measure throughput. Any ideas? Thanks, Stefan
Re: Hadoop RPC call response post processing
Hi Todd, Right, that is the code I'm looking into. Though Responder is inner private class and is created responder = new Responder(); It would be great if the Responder implementation could be configured. Do you have any idea how to overwrite the Responder? Thanks, Stefan On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote: Hi Stefan, Sounds interesting. Maybe you're looking for o.a.h.ipc.Server$Responder? -Todd On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf s...@101tec.com wrote: Hi All, I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor slot that allows me to handle a RPC call response writable after it was send over the wire. Does anybody has an idea how break into the RPC code from outside. All the interesting methods are private. :( Background: Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems that the garbage collect can get so busy that the jvm almost freezes for seconds. Things like zookeeper sessions time out in that cases. My idea is to create an object pool for writables. Borrowing an object from the pool is simple since this happen in our custom code, though we do know when the writable return was send over the wire and can be returned into the pool. A dirty hack would be to overwrite the write(out) method in the writable, assuming that is the last thing done with the writable, though turns out that this method is called in other cases too, e.g. to measure throughput. Any ideas? Thanks, Stefan -- Todd Lipcon Software Engineer, Cloudera
Re: Hadoop RPC call response post processing
Knowing the tenuring distribution will tell a lot about that exact issue. Ephemeral collections take on average less than one instruction per allocation and the allocation itself is generally only a single instruction. For ephemeral garbage, it is extremely unlikely that you can beat that. So the real question is whether you are actually creating so much garbage that you are over-whelming the collector or whether the data is much longer lived than it should be. *That* can cause lots of collection costs. To tell how long data lives, you need to get the tenuring distribution: -XX:+PrintTenuringDistribution Prints details about the tenuring distribution to standard out. It can be used to show this threshold and the ages of objects in the new generation. It is also useful for observing the lifetime distribution of an application. On Tue, Dec 28, 2010 at 11:59 AM, Stefan Groschupf s...@101tec.com wrote: I don't think the problem is allocation but garbage collection.
Re: Hadoop RPC call response post processing
On Tue, Dec 28, 2010 at 1:00 PM, Stefan Groschupf s...@101tec.com wrote: Hi Todd, Right, that is the code I'm looking into. Though Responder is inner private class and is created responder = new Responder(); It would be great if the Responder implementation could be configured. Do you have any idea how to overwrite the Responder? Nope, it's not currently pluggable, nor do I think there's any compelling reason to make it pluggable. It's coupled quite tightly to the implementation right now. Perhaps you can hack something in a git branch, and if it has good results on something like NNBench it could be a general contribution? -Todd On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote: Hi Stefan, Sounds interesting. Maybe you're looking for o.a.h.ipc.Server$Responder? -Todd On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf s...@101tec.com wrote: Hi All, I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor slot that allows me to handle a RPC call response writable after it was send over the wire. Does anybody has an idea how break into the RPC code from outside. All the interesting methods are private. :( Background: Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems that the garbage collect can get so busy that the jvm almost freezes for seconds. Things like zookeeper sessions time out in that cases. My idea is to create an object pool for writables. Borrowing an object from the pool is simple since this happen in our custom code, though we do know when the writable return was send over the wire and can be returned into the pool. A dirty hack would be to overwrite the write(out) method in the writable, assuming that is the last thing done with the writable, though turns out that this method is called in other cases too, e.g. to measure throughput. Any ideas? Thanks, Stefan -- Todd Lipcon Software Engineer, Cloudera -- Todd Lipcon Software Engineer, Cloudera
Re: Hadoop RPC call response post processing
Are you connecting to this JVM with RMI? RMI does a very nasty thing with garbage collection: it forces a blocking collection every 60 seconds. Really. You have to change this with a system property. On Tue, Dec 28, 2010 at 5:56 PM, Todd Lipcon t...@cloudera.com wrote: On Tue, Dec 28, 2010 at 1:00 PM, Stefan Groschupf s...@101tec.com wrote: Hi Todd, Right, that is the code I'm looking into. Though Responder is inner private class and is created responder = new Responder(); It would be great if the Responder implementation could be configured. Do you have any idea how to overwrite the Responder? Nope, it's not currently pluggable, nor do I think there's any compelling reason to make it pluggable. It's coupled quite tightly to the implementation right now. Perhaps you can hack something in a git branch, and if it has good results on something like NNBench it could be a general contribution? -Todd On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote: Hi Stefan, Sounds interesting. Maybe you're looking for o.a.h.ipc.Server$Responder? -Todd On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf s...@101tec.com wrote: Hi All, I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor slot that allows me to handle a RPC call response writable after it was send over the wire. Does anybody has an idea how break into the RPC code from outside. All the interesting methods are private. :( Background: Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems that the garbage collect can get so busy that the jvm almost freezes for seconds. Things like zookeeper sessions time out in that cases. My idea is to create an object pool for writables. Borrowing an object from the pool is simple since this happen in our custom code, though we do know when the writable return was send over the wire and can be returned into the pool. A dirty hack would be to overwrite the write(out) method in the writable, assuming that is the last thing done with the writable, though turns out that this method is called in other cases too, e.g. to measure throughput. Any ideas? Thanks, Stefan -- Todd Lipcon Software Engineer, Cloudera -- Todd Lipcon Software Engineer, Cloudera -- Lance Norskog goks...@gmail.com
Re: Hadoop RPC call response post processing
Hi Todd, Thanks for the feedback. Nope, it's not currently pluggable, nor do I think there's any compelling reason to make it pluggable. Well, one could argue with an interceptor / filter it would be very easy to add compression or encryption to RPC. But since the nutch days the code base was never architected in extendable or modular way. Perhaps you can hack something in a git branch, and if it has good results on something like NNBench it could be a general contribution? Thanks - I pass on that offer. The days waiting a half year to get a patch into the codebase are behind me. :) I think I will just replace hadoop RPC with netty. Cheers, Stefan
Hadoop RPC call response post processing
Hi All, I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor slot that allows me to handle a RPC call response writable after it was send over the wire. Does anybody has an idea how break into the RPC code from outside. All the interesting methods are private. :( Background: Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems that the garbage collect can get so busy that the jvm almost freezes for seconds. Things like zookeeper sessions time out in that cases. My idea is to create an object pool for writables. Borrowing an object from the pool is simple since this happen in our custom code, though we do know when the writable return was send over the wire and can be returned into the pool. A dirty hack would be to overwrite the write(out) method in the writable, assuming that is the last thing done with the writable, though turns out that this method is called in other cases too, e.g. to measure throughput. Any ideas? Thanks, Stefan
Re: Hadoop RPC call response post processing
Hi Stefan, Sounds interesting. Maybe you're looking for o.a.h.ipc.Server$Responder? -Todd On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf s...@101tec.com wrote: Hi All, I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor slot that allows me to handle a RPC call response writable after it was send over the wire. Does anybody has an idea how break into the RPC code from outside. All the interesting methods are private. :( Background: Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems that the garbage collect can get so busy that the jvm almost freezes for seconds. Things like zookeeper sessions time out in that cases. My idea is to create an object pool for writables. Borrowing an object from the pool is simple since this happen in our custom code, though we do know when the writable return was send over the wire and can be returned into the pool. A dirty hack would be to overwrite the write(out) method in the writable, assuming that is the last thing done with the writable, though turns out that this method is called in other cases too, e.g. to measure throughput. Any ideas? Thanks, Stefan -- Todd Lipcon Software Engineer, Cloudera
Re: Hadoop RPC call response post processing
I would be very surprised if allocation itself is the problem as opposed to good old fashioned excess copying. It is very hard to write an allocator faster than the java generational gc, especially if you are talking about objects that are ephemeral. Have you looked at the tenuring distribution? On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf s...@101tec.com wrote: Hi All, I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor slot that allows me to handle a RPC call response writable after it was send over the wire. Does anybody has an idea how break into the RPC code from outside. All the interesting methods are private. :( Background: Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems that the garbage collect can get so busy that the jvm almost freezes for seconds. Things like zookeeper sessions time out in that cases. My idea is to create an object pool for writables. Borrowing an object from the pool is simple since this happen in our custom code, though we do know when the writable return was send over the wire and can be returned into the pool. A dirty hack would be to overwrite the write(out) method in the writable, assuming that is the last thing done with the writable, though turns out that this method is called in other cases too, e.g. to measure throughput. Any ideas? Thanks, Stefan
