did you build it on top of zmq? i really don't see the need for people reinventing the low level rcp stuff over and over again. zmq comes with baked in request-response, pub-sub, and pipeline (distributed processing) communication. once you rely on protobuf + zmq for the rpc is it trivial to add clients in other languages, i had java, R and python talking to each other with minimal effort.
for rcp the comparison of speed is a bit of a moot point, since all the latency will be in the communication, not so much in the serialization, i suspect. but once you communicate using protobuf it also becomes really tempting to store in hadoop using protobuf instead of writables/sequencefiles, and from what i have heard (i have not tested this myself) it is a good deal slower in that situation. On Fri, Sep 23, 2011 at 8:14 AM, Stephan Gammeter < gamme...@vision.ee.ethz.ch> wrote: > I don't think protobuf are slower than writable actually, they do really > well in speed. I actually wrote some rpc code in C++ for protocolbuffers and > some swig wrappers to have clients in java. A simple c++ server can easily > handle about 20k qps in that setup and this is just with a naive > implementation where still some excess data copies happen during the > processing of requests. If i have time i would like to opensource it, but i > would need some help to get it running properly in other languages, so that > it can be truly cross language. (right now servers are only supported in > c++, clients are synchronous and asynchronous in c++, in java only > synchronous clients are supported) > > > On 21.09.2011 22:59, Koert Kuipers wrote: > > i would love an IDL, plus that modern serialization frameworks such as > protobuf/thrift support versioning (although i still have issues with > different versions of thrift not working nicely together, argh why is that). > the only downside is perhaps that they are a little slower than writables. > > On Wed, Sep 21, 2011 at 3:12 AM, Uma Maheswara Rao G 72686 < > mahesw...@huawei.com> wrote: > >> Hadoop has its RPC machanism mainly Writables to overcome some of the >> disadvantages on normal serializations. >> For more info: >> http://www.lexemetech.com/2008/07/rpc-and-serialization-with-hadoop.html >> >> Regards, >> Uma >> ----- Original Message ----- >> From: jie_zhou <jie_z...@xa.allyes.com> >> Date: Wednesday, September 21, 2011 12:12 pm >> Subject: A question about RPC >> To: hdfs-user@hadoop.apache.org >> >> > Dear: >> > >> > Nice to meet you! >> > >> > I am a beginner of hadoop. Recently, I have seen the source of RPC of >> > hadoop,but now I have a question. As we know,hadoop RPC make use >> > of Dynamic >> > proxy mechanism ,but >> > >> > why not use IDL such as CORBA, or AIDL of Android? >> > >> > Thanks for your early reply. >> > >> > Best Regards, >> > >> > jie >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > > >
