Re: [protobuf] does java api have something like DescriptorPool
On 12/09/2009 09:00 PM, Kenton Varda wrote: Actually I don't think we need DescriptorPool in Java. DescriptorPool was primarily written for the purpose of memory management, but Java handles that for us. If all you need is the mapping aspect, just build a MapString, ServiceDescriptor yourself and use it. This is close to what I do now. Next question is : can I get hold of the java class that is associated with a Descriptor. From the service descriptor, I can get to the MethodDescriptor I am interested in, and then I can get the Descriptor of the input type of the method, but at that point I would like to create a builder for the input message. Should I also maintain a MapString,Descriptor One thing I thought of would be to go back to the file descriptor associated with the input message type descriptor, check the two java options, and use java reflection. would this work ? It is a shame the java class name is not the same as the message type full name. True, in C++ there is the global pool which is automatically populated with everything compiled into the binary. This wouldn't work in Java because classes are not loaded until they are first accessed, so there is no way to populate this pool. Besides, singletons are evil. I honestly wish that I'd never introduced the global pool in C++; it has lead to too many subtle singleton problems. 2009/12/9 Jason Hsueh jas...@google.com mailto:jas...@google.com No, there isn't an equivalent to the DescriptorPool in Java. If you know the types that you want you can build a mapping yourself. Or if you'd be interested in porting the C++ DescriptorPool to java that would be great! 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr Hello, Given a service/method full name, I'd like to get hold of the ServiceDescriptor, or MethodDescriptor. In C++ I would use the DescriptorPool, but I don't see this in the java api. Is there a way ? Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/Gq7i : ohloh |- http://tr.im/FtUu : new package : highlight `- http://tr.im/EAD5 : LondonR slides -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] does java api have something like DescriptorPool
Is this for the RPC system? I think you're making this a lot harder than it needs to be. On the server side, when someone wants to export a service, they will construct a service implementation and pass it to your system. The com.google.protobuf.Service interface has methods getDescriptor(), getRequestPrototype(), and getResponsePrototype(), to get the ServiceDescriptor, request message class, and response message class respectively. So, you don't need to consult any tables. On the client side, your RPC implementation should merely provide the user with an object implementing com.google.protobuf.RpcChannel. The caller then calls MyServiceType.newStub(channel) to construct a type-safe stub. The RPC system does not need to know what kind of service it is. 2009/12/10 Romain François francoisrom...@free.fr On 12/09/2009 09:00 PM, Kenton Varda wrote: Actually I don't think we need DescriptorPool in Java. DescriptorPool was primarily written for the purpose of memory management, but Java handles that for us. If all you need is the mapping aspect, just build a MapString, ServiceDescriptor yourself and use it. This is close to what I do now. Next question is : can I get hold of the java class that is associated with a Descriptor. From the service descriptor, I can get to the MethodDescriptor I am interested in, and then I can get the Descriptor of the input type of the method, but at that point I would like to create a builder for the input message. Should I also maintain a MapString,Descriptor One thing I thought of would be to go back to the file descriptor associated with the input message type descriptor, check the two java options, and use java reflection. would this work ? It is a shame the java class name is not the same as the message type full name. True, in C++ there is the global pool which is automatically populated with everything compiled into the binary. This wouldn't work in Java because classes are not loaded until they are first accessed, so there is no way to populate this pool. Besides, singletons are evil. I honestly wish that I'd never introduced the global pool in C++; it has lead to too many subtle singleton problems. 2009/12/9 Jason Hsueh jas...@google.com mailto:jas...@google.com No, there isn't an equivalent to the DescriptorPool in Java. If you know the types that you want you can build a mapping yourself. Or if you'd be interested in porting the C++ DescriptorPool to java that would be great! 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr Hello, Given a service/method full name, I'd like to get hold of the ServiceDescriptor, or MethodDescriptor. In C++ I would use the DescriptorPool, but I don't see this in the java api. Is there a way ? Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/Gq7i : ohloh |- http://tr.im/FtUu : new package : highlight `- http://tr.im/EAD5 : LondonR slides -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protobuf rpc over http
2009/12/10 Romain François francoisrom...@free.fr On 12/09/2009 09:12 PM, Kenton Varda wrote: Coincidentally, last weekend I started working on an open source protobuf-based RPC system. Currently I am defining a socket-level protocol, but I also intend to support an HTTP-level protocol with optional JSON encoding to allow calls from Javascript. I stuck some totally undocumented code here: Thanks. My intention of having it over http is that it can communicate with other languages. I'd be good if we can synchronize our protocols. I need to make some changes based on what you said on another thread, and then I'll make my java basic server code available. http://pbcap.googlecode.com But some coworkers pointed out that the name is confusingly similar to pcap, so I'm planning to change it. Currently this is not an official Google project; I'm working on it in my spare time. 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 I would recommend against putting the service type name in the URL. This makes it impossible to export two services of the same type. Instead, you should allow the application to register services under any name it chooses. Fair enough. Maybe adding some protobuf specific headers : ProtobufService: {service full name} You don't need to send the service name at all. The server should already know what kind of service it is exporting. ProtobufMethod: {method full name} You do need the method name, though. Inventing new HTTP headers isn't usually a good idea as they may confuse proxies and such. or encode it as parameters of the url as you said. I'd also suggest making the method name be part of the query, like: POST /someservice?method={method name} This may be a matter of taste, but I feel like a service object should be a single HTTP resource, rather than have each method be a separate resource. Connection: close Why not allow pipelining? this was simpler to do a one shot service call, but indeed why not. Content-Length: {length of the serialized message} {raw bytes of the serialized message} - And a successful response looks like this: - HTTP/1.1 200 OK Content-length: {length of the serialized response} {raw bytes of the serialized response} - Since this is very early in this, I wondered if others would have views on this. http is quite verbose for sending protobuf message around, but it is likely to be implemented for a lot of languages. Regards, Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/Gq7i : ohloh |- http://tr.im/FtUu : new package : highlight `- http://tr.im/EAD5 : LondonR slides -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] does java api have something like DescriptorPool
On 12/10/2009 10:13 AM, Kenton Varda wrote: Is this for the RPC system? yes I think you're making this a lot harder than it needs to be. ah. On the server side, when someone wants to export a service, they will construct a service implementation and pass it to your system. The com.google.protobuf.Service interface has methods getDescriptor(), getRequestPrototype(), and getResponsePrototype(), to get the ServiceDescriptor, request message class, and response message class respectively. So, you don't need to consult any tables. Right. That's what I needed. Thanks. Sorry. On the client side, your RPC implementation should merely provide the user with an object implementing com.google.protobuf.RpcChannel. The caller then calls MyServiceType.newStub(channel) to construct a type-safe stub. The RPC system does not need to know what kind of service it is. I'm not using this since you said this will disapear at some point... 2009/12/10 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr On 12/09/2009 09:00 PM, Kenton Varda wrote: Actually I don't think we need DescriptorPool in Java. DescriptorPool was primarily written for the purpose of memory management, but Java handles that for us. If all you need is the mapping aspect, just build a MapString, ServiceDescriptor yourself and use it. This is close to what I do now. Next question is : can I get hold of the java class that is associated with a Descriptor. From the service descriptor, I can get to the MethodDescriptor I am interested in, and then I can get the Descriptor of the input type of the method, but at that point I would like to create a builder for the input message. Should I also maintain a MapString,Descriptor One thing I thought of would be to go back to the file descriptor associated with the input message type descriptor, check the two java options, and use java reflection. would this work ? It is a shame the java class name is not the same as the message type full name. True, in C++ there is the global pool which is automatically populated with everything compiled into the binary. This wouldn't work in Java because classes are not loaded until they are first accessed, so there is no way to populate this pool. Besides, singletons are evil. I honestly wish that I'd never introduced the global pool in C++; it has lead to too many subtle singleton problems. 2009/12/9 Jason Hsueh jas...@google.com mailto:jas...@google.com mailto:jas...@google.com mailto:jas...@google.com No, there isn't an equivalent to the DescriptorPool in Java. If you know the types that you want you can build a mapping yourself. Or if you'd be interested in porting the C++ DescriptorPool to java that would be great! 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr Hello, Given a service/method full name, I'd like to get hold of the ServiceDescriptor, or MethodDescriptor. In C++ I would use the DescriptorPool, but I don't see this in the java api. Is there a way ? Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/Gq7i : ohloh |- http://tr.im/FtUu : new package : highlight `- http://tr.im/EAD5 : LondonR slides -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protobuf rpc over http
2009/12/10 Romain François francoisrom...@free.fr Cool. Do you have your layout documented somewhere in your project. Embed service full name and method name in the http request as we do allows the request to be self-contained. What kenton suggests needs the client to know of the mapping between the service full name (in pb speech) and the http service name, which might then need an independant exchange before hand between the client and the server . No, that's not what I'm suggesting. The client doesn't start from I want to use FooService.. The client starts from I want to talk to http://example.com/foo-service, which I already know is of type FooService.. It doesn't make sense for the client to ask the server where to find a FooService, because the server might expose *multiple* FooServices. The client has to know which one it wants to talk to. The URL identifies that. client: how do you call this service server: foo client: sending the http request using foo Either way is fine. It would be good to all agree on what rpc over http means. On 12/10/2009 10:15 AM, Marc Gravell wrote: The layout in the original post looks a lot like how protobuf-net does RPC over http. It uses {root}/service/method, but it would be trivial to tweak it to use any similar pattern. I'd be happy to make tweaks to protobuf-net to make it play the same game, and try a few cross-implementation scenarios. Marc 2009/12/10 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr On 12/09/2009 09:12 PM, Kenton Varda wrote: Coincidentally, last weekend I started working on an open source protobuf-based RPC system. Currently I am defining a socket-level protocol, but I also intend to support an HTTP-level protocol with optional JSON encoding to allow calls from Javascript. I stuck some totally undocumented code here: Thanks. My intention of having it over http is that it can communicate with other languages. I'd be good if we can synchronize our protocols. I need to make some changes based on what you said on another thread, and then I'll make my java basic server code available. http://pbcap.googlecode.com But some coworkers pointed out that the name is confusingly similar to pcap, so I'm planning to change it. Currently this is not an official Google project; I'm working on it in my spare time. 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 I would recommend against putting the service type name in the URL. This makes it impossible to export two services of the same type. Instead, you should allow the application to register services under any name it chooses. Fair enough. Maybe adding some protobuf specific headers : ProtobufService: {service full name} ProtobufMethod: {method full name} or encode it as parameters of the url as you said. I'd also suggest making the method name be part of the query, like: POST /someservice?method={method name} This may be a matter of taste, but I feel like a service object should be a single HTTP resource, rather than have each method be a separate resource. Connection: close Why not allow pipelining? this was simpler to do a one shot service call, but indeed why not. Content-Length: {length of the serialized message} {raw bytes of the serialized message} - And a successful response looks like this: - HTTP/1.1 200 OK Content-length: {length of the serialized response} {raw bytes of the serialized response} - Since this is very early in this, I wondered if others would have views on this. http is quite verbose for sending protobuf message around, but it is likely to be implemented for a lot of languages. Regards, Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/Gq7i : ohloh |- http://tr.im/FtUu : new package : highlight `- http://tr.im/EAD5 : LondonR slides -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to
Re: [protobuf] protobuf rpc over http
On 12/10/2009 11:13 AM, Kenton Varda wrote: 2009/12/10 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr Cool. Do you have your layout documented somewhere in your project. Embed service full name and method name in the http request as we do allows the request to be self-contained. What kenton suggests needs the client to know of the mapping between the service full name (in pb speech) and the http service name, which might then need an independant exchange before hand between the client and the server . No, that's not what I'm suggesting. The client doesn't start from I want to use FooService.. The client starts from I want to talk to http://example.com/foo-service, which I already know is of type FooService.. It doesn't make sense for the client to ask the server where to find a FooService, because the server might expose *multiple* FooServices. The client has to know which one it wants to talk to. The URL identifies that. Ah. I think I understand now. For me the method full name uniquely identifies the method and the server is generic, serving a service/method if it knows about it. Anyway, for those interested my code is here: $ svn checkout svn://svn.r-forge.r-project.org/svnroot/rprotobuf/java $ ant $ ./start.sh Implementing a method involves subclassing the ProtobufMethodInvoker class which is generic (on both input and output type). So for example with this service: service EchoService { rpc Echo (Person) returns (Person); } I define the EchoInvoker class like this : package org.rproject.rprotobuf ; import com.example.tutorial.AddressBookProtos.Person ; import com.google.protobuf.Message ; @MethodImplementation(tutorial.EchoService.Echo) public class EchoInvoker extends ProtobufMethodInvokerPerson,Person{ public Person invoke(Person person){ return person ; } public Person getInputDefaultInstance(){ return com.example.tutorial.AddressBookProtos.Person.getDefaultInstance() ; } } The method invoker must be registered using the ProtobufMethodPool.register method. For example : register(tutorial.EchoService.Echo, new EchoInvoker() ) ; I'm planning on adding some scanning based on the @MethodImplementation annotation. Romain client: how do you call this service server: foo client: sending the http request using foo Either way is fine. It would be good to all agree on what rpc over http means. On 12/10/2009 10:15 AM, Marc Gravell wrote: The layout in the original post looks a lot like how protobuf-net does RPC over http. It uses {root}/service/method, but it would be trivial to tweak it to use any similar pattern. I'd be happy to make tweaks to protobuf-net to make it play the same game, and try a few cross-implementation scenarios. Marc 2009/12/10 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr On 12/09/2009 09:12 PM, Kenton Varda wrote: Coincidentally, last weekend I started working on an open source protobuf-based RPC system. Currently I am defining a socket-level protocol, but I also intend to support an HTTP-level protocol with optional JSON encoding to allow calls from Javascript. I stuck some totally undocumented code here: Thanks. My intention of having it over http is that it can communicate with other languages. I'd be good if we can synchronize our protocols. I need to make some changes based on what you said on another thread, and then I'll make my java basic server code available. http://pbcap.googlecode.com But some coworkers pointed out that the name is confusingly similar to pcap, so I'm planning to change it. Currently this is not an official Google project; I'm working on it in my spare time. 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr mailto:francoisrom...@free.fr A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 I would recommend against putting the service type name in the URL. This makes it impossible to export two services of the same type.
Re: [protobuf] protobuf rpc over http
For info only, protobuf-net currently uses: internal const string HTTP_RPC_VERSION_HEADER = pb-net-rpc; internal const string HTTP_RPC_MIME_TYPE = application/x-protobuf ; (version for my own internal purposes, in case I need to change the body layout) Re the sockets point also raised; there's a lot of difference between raw sockets and http; it would be good to get some kind of official http transport working - people can always add raw later...? I'd rather not get bogged down in trying to predict every nuance of sockets, when people can always (for now) do their own thing using protocol buffers just for serialization. Marc 2009/12/10 Romain François francoisrom...@free.fr On 12/10/2009 04:30 PM, Pavel Shramov wrote: On Wed, Dec 09, 2009 at 12:10:33PM +0100, Romain François wrote: Hello, Following Kenton's advice, I'm starting to look at implementing protobuf rpc over http. I have started to work on a basic java server (based on the com.sun.net.httpserver class). I will post this at some point when I am happier with it (currently it can only serve one dummy service that returns the input message as is) A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 Connection: close Content-Length: {length of the serialized message} {raw bytes of the serialized message} - I'm using method name encoded in query part of URL like /base/url?Service.Method That seems odd. Why not /base/url?service=Servicemethod=Method instead ? Also it's seem useful to provide Content-Type to distinguish between different encodings of message (for example JSON). Yep. Will add this. And a successful response looks like this: - HTTP/1.1 200 OK Content-length: {length of the serialized response} {raw bytes of the serialized response} - Also it may be useful to state that errors are transmitted as body of 500 Internal Server Error response. For my implementation You may see [1] and [2] for python and C++ HTTP client. I will. So this makes 3 very similar http based protocols, but slightly different. We should come to an agreement. :-) Pavel [1] http://grid.pp.ru/wiki/pbufrpc [2] http://grid.pp.ru/cgit/pbufrpc -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/Gq7i : ohloh |- http://tr.im/FtUu : new package : highlight `- http://tr.im/EAD5 : LondonR slides -- Regards, Marc -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protobuf rpc over http
On 12/10/2009 05:30 PM, Marc Gravell wrote: For info only, protobuf-net currently uses: internal const string HTTP_RPC_VERSION_HEADER = pb-net-rpc; internal const string HTTP_RPC_MIME_TYPE = application/x-protobuf; (version for my own internal purposes, in case I need to change the body layout) Re the sockets point also raised; there's a lot of difference between raw sockets and http; it would be good to get some kind of official http transport working - people can always add raw later...? Totally. From your posts and Pavels, what about : request : - POST /{root}/{service full name}/{method name} HTTP/1.0 Content-Length: {length of the serialized message} Content-Type: application/x-protobuf {raw bytes of the serialized message} - (and some other headers) successful response : - HTTP/1.0 200 OK Content-Length: {length of the serialized response} Content-Type: application/x-protobuf {raw bytes of the serialized response} - error : - HTTP/1.0 500 Internal Server Error Content-Length: {length of error message} Content-Type: text/html {some error message} - Romain I'd rather not get bogged down in trying to predict every nuance of sockets, when people can always (for now) do their own thing using protocol buffers just for serialization. Marc 2009/12/10 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr On 12/10/2009 04:30 PM, Pavel Shramov wrote: On Wed, Dec 09, 2009 at 12:10:33PM +0100, Romain François wrote: Hello, Following Kenton's advice, I'm starting to look at implementing protobuf rpc over http. I have started to work on a basic java server (based on the com.sun.net.httpserver class). I will post this at some point when I am happier with it (currently it can only serve one dummy service that returns the input message as is) A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 Connection: close Content-Length: {length of the serialized message} {raw bytes of the serialized message} - I'm using method name encoded in query part of URL like /base/url?Service.Method That seems odd. Why not /base/url?service=Servicemethod=Method instead ? Also it's seem useful to provide Content-Type to distinguish between different encodings of message (for example JSON). Yep. Will add this. And a successful response looks like this: - HTTP/1.1 200 OK Content-length: {length of the serialized response} {raw bytes of the serialized response} - Also it may be useful to state that errors are transmitted as body of 500 Internal Server Error response. For my implementation You may see [1] and [2] for python and C++ HTTP client. I will. So this makes 3 very similar http based protocols, but slightly different. We should come to an agreement. :-) Pavel [1] http://grid.pp.ru/wiki/pbufrpc [2] http://grid.pp.ru/cgit/pbufrpc -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |- http://tr.im/Gq7i : ohloh |- http://tr.im/FtUu : new package : highlight `- http://tr.im/EAD5 : LondonR slides -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Inheritance Support in protobuf-net wiht WCF
Hello, I have two classes: [DataContract] public class TestClass: BaseTestClass { [DataMember(Order = 1)] public string Str { get; set; } } [DataContract] [ProtoInclude(1,typeof(TestClass))] public class BaseT { [DataMember] public T ReturnType { get; set; } } I know, that I must use ProtoInclude attribute to allow serialization. Problem is, that I have about 100 classes that inherit from Base class, and I should write 100 attributes. Is any possibility to add this information not using Attribute on class but programmatically? In DataContractSerializerOperationBehavior when you override method CreateSerializer you may write code to add your own list of known types. But.. ProtoInclude has not only information about type but also tag. can anyone help? Bartosz Pierzchlewicz -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protobuf rpc over http
On Wed, Dec 09, 2009 at 12:10:33PM +0100, Romain François wrote: Hello, Following Kenton's advice, I'm starting to look at implementing protobuf rpc over http. I have started to work on a basic java server (based on the com.sun.net.httpserver class). I will post this at some point when I am happier with it (currently it can only serve one dummy service that returns the input message as is) A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 Connection: close Content-Length: {length of the serialized message} {raw bytes of the serialized message} - I'm using method name encoded in query part of URL like /base/url?Service.Method Also it's seem useful to provide Content-Type to distinguish between different encodings of message (for example JSON). And a successful response looks like this: - HTTP/1.1 200 OK Content-length: {length of the serialized response} {raw bytes of the serialized response} - Also it may be useful to state that errors are transmitted as body of 500 Internal Server Error response. For my implementation You may see [1] and [2] for python and C++ HTTP client. Pavel [1] http://grid.pp.ru/wiki/pbufrpc [2] http://grid.pp.ru/cgit/pbufrpc -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protobuf rpc over http
On Thu, Dec 10, 2009 at 05:06:09PM +0100, Romain François wrote: I'm using method name encoded in query part of URL like /base/url?Service.Method That seems odd. Why not /base/url?service=Servicemethod=Method instead ? For simplicity (from my point of view). Query string is here only to identify method to call and not to pass parameters to it. So why to bother? Also it's seem useful to provide Content-Type to distinguish between different encodings of message (for example JSON). Yep. Will add this. I'm using application/x-protobuf here as protobuf-net does. I will. So this makes 3 very similar http based protocols, but slightly different. We should come to an agreement. :-) Definitely we have :) But it seem impossible :) Pavel -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Protocol Buffers using Lzip
One compression algo that I thought would be particularly useful with PB's would be LZO. It lines up nicely with PB's goals of being fast and compact. Have you thought about allowing an integrated LZO stream? --Chris On Wed, Dec 9, 2009 at 12:21 PM, Kenton Varda ken...@google.com wrote: Thanks for writing this code! I'm sure someone will find it useful. That said, I'm wary of adding random stuff to the protobuf library. gzip made sense because practically everyone has zlib, but lzlib is less universal. Also, if we have lzip support built-in, should we also support bzip, rar, etc.? Also note that libprotobuf is already much larger (in terms of binary footprint) than it should be, so making it bigger is a tricky proposition. And finally, on a non-technical note, adding any code to the protobuf distribution puts maintenance work on me, and I'm overloaded as it is. Usually I recommend that people set up a googlecode project to host code like this, but lzip_stream may be a bit small to warrant that. Maybe a project whose goal is to provide protobuf adaptors for many different compression formats? Or a project for hosting random protobuf-related utility code in general? On Tue, Dec 8, 2009 at 4:17 AM, Jacob Rief jacob.r...@gmail.com wrote: Hello Brian, hello Kenton, hello list, as an alternative to GzipInputStream and GzipOutputStream I have written a compression and an uncompression stream class which are stackable into Protocol Buffers streams. They are named LzipInputStream and LzipOutputStream and use the Lempel-Ziv-Markov chain algorithm, as implemented by LZIP http://www.nongnu.org/lzip/lzip.html An advantage for using Lzip instead of Gzip is, that Lzip supports multi member compression. So one can jump into the stream at any position, forward up to the next synchronization boundary and start reading from there. Using the default compression level, Lzip has a better compression ratio at the cost of being slower than Gzip, but when Lzip is used with a low compression level, speed and output size of Lzip are comparable to that of Gzip. I would like to donate these classes to the ProtoBuf software repository. They will be released under an OSS license, compatible to LZIP and Google's. Could someone please check them and tell me in what kind of repository I can publish them. In Google's license agreements there is a passage telling: Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. Since I have to use the name google in the C++ namespace of LzipIn/OutputStream, hereby I ask for permission to do so. Comments are appreciated, Jacob -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- Chris -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: protobuf rpc over http
Re the sockets point also raised; there's a lot of difference between raw sockets and http; it would be good to get some kind of official http transport working - people can always add raw later...? This is exactly the point I was trying to raise: if you bind Protocol Buffer RPC to a transport protocol then you can't easily use different transports as they might lack certain features. I mentioned TCP sockets just as an example. Here is my line of thought behind using a more generic (less HTTP- dependent) approach: 1) Not all protobufs target applications have an HTTP stack available. I know some applications that could use protobuf RPC but don't have HTTP available; 2) PB is a very efficient binary format so tying it up to a less than optimal text-based HTTP seems to be counterproductive; 3) It is a common practice to make RPC protocols self contained, and having a message carry all the information needed for its delivery (JSON RPC, SOAP, COBRA, Java RMI, etc.); 4) To achieve #3, no service or method name can be coded externally, no custom transport protocol features can be used either (HTTP headers); 5) Using protobuf to define header messages feels right as its easy to parse with the existing tools and it allows for a greater extensibility and soft versioning with optional fields; 6) Self contained RPC messages don't just enable using different transports. You can do a whole lot more: safely and easily persist messages, batch them which can greatly improve throughput; 7) It simplifies the RPC protocol specification. Header messages can be declared as a .proto file, people can compile it for their own platforms. It seems to be in line with the general protobuf philosophy. That's my 2c. Regards, On Dec 10, 10:30 am, Marc Gravell marc.grav...@gmail.com wrote: For info only, protobuf-net currently uses: internal const string HTTP_RPC_VERSION_HEADER = pb-net-rpc; internal const string HTTP_RPC_MIME_TYPE = application/x-protobuf ; (version for my own internal purposes, in case I need to change the body layout) Re the sockets point also raised; there's a lot of difference between raw sockets and http; it would be good to get some kind of official http transport working - people can always add raw later...? I'd rather not get bogged down in trying to predict every nuance of sockets, when people can always (for now) do their own thing using protocol buffers just for serialization. Marc 2009/12/10 Romain François francoisrom...@free.fr On 12/10/2009 04:30 PM, Pavel Shramov wrote: On Wed, Dec 09, 2009 at 12:10:33PM +0100, Romain François wrote: Hello, Following Kenton's advice, I'm starting to look at implementing protobuf rpc over http. I have started to work on a basic java server (based on the com.sun.net.httpserver class). I will post this at some point when I am happier with it (currently it can only serve one dummy service that returns the input message as is) A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 Connection: close Content-Length: {length of the serialized message} {raw bytes of the serialized message} - I'm using method name encoded in query part of URL like /base/url?Service.Method That seems odd. Why not /base/url?service=Servicemethod=Method instead ? Also it's seem useful to provide Content-Type to distinguish between different encodings of message (for example JSON). Yep. Will add this. And a successful response looks like this: - HTTP/1.1 200 OK Content-length: {length of the serialized response} {raw bytes of the serialized response} - Also it may be useful to state that errors are transmitted as body of 500 Internal Server Error response. For my implementation You may see [1] and [2] for python and C++ HTTP client. I will. So this makes 3 very similar http based protocols, but slightly different. We should come to an agreement. :-) Pavel [1]http://grid.pp.ru/wiki/pbufrpc [2]http://grid.pp.ru/cgit/pbufrpc -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |-http://tr.im/Gq7i: ohloh |-http://tr.im/FtUu: new package : highlight `-http://tr.im/EAD5: LondonR slides -- Regards, Marc -- Mikhail Opletayev http://dataflow-software.com -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protobuf rpc over http
On Thu, Dec 10, 2009 at 05:44:48PM +0100, Romain François wrote: - POST /{root}/{service full name}/{method name} HTTP/1.0 Content-Length: {length of the serialized message} Content-Type: application/x-protobuf {raw bytes of the serialized message} - I think that using query part for service and method is better than encoding it in URL. Btw it's possible to define something like 'calling conventions' for clients and tell client that it must use this or that. Pavel -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protobuf rpc over http
On Thu, Dec 10, 2009 at 06:23:14PM +0100, Romain François wrote: What about then if you want to control the kind of output that is returned back (pb or json). I would then add encoding=pb or encoding=json. How do you do this ? Everything is invented before us [1] :) To be fair in my implementation I send response encoded as request. Request encoding is read from Content-Type header and defaults to PB. I don't have any strong opinions. I think the best format is : /base/url/{service}?method={method} where service is the service full name, and method the method name within the service. So we have both URL and query encoding at once :) Pavel -- [1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3 -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Inheritance Support in protobuf-net wiht WCF
Hi Bartosz; At the moment, it only supports the reflection/attribute based approach (directly comparably to the [KnownType] attribute used in vanilla WCF / DataContractSerializer). I have some plans and prototype code to rectify this (runtime models), but it is lots of work, and it isn't anything I can promise on any fixed timescale I'm afraid. In order to shim inheritance into the protocol buffers format (without doing anything /too/ evil) it needs to know a unique tag per subclass. I can't help thinking that the most pragmatic option here is just to add a few attributes. Hunting for subclasses on the fly would be messy, and I can't include the type metadata without breaking completely from the standard protocol buffers wire format. Marc Gravell (protobuf-net) 2009/12/10 Bartosz Pierzchlewicz bar...@extranet.one.pl Hello, I have two classes: [DataContract] public class TestClass: BaseTestClass { [DataMember(Order = 1)] public string Str { get; set; } } [DataContract] [ProtoInclude(1,typeof(TestClass))] public class BaseT { [DataMember] public T ReturnType { get; set; } } I know, that I must use ProtoInclude attribute to allow serialization. Problem is, that I have about 100 classes that inherit from Base class, and I should write 100 attributes. Is any possibility to add this information not using Attribute on class but programmatically? In DataContractSerializerOperationBehavior when you override method CreateSerializer you may write code to add your own list of known types. But.. ProtoInclude has not only information about type but also tag. can anyone help? Bartosz Pierzchlewicz -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- Regards, Marc -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: protobuf rpc over http
Disclaimer: I'd be happy if once one of raw async RPC would be 'standart' but now maybe HTTP one will be. They just suppliment each other. On Thu, Dec 10, 2009 at 09:41:15AM -0800, Mikhail Opletayev wrote: Re the sockets point also raised; there's a lot of difference between raw sockets and http; it would be good to get some kind of official http transport working - people can always add raw later...? This is exactly the point I was trying to raise: if you bind Protocol Buffer RPC to a transport protocol then you can't easily use different transports as they might lack certain features. I mentioned TCP sockets just as an example. Here is my line of thought behind using a more generic (less HTTP- dependent) approach: Nobody argues that non-HTTP RPC must exists. But raw transport needs some kind of envelope (or header as in dataflow's pbuf-rpc) and thus a bit more complicated (as protocol). 1) Not all protobufs target applications have an HTTP stack available. I know some applications that could use protobuf RPC but don't have HTTP available; Then it's not place for RPC over HTTP :) But there is place for it. 2) PB is a very efficient binary format so tying it up to a less than optimal text-based HTTP seems to be counterproductive; Sometimes You need slow non-efficient protocol for stateless calls or when they are rare and must survive server restart. 3) It is a common practice to make RPC protocols self contained, and having a message carry all the information needed for its delivery (JSON RPC, SOAP, COBRA, Java RMI, etc.); BTW have you seen/used non-HTTP SOAP transports? :) Not on slides but in real life? 4) To achieve #3, no service or method name can be coded externally, no custom transport protocol features can be used either (HTTP headers); 5) Using protobuf to define header messages feels right as its easy to parse with the existing tools and it allows for a greater extensibility and soft versioning with optional fields; 6) Self contained RPC messages don't just enable using different transports. You can do a whole lot more: safely and easily persist messages, batch them which can greatly improve throughput; 7) It simplifies the RPC protocol specification. Header messages can be declared as a .proto file, people can compile it for their own platforms. It seems to be in line with the general protobuf philosophy. Simpliest RPC implementation is one over HTTP since only part that needs external specification is 'calling conventions' for mathod to URL translation. Pavel P.S. Sorry for flaming :) -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protobuf rpc over http
On Thu, Dec 10, 2009 at 07:56:49PM +0100, Romain François wrote: On 12/10/2009 06:31 PM, Pavel Shramov wrote: On Thu, Dec 10, 2009 at 06:23:14PM +0100, Romain François wrote: What about then if you want to control the kind of output that is returned back (pb or json). I would then addencoding=pb or encoding=json. How do you do this ? Everything is invented before us [1] :) Fair enough. I guess we could set Accept-encoding for that. To be fair in my implementation I send response encoded as request. Request encoding is read from Content-Type header and defaults to PB. I don't have any strong opinions. I think the best format is : /base/url/{service}?method={method} where service is the service full name, and method the method name within the service. So we have both URL and query encoding at once :) The rationale is this: it seems about right for a service to be bound to an url (whether the url uses the actual service full name or soime other key as per kenton's emails) and the method smells more like a parameter. My 'design' was affected by SOAP (Web-Services) concept of port types -- one service may implement different 'ports'. For example (not real life but like it) you have arigmetic service which consists of two ports: one of sum/substract functions and another of mul/div ones. It's one service with one endpoint but implementing two different sets of functions. In my opinion URL (path without query part) is more similar to host:port combination in raw transport (or endpoint int Web-Services terminology). So xxx?method=methodservice=service is more close to me. It is easy enough for me to encode the request in your format if I wanted to be able to interchange with your server, but is it not better to all use the same ... It is so we'll trying to find common point... As last resort we may define set of calling conventions for clients. Pavel -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protobuf rpc over http
Re the whole what should an endpoint url look like thing - I had a similar discussion with a user re protobuf-net; in the end it was quicker to just *default* to the former (since it doesn't need any extra specification), but *support* both - so the code detects key strings in the supplied url and works some magic. i.e. if I give it the endpoint: http://foo/foocorp; then it uses http://foo/foocorp/myservice/somemethod; but if I give it the endpoint http://foo/foocorp?svc={service}act={action} it uses http://foo/foocorp?svc=myserviceaction=somemethod; And so I can officially say I don't care about this discussion - either suits me without any code-changes ;-p Marc So xxx?method=methodservice=service is more close to me. It is easy enough for me to encode the request in your format if I wanted to be able to interchange with your server, but is it not better to all use the same ... It is so we'll trying to find common point... As last resort we may define set of calling conventions for clients. Pavel -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- Regards, Marc -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: protobuf rpc over http
On Thu, Dec 10, 2009 at 8:13 AM, Mikhail Opletayev opleta...@gmail.comwrote: It's great news that you working on a standard way to communicate between Protocol Buffers implementations! You don't need to send the service name at all. The server should already know what kind of service it is exporting. I think its handy to export several services from the same end point, especially if you are running RPC over something else than HTTP. If you were to run Protocol Buffers RPC over plain sockets you'd probably want to publish a bunch of services on the same port. This is exactly my point. If you use the service type name to identify the service, then you can only export one service of each type. Instead, some other name -- having nothing to do with the type name -- should be used to identify the service. Actually, the implementation I'm working on doesn't even identify services by names. Instead, when you first connect on a port, you connect to the default service for that port. However, the default service can send back pointers to other services in RPC responses. So, the default service may have a method which looks up other services by name, but this is up to the application. In Dataflow implementation we use one field (method) but we require the method name to be in serviceName.methodName format. For the same reason we decided against using HTTP headers to transfer the RPC metadata as it binds you to the transport protocol. That's why send content length and header length as the first 2 values in the coded stream, then the header message, then the actual data message: http://www.dataflow-software.com/docs/pbuf-rpc.html On Dec 10, 3:15 am, Kenton Varda ken...@google.com wrote: 2009/12/10 Romain François francoisrom...@free.fr On 12/09/2009 09:12 PM, Kenton Varda wrote: Coincidentally, last weekend I started working on an open source protobuf-based RPC system. Currently I am defining a socket-level protocol, but I also intend to support an HTTP-level protocol with optional JSON encoding to allow calls from Javascript. I stuck some totally undocumented code here: Thanks. My intention of having it over http is that it can communicate with other languages. I'd be good if we can synchronize our protocols. I need to make some changes based on what you said on another thread, and then I'll make my java basic server code available. http://pbcap.googlecode.com But some coworkers pointed out that the name is confusingly similar to pcap, so I'm planning to change it. Currently this is not an official Google project; I'm working on it in my spare time. 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 I would recommend against putting the service type name in the URL. This makes it impossible to export two services of the same type. Instead, you should allow the application to register services under any name it chooses. Fair enough. Maybe adding some protobuf specific headers : ProtobufService: {service full name} You don't need to send the service name at all. The server should already know what kind of service it is exporting. ProtobufMethod: {method full name} You do need the method name, though. Inventing new HTTP headers isn't usually a good idea as they may confuse proxies and such. or encode it as parameters of the url as you said. I'd also suggest making the method name be part of the query, like: POST /someservice?method={method name} This may be a matter of taste, but I feel like a service object should be a single HTTP resource, rather than have each method be a separate resource. Connection: close Why not allow pipelining? this was simpler to do a one shot service call, but indeed why not. Content-Length: {length of the serialized message} {raw bytes of the serialized message} - And a successful response looks like this: - HTTP/1.1 200 OK Content-length: {length of the serialized response} {raw bytes of the serialized response} - Since this is very early in this, I wondered if others would have views on this. http is quite verbose for sending protobuf message around, but it is likely to be implemented for a lot of languages. Regards, Romain -- Romain Francois Professional R Enthusiast +33(0) 6 28 91 30 30 http://romainfrancois.blog.free.fr |-http://tr.im/Gq7i: ohloh |-http://tr.im/FtUu: new package : highlight `-http://tr.im/EAD5:
Re: [protobuf] protoc generated .h file has errors in VS2008
The errors suggest to me that somewhere you have #defined Arc as a macro. Is this possible? On Thu, Dec 10, 2009 at 9:21 AM, bayWeiss bzwr...@yahoo.com wrote: I cant post the whole generated .h file, but below is a snippet where for some reason the compiler cant identify the Arc class. The declaration for Arc is definitely above the below c++ snippet, but it acts as if it cant recognize it. ---the errors are from VS2008-- error C4430: missing type specifier - int assumed. Note: C++ does not support default-int c:\googleprototest\googleprototestcppserver \person.pb.h669 GoogleProtoTestCPPServer error C2143: syntax error : missing ';' before '' c:\googleprototest \googleprototestcppserver\person.pb.h 669 error C2838: 'Arc' : illegal qualified name in member declaration c: \googleprototest\googleprototestcppserver\person.pb.h 669 error C4430: missing type specifier - int assumed. Note: C++ does not support default-int c:\googleprototest\googleprototestcppserver \person.pb.h669 -here is the generated header snippet from 2.2.0a protoc, within the Entity class declaration // optional .Square square = 3; inline bool has_square() const; inline void clear_square(); static const int kSquareFieldNumber = 3; inline const ::Square square() const; inline ::Square* mutable_square(); // optional .Arc arc = 4; inline bool has_arc() const; inline void clear_arc(); static const int kArcFieldNumber = 4; inline const ::Arc arc() const; // - the problem is here, line 669 inline ::Arc* mutable_arc(); // and here as well ---here is the .proto file- message Point { required double X = 1; required double Y = 2; } message Circle { required Point point1 = 1; required Point point2 = 2; required Point point3 = 3; required Point point4 = 4; } message Square { repeated Point pointList = 1; } message Arc { required Point center = 1; required double startAngle = 2; required double endAngle = 3; } message Parameter { required double somethingHere = 1; } message Entity { enum Type { CIRCLE = 1; SQUARE = 2; ARC = 3; } // Identifies which entity it is. required Type type = 1; optional Circle circle = 2; optional Square square = 3; optional Arc arc = 4; } message Job { repeated Entity entity = 1; } message Initialize { } message Start { } message MainMessage { enum Type { NEW_JOB = 1; INITIALIZE = 2; START = 3;} // Identifies which field is filled in. required Type type = 1; // One of the following will be filled in. optional Job job = 2; optional Initialize initialize = 3; optional Start start = 4; } -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: protobuf rpc over http
Interesting. Essentially a discovery service for protobuf RPC. I am not quite sure what you mean by pointers to other services. Is it going to reference them by name or a more complex structure containing full endpoint information? Also, is it going to be an extension to pbcap or something completely new? The reason I am asking is because some patterns in pbcap (such as wrapping up everything into a global Stream message) are rather questionable and not without consequences. Regards, On Dec 10, 4:22 pm, Kenton Varda ken...@google.com wrote: On Thu, Dec 10, 2009 at 8:13 AM, Mikhail Opletayev opleta...@gmail.comwrote: It's great news that you working on a standard way to communicate between Protocol Buffers implementations! You don't need to send the service name at all. The server should already know what kind of service it is exporting. I think its handy to export several services from the same end point, especially if you are running RPC over something else than HTTP. If you were to run Protocol Buffers RPC over plain sockets you'd probably want to publish a bunch of services on the same port. This is exactly my point. If you use the service type name to identify the service, then you can only export one service of each type. Instead, some other name -- having nothing to do with the type name -- should be used to identify the service. Actually, the implementation I'm working on doesn't even identify services by names. Instead, when you first connect on a port, you connect to the default service for that port. However, the default service can send back pointers to other services in RPC responses. So, the default service may have a method which looks up other services by name, but this is up to the application. In Dataflow implementation we use one field (method) but we require the method name to be in serviceName.methodName format. For the same reason we decided against using HTTP headers to transfer the RPC metadata as it binds you to the transport protocol. That's why send content length and header length as the first 2 values in the coded stream, then the header message, then the actual data message: http://www.dataflow-software.com/docs/pbuf-rpc.html On Dec 10, 3:15 am, Kenton Varda ken...@google.com wrote: 2009/12/10 Romain François francoisrom...@free.fr On 12/09/2009 09:12 PM, Kenton Varda wrote: Coincidentally, last weekend I started working on an open source protobuf-based RPC system. Currently I am defining a socket-level protocol, but I also intend to support an HTTP-level protocol with optional JSON encoding to allow calls from Javascript. I stuck some totally undocumented code here: Thanks. My intention of having it over http is that it can communicate with other languages. I'd be good if we can synchronize our protocols. I need to make some changes based on what you said on another thread, and then I'll make my java basic server code available. http://pbcap.googlecode.com But some coworkers pointed out that the name is confusingly similar to pcap, so I'm planning to change it. Currently this is not an official Google project; I'm working on it in my spare time. 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 I would recommend against putting the service type name in the URL. This makes it impossible to export two services of the same type. Instead, you should allow the application to register services under any name it chooses. Fair enough. Maybe adding some protobuf specific headers : ProtobufService: {service full name} You don't need to send the service name at all. The server should already know what kind of service it is exporting. ProtobufMethod: {method full name} You do need the method name, though. Inventing new HTTP headers isn't usually a good idea as they may confuse proxies and such. or encode it as parameters of the url as you said. I'd also suggest making the method name be part of the query, like: POST /someservice?method={method name} This may be a matter of taste, but I feel like a service object should be a single HTTP resource, rather than have each method be a separate resource. Connection: close Why not allow pipelining? this was simpler to do a one shot service call, but indeed why not. Content-Length: {length of the serialized message} {raw bytes of the serialized message} - And a successful response looks like this: - HTTP/1.1 200 OK Content-length: {length of the serialized
Re: [protobuf] Re: protobuf rpc over http
BTW, I haven't defined how pbcap/Captain Proto will work over HTTP yet. So, I'm only talking about the raw-socket protocol. On Thu, Dec 10, 2009 at 2:42 PM, Kenton Varda ken...@google.com wrote: On Thu, Dec 10, 2009 at 2:37 PM, Mikhail Opletayev opleta...@gmail.comwrote: Interesting. Essentially a discovery service for protobuf RPC. I am not quite sure what you mean by pointers to other services. Is it going to reference them by name or a more complex structure containing full endpoint information? Currently it references them by an ID number that is tied to the particular connection. So, each time a new service object is returned on the connection, a new ID number is assigned to it. Also, is it going to be an extension to pbcap or something completely new? Not an extension -- this *is* pbcap. It supports this already. (Note that I'm planning to change the name to Captain Proto, aka capnproto, to avoid the confusion with pcap.) The reason I am asking is because some patterns in pbcap (such as wrapping up everything into a global Stream message) are rather questionable and not without consequences. What do you mean? The global Stream message exists only to define the protocol. It is not actually used at runtime -- individual messages are read from the stream one at a time. Regards, On Dec 10, 4:22 pm, Kenton Varda ken...@google.com wrote: On Thu, Dec 10, 2009 at 8:13 AM, Mikhail Opletayev opleta...@gmail.com wrote: It's great news that you working on a standard way to communicate between Protocol Buffers implementations! You don't need to send the service name at all. The server should already know what kind of service it is exporting. I think its handy to export several services from the same end point, especially if you are running RPC over something else than HTTP. If you were to run Protocol Buffers RPC over plain sockets you'd probably want to publish a bunch of services on the same port. This is exactly my point. If you use the service type name to identify the service, then you can only export one service of each type. Instead, some other name -- having nothing to do with the type name -- should be used to identify the service. Actually, the implementation I'm working on doesn't even identify services by names. Instead, when you first connect on a port, you connect to the default service for that port. However, the default service can send back pointers to other services in RPC responses. So, the default service may have a method which looks up other services by name, but this is up to the application. In Dataflow implementation we use one field (method) but we require the method name to be in serviceName.methodName format. For the same reason we decided against using HTTP headers to transfer the RPC metadata as it binds you to the transport protocol. That's why send content length and header length as the first 2 values in the coded stream, then the header message, then the actual data message: http://www.dataflow-software.com/docs/pbuf-rpc.html On Dec 10, 3:15 am, Kenton Varda ken...@google.com wrote: 2009/12/10 Romain François francoisrom...@free.fr On 12/09/2009 09:12 PM, Kenton Varda wrote: Coincidentally, last weekend I started working on an open source protobuf-based RPC system. Currently I am defining a socket-level protocol, but I also intend to support an HTTP-level protocol with optional JSON encoding to allow calls from Javascript. I stuck some totally undocumented code here: Thanks. My intention of having it over http is that it can communicate with other languages. I'd be good if we can synchronize our protocols. I need to make some changes based on what you said on another thread, and then I'll make my java basic server code available. http://pbcap.googlecode.com But some coworkers pointed out that the name is confusingly similar to pcap, so I'm planning to change it. Currently this is not an official Google project; I'm working on it in my spare time. 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 I would recommend against putting the service type name in the URL. This makes it impossible to export two services of the same type. Instead, you should allow the application to register services under any name it chooses. Fair enough. Maybe adding some protobuf specific headers : ProtobufService: {service full name} You don't need to send the service name at all. The server should already know what kind of service it is exporting. ProtobufMethod: {method full name}
[protobuf] Re: protobuf rpc over http
Interesting. Essentially a discovery service for protobuf RPC. I am not quite sure what you mean by pointers to other services. Is it going to reference them by name or a more complex structure containing full endpoint information? Also, is it going to be an extension to pbcap or something completely new? The reason I am asking is because some patterns in pbcap (such as wrapping up everything into a global Stream message) are rather questionable and not without consequences. Regards, On Dec 10, 4:22 pm, Kenton Varda ken...@google.com wrote: On Thu, Dec 10, 2009 at 8:13 AM, Mikhail Opletayev opleta...@gmail.comwrote: It's great news that you working on a standard way to communicate between Protocol Buffers implementations! You don't need to send the service name at all. The server should already know what kind of service it is exporting. I think its handy to export several services from the same end point, especially if you are running RPC over something else than HTTP. If you were to run Protocol Buffers RPC over plain sockets you'd probably want to publish a bunch of services on the same port. This is exactly my point. If you use the service type name to identify the service, then you can only export one service of each type. Instead, some other name -- having nothing to do with the type name -- should be used to identify the service. Actually, the implementation I'm working on doesn't even identify services by names. Instead, when you first connect on a port, you connect to the default service for that port. However, the default service can send back pointers to other services in RPC responses. So, the default service may have a method which looks up other services by name, but this is up to the application. In Dataflow implementation we use one field (method) but we require the method name to be in serviceName.methodName format. For the same reason we decided against using HTTP headers to transfer the RPC metadata as it binds you to the transport protocol. That's why send content length and header length as the first 2 values in the coded stream, then the header message, then the actual data message: http://www.dataflow-software.com/docs/pbuf-rpc.html On Dec 10, 3:15 am, Kenton Varda ken...@google.com wrote: 2009/12/10 Romain François francoisrom...@free.fr On 12/09/2009 09:12 PM, Kenton Varda wrote: Coincidentally, last weekend I started working on an open source protobuf-based RPC system. Currently I am defining a socket-level protocol, but I also intend to support an HTTP-level protocol with optional JSON encoding to allow calls from Javascript. I stuck some totally undocumented code here: Thanks. My intention of having it over http is that it can communicate with other languages. I'd be good if we can synchronize our protocols. I need to make some changes based on what you said on another thread, and then I'll make my java basic server code available. http://pbcap.googlecode.com But some coworkers pointed out that the name is confusingly similar to pcap, so I'm planning to change it. Currently this is not an official Google project; I'm working on it in my spare time. 2009/12/9 Romain François francoisrom...@free.fr mailto:francoisrom...@free.fr A request looks like this : - POST /{service full name}/{method name} HTTP/1.0 I would recommend against putting the service type name in the URL. This makes it impossible to export two services of the same type. Instead, you should allow the application to register services under any name it chooses. Fair enough. Maybe adding some protobuf specific headers : ProtobufService: {service full name} You don't need to send the service name at all. The server should already know what kind of service it is exporting. ProtobufMethod: {method full name} You do need the method name, though. Inventing new HTTP headers isn't usually a good idea as they may confuse proxies and such. or encode it as parameters of the url as you said. I'd also suggest making the method name be part of the query, like: POST /someservice?method={method name} This may be a matter of taste, but I feel like a service object should be a single HTTP resource, rather than have each method be a separate resource. Connection: close Why not allow pipelining? this was simpler to do a one shot service call, but indeed why not. Content-Length: {length of the serialized message} {raw bytes of the serialized message} - And a successful response looks like this: - HTTP/1.1 200 OK Content-length: {length of the serialized
[protobuf] Re: protobuf rpc over http
Sorry for the double post :/ I understand that you are talking about raw raw-sockets. From my perspective, it's a good thing as it doesn't tie the RPC protocol to any specific transport protocol. Currently it references them by an ID number that is tied to the particular connection. So, each time a new service object is returned on the connection, a new ID number is assigned to it. Will there be a way to directly connect to the service or a client must go thru the discovery service each time a connection is open? What do you mean? The global Stream message exists only to define the protocol. It is not actually used at runtime -- individual messages are read from the stream one at a time. From pbcap.proto it looked to me as all requests and responses must be wrapped inside a Stream message. I guess I'll need to take a look at the implementation before I ask more questions, I only had time to take a quick peak earlier today. P.S. I hope my questions don't derail this thread too much as the original question was about HTTP RPC. On Dec 10, 4:42 pm, Kenton Varda ken...@google.com wrote: On Thu, Dec 10, 2009 at 2:37 PM, Mikhail Opletayev opleta...@gmail.comwrote: Interesting. Essentially a discovery service for protobuf RPC. I am not quite sure what you mean by pointers to other services. Is it going to reference them by name or a more complex structure containing full endpoint information? Currently it references them by an ID number that is tied to the particular connection. So, each time a new service object is returned on the connection, a new ID number is assigned to it. Also, is it going to be an extension to pbcap or something completely new? Not an extension -- this *is* pbcap. It supports this already. (Note that I'm planning to change the name to Captain Proto, aka capnproto, to avoid the confusion with pcap.) The reason I am asking is because some patterns in pbcap (such as wrapping up everything into a global Stream message) are rather questionable and not without consequences. What do you mean? The global Stream message exists only to define the protocol. It is not actually used at runtime -- individual messages are read from the stream one at a time. Regards, On Dec 10, 4:22 pm, Kenton Varda ken...@google.com wrote: On Thu, Dec 10, 2009 at 8:13 AM, Mikhail Opletayev opleta...@gmail.com wrote: It's great news that you working on a standard way to communicate between Protocol Buffers implementations! You don't need to send the service name at all. The server should already know what kind of service it is exporting. I think its handy to export several services from the same end point, especially if you are running RPC over something else than HTTP. If you were to run Protocol Buffers RPC over plain sockets you'd probably want to publish a bunch of services on the same port. This is exactly my point. If you use the service type name to identify the service, then you can only export one service of each type. Instead, some other name -- having nothing to do with the type name -- should be used to identify the service. Actually, the implementation I'm working on doesn't even identify services by names. Instead, when you first connect on a port, you connect to the default service for that port. However, the default service can send back pointers to other services in RPC responses. So, the default service may have a method which looks up other services by name, but this is up to the application. In Dataflow implementation we use one field (method) but we require the method name to be in serviceName.methodName format. For the same reason we decided against using HTTP headers to transfer the RPC metadata as it binds you to the transport protocol. That's why send content length and header length as the first 2 values in the coded stream, then the header message, then the actual data message: http://www.dataflow-software.com/docs/pbuf-rpc.html On Dec 10, 3:15 am, Kenton Varda ken...@google.com wrote: 2009/12/10 Romain François francoisrom...@free.fr On 12/09/2009 09:12 PM, Kenton Varda wrote: Coincidentally, last weekend I started working on an open source protobuf-based RPC system. Currently I am defining a socket-level protocol, but I also intend to support an HTTP-level protocol with optional JSON encoding to allow calls from Javascript. I stuck some totally undocumented code here: Thanks. My intention of having it over http is that it can communicate with other languages. I'd be good if we can synchronize our protocols. I need to make some changes based on what you said on another thread, and then I'll make my java basic server code available. http://pbcap.googlecode.com But some coworkers pointed out that the name
[protobuf] Re: protoc generated .h file has errors in VS2008
Ok, some more info.. thanks to your help So I renamed Arc to ArcShape in the protofile, for example, and it all works fine. Funny thing is, if I rename Square to Polyline, I get the same errors. If I rename Polyline to PolyLine.. all is ok. Using Ellipse throws errors also... So, I guess I cant pin down exactly where the conflict is, but there must be some geometry objects included with the c++ console project type i created... thanks! On Dec 10, 4:36 pm, Kenton Varda ken...@google.com wrote: The errors suggest to me that somewhere you have #defined Arc as a macro. Is this possible? On Thu, Dec 10, 2009 at 9:21 AM, bayWeiss bzwr...@yahoo.com wrote: I cant post the whole generated .h file, but below is a snippet where for some reason the compiler cant identify the Arc class. The declaration for Arc is definitely above the below c++ snippet, but it acts as if it cant recognize it. ---the errors are from VS2008-- error C4430: missing type specifier - int assumed. Note: C++ does not support default-int c:\googleprototest\googleprototestcppserver \person.pb.h 669 GoogleProtoTestCPPServer error C2143: syntax error : missing ';' before '' c:\googleprototest \googleprototestcppserver\person.pb.h 669 error C2838: 'Arc' : illegal qualified name in member declaration c: \googleprototest\googleprototestcppserver\person.pb.h 669 error C4430: missing type specifier - int assumed. Note: C++ does not support default-int c:\googleprototest\googleprototestcppserver \person.pb.h 669 -here is the generated header snippet from 2.2.0a protoc, within the Entity class declaration // optional .Square square = 3; inline bool has_square() const; inline void clear_square(); static const int kSquareFieldNumber = 3; inline const ::Square square() const; inline ::Square* mutable_square(); // optional .Arc arc = 4; inline bool has_arc() const; inline void clear_arc(); static const int kArcFieldNumber = 4; inline const ::Arc arc() const; // - the problem is here, line 669 inline ::Arc* mutable_arc(); // and here as well ---here is the .proto file- message Point { required double X = 1; required double Y = 2; } message Circle { required Point point1 = 1; required Point point2 = 2; required Point point3 = 3; required Point point4 = 4; } message Square { repeated Point pointList = 1; } message Arc { required Point center = 1; required double startAngle = 2; required double endAngle = 3; } message Parameter { required double somethingHere = 1; } message Entity { enum Type { CIRCLE = 1; SQUARE = 2; ARC = 3; } // Identifies which entity it is. required Type type = 1; optional Circle circle = 2; optional Square square = 3; optional Arc arc = 4; } message Job { repeated Entity entity = 1; } message Initialize { } message Start { } message MainMessage { enum Type { NEW_JOB = 1; INITIALIZE = 2; START = 3;} // Identifies which field is filled in. required Type type = 1; // One of the following will be filled in. optional Job job = 2; optional Initialize initialize = 3; optional Start start = 4; } -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.- Hide quoted text - - Show quoted text - -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: protoc generated .h file has errors in VS2008
Maybe ask the compiler for the pre-processed output (sorry, I don't know the VS2008 flag), and search through that for the first appearance of those identifiers. That'll tell you what header they come from. Adding a package should fix this kind of problem by putting all of the messages in a c++ namespace. On Thu, Dec 10, 2009 at 3:36 PM, bayWeiss bzwr...@yahoo.com wrote: Ok, some more info.. thanks to your help So I renamed Arc to ArcShape in the protofile, for example, and it all works fine. Funny thing is, if I rename Square to Polyline, I get the same errors. If I rename Polyline to PolyLine.. all is ok. Using Ellipse throws errors also... So, I guess I cant pin down exactly where the conflict is, but there must be some geometry objects included with the c++ console project type i created... thanks! On Dec 10, 4:36 pm, Kenton Varda ken...@google.com wrote: The errors suggest to me that somewhere you have #defined Arc as a macro. Is this possible? On Thu, Dec 10, 2009 at 9:21 AM, bayWeiss bzwr...@yahoo.com wrote: I cant post the whole generated .h file, but below is a snippet where for some reason the compiler cant identify the Arc class. The declaration for Arc is definitely above the below c++ snippet, but it acts as if it cant recognize it. ---the errors are from VS2008-- error C4430: missing type specifier - int assumed. Note: C++ does not support default-int c:\googleprototest\googleprototestcppserver \person.pb.h669 GoogleProtoTestCPPServer error C2143: syntax error : missing ';' before '' c:\googleprototest \googleprototestcppserver\person.pb.h 669 error C2838: 'Arc' : illegal qualified name in member declaration c: \googleprototest\googleprototestcppserver\person.pb.h 669 error C4430: missing type specifier - int assumed. Note: C++ does not support default-int c:\googleprototest\googleprototestcppserver \person.pb.h669 -here is the generated header snippet from 2.2.0a protoc, within the Entity class declaration // optional .Square square = 3; inline bool has_square() const; inline void clear_square(); static const int kSquareFieldNumber = 3; inline const ::Square square() const; inline ::Square* mutable_square(); // optional .Arc arc = 4; inline bool has_arc() const; inline void clear_arc(); static const int kArcFieldNumber = 4; inline const ::Arc arc() const; // - the problem is here, line 669 inline ::Arc* mutable_arc(); // and here as well ---here is the .proto file- message Point { required double X = 1; required double Y = 2; } message Circle { required Point point1 = 1; required Point point2 = 2; required Point point3 = 3; required Point point4 = 4; } message Square { repeated Point pointList = 1; } message Arc { required Point center = 1; required double startAngle = 2; required double endAngle = 3; } message Parameter { required double somethingHere = 1; } message Entity { enum Type { CIRCLE = 1; SQUARE = 2; ARC = 3; } // Identifies which entity it is. required Type type = 1; optional Circle circle = 2; optional Square square = 3; optional Arc arc = 4; } message Job { repeated Entity entity = 1; } message Initialize { } message Start { } message MainMessage { enum Type { NEW_JOB = 1; INITIALIZE = 2; START = 3;} // Identifies which field is filled in. required Type type = 1; // One of the following will be filled in. optional Job job = 2; optional Initialize initialize = 3; optional Start start = 4; } -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com protobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.- Hide quoted text - - Show quoted text - -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups
Re: [protobuf] Re: protobuf rpc over http
On Dec 10, 2009, at 18:33 , Kenton Varda wrote: client: Open service Foo. client: When request 1 is complete, call method Bar() on the result. I'm curious: Why have the Open request at all? Why not just cal method Foo.Bar()? Are you planning on supporting some concept of sessions? Evan -- Evan Jones http://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: protobuf rpc over http
On Thu, Dec 10, 2009 at 4:07 PM, Evan Jones ev...@mit.edu wrote: On Dec 10, 2009, at 18:33 , Kenton Varda wrote: client: Open service Foo. client: When request 1 is complete, call method Bar() on the result. I'm curious: Why have the Open request at all? Why not just cal method Foo.Bar()? Are you planning on supporting some concept of sessions? Sorry, but I think you missed the whole point of the conversation. The idea is that when you make a connection (using my RPC system), you initially only get access to some default service. Typically that default service implements a method called Open() which returns references to other services. This way, the protocol itself does not need to have any concept of naming RPC services -- this can all be pushed to a higher layer. Evan -- Evan Jones http://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.