Re: [protobuf] Re: How to read continuous stream of messages from TCP
Have you tried using https://grpc.io/? Streaming is supported there:
https://grpc.io/docs/guides/concepts/
*From: *
*Date: *Mon, May 6, 2019 at 8:27 PM
*To: *Protocol Buffers
Hi,
>
>
> this problem can be solved in below way also.
>
> 1. By defining new message type.
>
> new_message {
> message a1:1;
> message a2:2;
> ...
> message a32:32
> }
>
> In proto3, all the fields are optional by default.
>
> so no need to encode everything. But the problem is we can define more
> then 32 messages because of 5-bit field value.
>
> But this problem can also be solved by using struct with in struct.
>
> new_mesasge {
>message 1-32 : 1;
>message 33-64 :2;
> ...
> }
>
> This way we no need to worry about Length encoding type. By using protobuf
> only w can solve this problem
>
> Thanks
> Satheesh
>
>
> On Tuesday, February 28, 2012 at 3:57:17 AM UTC+5:30, waynix wrote:
>>
>> Hello All;
>>
>> New to protobuf and did some limited search for my question. So my
>> apology if this has already been talked about.
>>
>> I naively thought that ParseFromFileDescriptor/ParseFromIstream would
>> block on an TCP socket and return when a valid message is received.
>> Read some old posts from 2010 and realized it's not that easy due to
>> mesages not being self-delimiting. And the suggestion from Jason
>> Hsueh was as follows:
>> "
>> One approach to writing multiple messages to the same stream is to use
>> a
>> length-delimited format: write the size of the message, then serialize
>> the
>> message itself. On the receiver side, you would set up a
>> FileInputStream,
>> and wrap a CodedInputStream around that. You can read the size of the
>> messages from the stream and then use PushLimit and PopLimit to
>> control how
>> much data is read.
>> "
>>
>> My quesions are:
>>
>> 1. Is this still the way to do it? Seems quite cumbersome (to lazy
>> me ;-). Is there a wrapper built in to do this?
>> 2. If I understand Jason's suggestion riht, the length is really not
>> part of the message, and the sender has to explcitly set it, instead
>> of having protobuf encode it in. Which means a generic third party
>> sender using my .proto file would not be sufficient. Plus how would
>> they know the length before encoding the message proper? Filling it in
>> after the fact would change the length again? or I am totally
>> missing it.
>>
>> 3. A related quesiton is in general do I have to manage reading of the
>> socket, or for that matter any istream, and spoon feed the protobuf
>> parser until it says OK, that's a whole message?
>>
>> Thanks a lot.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit
https://groups.google.com/d/msgid/protobuf/CAEWFqPZy9_nNANYWRygRbd3iE68S1oiF8ZxbrYsy8pcfgE9iYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
smime.p7s
Description: S/MIME Cryptographic Signature
[protobuf] Re: How to read continuous stream of messages from TCP
Hi,
this problem can be solved in below way also.
1. By defining new message type.
new_message {
message a1:1;
message a2:2;
...
message a32:32
}
In proto3, all the fields are optional by default.
so no need to encode everything. But the problem is we can define more then
32 messages because of 5-bit field value.
But this problem can also be solved by using struct with in struct.
new_mesasge {
message 1-32 : 1;
message 33-64 :2;
...
}
This way we no need to worry about Length encoding type. By using protobuf
only w can solve this problem
Thanks
Satheesh
On Tuesday, February 28, 2012 at 3:57:17 AM UTC+5:30, waynix wrote:
>
> Hello All;
>
> New to protobuf and did some limited search for my question. So my
> apology if this has already been talked about.
>
> I naively thought that ParseFromFileDescriptor/ParseFromIstream would
> block on an TCP socket and return when a valid message is received.
> Read some old posts from 2010 and realized it's not that easy due to
> mesages not being self-delimiting. And the suggestion from Jason
> Hsueh was as follows:
> "
> One approach to writing multiple messages to the same stream is to use
> a
> length-delimited format: write the size of the message, then serialize
> the
> message itself. On the receiver side, you would set up a
> FileInputStream,
> and wrap a CodedInputStream around that. You can read the size of the
> messages from the stream and then use PushLimit and PopLimit to
> control how
> much data is read.
> "
>
> My quesions are:
>
> 1. Is this still the way to do it? Seems quite cumbersome (to lazy
> me ;-). Is there a wrapper built in to do this?
> 2. If I understand Jason's suggestion riht, the length is really not
> part of the message, and the sender has to explcitly set it, instead
> of having protobuf encode it in. Which means a generic third party
> sender using my .proto file would not be sufficient. Plus how would
> they know the length before encoding the message proper? Filling it in
> after the fact would change the length again? or I am totally
> missing it.
>
> 3. A related quesiton is in general do I have to manage reading of the
> socket, or for that matter any istream, and spoon feed the protobuf
> parser until it says OK, that's a whole message?
>
> Thanks a lot.
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: How to read continuous stream of messages from TCP
On Mar 8, 2012, at 2:30 , waynix wrote: Since this is so common an issue and the suggested solution is almost de facto standard, (saw this after my initial post: http://code.google.com/apis/protocolbuffers/docs/techniques.html), it begs the question of why not build it into protobuf proper. Yeah, I would agree that something simple probably should have been included. The reasoning here is that this allows people to use protocol buffers with whatever other systems they might already be using (eg. HTTP, databases, files, RPC protocols, whatever), without being tied to a specific implementation. Compare the protocol buffer API to Thrift, for example, where the message serialization/deserialization is tied pretty tightly to the RPC system. There were proposals to possibly add a protocol buffer utils API, or a streaming API, but neither of those went anywhere. The closest thing is writeDelimitedTo / mergeDelimitedFrom in the Java API: http://code.google.com/apis/protocolbuffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo(java.io.OutputStream) Evan -- http://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: How to read continuous stream of messages from TCP
Thank you so much Evan for your response. While I look at your old posts and dig into coded stream (as I said, new to protobuf), it seems to me this approach is external to protobuf, in the sense that I have to tell the third party what my delimiter is (it's a length as in you sample code) or I have to give them a wrapper library on top of protobuf, to read that much data before invoking protobuf proper. Granted it's no big deal coding wise but it's an extra piece of deliverable. I can't simply hand my .proto file to customer, refer them to google's protobuf site and wash my hands off. Since this is so common an issue and the suggested solution is almost de facto standard, (saw this after my initial post: http://code.google.com/apis/protocolbuffers/docs/techniques.html), it begs the question of why not build it into protobuf proper. So that ParseFromIstream would block until reading that much data as indicated in the delimiter and decode and return. I see StartGroup and StopProup being deprecated, (Poof! there goes 25% of precious wire types ;-)). Makes you wonder if they were originally intended as delimiters. Which would not be a bad idea. You could have a StartGroup as an optional field in front of each of your top level message, which would be defined as some meta data or some header about the ensuing message, the simplest could be a length. The end group could be some CRC for unreliable media such as serial transmission. I know CRC may sound overreaching, but a default startGroup type being a length should be simple and generic enough and would solve a big problem. For more elaborate and custom Start/End definition, you could have some custom callback mechanism to interpret them. Thanks again. On Mar 6, 3:08 pm, Evan Jones ev...@csail.mit.edu wrote: On Feb 27, 2012, at 17:27 , waynix wrote: 1. Is this still the way to do it? Seems quite cumbersome (to lazy me ;-). Is there a wrapper built in to do this? Yes. Sadly there is no wrapper included in the library. 2. If I understand Jason's suggestion riht, the length is really not part of the message, and the sender has to explcitly set it, instead of having protobuf encode it in. Which means a generic third party sender using my .proto file would not be sufficient. Plus how would they know the length before encoding the message proper? Filling it in after the fact would change the length again? or I am totally missing it. As long as both sides encode the length in the same way , just having the right .proto will do the trick. 3. A related quesiton is in general do I have to manage reading of the socket, or for that matter any istream, and spoon feed the protobuf parser until it says OK, that's a whole message? Basically yes. There is a sketch of some example code here: https://groups.google.com/forum/?fromgroups#!searchin/protobuf/sequen... Good luck, Evan --http://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
