Re: [protobuf] Efficiently reading/writing multiple messages in C++
The fast path only works if the optimize_for option is set to SPEED in the proto file. If that's not the case, just use approach 2. If you enabled the optimize_for option and want to take advantage of the fast path, 3) seems to be a reasonable approach. The code should be similar to the implementation of WireFormatLite::WriteMessageMaybeToArray(). Maybe we can just add a WireFormatLite::WriteMessageNoTagMaybeToArray(). On Wed, Jun 26, 2013 at 8:17 AM, Hochhaus, Andrew ahochh...@samegoal.comwrote: Does anyone have any insight on Evan's related question below? On Thu, Jun 17, 2010 at 5:05 PM, Evan Jones ev...@mit.edu wrote: Related question: If I am writing length-prefixed values, what is the best way to do this? I think there are 3 approaches: 1. out.WriteVarint32(msg.ByteSize()); msg.SerializeToCodedStream(out); This approach will choose between the ToArray and regular versions of the Serialize method, as appropriate. However, it computes the ByteSize twice. 2. out.WriteVarint32(msg.ByteSize()); msg.SerializeWithCachedSizes(out); This computes the ByteSize once, but never uses the fast path ToArray method. 3. Some custom code that calls ByteSize once, then chooses between the fast and slow methods, as appropriate. Is there actually any significant difference in performance between SerializeWithCachedSizes and SerializeWithCachedSizesToArray, so it would be worth me actually doing this? -Andy -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] Increasing memory usage with Protobuf
Hi there
We have an issue with increasing memory usage of an application using
Protobuf. The application has been tested with Valgrind in Ubuntu to find
the reason for this increase. The results shown with Massif Visualiser is
shown below:
https://lh6.googleusercontent.com/-wmtsYjFOW_M/Ucrxa0J4E8I/ABw/9itx_i5csOI/s1600/Memory+usage.png
Here is a snapshot of the part of code highlighted by Valgrind:
https://lh4.googleusercontent.com/-C0QgKsXxnJo/Ucr4wEPtvdI/ACI/9cf44aekIsc/s1600/Snapshot%2360-Protobuf.png
The figure is suggesting that the issue is related to Protobuf library
functions, and further studies of the source code have strongened these
suspicions.
Here are the functions in ProtobufReceiver:
void ProtobufReceiver::runClient()
{
string receivedData;
while(!terminationRequested)
{
string tmpData;
*socket tmpData;
receivedData += tmpData;
while(parseContent(receivedData));
}
}
bool ProtobufReceiver::parseContent(string receivedData){
if(dataSize = packetSize){DataList *dl = new DataList();
string dataString = receivedData.substr(packetSize-messageSize, messageSize);
dl-ParseFromString(dataString);
...
...
if(dataSize packetSize){ retval = true;
receivedData = receivedData.substr(packetSize);}else{
receivedData.clear();}
}
}
The DataList is the protobuf message, and even though this is not seen deleted
here, it has been verified that every object is correctly deleted later in a
getMessage() function.
The memory usage has been shown to increase up to values of around 1GB, before
flattening out (increases at an almost negligible rate). At this stage, the
program behaviour still seems normal.
No crashes have been observed because of this, but it would still be nice to
figure out the reason for this behaviour.
Also, a strange thing is that when running the program on a newer version of
the HP EliteBook series, the issue seems to not appear. The program has been
runned on a few other computers
(mine is an EliteBook 8740w), on both Windows 7 and Ubuntu, with the same
memory issues. I guess this could imply that the issue is HW dependant.
We are using Protobuf version 2.4.1.
Does anybody have a clue what's going on here? Any feedback would be
appreciated.
Regards
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] Define messages at runtime (Java)
Hello, I am looking into protobufs and I have the following question: 1) Is it possible to define message structures at runtime? I have an application that sends out protobufs, but does not know how the message looks like at first (because they can change over time). So before sending, it would compose the message according to a structure (given field names and types). 2) Also I would like to send out the message structure to the receiving application. This way the receiving application could adapt to the messages sent by the first application. Is it possible to send such message structures with protobufs, as well? Of course I would create a message type for the exchange of message structure and use protoc for this (this message type will not change over time). So are those to cases possible? I have found https://groups.google.com/forum/#!topic/protobuf/bWlJaaMmx4Q but I have no idea how to create a message. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Byte Offsets into File with Multiple Messages
A little bit more info: So, if I loop over the entries in parsing the file and create a new CodedInputStream per entry (so that I don't hit any byte limit warnings) AND don't use seekg, everything is ok. Strangely, if I also put in the construction of a new IFStreamReader (which is an exact copy except for the name from IStreamInputStream) into the same loop, thus making a new ZeroCopyingInputStream and a new CodedInputStream per message, the parsing fails. So, in this case, constructing the new IStreamInputStream resets some state of some sort and causes problems. Another problem is when I use seekg to change the underlying ifstream. Even when only using a single IFStreamReader (a solution that worked previously) and using seekg to skip to the same byte location where the stream should be at, causes problems. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
