[protobuf] Max number of repeated fields

[Øyvind Bakken]

We have a protobuf message defined in the following way:

message ModuleEntry 
{
optional uint32 id = 1;
optional string name = 2;
optional uint32 type = 3;
repeated ParameterEntry parameter = 4;
repeated ModuleEntry subModules = 5;
optional uint32 usage = 6;
optional string description = 7;
}

The message ModuleEntry can include several other ModuleEntrys 
(submodules). One of these submodules can again include several 
ParameterEntrys:

message ParameterEntry
{
required uint32 id = 1;
optional ParameterDataType dataType = 2;
optional ParameterType type = 3 [default = MEASUREMENT];
optional string name = 4;
optional int32 integerData = 5;
optional double doubleData = 6;
optional string unit = 7;

enum DynamicSimulation
{
STATIC = 0;
DYNAMIC = 1;
}
enum ParameterType
{
MEASUREMENT = 0;
CONFIGURABLE = 1;
INPUT = 2;
}
}

Our issue is appearing when we are increasing the number of ParameterEntrys 
for a submodule over a certain limit (from 18 to 19). The program then 
crashes with the following error message:

 libprotobuf FATAL ../build/include/google/protobuf/repeated_field.h:659] 
CHECK failed: (index)  (size()): terminate called after throwing an 
instance of 'google::protobuf::FatalException'

what(): CHECK failed: (index)  (size()): 

I have searched our code to find if we have set any constraints on array 
size or similar, but it does not seem so. Is there any limitation to 
Protobuf that is causing this behavior, i.e. a max number of repeated 
fields? It seems that the number of repeated fields for messages that are 
not submodules can be higher though, without crashes occuring. Also, 
decreasing the number of the given submodule, or the number of its parent 
module (to decrease the total data size) have no effect.


We are using Protobuf version 2.4.1. (I guess this version is kind of 
outdated, is it recommendable to upgrade to the latest version?)


Regards

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

[protobuf] Increasing memory usage with Protobuf

[Øyvind Bakken]

Hi there

We have an issue with increasing memory usage of an application using 
Protobuf. The application has been tested with Valgrind in Ubuntu to find 
the reason for this increase. The results shown with Massif Visualiser is 
shown below:

https://lh6.googleusercontent.com/-wmtsYjFOW_M/Ucrxa0J4E8I/ABw/9itx_i5csOI/s1600/Memory+usage.png

Here is a snapshot of the part of code highlighted by Valgrind:

https://lh4.googleusercontent.com/-C0QgKsXxnJo/Ucr4wEPtvdI/ACI/9cf44aekIsc/s1600/Snapshot%2360-Protobuf.png

The figure is suggesting that the issue is related to Protobuf library 
functions, and further studies of the source code have strongened these 
suspicions. 

Here are the functions in ProtobufReceiver:

void ProtobufReceiver::runClient() 

{

string receivedData;

while(!terminationRequested)

{

string tmpData;

*socket  tmpData;

receivedData += tmpData;


while(parseContent(receivedData));

}

}

bool ProtobufReceiver::parseContent(string receivedData){




if(dataSize = packetSize){DataList *dl = new DataList();   
 string dataString = receivedData.substr(packetSize-messageSize, messageSize);  
  dl-ParseFromString(dataString);

...
...
   
if(dataSize  packetSize){   retval = true;   
receivedData = receivedData.substr(packetSize);}else{   
receivedData.clear();}
}

}

The DataList is the protobuf message, and even though this is not seen deleted 
here, it has been verified that every object is correctly deleted later in a 
getMessage() function.

The memory usage has been shown to increase up to values of around 1GB, before 
flattening out (increases at an almost negligible rate). At this stage, the 
program behaviour still seems normal.
No crashes have been observed because of this, but it would still be nice to 
figure out the reason for this behaviour. 

Also, a strange thing is that when running the program on a newer version of 
the HP EliteBook series, the issue seems to not appear. The program has been 
runned on a few other computers 
(mine is an EliteBook 8740w), on both Windows 7 and Ubuntu, with the same 
memory issues. I guess this could imply that the issue is HW dependant.

We are using Protobuf version 2.4.1.

Does anybody have a clue what's going on here? Any feedback would be 
appreciated.

Regards










-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.