[protobuf] Max number of repeated fields
We have a protobuf message defined in the following way: message ModuleEntry { optional uint32 id = 1; optional string name = 2; optional uint32 type = 3; repeated ParameterEntry parameter = 4; repeated ModuleEntry subModules = 5; optional uint32 usage = 6; optional string description = 7; } The message ModuleEntry can include several other ModuleEntrys (submodules). One of these submodules can again include several ParameterEntrys: message ParameterEntry { required uint32 id = 1; optional ParameterDataType dataType = 2; optional ParameterType type = 3 [default = MEASUREMENT]; optional string name = 4; optional int32 integerData = 5; optional double doubleData = 6; optional string unit = 7; enum DynamicSimulation { STATIC = 0; DYNAMIC = 1; } enum ParameterType { MEASUREMENT = 0; CONFIGURABLE = 1; INPUT = 2; } } Our issue is appearing when we are increasing the number of ParameterEntrys for a submodule over a certain limit (from 18 to 19). The program then crashes with the following error message: libprotobuf FATAL ../build/include/google/protobuf/repeated_field.h:659] CHECK failed: (index) (size()): terminate called after throwing an instance of 'google::protobuf::FatalException' what(): CHECK failed: (index) (size()): I have searched our code to find if we have set any constraints on array size or similar, but it does not seem so. Is there any limitation to Protobuf that is causing this behavior, i.e. a max number of repeated fields? It seems that the number of repeated fields for messages that are not submodules can be higher though, without crashes occuring. Also, decreasing the number of the given submodule, or the number of its parent module (to decrease the total data size) have no effect. We are using Protobuf version 2.4.1. (I guess this version is kind of outdated, is it recommendable to upgrade to the latest version?) Regards -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Max number of repeated fields
On Thu, Jan 30, 2014 at 6:48 AM, Øyvind Bakken oyv...@gmail.com wrote: We have a protobuf message defined in the following way: message ModuleEntry { optional uint32 id = 1; optional string name = 2; optional uint32 type = 3; repeated ParameterEntry parameter = 4; repeated ModuleEntry subModules = 5; optional uint32 usage = 6; optional string description = 7; } The message ModuleEntry can include several other ModuleEntrys (submodules). One of these submodules can again include several ParameterEntrys: message ParameterEntry { required uint32 id = 1; optional ParameterDataType dataType = 2; optional ParameterType type = 3 [default = MEASUREMENT]; optional string name = 4; optional int32 integerData = 5; optional double doubleData = 6; optional string unit = 7; enum DynamicSimulation { STATIC = 0; DYNAMIC = 1; } enum ParameterType { MEASUREMENT = 0; CONFIGURABLE = 1; INPUT = 2; } } Our issue is appearing when we are increasing the number of ParameterEntrys for a submodule over a certain limit (from 18 to 19). The program then crashes with the following error message: libprotobuf FATAL ../build/include/google/protobuf/repeated_field.h:659] CHECK failed: (index) (size()): terminate called after throwing an instance of 'google::protobuf::FatalException' what(): CHECK failed: (index) (size()): You should check the stack trace and see why it's called with an out-of-band index. I have searched our code to find if we have set any constraints on array size or similar, but it does not seem so. Is there any limitation to Protobuf that is causing this behavior, i.e. a max number of repeated fields? There is no such limit in protobuf. It seems that the number of repeated fields for messages that are not submodules can be higher though, without crashes occuring. Also, decreasing the number of the given submodule, or the number of its parent module (to decrease the total data size) have no effect. We are using Protobuf version 2.4.1. (I guess this version is kind of outdated, is it recommendable to upgrade to the latest version?) Regards -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Max number of repeated fields?
std::bad_alloc comes from your system's memory allocator, not from protocol buffers. I would guess that the round number is because it creates buckets for each allocation size, and you aren't allocating anything else that is the same size as B. So 0x100 is your system's bucket limit. You could try using tcmalloc (Google's memory allocator) instead. http://code.google.com/p/google-perftools/ Alternatively, if message B only has a single int32 field, then you could also try changing message A like: message A { repeated int32 b = 1 [packed=true]; } This would be far more efficient, both in memory and on the wire. On Thu, Aug 26, 2010 at 9:34 AM, Louis Marascio maras...@gmail.com wrote: Hi there, I have a simple message defined with a repeated field. The repeated field type is another message type. For example: Message A { repeated B b = 1; } Message B { required uint32 x = 1; } I'm building up a single A message in memory with quite a few B's. I am constantly receveiving a std::bad_alloc exception while trying to allocate the 16777217 or 0x101 'th msg. This number seems oddly round. Is there some builtin limit to the number of repeated values a repeated field may contain? Thanks, Louis -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Max number of repeated fields?
Hi there, I have a simple message defined with a repeated field. The repeated field type is another message type. For example: Message A { repeated B b = 1; } Message B { required uint32 x = 1; } I'm building up a single A message in memory with quite a few B's. I am constantly receveiving a std::bad_alloc exception while trying to allocate the 16777217 or 0x101 'th msg. This number seems oddly round. Is there some builtin limit to the number of repeated values a repeated field may contain? Thanks, Louis -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.