[protobuf] Bytes type limit of 4 GB

2021-06-11 Thread David Yat Sin
Hi, I am trying to store large buffers that can potentially grow to 32 GB, but when using the bytes type, I see the unpack fails when the size is just below 4GB. Is that a known limitation? Would you have a recommendation on how to pack buffers > 32 GB. proto file: ---

Re: [protobuf] Bytes type limit of 4 GB

2021-06-18 Thread 'Adam Cozzette' via Protocol Buffers
Actually protobuf messages must be strictly less than 2 GiB in size. It looks like you're using protobuf-c and perhaps that implementation will allow somewhat larger sizes, but the implementations maintained by Google enforce a 2 GiB limit. There is no real workaround except to break up the data in

Re: [protobuf] Bytes type limit of 4 GB

2021-06-18 Thread David Yat Sin
Ok. Thanks Adam for clarifying. Regards, David On Fri, Jun 18, 2021 at 6:42 PM Adam Cozzette wrote: > Actually protobuf messages must be strictly less than 2 GiB in size. It > looks like you're using protobuf-c and perhaps that implementation will > allow somewhat larger sizes, but the impleme