Thanks, that worked! Terri
On Mon, May 24, 2010 at 4:46 PM, Kenton Varda <ken...@google.com> wrote: > My guess is that you're using a single CodedInputStream to read all your > input, repeatedly calling message.ParseFromCodedStream(). Instead, create a > new CodedInputStream for each message. If you construct it on the stack, > there is no significant overhead to doing this: > while (true) { > CodedInputStream stream(&input); > // read one message, or break if at EOF > } > > On Mon, May 24, 2010 at 12:21 PM, Terri <terri.k...@gmail.com> wrote: >> >> Hi, >> >> I've been struggling to figure out just exactly how to do the many >> smaller messages approach. I've implemented this strategy, which is >> working except for a byte limit problem: >> >> >> http://groups.google.com/group/protobuf/browse_thread/thread/038cc4ad000b4265/95981da7e07ce197?hide_quotes=no >> >> I also raised the byte limit using SetTotalBytesLimit to maxint. >> >> I use a python program to read my data form disk and package it up >> into messages that are roughly 110 bytes each. Then I pipe it to a C++ >> program that reads messages and crunches. But, I still have a problem >> because the total number of bytes of all my smaller messages is >> greater than maxint and the C++ fails to read when it hits the limit. >> >> I like the protobuf approach to passing data, I just need to remove >> that limit. >> >> What can I do? >> >> Thanks, >> Terri >> >> On May 17, 7:00 pm, Jason Hsueh <jas...@google.com> wrote: >> > There is a default byte size limit of 64MB when parsing protocol buffers >> > - >> > if a message is larger than that, it will fail to parse. This can be >> > configured if you really need to parse larger messages, but it is >> > generally >> > not recommended. Additionally, ByteSize() returns a 32-bit integer, so >> > there's an implicit limit on the size of data that can be serialized. >> > >> > You can certainly use protocol buffers in large data sets, but it's not >> > recommended to have your entire data set be represented by a single >> > message. >> > Instead, see if you can break it up into smaller messages. >> > >> > >> > >> > On Mon, May 17, 2010 at 1:05 PM, sanikumbh <saniku...@gmail.com> wrote: >> > > I wanted to get some opinion on large data sets and protocol buffers. >> > > Protocol Buffer project page by google says that for data > 1 >> > > megabytes, one should consider something different but they don’t >> > > mention what would happen if one crosses this limit. Are there any >> > > known failure modes when it comes to the large data sets? >> > > What are your observations, recommendations from your experience on >> > > this front? >> > >> > > -- >> > > You received this message because you are subscribed to the Google >> > > Groups >> > > "Protocol Buffers" group. >> > > To post to this group, send email to proto...@googlegroups.com. >> > > To unsubscribe from this group, send email to >> > > >> > > protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com> >> > > . >> > > For more options, visit this group at >> > >http://groups.google.com/group/protobuf?hl=en. >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Protocol Buffers" group. >> > To post to this group, send email to proto...@googlegroups.com. >> > To unsubscribe from this group, send email to >> > protobuf+unsubscr...@googlegroups.com. >> > For more options, visit this group >> > athttp://groups.google.com/group/protobuf?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To post to this group, send email to proto...@googlegroups.com. >> To unsubscribe from this group, send email to >> protobuf+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/protobuf?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.