Re: [protobuf] Issues with Large Coded Stream Files?

To be clear, I do not encode the entire file! Each file contains many small messages, each of which is stored as a length delimited record. It is just that there are quite a few messages bundled in one file. I'm assuming that Evan's assessment still stand? Cheers, Nader On 6/3/2010 15:05 Kenton

Re: [protobuf] Issues with Large Coded Stream Files?

It is not a single object; I am writing into a coded output stream file which could grow to much larger than 2GB (it's more like 100GB). I also have to read from this file. Is there a performance hit in the above-mentioned scenario? Nader On 6/3/2010 15:03 Evan Jones writes: > On Jun 3, 2010, a

Re: [protobuf] Issues with Large Coded Stream Files?

On Thu, Jun 3, 2010 at 3:15 PM, Nader Salehi wrote: > To be clear, I do not encode the entire file! Each file contains many > small messages, each of which is stored as a length delimited record. > It is just that there are quite a few messages bundled in one file. > Right, that's what I was as

Re: [protobuf] Issues with Large Coded Stream Files?

Note that writing a 100GB file using CodedStream is probably a bad idea because: - Readers will have to read the entire file sequentially; they will not be able to seek to particular parts. - One bit of corruption anywhere in the file could potentially render the entire rest of the file unreadable.

Re: [protobuf] Issues with Large Coded Stream Files?

On Jun 3, 2010, at 15:29 , Nader Salehi wrote: It is not a single object; I am writing into a coded output stream file which could grow to much larger than 2GB (it's more like 100GB). I also have to read from this file. Is there a performance hit in the above-mentioned scenario? No, this shoul

Re: [protobuf] Issues with Large Coded Stream Files?

On Jun 3, 2010, at 14:18 , Nader Salehi wrote: I was told that coded streams have issues when they are larger than 2GB. Is it true, and, if so, what are the issues? If you have a single object that is 2GB in size, there are 32-bit integers that will overflow. However, provided that you cal

[protobuf] Issues with Large Coded Stream Files?

I was told that coded streams have issues when they are larger than 2GB. Is it true, and, if so, what are the issues? Cheers, Nader -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.