On Mon, Mar 14, 2011 at 10:34, ksamdev <samvel.khalat...@gmail.com> wrote: > Thanks for a quick reply. > Honestly, I fill a set of histograms for each event. I've added this feature > only recently and have a version of the code without histograms. > Here is the same performance measurement without histograms: > READING > ======= > Read ProtoBuf > Processed events: 1000000 > real 0m2.510s > user 0m2.105s > sys 0m0.298s > ---===--- > READING (MULTITHREADS) > ===================== > process files > init threads > start threads > run threads > Thread read 1000000 events > real 0m2.358s > user 0m2.085s > sys 0m0.236s > Again, the same situation. > My file is 384MB. I've already tested the use case with files above 1GB. It > turs out that ProtoBuf has a "int" limitation on file size. Anyway, I am a > way below the limit. The messages are pretty short (~400B).
Wait, where do you get the 1GB limit ? There is no real limit on the file size (in fact, the protocol buffer library does not do anything with files directly). (and: don't try to load everything in memory at once as a gigantic repeated message, that will give you a lot of malloc overhead). Another thing that could be limiting your throughput in your library is memory allocation: the system libraries are often not very good in handling threads well; I am usually using tcmalloc ( http://code.google.com/p/google-perftools/ ) which is handling that case pretty well. Also: if you read a lot of messages, it is a good idea to Clear() and re-use a message instead of allocating a new one (protocol buffers attempt to re-use memory internally). -h > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to protobuf@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
