On Tue, Jan 4, 2011 at 5:25 PM, Igor Gatis <igorga...@gmail.com> wrote:
> A while ago, a colleague had a "memory leak" reusing a PB message which > contained a repeated field. If I'm not mistaken the problem was that > pb_message::Clear() calls vector<something>::clear() and string::clear() > which does not really release the memory allocated. I can't really tell for > sure actually. > > @Kenton, does that make any sense? If yes, is there a way to avoid it? > As Evan says, this is by design. The memory is not "leaked" -- it will be reused when the message object is reused, and deleted when the message object is deleted. The actual problem that your colleague probably observed is that if you happen to parse one message which is much larger than usual, the object will allocate a bunch of memory for that one large message, and then will keep it around even after parsing smaller messages. So your memory usage is determined by the largest message you parse, rather than by the average. You can also run into problems if you have a message type whose instances vary widely in "shape". E.g. if type Foo has optional fields of type Bar and Baz, and you parse one instance of Foo that contains a Bar, then reuse the Foo to parse a message containing a Baz, then the Foo has allocated both Bar and Baz and will hold on to them. Thus the Foo is actually using more memory than was needed for either of the two messages it parsed. In practice these problems can manifest as memory usage that monotonically increases over the life of the process, although the rate of increase slows over time. A way to avoid this problem is to call SpaceUsed() to find out how much memory the object is using at any particular time. Once it crosses some threshold, delete the object and create a new one. Another approach is to reuse each objects at most N times -- this saves most of the allocation costs while preventing memory usage from growing without bound. Of course, all of this applies *only* to C++. Java protobuf objects are not reusable (since they are immutable), and in Python memory is discarded on Clear(). -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
