On Sun, Feb 7, 2010 at 10:24 AM, Brice Figureau < [email protected]> wrote:
> Hi, > > Following one of my blog post[1] about Puppetd memory consumption, I > decided to also try to add my stone to the edifice. > > Since #2892 is almost fixed by now, I decided to spend my time on > something else: > 1) json catalog (and its deserialization) > 2) sourced file content > > Those two things can consume memory for a short time (memory which > unfortunately > is always consumed, see my blog post for some explanations). > > So this patch brings to puppet: > > 1) support for a new Json parser: Yajl-ruby[1], which is faster and uses > less memory than the regular json parser. Yajl-ruby is available as a gem. > To activate the support it is needed to use > --preferred_serialization_format=yajl > > 2) use Yajl-ruby in "streaming" mode: puppetd never reads the whole > response > body in ram, but instead feeds Yajl-ruby chunk by chunk. This way, the json > serialized catalog is never fully stored in RAM. > > 3) allow the file resource source system to stream file content to disk > instead > of reading the whole response body in RAM and then dumping it on file. > > This patch has been lightly tested and seems to work, YMMV, though. I also > tried for the patch to be as minimal as possible. > > I took some figures while puppetd was sourcing a 100MiB file: > > Start Peak > 0.25.4 VSZ 612860 925772 > patched VSZ 611936 644104 > > 0.25.4 RES 12988 298104 > patched RES 11704 43536 > > Which I think speaks for itself :-) > :-O indeed! > > Now to be clear: this patch is a quick and dirty hack, I don't expect it to > be > merged as is, so I'm waiting reviews, tests, comments, flames. I even think > this patch brings a lot whole new serie of issues and bugs :-) > > Note: this patch introduces a new issue, regarding #2892, since we don't > have > anymore the whole serialized catalog in memory, it will be hard dump it as > is > to disk for the local cache. Something clever will need to be done (like > streaming > the cache...). > > Thanks, > Brice > > [1]: > http://www.masterzen.fr/2010/01/28/puppet-memory-usage-not-a-fatality/ > [2]: > http://github.com/brianmario/yajl-ruby > > Brice Figureau (5): > Add stream format capability > Add Yajl-ruby json parser format > Add a way to capture checksums on a "stream" > Let the rest client know how to handle stream response > Allows puppet:// content client streaming > > lib/puppet/feature/yajl.rb | 24 ++++ > lib/puppet/file_serving/content_stream.rb | 23 +++ > lib/puppet/indirector/file_content/rest.rb | 10 ++- > lib/puppet/indirector/rest.rb | 14 ++- > lib/puppet/network/deferred_response.rb | 94 +++++++++++++ > lib/puppet/network/format.rb | 4 + > lib/puppet/network/format_handler.rb | 23 +++- > lib/puppet/network/formats.rb | 112 +++++++++++++++ > lib/puppet/network/response_stream.rb | 45 ++++++ > lib/puppet/resource.rb | 2 +- > lib/puppet/type/file.rb | 19 +++- > lib/puppet/util/checksum_stream.rb | 20 +++ > lib/puppet/util/checksums.rb | 12 ++ > spec/integration/network/deferred_response.rb | 34 +++++ > spec/integration/network/formats.rb | 82 +++++++++++- > spec/unit/file_serving/content_stream.rb | 29 ++++ > spec/unit/indirector/file_content/rest.rb | 30 ++++- > spec/unit/indirector/rest.rb | 45 ++++--- > spec/unit/network/deferred_response.rb | 182 > +++++++++++++++++++++++++ > spec/unit/network/format.rb | 4 + > spec/unit/network/format_handler.rb | 33 +++++ > spec/unit/network/formats.rb | 110 +++++++++++++++- > spec/unit/network/response_stream.rb | 60 ++++++++ > spec/unit/type/file.rb | 32 +++++ > spec/unit/util/checksum_stream.rb | 23 +++ > spec/unit/util/checksums.rb | 12 ++ > 26 files changed, 1040 insertions(+), 38 deletions(-) > create mode 100644 lib/puppet/feature/yajl.rb > create mode 100644 lib/puppet/file_serving/content_stream.rb > create mode 100644 lib/puppet/network/deferred_response.rb > create mode 100644 lib/puppet/network/response_stream.rb > create mode 100644 lib/puppet/util/checksum_stream.rb > create mode 100644 spec/integration/network/deferred_response.rb > create mode 100644 spec/unit/file_serving/content_stream.rb > create mode 100644 spec/unit/network/deferred_response.rb > create mode 100644 spec/unit/network/response_stream.rb > create mode 100644 spec/unit/util/checksum_stream.rb > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<puppet-dev%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en. > > -- nigel -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
