Hi,
Following one of my blog post[1] about Puppetd memory consumption, I
decided to also try to add my stone to the edifice.
Since #2892 is almost fixed by now, I decided to spend my time on
something else:
1) json catalog (and its deserialization)
2) sourced file content
Those two things can consume memory for a short time (memory which unfortunately
is always consumed, see my blog post for some explanations).
So this patch brings to puppet:
1) support for a new Json parser: Yajl-ruby[1], which is faster and uses
less memory than the regular json parser. Yajl-ruby is available as a gem.
To activate the support it is needed to use
--preferred_serialization_format=yajl
2) use Yajl-ruby in "streaming" mode: puppetd never reads the whole response
body in ram, but instead feeds Yajl-ruby chunk by chunk. This way, the json
serialized catalog is never fully stored in RAM.
3) allow the file resource source system to stream file content to disk instead
of reading the whole response body in RAM and then dumping it on file.
This patch has been lightly tested and seems to work, YMMV, though. I also
tried for the patch to be as minimal as possible.
I took some figures while puppetd was sourcing a 100MiB file:
Start Peak
0.25.4 VSZ 612860 925772
patched VSZ 611936 644104
0.25.4 RES 12988 298104
patched RES 11704 43536
Which I think speaks for itself :-)
Now to be clear: this patch is a quick and dirty hack, I don't expect it to be
merged as is, so I'm waiting reviews, tests, comments, flames. I even think
this patch brings a lot whole new serie of issues and bugs :-)
Note: this patch introduces a new issue, regarding #2892, since we don't have
anymore the whole serialized catalog in memory, it will be hard dump it as is
to disk for the local cache. Something clever will need to be done (like
streaming
the cache...).
Thanks,
Brice
[1]:
http://www.masterzen.fr/2010/01/28/puppet-memory-usage-not-a-fatality/
[2]:
http://github.com/brianmario/yajl-ruby
Brice Figureau (5):
Add stream format capability
Add Yajl-ruby json parser format
Add a way to capture checksums on a "stream"
Let the rest client know how to handle stream response
Allows puppet:// content client streaming
lib/puppet/feature/yajl.rb | 24 ++++
lib/puppet/file_serving/content_stream.rb | 23 +++
lib/puppet/indirector/file_content/rest.rb | 10 ++-
lib/puppet/indirector/rest.rb | 14 ++-
lib/puppet/network/deferred_response.rb | 94 +++++++++++++
lib/puppet/network/format.rb | 4 +
lib/puppet/network/format_handler.rb | 23 +++-
lib/puppet/network/formats.rb | 112 +++++++++++++++
lib/puppet/network/response_stream.rb | 45 ++++++
lib/puppet/resource.rb | 2 +-
lib/puppet/type/file.rb | 19 +++-
lib/puppet/util/checksum_stream.rb | 20 +++
lib/puppet/util/checksums.rb | 12 ++
spec/integration/network/deferred_response.rb | 34 +++++
spec/integration/network/formats.rb | 82 +++++++++++-
spec/unit/file_serving/content_stream.rb | 29 ++++
spec/unit/indirector/file_content/rest.rb | 30 ++++-
spec/unit/indirector/rest.rb | 45 ++++---
spec/unit/network/deferred_response.rb | 182 +++++++++++++++++++++++++
spec/unit/network/format.rb | 4 +
spec/unit/network/format_handler.rb | 33 +++++
spec/unit/network/formats.rb | 110 +++++++++++++++-
spec/unit/network/response_stream.rb | 60 ++++++++
spec/unit/type/file.rb | 32 +++++
spec/unit/util/checksum_stream.rb | 23 +++
spec/unit/util/checksums.rb | 12 ++
26 files changed, 1040 insertions(+), 38 deletions(-)
create mode 100644 lib/puppet/feature/yajl.rb
create mode 100644 lib/puppet/file_serving/content_stream.rb
create mode 100644 lib/puppet/network/deferred_response.rb
create mode 100644 lib/puppet/network/response_stream.rb
create mode 100644 lib/puppet/util/checksum_stream.rb
create mode 100644 spec/integration/network/deferred_response.rb
create mode 100644 spec/unit/file_serving/content_stream.rb
create mode 100644 spec/unit/network/deferred_response.rb
create mode 100644 spec/unit/network/response_stream.rb
create mode 100644 spec/unit/util/checksum_stream.rb
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en.
