Issue #14759 has been updated by Rich Rauenzahn.
So, I had to use the PSON parser to get this to reproduce:
I added this to one of our nodes temporarily:
$json = '[{"lookforme9": 9, "lookforme8": 8, "lookforme3": 3, "lookforme2":
2, "lookforme1": 1, "lookforme0": 0, "lookforme7": 7, "lookforme6": 6, "lookf
orme5": 5, "lookforme4": 4}, {"lookforme9": 19, "lookforme8": 18,
"lookforme3": 13, "lookforme2": 12, "lookforme1": 11, "lookforme0": 10,
"lookforme7": 17, "
lookforme6": 16, "lookforme5": 15, "lookforme4": 14}, {"lookforme9": 29,
"lookforme8": 28, "lookforme3": 23, "lookforme2": 22, "lookforme1": 21,
"lookforme0"
: 20, "lookforme7": 27, "lookforme6": 26, "lookforme5": 25, "lookforme4":
24}, {"lookforme9": 39, "lookforme8": 38, "lookforme3": 33, "lookforme2": 32,
"look
forme1": 31, "lookforme0": 30, "lookforme7": 37, "lookforme6": 36,
"lookforme5": 35, "lookforme4": 34}, {"lookforme9": 49, "lookforme8": 48,
"lookforme3": 43
, "lookforme2": 42, "lookforme1": 41, "lookforme0": 40, "lookforme7": 47,
"lookforme6": 46, "lookforme5": 45, "lookforme4": 44}, {"lookforme9": 59,
"lookform
e8": 58, "lookforme3": 53, "lookforme2": 52, "lookforme1": 51,
"lookforme0": 50, "lookforme7": 57, "lookforme6": 56, "lookforme5": 55,
"lookforme4": 54}, {"l
ookforme9": 69, "lookforme8": 68, "lookforme3": 63, "lookforme2": 62,
"lookforme1": 61, "lookforme0": 60, "lookforme7": 67, "lookforme6": 66,
"lookforme5": 6
5, "lookforme4": 64}, {"lookforme9": 79, "lookforme8": 78, "lookforme3":
73, "lookforme2": 72, "lookforme1": 71, "lookforme0": 70, "lookforme7": 77,
"lookfor
me6": 76, "lookforme5": 75, "lookforme4": 74}, {"lookforme9": 89,
"lookforme8": 88, "lookforme3": 83, "lookforme2": 82, "lookforme1": 81,
"lookforme0": 80, "
lookforme7": 87, "lookforme6": 86, "lookforme5": 85, "lookforme4": 84},
{"lookforme9": 99, "lookforme8": 98, "lookforme3": 93, "lookforme2": 92,
"lookforme1"
: 91, "lookforme0": 90, "lookforme7": 97, "lookforme6": 96, "lookforme5":
95, "lookforme4": 94}]'
$items = our_json_parse($json)
notice($items)
define iterate_list {
$x = $name['lookforme0']
file { "/tmp/tmp.$x":
ensure => file,
content => "hello world",
}
}
iterate_list { $items:
}
our_json_parse is basically this, although we add checks for exceptions, etc.
(untested...)
module Puppet::Parser::Functions
newfunction(:our_json_parse, :type => :rvalue, :doc => "
Parse json string and return result.
") do |args|
begin
PSON.parse(json)
end
end
Here is the state file, which has duplicates:
$ sudo grep lookforme /var/lib/puppet/state/state.yaml
"Ournode::Iterate_list[lookforme595lookforme292lookforme797lookforme898lookforme191lookforme999lookforme494lookforme393lookforme696lookforme090]":
"Ournode::Iterate_list[lookforme747lookforme848lookforme141lookforme949lookforme444lookforme343lookforme646lookforme040lookforme545lookforme242]":
"Ournode::Iterate_list[lookforme424lookforme525lookforme626lookforme727lookforme020lookforme121lookforme222lookforme323lookforme828lookforme929]":
"Ournode::Iterate_list[lookforme444lookforme545lookforme646lookforme747lookforme040lookforme141lookforme242lookforme343lookforme848lookforme949]":
"Ournode::Iterate_list[lookforme313lookforme616lookforme010lookforme515lookforme212lookforme717lookforme818lookforme111lookforme919lookforme414]":
"Ournode::Iterate_list[lookforme494lookforme595lookforme696lookforme797lookforme090lookforme191lookforme292lookforme393lookforme898lookforme999]":
"Ournode::Iterate_list[lookforme434lookforme535lookforme636lookforme737lookforme030lookforme131lookforme232lookforme333lookforme838lookforme939]":
"Ournode::Iterate_list[lookforme535lookforme232lookforme737lookforme838lookforme131lookforme939lookforme434lookforme333lookforme636lookforme030]":
"Ournode::Iterate_list[lookforme151lookforme959lookforme454lookforme353lookforme656lookforme050lookforme555lookforme252lookforme757lookforme858]":
"Ournode::Iterate_list[lookforme969lookforme464lookforme363lookforme666lookforme060lookforme565lookforme262lookforme767lookforme868lookforme161]":
"Ournode::Iterate_list[lookforme414lookforme515lookforme616lookforme717lookforme010lookforme111lookforme212lookforme313lookforme818lookforme919]":
"Ournode::Iterate_list[lookforme464lookforme565lookforme666lookforme767lookforme060lookforme161lookforme262lookforme363lookforme868lookforme969]":
"Ournode::Iterate_list[lookforme676lookforme070lookforme575lookforme272lookforme777lookforme878lookforme171lookforme979lookforme474lookforme373]":
"Ournode::Iterate_list[lookforme020lookforme525lookforme222lookforme727lookforme828lookforme121lookforme929lookforme424lookforme323lookforme626]":
"Ournode::Iterate_list[lookforme080lookforme585lookforme282lookforme787lookforme888lookforme181lookforme989lookforme484lookforme383lookforme686]":
"Ournode::Iterate_list[lookforme44lookforme55lookforme66lookforme77lookforme00lookforme11lookforme22lookforme33lookforme88lookforme99]":
"Ournode::Iterate_list[lookforme454lookforme555lookforme656lookforme757lookforme050lookforme151lookforme252lookforme353lookforme858lookforme959]":
"Ournode::Iterate_list[lookforme99lookforme44lookforme33lookforme66lookforme00lookforme55lookforme22lookforme77lookforme88lookforme11]":
"Ournode::Iterate_list[lookforme484lookforme585lookforme686lookforme787lookforme080lookforme181lookforme282lookforme383lookforme888lookforme989]":
"Ournode::Iterate_list[lookforme474lookforme575lookforme676lookforme777lookforme070lookforme171lookforme272lookforme373lookforme878lookforme979]":
$ sudo grep lookforme /var/lib/puppet/state/state.yaml | wc -l
20
The json was generated with:
#!/usr/bin/python
import json
keys = []
for k in range(0,10):
keys.append("lookforme%d" % k)
items = []
v = 0
for count in range(0,10):
d = {}
for key in keys:
d[key] = v
v += 1
items.append(d)
print json.dumps(items)
----------------------------------------
Bug #14759: state.yaml balooning when using dictionary as resource $name
https://projects.puppetlabs.com/issues/14759#change-64433
Author: Rich Rauenzahn
Status: Needs More Information
Priority: Normal
Assignee: Rich Rauenzahn
Category:
Target version:
Affected Puppet version:
Keywords:
Branch:
We had some puppet processes ballooning and I spent some time trying to figure
out why. Turns out our state.yaml is something like 500MB after the run.
Examining the state.yaml shows a particular resource ballooning that we use for
iterating over a moderately large list of dictionaries.
We use a custom function to generate a list of dictionaries, and then we pass
the dictionary to the resource, the resource picks apart the dictionary,
filters based on some keys, etc., and creates other resources.
I verified the custom function is behaving appropriately -- it creates
something around 180 items. The state.yaml balloons it up to about 400,000 -
kind of a cartesian product I think.
oversimplified example:
$list = generate_list()
define process_list {
if $list['foo'] == 'xxxx' {
file {
...
}
}
}
What I found in the state.yaml was each element (a dict) of $list stringified
and listed as Process_List[] multiple times, but the stringification listed the
dictionary in different orders, causing the same entry to be duplicated, but
represented as a different strinfication of the dict.
My guess is that internally Puppet is stringifying the dictionary in multiple
locations and this causes duplicate resources to be defined when creating the
state to be saved in the state.yaml (I don't think it is doing that for
determining resources to be created; otherwise we'd be seeing duplication
errors).
Anyway, I know it is kind of an odd case we're using here, but the ballooning
is so severe (~4GB) I wanted to report it.
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en.
