Jira (PDB-1721) Possible Memory leak with PuppetDB?
Title: Message Title Simon Oxwell commented on PDB-1721 Re: Possible Memory leak with PuppetDB? Wyatt Alt - the memory dump had 381 instances of org.postgresql.jdbc4.Jdbc4PreparedStatement. Which is a definite improvement. Seems a little high if we're not caching them? Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-1721) Possible Memory leak with PuppetDB?
Title: Message Title Simon Oxwell commented on PDB-1721 Re: Possible Memory leak with PuppetDB? Hi Wyatt Alt - sorry for the lack of updates. We set the cache size to 0 and restarted puppetdb on the 18th August. Just looking at the dashboard, it looks like JVM usage is peaking around 320MB, but GC-ing backing down to 160MB. We've taken a memory dump to look at, but I think we're not going to find any lingering shared statements this time. Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-1721) Possible Memory leak with PuppetDB?
Title: Message Title Simon Oxwell commented on PDB-1721 Re: Possible Memory leak with PuppetDB? Wyatt Alt I see that 3.0.2 is out and this issue is referenced in the release notes. Hurrah! But, the upgrade path to PuppetDB 3.x is pretty involved. I don't suppose there's any chance of a backport to the 2.3.x branch? Add Comment This message was sent by Atlassian JIRA (v6.4.5#64020-sha1:78acd6c) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-1721) Possible Memory leak with PuppetDB?
Title: Message Title Simon Oxwell commented on PDB-1721 Re: Possible Memory leak with PuppetDB? Wyatt Alt Thanks for the update. Not entirely sure how large our biggest structured facts are, but probably not quite as large Richard Chatterton package lists. Richard Chatterton - thanks for additions to the ticket. Glad to know its not just my site. Add Comment This message was sent by Atlassian JIRA (v6.4.5#64020-sha1:78acd6c) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-1721) Possible Memory leak with PuppetDB?
Title: Message Title Simon Oxwell created an issue PuppetDB / PDB-1721 Possible Memory leak with PuppetDB? Issue Type: Bug Affects Versions: PDB 2.3.4 Assignee: Unassigned Created: 2015/07/02 12:36 AM Environment: CentOS7, Postgres 9.2.7 Priority: Normal Reporter: Simon Oxwell We're seeing PuppetDB regularly run out of memory every 3-4 weeks still. I did report something similar back in PDB-1484 , but that was when we had troubles with 'replace facts' failing ( PDB-1448 ). We're getting this in our logs, as a first sign: 2015-07-02 08:05:05,553 ERROR [c.p.p.command] [08271ea0-ad7c-42c7-a0e5-3810fa668118]
Jira (PDB-1448) 'replace facts' failing due to foreign key constraint issue
Title: Message Title Simon Oxwell commented on PDB-1448 Re: 'replace facts' failing due to foreign key constraint issue Yep - looking good. Thanks! (now to figure out how to clear the DLO) Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-1484) Memory leak when 'replace facts' failing
Title: Message Title Simon Oxwell commented on PDB-1484 Re: Memory leak when 'replace facts' failing Hi Kenneth, Pretty sure that this is only happening because the failures we're seeing as a result of PDB-1448 , which is resulting in a lot of failed SQL. I've updated to 2.3.4 and my SQL failures have gone, so I'm not expecting the problem to repeat. Should know in a couple of days. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-1448) 'replace facts' failing due to foreign key constraint issue
Title: Message Title Simon Oxwell commented on PDB-1448 Re: 'replace facts' failing due to foreign key constraint issue Hi Wyatt, I've put the new packages in, and it seems to be going well so far! Will update again in a few hours. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-1484) Memory leak when 'replace facts' failing
Title: Message Title Simon Oxwell commented on PDB-1484 Re: Memory leak when 'replace facts' failing Yes, two, but they're on the large size (~900M). Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-1484) Memory leak when 'replace facts' failing
Title: Message Title Simon Oxwell created an issue PuppetDB / PDB-1484 Memory leak when 'replace facts' failing Issue Type: Bug Affects Versions: PDB 2.3.3 Assignee: Unassigned Created: 2015/05/05 11:51 PM Environment: CentOS 7, Postgres 9.2.7 Priority: Normal Reporter: Simon Oxwell Hi, Just as an aside to PDB-1448 , we're also seeing PuppetDB run out of heap. Looking at the memory dumps, we see a very large number of Postgres Prepared Statements, taking up huge amounts of memory. For example, from today we had: 4,881 instances of org.postgresql.jdbc4.Jdbc4PreparedStatement with retained size of 500,866,312 bytes and one from a month ago: 4,614 instances of org.postgresql.jdbc4.Jdbc4PreparedStatement with retained size of 617,294,101 bytes. Our heap size is 750M, and 109 puppet agents. My guess is that when the 'replace facts' fails with the foreign key constraint violation, it doesn't clean up the prepared statement objects.
Jira (PDB-1448) 'replace facts' failing due to foreign key constraint issue
Title: Message Title Simon Oxwell commented on PDB-1448 Re: 'replace facts' failing due to foreign key constraint issue Hi Wyatt, Ah, great. That's awesome. Glad to be of help. Look forward to trying out the fix. Simon Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-1448) 'replace facts' failing due to foreign key constraint issue
Title: Message Title Simon Oxwell commented on PDB-1448 Re: 'replace facts' failing due to foreign key constraint issue Hi Wyatt - sorry I haven't been able to get this over the last few days, had other operational priorities. I've now got a new server playing up, and it's not being blocked by a structured fact: puppetdb=# DELETE FROM fact_values fv puppetdb-#WHERE fv.id in (2347921,5398,5421,1525955,2374767,2301540,2373170,5416,2301540,5407,2374770,2374768,2374770,2374767,2374769,5398,2374771) puppetdb-# AND NOT EXISTS (SELECT 1 FROM facts f puppetdb(#WHERE f.fact_value_id in (2347921,5398,5421,1525955,2374767,2301540,2373170,5416,2301540,5407,2374770,2374768,2374770,2374767,2374769,5398,2374771) puppetdb(# AND f.fact_value_id = fv.id puppetdb(# AND (f.factset_id, f.fact_path_id) NOT in ((93,148),(93,79),(93,4731),(93,4723),(93,131),(93,68),(93,48),(93,4720),(93,27),(93,4836),(93,127),(93,86),(93,40),(93,103),(93,142),(93,92),(93,105))); ERROR: update or delete on table fact_values violates foreign key constraint fact_value_id_fk on table facts DETAIL: Key (id)=(5407) is still referenced from table facts.
Jira (PDB-1448) 'replace facts' failing due to foreign key constraint issue
Title: Message Title Simon Oxwell commented on PDB-1448 Re: 'replace facts' failing due to foreign key constraint issue Hi Wyatt, I've now got 5 servers affected. I've collected the blocking fact values ids from the postgres logs, and they all seem to be related to our structured collections: puppetdb=# select factset_id,fact_value_id,value_hash,name,path from facts f inner join fact_values fv on f.fact_value_id=fv.id inner join fact_paths fp on f.fact_path_id=fp.id where f.fact_value_id in (807,810,1788,2709,2731,5697,1524348,1546444,1547903,1549732,1552056) order by fact_value_id; factset_id | fact_value_id |value_hash|name| path +---+--++-- 4 | 807 | eedbf60502cdb6b60552f2044fe16fcf6483e077 | funnelback_collections_profiles_structured | funnelback_collections_profiles_structured#~59#~collection 4 | 807 | eedbf60502cdb6b60552f2044fe16fcf6483e077 | funnelback_collections_structured | funnelback_collections_structured#~59 4 | 810 | cd339c27e31eff5c2c390c4c4e688d8ff1f60763 | funnelback_collections_profiles_structured | funnelback_collections_profiles_structured#~586#~collection 4 | 810 | cd339c27e31eff5c2c390c4c4e688d8ff1f60763 | funnelback_collections_structured | funnelback_collections_structured#~586
Jira (PDB-1448) 'replace facts' failing due to foreign key constraint issue
Title: Message Title Simon Oxwell commented on PDB-1448 Re: 'replace facts' failing due to foreign key constraint issue Hi Wyatt, Currently I have four servers that aren't updating, so I've got four different 'replace facts' errors repeating in the logs, so there's four distinct fact_set_id's from the NOT EXISTS IN subquery that match those four servers. The fact _value_id's listed are consistent for each server. The query outputs: puppetdb=# select * from facts f inner join fact_values fv on f.fact_value_id=fv.id inner join fact_paths fp on f.fact_path_id=fp.id where f.fact_value_id=1524348; factset_id | fact_path_id | fact_value_id | id| value_type_id |value_hash| value_integer | value_float | value_string | value_boolean | value_json | id | depth |name| path +--+---+-+---+--+---+-+--+---++-+---++--- 36 | 139 | 1524348 | 1524348 | 0 | 5c11c98386f3c4e9fb8a59f6c8540d4db4cbf625 | | | cang-test| | | 139 | 2 | funnelback_collections_profiles_structured | funnelback_collections_profiles_structured#~4#~collection 36 | 46 | 1524348 | 1524348 | 0 | 5c11c98386f3c4e9fb8a59f6c8540d4db4cbf625 | | | cang-test| | | 46 | 1 | funnelback_collections_structured | funnelback_collections_structured#~4 (2 rows)
Jira (PDB-1448) 'replace facts' failing due to foreign key constraint issue
Title: Message Title
Simon Oxwell commented on PDB-1448
Re: 'replace facts' failing due to foreign key constraint issue
Hi Wyatt,
Here's the results of the two queries (I took the liberty of changing the second one from funnelback_collections_profiles_profiles_structured to funnelback_collections_structured to match the query from the previous update, otherwise it returned nothing)
I've anonymized some of the names too (to hide our server and client names), but consistently across the output of both commands.
curl -X GET http://localhost:8080/v4/nodes/server-name.search/facts/funnelback_collections_profiles_structured
[ {
value : [ {
collection : client1-intranet,
profiles : [ _default, _default_preview ]
}, {
collection : client2-youtube,
profiles : [ _default, _default_preview ]
}, {
Jira (PDB-1448) 'replace facts' failing due to foreign key constraint issue
Title: Message Title Simon Oxwell created an issue PuppetDB / PDB-1448 'replace facts' failing due to foreign key constraint issue Issue Type: Bug Affects Versions: PDB 2.3.3 Assignee: Unassigned Created: 2015/04/22 9:53 PM Environment: CentOS 7, Postgres 9.2.7 Priority: Normal Reporter: Simon Oxwell Hi, We're seeing an issue with our puppetdb since it was upgraded to 2.3.1, with some nodes not updating. We get errors like the following the the puppetdb.log 015-04-23 14:45:10,223 ERROR [c.p.p.command] [b7ba3fa5-c61b-4461-a2b5-4317d00a92c8] [replace facts] Retrying after attempt 4, due to: java.sql.BatchUpdateException: Batch entry 0 DELETE FROM fact_values fv
