Re: [Puppet Users] PuppetDB errors
Thanks Wyatt, I have sent the tarballs from a couple of instances where we are seeing this to your inbox. Cheers Paul On Friday, 21 November 2014 17:32:16 UTC, Wyatt Alt wrote: Manually deleting that row (id=22128) in fact_values might also do the trick if it's only one affected node/value, but wouldn't tell much about how you got to this state so could be leaving something broken. It would be interesting to get the discarded tarball in any case. On Fri, Nov 21, 2014 at 8:57 AM, Wyatt Alt wy...@puppetlabs.com javascript: wrote: Paul, Thanks for that. It looks like 22128 has not been cleaned up properly for some reason, and it's likely being skipped over in the gc process because the facts cleanup identifies values to delete from that column in the facts table. Are there failed commands in your /var/lib/puppetdb/mq/discarded directory you can tie this to? I'd like to take a look at that if you can send me a tarball. This issue would likely disappear if you dropped and recreated the puppetdb database and allowed the nodes to repopulate it, but that could take some maneuvering depending on your situation/data needs. Wyatt On 11/20/14 12:29 AM, Paul Seymour wrote: On Wednesday, 19 November 2014 17:56:23 UTC, Wyatt Alt wrote: Hey Paul, That's some kind of DB corruption. Shouldn't be happening. Do you have any sense of when this started or whether it was tied to a recent upgrade? Is there a stacktrace in your logs that you could gist? Does it happen on every agent run or only occasionally? Is it always the same path_id like in the other ticket? Also would you mind reporting the output of this? select * from facts where fact_value_id in (50319,22128); Thanks Wyatt, I cannot be sure if this is tied to an upgrade specifically but I just happened to be keeping an eye on the logs a little more lately (working on an ELK thing to monitoring them). A stack trace is available at https://gist.github.com/PsychoSid/b1a479d7d6fec7477263 The output from the select is:- $ psql psql (9.3.5) Type help for help. postgres=# \c puppetdb You are now connected to database puppetdb as user postgres. puppetdb=# select * from facts where fact_value_id in (50319,22128); factset_id | fact_value_id +--- 22 | 50319 (1 row) Hope this helps. Cheers Paul -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7baf1653-d0b4-4fcf-8bcb-039b43af99a5%40googlegroups.com https://groups.google.com/d/msgid/puppet-users/7baf1653-d0b4-4fcf-8bcb-039b43af99a5%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/abee638f-5e75-410a-8fe2-7ff5acc8b7b2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] PuppetDB errors
Paul, Thanks for that. It looks like 22128 has not been cleaned up properly for some reason, and it's likely being skipped over in the gc process because the facts cleanup identifies values to delete from that column in the facts table. Are there failed commands in your /var/lib/puppetdb/mq/discarded directory you can tie this to? I'd like to take a look at that if you can send me a tarball. This issue would likely disappear if you dropped and recreated the puppetdb database and allowed the nodes to repopulate it, but that could take some maneuvering depending on your situation/data needs. Wyatt On 11/20/14 12:29 AM, Paul Seymour wrote: On Wednesday, 19 November 2014 17:56:23 UTC, Wyatt Alt wrote: Hey Paul, That's some kind of DB corruption. Shouldn't be happening. Do you have any sense of when this started or whether it was tied to a recent upgrade? Is there a stacktrace in your logs that you could gist? Does it happen on every agent run or only occasionally? Is it always the same path_id like in the other ticket? Also would you mind reporting the output of this? select * from facts where fact_value_id in (50319,22128); Thanks Wyatt, I cannot be sure if this is tied to an upgrade specifically but I just happened to be keeping an eye on the logs a little more lately (working on an ELK thing to monitoring them). A stack trace is available at https://gist.github.com/PsychoSid/b1a479d7d6fec7477263 The output from the select is:- $ psql psql (9.3.5) Type help for help. postgres=# \c puppetdb You are now connected to database puppetdb as user postgres. puppetdb=# select * from facts where fact_value_id in (50319,22128); factset_id | fact_value_id +--- 22 | 50319 (1 row) Hope this helps. Cheers Paul -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com mailto:puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7baf1653-d0b4-4fcf-8bcb-039b43af99a5%40googlegroups.com https://groups.google.com/d/msgid/puppet-users/7baf1653-d0b4-4fcf-8bcb-039b43af99a5%40googlegroups.com?utm_medium=emailutm_source=footer. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/546F6EF0.6060402%40puppetlabs.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] PuppetDB errors
Manually deleting that row (id=22128) in fact_values might also do the trick if it's only one affected node/value, but wouldn't tell much about how you got to this state so could be leaving something broken. It would be interesting to get the discarded tarball in any case. On Fri, Nov 21, 2014 at 8:57 AM, Wyatt Alt wy...@puppetlabs.com wrote: Paul, Thanks for that. It looks like 22128 has not been cleaned up properly for some reason, and it's likely being skipped over in the gc process because the facts cleanup identifies values to delete from that column in the facts table. Are there failed commands in your /var/lib/puppetdb/mq/discarded directory you can tie this to? I'd like to take a look at that if you can send me a tarball. This issue would likely disappear if you dropped and recreated the puppetdb database and allowed the nodes to repopulate it, but that could take some maneuvering depending on your situation/data needs. Wyatt On 11/20/14 12:29 AM, Paul Seymour wrote: On Wednesday, 19 November 2014 17:56:23 UTC, Wyatt Alt wrote: Hey Paul, That's some kind of DB corruption. Shouldn't be happening. Do you have any sense of when this started or whether it was tied to a recent upgrade? Is there a stacktrace in your logs that you could gist? Does it happen on every agent run or only occasionally? Is it always the same path_id like in the other ticket? Also would you mind reporting the output of this? select * from facts where fact_value_id in (50319,22128); Thanks Wyatt, I cannot be sure if this is tied to an upgrade specifically but I just happened to be keeping an eye on the logs a little more lately (working on an ELK thing to monitoring them). A stack trace is available at https://gist.github.com/PsychoSid/b1a479d7d6fec7477263 The output from the select is:- $ psql psql (9.3.5) Type help for help. postgres=# \c puppetdb You are now connected to database puppetdb as user postgres. puppetdb=# select * from facts where fact_value_id in (50319,22128); factset_id | fact_value_id +--- 22 | 50319 (1 row) Hope this helps. Cheers Paul -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7baf1653-d0b4-4fcf-8bcb-039b43af99a5%40googlegroups.com https://groups.google.com/d/msgid/puppet-users/7baf1653-d0b4-4fcf-8bcb-039b43af99a5%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJDiH3EtXEdC-5MvuC-aOZxAYAvk9ja3ZOShtd6p5J6nPKoT_w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] PuppetDB errors
On Wednesday, 19 November 2014 17:56:23 UTC, Wyatt Alt wrote: Hey Paul, That's some kind of DB corruption. Shouldn't be happening. Do you have any sense of when this started or whether it was tied to a recent upgrade? Is there a stacktrace in your logs that you could gist? Does it happen on every agent run or only occasionally? Is it always the same path_id like in the other ticket? Also would you mind reporting the output of this? select * from facts where fact_value_id in (50319,22128); Thanks Wyatt, I cannot be sure if this is tied to an upgrade specifically but I just happened to be keeping an eye on the logs a little more lately (working on an ELK thing to monitoring them). A stack trace is available at https://gist.github.com/PsychoSid/b1a479d7d6fec7477263 The output from the select is:- $ psql psql (9.3.5) Type help for help. postgres=# \c puppetdb You are now connected to database puppetdb as user postgres. puppetdb=# select * from facts where fact_value_id in (50319,22128); factset_id | fact_value_id +--- 22 | 50319 (1 row) Hope this helps. Cheers Paul -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7baf1653-d0b4-4fcf-8bcb-039b43af99a5%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] PuppetDB errors
Hello, Getting a couple of errors thrown with PuppetDB 2014-11-19 08:03:49,120 ERROR [c.p.p.command] [22329b48-7c20-4ec4-beb2-15cd02e11bff] [replace facts] Retrying after attempt 8, due to: org.postgresql.util.PSQLException: ERROR: update or delete on table fact_paths violates foreign key constraint fact_values_path_id_fk on table fact_values Detail: Key (id)=(512) is still referenced from table fact_values. org.postgresql.util.PSQLException: ERROR: update or delete on table fact_paths violates foreign key constraint fact_values_path_id_fk on table fact_values Detail: Key (id)=(512) is still referenced from table fact_values. Which is similar to PDB-1031 albeit with a different fact (ours is a custom one not _timestamp). puppetdb=# select * from fact_paths where id = 512; id | value_type_id | depth | name | path -+---+---++ 512 | 0 | 0 | XX_ethernet_eth0_intralink_unit_id | XX_ethernet_eth0_intralink_unit_id (1 row) puppetdb=# select * from fact_values where path_id = 512; id | path_id | value_type_id |value_hash | value_integer | value_float | value_string | value_boolean | value_json ---+-+---+--+---+-+--+---+ 50319 | 512 | 0 | 80e54cb487aed67b98d7dc92feaac988bf95bcc7 | | | 107827 | | 22128 | 512 | 0 | 47e3876c8d334accbeb827775e728643581bbb83 | | | 107292 | | Is it just noise or something to be concerned about ? This is with v2.2.1 and PostGres 9.3.5 Thanks Paul -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/08bf4cb9-6c80-489d-a0ef-26627f395f5e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] PuppetDB errors
Hey Paul, That's some kind of DB corruption. Shouldn't be happening. Do you have any sense of when this started or whether it was tied to a recent upgrade? Is there a stacktrace in your logs that you could gist? Does it happen on every agent run or only occasionally? Is it always the same path_id like in the other ticket? Also would you mind reporting the output of this? select * from facts where fact_value_id in (50319,22128); Wyatt On 11/19/14 12:15 AM, Paul Seymour wrote: Hello, Getting a couple of errors thrown with PuppetDB 2014-11-19 08:03:49,120 ERROR [c.p.p.command] [22329b48-7c20-4ec4-beb2-15cd02e11bff] [replace facts] Retrying after attempt 8, due to: org.postgresql.util.PSQLException: ERROR: update or delete on table fact_paths violates foreign key constraint fact_values_path_id_fk on table fact_values Detail: Key (id)=(512) is still referenced from table fact_values. org.postgresql.util.PSQLException: ERROR: update or delete on table fact_paths violates foreign key constraint fact_values_path_id_fk on table fact_values Detail: Key (id)=(512) is still referenced from table fact_values. Which is similar to PDB-1031 albeit with a different fact (ours is a custom one not _timestamp). puppetdb=# select * from fact_paths where id = 512; id | value_type_id | depth | name | path -+---+---++ 512 | 0 | 0 | XX_ethernet_eth0_intralink_unit_id | XX_ethernet_eth0_intralink_unit_id (1 row) puppetdb=# select * from fact_values where path_id = 512; id | path_id | value_type_id | value_hash| value_integer | value_float | value_string | value_boolean | value_json ---+-+---+--+---+-+--+---+ 50319 | 512 | 0 | 80e54cb487aed67b98d7dc92feaac988bf95bcc7 | | | 107827 | | 22128 | 512 | 0 | 47e3876c8d334accbeb827775e728643581bbb83 | | | 107292 | | Is it just noise or something to be concerned about ? This is with v2.2.1 and PostGres 9.3.5 Thanks Paul -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com mailto:puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/08bf4cb9-6c80-489d-a0ef-26627f395f5e%40googlegroups.com https://groups.google.com/d/msgid/puppet-users/08bf4cb9-6c80-489d-a0ef-26627f395f5e%40googlegroups.com?utm_medium=emailutm_source=footer. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/546CD9C0.2050308%40puppetlabs.com. For more options, visit https://groups.google.com/d/optout.
