Oh - and a copy of the current dead letter queue would be nice, its normally stored in:
/var/lib/puppetdb/mq/discarded/* This should also contain the full exceptions for the failed SQL as I mentioned earlier, so perhaps a glance into those now and letting me know what the prevalent failure is would be handy. I can organise a secure space on a Puppetlabs support storage area to upload this data if you are willing. Just contact me privately to organise it. ken. On Fri, Mar 1, 2013 at 2:25 PM, Ken Barber <k...@puppetlabs.com> wrote: > So I've been pondering this issue of yours, and I keep coming back to > that error in my mind: > > ERROR: insert or update on table "certname_catalogs" violates foreign > key constraint "certname_catalogs_catalog_fkey" > > Regardless of the other issues, 512 GB db - yes its big, but so what? > That shouldn't cause an integrity violation. It could be a big hint to > something else, but I'm not entirely sure this is the cause. > > This error - can we try to track down the corresponding PuppetDB error > message that goes with it in the puppetdb.log or in perhaps the dead > letter queue for activemq? If postgresql is throwing the error - then > there must be a corresponding exception in puppetdb.log. > > Its just that when I look at the code ... it doesn't make sense that > the insert fails here - "Key > (catalog)=(d1c89bbef78a55edcf560c432d965cfe1263059c) is not present in > table "catalogs". ... but I can't yet see a fault in the logic inside > the code yet. > > Still its going to be hard to replicate without the data on my side. > > As far as your issue ... you have a couple of avenues: > > * Drop the existing database, allow it to be recreated from scratch - > run noop a serveral times across your entire infrastructure to > repopulate it. Has the advantage of being an effective vacuum at the > same time :-). > * Rollback the PuppetDB version and the database. This will at least > get you back to a known working state, but will still leave you in a > place where you need to recover. > > Either case - you would want to take a backup of the existing database > for diagnostic purposes. But I really feel like there is something > structurally wrong here - perhaps the migration scripts have failed > during the database upgrade? > > But I'm guessing you can't stay broken for very long - so I would take > a snapshot of the current broken state so we can take a look and try > to replicate it, and make moves on getting back to a working state as > suggested above. > > I'll need: > > * PuppetDB logs > * Backup of your current DB > * Backup of your old DB > * The broken KahaDB queues > > ken. > > On Fri, Mar 1, 2013 at 9:19 AM, ak0ska <akos.he...@gmail.com> wrote: >> >> >> On Thursday, February 28, 2013 6:09:35 PM UTC+1, Ken Barber wrote: >>> >>> FYI, I just upgraded only the PuppetDB part to 1.1.1, using the old >>> 1.0.2 terminus I get no errors: >>> >>> 2013-02-28 17:06:27,711 WARN [qtp1478462104-39] [http.server] Use of >>> unversioned APIs is deprecated; please use /v1/commands >>> 2013-02-28 17:06:28,284 INFO [command-proc-44] [puppetdb.command] >>> [7a4a1f70-1e85-4256-8f95-21008b6c1199] [replace facts] puppetdb2.vm >>> 2013-02-28 17:06:30,724 WARN [qtp1478462104-32] [http.server] Use of >>> unversioned APIs is deprecated; please use /v1/commands >>> 2013-02-28 17:06:32,693 INFO [command-proc-44] [puppetdb.command] >>> [a87001c5-f064-4a2d-9f9e-de903f62f824] [replace catalog] puppetdb2.vm >>> >> >> As I was saying, we're not entirely sure that this behaviour was causes by >> the upgrade. We only noticed it one week after the upgrade (so unfortunately >> the postgres logs from before the update were rotated, that's the default >> setting), and it might be a coincidence. Then again. maybe not. :) >> >>> >>> If at all possible - I wouldn't mind a full copy of your puppetdb.log >>> ... to dig a bit deeper. And I know I told you to clear the KahaDB >>> queue (I always make this mistake) but I don't suppose you kept an old >>> copy of it? >> >> >> We kept the KahaDB backup both times we flushed the queue. The sizes in >> /var/lib/puppetdb/mq/localhost/ are as follows: >> >> 1.3G ./KahaDB >> 637M ./KahaDB.old2 >> 1.1G ./scheduler >> 6.2G ./KahaDB.old >> >> The big one is a 25 000 queue. >> >> I would happily send the logs over, but this is something I have discuss >> with my colleagues. >> >> Again, thank you for helping! :) >> >> ak0ska >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscr...@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
