Ok, so yea, it was a good samaritan updating all the puppet agents via
satellite. No big deal on the clients, but that does break the server in a
very annoying way.
Thank you for the assistance!
On Thursday, November 9, 2023 at 12:23:05 PM UTC-8 James Millsap wrote:
> Migrating/upgrading this whole stack is on my list.
>
> A newer version of the puppet agent was installed on this server right
> before everything started failing. Main difference appears to be hiera
> 3.12 vs the 3.4 that was part of the 5.x agent. No idea how that happened
> yet, as no one was logged in and no remote execution was logged. I may
> just restore the whole VM and call it a day.
>
> Just in case, here is the output of --trace
>
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/rest.rb:36:in
>
> `rescue in find'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/rest.rb:7:in
>
> `find'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in
>
> `find'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:725:in
> `block in retrieve_new_catalog'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:567:in `block
> in thinmark'
> /opt/puppetlabs/puppet/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:566:in
> `thinmark'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:724:in
> `retrieve_new_catalog'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:82:in
> `retrieve_catalog'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:267:in
> `prepare_and_retrieve_catalog'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:428:in
> `run_internal'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:339:in
> `run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:83:in `block
> (6 levels) in run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in
> `override'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:82:in `block
> (5 levels) in run'
> /opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
> /opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:81:in `block
> (4 levels) in run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent/locker.rb:21:in
> `lock'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:71:in `block
> (3 levels) in run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:162:in
> `with_client'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:67:in `block
> (2 levels) in run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:127:in
> `run_in_fork'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:66:in `block
> in run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:172:in
> `controlled_run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:47:in `run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:433:in
>
> `onetime'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:393:in
>
> `block in run_command'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in
> `override'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:390:in
>
> `run_command'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:421:in
> `block in run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:756:in
> `exit_on_fail'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:421:in
> `run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in
>
> `run'
> /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in
> `execute'
> /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
>
> Warning: Not using cache on failed catalog
> Error: Could not retrieve catalog; skipping run
>
> On Thursday, November 9, 2023 at 8:25:05 AM UTC-8 Martin Alfke wrote:
>
>>
>> On 8. Nov 2023, at 20:17, James Millsap <jmill...@gmail.com> wrote:
>>
>> Thank you very much for the reply! We are running these versions...
>>
>> puppetserver version: 5.3.11
>> ruby 2.0.0p648
>> psych (2.0.0)
>> hiera 3.12.0
>>
>>
>> That is super outdated Puppet version.
>> Can you please check the puppetserver log file.
>> There should be a ruby stack trace which might give you a better idea on
>> what is broken.
>> Did you update any module recently?
>>
>>
>> On Wednesday, November 8, 2023 at 11:01:16 AM UTC-8 ben.ford wrote:
>>
>>> Hey James! That error sounds like you have an out-of-date Psych YAML
>>> gem.
>>>
>>> What versions of Puppet, Ruby, Hiera, and Psych are you using?
>>>
>>>
>>>
>>> On Nov 8, 2023, at 8:54 AM, James Millsap <jmill...@gmail.com> wrote:
>>>
>>> We have a fairly mature puppet instance along with foreman for
>>> classification. For some reason, yesterday, everything started failing
>>> with this error:
>>>
>>> Info: Using environment 'production'
>>> Info: Retrieving pluginfacts
>>> Info: Retrieving plugin
>>> Info: Loading facts
>>> Error: Could not retrieve catalog from remote server: Error 500 on
>>> SERVER: Server Error: Evaluation Error: Error while evaluating a Function
>>> Call, undefined method `safe_load' for Puppet::Util::Yaml:Module (file:
>>> /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 3,
>>> column: 1) on node hostname.company.com
>>> Warning: Not using cache on failed catalog
>>> Error: Could not retrieve catalog; skipping run
>>>
>>> Line 3 of the site.pp is just this:
>>> hiera_include('classes', '')
>>>
>>> Which has not changed in many years. There is either a bug that
>>> appeared from nowhere, or a syntax error buried somewhere. I have been
>>> going through our repository and rolling back every yaml change, but so far
>>> nothing has worked. If anyone has a clever way to get puppet to give me
>>> some more information so I can find this, I would be eternally grateful.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Puppet Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to puppet-users...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/puppet-users/cb051ca4-92a5-43c7-a7d0-9e8942b0af73n%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/puppet-users/cb051ca4-92a5-43c7-a7d0-9e8942b0af73n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>> *CAUTION:* This email originated from outside of the organization. Do
>>> not click on links or open attachments unless you recognize the sender and
>>> know the content is safe.
>>>
>>>
>>>
>>> This e-mail may contain information that is privileged or confidential.
>>> If you are not the intended recipient, please delete the e-mail and any
>>> attachments and notify us immediately.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to puppet-users...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/bf1b9d65-aa35-4f09-a445-6e0bce2ede12n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/puppet-users/bf1b9d65-aa35-4f09-a445-6e0bce2ede12n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/41abf246-fe52-47f3-9f7e-26344a9191f6n%40googlegroups.com.
