Jira (PUP-10775) Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog`

[Tim Meusel (Jira)]

Title: Message Title

Tim Meusel created an issue   Puppet / PUP-10775 Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog` Issue Type: Bug Assignee: Unassigned Created: 2020/11/05 9:17 AM Priority: Normal Reporter: Tim Meusel Hey Josh Cooper, I did further debugging for https://tickets.puppetlabs.com/browse/PUP-10772. while doing so I discovered that I've 5 Ubuntu 16.04 machines, all of them currently show the `Failed to deserialize Puppet::Resource::Catalog` error message that I show later my client: root~ # facter -p os pup{ architecture => "amd64", distro => { codename => "xenial", description => "Ubuntu 16.04.7 LTS", id => "Ubuntu", release => { full => "16.04", major => "16.04" } }, family => "Debian", hardware => "x86_64", name => "Ubuntu", release => { full => "16.04", major => "16.04" }, selinux => { enabled => false } } root~ # puppet --version 6.19.1 root ~ # preferred_serialization_format is not set: root ~ # cat /etc/puppetlabs/puppet/puppet.conf   [main] certname = * codedir = /etc/puppetlabs/code hostprivkey = $privatekeydir/$certname.pem { mode = 640 } logdir = /var/log/puppetlabs/puppet pluginfactsource = puppet:///pluginfacts pluginsource = puppet:///plugins privatekeydir = $ssldir/private_keys { group = service } rundir = /var/run/puppetlabs server = * show_diff = true ssldir = /etc/puppetlabs/puppet/ssl vardir = /opt/puppetlabs/puppet/cache   [agent] classfile = $statedir/classes.txt default_schedules = false environment = production localconfig = $vardir/localconfig masterport = 8140 noop = false report = true runinterval = 7200 splay = true splaylimit = 7200 usecacheonfailure = true root ~ # but msgpack is installed: root ~ # /opt/puppetlabs/puppet/bin/gem list msgpack   *** LOCAL GEMS ***   msgpack (1.3.3, 1.3.1) root ~ # during a puppetrun to a 6.14.1 puppetserver with msgpack installed I see the following messages: starting SSL for *:8140... SSL established <- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+json, application/json, application/vnd.puppet.rich+msgpack, application/x-msgpack, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 40356\r\n\r\n" ... Info: Unable to serialize catalog to json, retrying with pson ... the agent run works fine. Now the same agent connects to a puppetserver without msgpack installed: ... starting SSL for *:8140... SSL established <- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+json, application/json, application/vnd.puppet.rich+msgpack, application/x-msgpack, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 40357\r\n\r\n" ... -> "Content-Type: text/pson;charset=iso-8859-1\r\n" -> "Transfer-Encoding: chunked\r\n" -> "Connection: keep-alive\r\n" -> "X-Puppet-Version: 6.19.1\r\n" -> "Vary: Accept-Encoding, User-Agent\r\n" -> "Content-Encoding: gzip\r\n" ... Error: Could not retrieve catalog from remote server: Failed to deserialize Puppet::Resource::Catalog from pson: expected value in object at '"ELF\u0002\u0001\u0'! Wrapped exception: expected value in object at '"ELF\u0002\u0001\u0'! Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run now it fails? Sadly I don't know since when this is the case. I suspect someone added a new file resource with a binary that's read in with file(). But this is really just a guess. Our codebase is pretty huge and digging through all of the modules takes some time. I cannot reproduce this on other agents with a different operating system. Sadly the code on those Ubuntu 16.04 boxes does only work on this specific OS. I don't know if this Ubuntu specific or indeed some puppet code. Third run. msgpack installed again on the puppetserver. also enabled preferred_serialization_format = msgpack on the agent: starting SSL for *:8140... SSL established <- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+msgpack, application/x-msgpack, application/vnd.puppet.rich+json, application/json, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 40356\r\n\r\n" ... -> "Date: Thu, 05 Nov 2020 17:14:27 GMT\r\n" -> "Content-Type: application/vnd.puppet.rich+msgpack;charset=iso-8859-1\r\n" -> "Transfer-Encoding: chunked\r\n" -> "Connection: keep-alive\r\n" -> "X-Puppet-Version: 6.19.1\r\n" -> "Vary: Accept-Encoding, User-Agent\r\n" -> "Content-Encoding: gzip\r\n" ... Info: Unable to serialize catalog to json, retrying with pson .... the agent run works as well with this setup. but I think the info message is wrong. While it might try pson, afterwards msgpack is used to decode the payload? Or the msgpack serialized to msgpack? Add Comment   This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.377304.1604596654000.72973.1604596680053%40Atlassian.JIRA.