Re: [Puppet Users] Could not retrieve file metadata ... end of file reached
I am not convinced that this is to do with an agent being busy then attempting the next connection. Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Error: Could not retrieve catalog from remote server: end of file reached Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run The above all happened within a few seconds. I might argue within 4-5 seconds. Unless of course I'd misunderstood your hypothesis? On 13 March 2015 at 05:28, Josh Cooper j...@puppetlabs.com wrote: On Thu, Mar 12, 2015 at 4:21 AM, James Green james.mk.gr...@gmail.com wrote: I was running puppet agent -t --noop repeatedly with this error. Running again, this time omitting --noop, it succeeded. Not entirely clear what this tells me... On 12 March 2015 at 10:58, James Green james.mk.gr...@gmail.com wrote: Error: /Stage[main]/Our_unattended_upgrades/File[/etc/apt/apt.conf.d/20auto-upgrades]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/our_unattended_upgrades/etc/apt/apt.conf.d/20auto-upgrades: end of file reached Wrapped exception: end of file reached Versions on the server: ii facter 2.4.1-1puppetlabs1 all Ruby module for collecting simple facts about a host operating system ii hiera1.3.4-1puppetlabs1 all A simple pluggable Hierarchical Database. ii puppet 3.7.4-1puppetlabs1 all Centralized configuration management - agent startup and compatibility scripts ii puppet-common3.7.4-1puppetlabs1 all Centralized configuration management ii puppetdb 2.2.2-1puppetlabs1 all PuppetDB Centralized Storage. ii puppetdb-terminus2.2.2-1puppetlabs1 all Connect Puppet to PuppetDB by setting up a terminus for PuppetDB. ii puppetlabs-release 1.0-11 all Package to install Puppet Labs gpg key and apt repo ii puppetmaster-common 3.7.4-1puppetlabs1 all Puppet master common scripts ii puppetmaster-passenger 3.7.4-1puppetlabs1 all Centralised configuration management - master setup to run under mod passenger Are we looking at a known bug or are we really going to need to debug this? James On 11 March 2015 at 09:47, James Green james.mk.gr...@gmail.com wrote: Hi, Sorry for the delayed response. In our case we are using Passenger. On 5 March 2015 at 15:24, Henrik Lindberg henrik.lindb...@cloudsmith.com wrote: On 2015-05-03 12:02, James Green wrote: We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large. If you are using webrick then it is most likely a concurrency problem (more than one agent calling in at the same time). Webrick is not recommended for production use because of this. - henrik -- Visit my Blog Puppet on the Edge http://puppet-on-the-edge.blogspot.se/ -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/puppet-users/md9sfc%24r1e%241%40ger.gmane.org. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bay6EoB1zCZ587s%2B-gWno%3DGdHBR%3DKkhmD1aVHHXn66TsVw%40mail.gmail.com https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bay6EoB1zCZ587s%2B-gWno%3DGdHBR%3DKkhmD1aVHHXn66TsVw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. The fact that you are on puppet 3.7 and the error occasionally occurs, but not always, makes me think it's https://tickets.puppetlabs.com/browse/PUP-3238. Starting in puppet 3.7, we've added support for persistent HTTP connections, see https://docs.puppetlabs.com/puppet/latest/reference/subsystem_agent_master_comm.html#persistent-connections--keepalive . This eliminates the TCP/SSL handshake for every file metadata and content request, and greatly reduces the load on the master. But there is a possibility that a loadbalancer or puppetmaster may close an idle connection. Unfortunately, ruby does not surface socket errors when writing a request, only when reading. So often this issue manifests itself with end of file reached. To confirm if this is the case, make sure the puppet agent's http_keep_alive setting is
Re: [Puppet Users] Could not retrieve file metadata ... end of file reached
On Tue, Mar 17, 2015 at 4:52 AM, James Green james.mk.gr...@gmail.com wrote: I am not convinced that this is to do with an agent being busy then attempting the next connection. Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Error: Could not retrieve catalog from remote server: end of file reached Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run The above all happened within a few seconds. I might argue within 4-5 seconds. Unless of course I'd misunderstood your hypothesis? Puppet's persistent http connection feature will attempt to reuse the connection for the duration of the run. There are several opportunities for the connection to remain idle while the puppet agent is busy doing other work, e.g. downloading a package from a repo, executing a long running process, evaluating facts (like your example above), etc. If the connection is idle longer than the master (or load-balancer's) keep alive or idle timeouts, then the server side will close the connection. For example, apache 2.4 defaults HTTP KeepAliveTimeout is 5 seconds[1], and I've seen customer environments where haproxy idle timeouts were set to 60 seconds. One way to check if the server side is closing the connection, is to disable keep alive on the agent as I mentioned earlier. If the problem seems to go away, then I'd try re-enabling keep alive on the agent, and running tcpdump during an agent run. You'll likely see the master/load-balancer sending a TCP RST some exact number of seconds after the agent last wrote to the socket. To resolve the issue, make sure your server side keep alive timeouts are greater than the agent's, either by increasing the server's timeout, or decreasing the agent's. [1] http://httpd.apache.org/docs/2.4/mod/core.html#keepalivetimeout On 13 March 2015 at 05:28, Josh Cooper j...@puppetlabs.com wrote: On Thu, Mar 12, 2015 at 4:21 AM, James Green james.mk.gr...@gmail.com wrote: I was running puppet agent -t --noop repeatedly with this error. Running again, this time omitting --noop, it succeeded. Not entirely clear what this tells me... On 12 March 2015 at 10:58, James Green james.mk.gr...@gmail.com wrote: Error: /Stage[main]/Our_unattended_upgrades/File[/etc/apt/apt.conf.d/20auto-upgrades]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/our_unattended_upgrades/etc/apt/apt.conf.d/20auto-upgrades: end of file reached Wrapped exception: end of file reached Versions on the server: ii facter 2.4.1-1puppetlabs1 all Ruby module for collecting simple facts about a host operating system ii hiera1.3.4-1puppetlabs1 all A simple pluggable Hierarchical Database. ii puppet 3.7.4-1puppetlabs1 all Centralized configuration management - agent startup and compatibility scripts ii puppet-common3.7.4-1puppetlabs1 all Centralized configuration management ii puppetdb 2.2.2-1puppetlabs1 all PuppetDB Centralized Storage. ii puppetdb-terminus2.2.2-1puppetlabs1 all Connect Puppet to PuppetDB by setting up a terminus for PuppetDB. ii puppetlabs-release 1.0-11 all Package to install Puppet Labs gpg key and apt repo ii puppetmaster-common 3.7.4-1puppetlabs1 all Puppet master common scripts ii puppetmaster-passenger 3.7.4-1puppetlabs1 all Centralised configuration management - master setup to run under mod passenger Are we looking at a known bug or are we really going to need to debug this? James On 11 March 2015 at 09:47, James Green james.mk.gr...@gmail.com wrote: Hi, Sorry for the delayed response. In our case we are using Passenger. On 5 March 2015 at 15:24, Henrik Lindberg henrik.lindb...@cloudsmith.com wrote: On 2015-05-03 12:02, James Green wrote: We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large. If you are using webrick then it is most likely a concurrency problem (more than one agent calling in at the same time). Webrick is not recommended for production use because of this. - henrik -- Visit my Blog Puppet on the Edge http://puppet-on-the-edge.blogspot.se/ -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/puppet-users/md9sfc%24r1e%241%40ger.gmane.org. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet
Re: [Puppet Users] Could not retrieve file metadata ... end of file reached
Error: /Stage[main]/Our_unattended_upgrades/File[/etc/apt/apt.conf.d/20auto-upgrades]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/our_unattended_upgrades/etc/apt/apt.conf.d/20auto-upgrades: end of file reached Wrapped exception: end of file reached Versions on the server: ii facter 2.4.1-1puppetlabs1 all Ruby module for collecting simple facts about a host operating system ii hiera1.3.4-1puppetlabs1 all A simple pluggable Hierarchical Database. ii puppet 3.7.4-1puppetlabs1 all Centralized configuration management - agent startup and compatibility scripts ii puppet-common3.7.4-1puppetlabs1 all Centralized configuration management ii puppetdb 2.2.2-1puppetlabs1 all PuppetDB Centralized Storage. ii puppetdb-terminus2.2.2-1puppetlabs1 all Connect Puppet to PuppetDB by setting up a terminus for PuppetDB. ii puppetlabs-release 1.0-11 all Package to install Puppet Labs gpg key and apt repo ii puppetmaster-common 3.7.4-1puppetlabs1 all Puppet master common scripts ii puppetmaster-passenger 3.7.4-1puppetlabs1 all Centralised configuration management - master setup to run under mod passenger Are we looking at a known bug or are we really going to need to debug this? James On 11 March 2015 at 09:47, James Green james.mk.gr...@gmail.com wrote: Hi, Sorry for the delayed response. In our case we are using Passenger. On 5 March 2015 at 15:24, Henrik Lindberg henrik.lindb...@cloudsmith.com wrote: On 2015-05-03 12:02, James Green wrote: We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large. If you are using webrick then it is most likely a concurrency problem (more than one agent calling in at the same time). Webrick is not recommended for production use because of this. - henrik -- Visit my Blog Puppet on the Edge http://puppet-on-the-edge.blogspot.se/ -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/puppet-users/md9sfc%24r1e%241%40ger.gmane.org. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMH6%2BazZoSjsGYKP%3D19q4p%2Bp6ASeSQKLVJ3Apd9NyHp52uo_pA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Could not retrieve file metadata ... end of file reached
On Thu, Mar 12, 2015 at 4:21 AM, James Green james.mk.gr...@gmail.com wrote: I was running puppet agent -t --noop repeatedly with this error. Running again, this time omitting --noop, it succeeded. Not entirely clear what this tells me... On 12 March 2015 at 10:58, James Green james.mk.gr...@gmail.com wrote: Error: /Stage[main]/Our_unattended_upgrades/File[/etc/apt/apt.conf.d/20auto-upgrades]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/our_unattended_upgrades/etc/apt/apt.conf.d/20auto-upgrades: end of file reached Wrapped exception: end of file reached Versions on the server: ii facter 2.4.1-1puppetlabs1 all Ruby module for collecting simple facts about a host operating system ii hiera1.3.4-1puppetlabs1 all A simple pluggable Hierarchical Database. ii puppet 3.7.4-1puppetlabs1 all Centralized configuration management - agent startup and compatibility scripts ii puppet-common3.7.4-1puppetlabs1 all Centralized configuration management ii puppetdb 2.2.2-1puppetlabs1 all PuppetDB Centralized Storage. ii puppetdb-terminus2.2.2-1puppetlabs1 all Connect Puppet to PuppetDB by setting up a terminus for PuppetDB. ii puppetlabs-release 1.0-11 all Package to install Puppet Labs gpg key and apt repo ii puppetmaster-common 3.7.4-1puppetlabs1 all Puppet master common scripts ii puppetmaster-passenger 3.7.4-1puppetlabs1 all Centralised configuration management - master setup to run under mod passenger Are we looking at a known bug or are we really going to need to debug this? James On 11 March 2015 at 09:47, James Green james.mk.gr...@gmail.com wrote: Hi, Sorry for the delayed response. In our case we are using Passenger. On 5 March 2015 at 15:24, Henrik Lindberg henrik.lindb...@cloudsmith.com wrote: On 2015-05-03 12:02, James Green wrote: We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large. If you are using webrick then it is most likely a concurrency problem (more than one agent calling in at the same time). Webrick is not recommended for production use because of this. - henrik -- Visit my Blog Puppet on the Edge http://puppet-on-the-edge.blogspot.se/ -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/puppet-users/md9sfc%24r1e%241%40ger.gmane.org. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bay6EoB1zCZ587s%2B-gWno%3DGdHBR%3DKkhmD1aVHHXn66TsVw%40mail.gmail.com https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bay6EoB1zCZ587s%2B-gWno%3DGdHBR%3DKkhmD1aVHHXn66TsVw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. The fact that you are on puppet 3.7 and the error occasionally occurs, but not always, makes me think it's https://tickets.puppetlabs.com/browse/PUP-3238. Starting in puppet 3.7, we've added support for persistent HTTP connections, see https://docs.puppetlabs.com/puppet/latest/reference/subsystem_agent_master_comm.html#persistent-connections--keepalive . This eliminates the TCP/SSL handshake for every file metadata and content request, and greatly reduces the load on the master. But there is a possibility that a loadbalancer or puppetmaster may close an idle connection. Unfortunately, ruby does not surface socket errors when writing a request, only when reading. So often this issue manifests itself with end of file reached. To confirm if this is the case, make sure the puppet agent's http_keep_alive setting is less than the keep alive on your master. See https://docs.puppetlabs.com/references/3.7.latest/configuration.html#httpkeepalivetimeout. You can also disable keepalive on the client by setting the timeout to 0. Most likely your catalog is performing a long'ish operation, e.g. installing a package, the master closes the idle connection, the agent then tries to retrieve a file whose source is a puppet URI, which
Re: [Puppet Users] Could not retrieve file metadata ... end of file reached
I was running puppet agent -t --noop repeatedly with this error. Running again, this time omitting --noop, it succeeded. Not entirely clear what this tells me... On 12 March 2015 at 10:58, James Green james.mk.gr...@gmail.com wrote: Error: /Stage[main]/Our_unattended_upgrades/File[/etc/apt/apt.conf.d/20auto-upgrades]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/our_unattended_upgrades/etc/apt/apt.conf.d/20auto-upgrades: end of file reached Wrapped exception: end of file reached Versions on the server: ii facter 2.4.1-1puppetlabs1 all Ruby module for collecting simple facts about a host operating system ii hiera1.3.4-1puppetlabs1 all A simple pluggable Hierarchical Database. ii puppet 3.7.4-1puppetlabs1 all Centralized configuration management - agent startup and compatibility scripts ii puppet-common3.7.4-1puppetlabs1 all Centralized configuration management ii puppetdb 2.2.2-1puppetlabs1 all PuppetDB Centralized Storage. ii puppetdb-terminus2.2.2-1puppetlabs1 all Connect Puppet to PuppetDB by setting up a terminus for PuppetDB. ii puppetlabs-release 1.0-11 all Package to install Puppet Labs gpg key and apt repo ii puppetmaster-common 3.7.4-1puppetlabs1 all Puppet master common scripts ii puppetmaster-passenger 3.7.4-1puppetlabs1 all Centralised configuration management - master setup to run under mod passenger Are we looking at a known bug or are we really going to need to debug this? James On 11 March 2015 at 09:47, James Green james.mk.gr...@gmail.com wrote: Hi, Sorry for the delayed response. In our case we are using Passenger. On 5 March 2015 at 15:24, Henrik Lindberg henrik.lindb...@cloudsmith.com wrote: On 2015-05-03 12:02, James Green wrote: We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large. If you are using webrick then it is most likely a concurrency problem (more than one agent calling in at the same time). Webrick is not recommended for production use because of this. - henrik -- Visit my Blog Puppet on the Edge http://puppet-on-the-edge.blogspot.se/ -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/puppet-users/md9sfc%24r1e%241%40ger.gmane.org. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bay6EoB1zCZ587s%2B-gWno%3DGdHBR%3DKkhmD1aVHHXn66TsVw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Could not retrieve file metadata ... end of file reached
Hi, Sorry for the delayed response. In our case we are using Passenger. On 5 March 2015 at 15:24, Henrik Lindberg henrik.lindb...@cloudsmith.com wrote: On 2015-05-03 12:02, James Green wrote: We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large. If you are using webrick then it is most likely a concurrency problem (more than one agent calling in at the same time). Webrick is not recommended for production use because of this. - henrik -- Visit my Blog Puppet on the Edge http://puppet-on-the-edge.blogspot.se/ -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/puppet-users/md9sfc%24r1e%241%40ger.gmane.org. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bays%2B5UYx4Lk51R0ZXrJNL6bCWtnTjGOPCyKprUr5F6Ygg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Could not retrieve file metadata ... end of file reached
On 2015-05-03 12:02, James Green wrote: We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large. If you are using webrick then it is most likely a concurrency problem (more than one agent calling in at the same time). Webrick is not recommended for production use because of this. - henrik -- Visit my Blog Puppet on the Edge http://puppet-on-the-edge.blogspot.se/ -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/md9sfc%24r1e%241%40ger.gmane.org. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Could not retrieve file metadata ... end of file reached
We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large. Looks like server and agents are 3.7.4, both Ubuntu. Can't see much in the way of server logs. Any ideas where to go to debug this? A few minutes on, and the agent in this case has now succeeded... Thanks, James -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bax-gWpyZLWHpB19_EWtD3OnG8U%3DHEvkJFtyK1HGzkAHHA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Could not retrieve file metadata ... end of file reached
It seems like other people have had this issue, and maybe it's when the puppetmaster process restarts in the middle of somebody's agent run? https://projects.puppetlabs.com/issues/3019 https://projects.puppetlabs.com/issues/4808 In your place I would probably do the following: a) turn up the log level a bit on the master and agent b) packet capture on agent-master connections (You have the keys/certs for both ends so you can decrypt the traffic: http://wiki.wireshark.org/SSL) Then when it happens again I would correlate the time in the logs with the packet capture to find the message that discussed the end of file. Then I would see if the agent actually received an EOF character or similar oddity from the master. If the logs are detailed enough I might be able to track that down to something happening on the master. I tried grepping through my local puppet installation for those error messages but I'm not skilled enough to tell how the things I found are relevant. On Thu, Mar 05, 2015 at 11:02:23AM +, James Green wrote: We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large. Looks like server and agents are 3.7.4, both Ubuntu. Can't see much in the way of server logs. Any ideas where to go to debug this? A few minutes on, and the agent in this case has now succeeded... Thanks, James -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to [1]puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit [2]https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bax-gWpyZLWHpB19_EWtD3OnG8U%3DHEvkJFtyK1HGzkAHHA%40mail.gmail.com. For more options, visit [3]https://groups.google.com/d/optout. References Visible links 1. mailto:puppet-users+unsubscr...@googlegroups.com 2. https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bax-gWpyZLWHpB19_EWtD3OnG8U%3DHEvkJFtyK1HGzkAHHA%40mail.gmail.com?utm_medium=emailutm_source=footer 3. https://groups.google.com/d/optout -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20150305141954.GA12691%40iniquitous.heresiarch.ca. For more options, visit https://groups.google.com/d/optout.
