Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Ciprian Badescu commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails Vadym Chepkov, we created a new ticket to track Windows specific error (https://tickets.puppetlabs.com/browse/PUP-11134) Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.68279.1624980420026%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Vadym Chepkov commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails I have upgraded non-prod infrastructure to 2019.8.7 and agents to 6.23.0. RedHat Linux agents produce the following log entry, but run succeeds : Jun 26 13:05:38 sdltstx-jira902 puppet-agent[23647]: Unable to connect to server from server_list setting: Request to https://infdevx-puppet202.example.com:8140/status/v1/simple/master failed after 0.007 seconds: Failed to open TCP connection to infdevx-example202.bnaint.com:8140 (Connection refused - connect(2) for "infdevx-example202.bnaint.com" port 8140) Trying with next server from server_list. Windows nodes, unfortunately, still fail Could not retrieve catalog from remote server: Request to https://infdevx-puppet202.example.com:8140/puppet/v3/catalog/infdevw-mdt001.example.com?environment=windows_development failed after 1.027 seconds: Failed to open TCP connection to infdevx-puppet202.example.com:8140 (No connection could be made because the target machine actively refused it. - connect(2) for "infdevx-puppet202.example.com" port 8140) Wrapped exception: Failed to open TCP connection to infdevx-puppet202.example.com:8140 (No connection could be made because the target machine actively refused it. - connect(2) for "infdevx-puppet202.example.com" port 8140)
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Claire Cadman updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Claire Cadman Labels: doc-reviewed jira_escalated platform_7.2 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.35023.1621346760030%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Josh Cooper updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Josh Cooper Fix Version/s: PUP 7.7.0 Fix Version/s: PUP 6.23.0 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.17276.1619197320110%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Josh Cooper commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails Merged to 6.x in https://github.com/puppetlabs/puppet/commit/cd615a0abbe321549fcc85a3a723f5969c269ff2 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.17277.1619197320157%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Dorin Pleava commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails I think now I understand what the issue was: When puppet processes server_list and tries to find a functional server, it go through each server, and if it cannot connect it throws an error, but it still moves on to the next server in server_list. Now it only throws a warning for each server it cannot connect to, and if no server from server_list is functional, then it throws an error. [root@twin-federalism puppet]# puppet config print server_list full-maturation.delivery.puppetlabs.net,crash-leapfrog.delivery.puppetlabs.net [root@twin-federalism puppet]# puppet agent -t Warning: Unable to connect to server from server_list setting: Request to https://full-maturation.delivery.puppetlabs.net:8140/status/v1/simple/server failed after 0.002 seconds: Failed to open TCP connection to full-maturation.delivery.puppetlabs.net:8140 (Connection refused - connect(2) for "full-maturation.delivery.puppetlabs.net" port 8140) Trying with next server from server_list. Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for twin-federalism.delivery.puppetlabs.net Info: Applying configuration version '1618915174'
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Mihai Buzgau updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Mihai Buzgau Sprint: NW - 2021-01-20, NW - 2021-02-03 , NW-2021-04-28 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.7503.1618321260094%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Mihai Buzgau updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Mihai Buzgau Fix Version/s: PUP 6.22.0 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.5881.1618212180030%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Luchian Nemes updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Luchian Nemes Fix Version/s: PUP 6.22.0 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.5005.1617956040029%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Vadym Chepkov commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails Something occurred to me. Is it possible the problem is on 'presentation' side and not on functionality? I looked through puppetserver.log on the replica and I do see nodes connecting to it during primary shutdown: 2021-03-03T07:27:53.295-05:00 INFO [qtp1835431929-14632] [puppetserver] Puppet Not using expired facts for pubtstx-web104.example.com from cache; expired at 2020-09-01 11:55:17 -0400 2021-03-03T07:27:53.356-05:00 INFO [qtp1835431929-14632] [puppetserver] Puppet Caching facts for pubtstx-web104.example.com 2021-03-03T07:27:55.563-05:00 ERROR [clojure-agent-send-off-pool-13553] [p.e.file-sync-errors] File Sync failure during sync or fetch phase: Couldn't connect to server (https://infdevx-puppet202.example.com:8140/file-sync/v1/latest-commits): (Connection refused). So, maybe the problem is with how agent handles the exception? Ideally, agent shouldn't through an error in the log and into submitted report if it was able to recover? At the end the problem manifests itself with splunk and report processor sounding false alerts Add Comment
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Vadym Chepkov updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Vadym Chepkov Attachment: puppet-agents.txt Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.155988.1614775560074%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Vadym Chepkov updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Vadym Chepkov Attachment: puppet.txt Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.155989.1614775560124%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Vadym Chepkov updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Vadym Chepkov Attachment: puppet-agents.txt Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.155974.1614775020041%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Vadym Chepkov commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails I have upgraded non-prod environment and non-prod nodes and problem persists in PE2019.8.5 with puppet 6.21.1. The root cause, which was in the ticket description maybe not accurate, but they method Jarret Lavallee used is still valid. I extracted events from splunk after I shutdown pe-pupeptserver on the primary puppet-agents.txt Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.155975.1614775020100%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Nick Walker commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails Vadym Chepkov reports this is still happening in his install. He's going to report back when he upgrades to PE 2019.8.5 if this is still happening. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.153260.1614367620120%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Vadym Chepkov commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails That's not my experience, I have 6.19.1 in PE2019.8.4 and each time I have to restart primary server for patching or other maintenance, dozens of agents fail, which defeats the purpose of HA. When we had PE 2018.1.x, agents have worked without failures Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.145249.1613667420322%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Josh Cooper commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails Jarret Lavallee sounds good. Also to summarize this issue, the current 6.x behavior matches how 5.x worked. It just so happened that 6.14.0 would process server_list for every REST request, regardless of whether the previous request succeeded or not. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.124632.1611678600261%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Jarret Lavallee commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails Josh Cooper and Dorin Pleava Thank you for looking into this deeper and providing some great analysis. I think you are correct with the desired behavior and we should close this ticket. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.124613.1611677640119%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Dorin Pleava commented on PUP-10844 Re: Agent failures with server_list when one puppetserver fails After some digging around on versions 5.5 21, 6.13.0, 6.14.0 and the newly released 6.20.0, only 6.14.0 did some things different. On 6.14.0, when a server failed midrun, the current running part would fail (like Retrieving pluginfacts, or Retrieving facts) and the code would continue to the next part(Retrieving locales) where it would check again for an available server, choosing the next functional server from server_list. I think this is not the intended functionality, as it could cause some sort of mix between catalogs from different servers. [root@blue-bumper ~]# puppet --version 6.14.0 [root@blue-bumper ~]# puppet agent -t --debug ... Debug: Creating new connection for https://past-medication.delivery.puppetlabs.net:8140 Debug: Starting connection for https://past-medication.delivery.puppetlabs.net:8140 Error: Could not retrieve catalog from remote server: Request to https://past-medication.delivery.puppetlabs.net:8140/puppet/v3/catalog/blue-bumper.delivery.puppetlabs.net?environment=production failed after 0.001 seconds: Failed to open TCP connection to past-medication.delivery.puppetlabs.net:8140 (Connection refused - connect(2) for "past-medication.delivery.puppetlabs.net" port 8140) Wrapped exception: Failed to open TCP connection to past-medication.delivery.puppetlabs.net:8140 (Connection refused - connect(2) for "past-medication.delivery.puppetlabs.net" port 8140)
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Mihai Buzgau updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Mihai Buzgau Sprint: NW - 2021-01-20 , NW - 2021-02-03 Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.119242.1611138540145%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Dorin Pleava assigned an issue to Dorin Pleava Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Dorin Pleava Assignee: Dorin Pleava Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.111680.1610376300184%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Mihai Buzgau updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Mihai Buzgau Story Points: 5 Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.108750.1609926180033%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Gheorghe Popescu updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Gheorghe Popescu Team: Coremunity Night's Watch Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.107797.1609861920048%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Gheorghe Popescu updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Gheorghe Popescu Sprint: NW - 2021-01-20 Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.107798.1609861920092%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Josh Cooper updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: Josh Cooper Labels: jira_escalated platform_7.2 Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.106599.1609784940183%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title zendesk.jira updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: zendesk.jira Zendesk Ticket Count: 1 Zendesk Ticket IDs: 40535 Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.105247.1608653280038%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title zendesk.jira updated an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Change By: zendesk.jira Labels: jira_escalated Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.381638.1608653215000.105248.1608653280080%40Atlassian.JIRA.
Jira (PUP-10844) Agent failures with server_list when one puppetserver fails
Title: Message Title Jarret Lavallee created an issue Puppet / PUP-10844 Agent failures with server_list when one puppetserver fails Issue Type: Bug Affects Versions: PUP 6.15.0 Assignee: Unassigned Created: 2020/12/22 8:06 AM Priority: Normal Reporter: Jarret Lavallee Puppet Version: 6.15.0+ Puppet Server Version: 6.x OS Name/Version: Any After the changes in 6.15.0 the server_list setting has different behavior. Previously when server_list was configured and the first puppetserver in the list failed, the agent would continue to run by connecting to the next puppetserver on the list. In 6.15.0, if the primary puppetserver fails while an agent is running, it results in a failed agent run. Desired Behavior: When the first puppetserver in the server_list goes offline, the agents should automatically try to connect to the second puppetserver in the server_list even if it is mid agent run. Actual Behavior: The agent run fails if the first puppetserver in the server_list goes offline while the agent is in the middle of a run. Some failures are below. Could not evaluate: Could not retrieve file metadata for puppet:///pe_packages/2019.8.1/windows-x86_64/puppet-agent-x64.msi: Request to https://primary.example.com:8140/puppet/v3/file_metadata/pe_packages/2019.8.1/windows-x86_64/puppet-agent-x64.msi?links=manage_type=sha256lite_permissions=ignore=windows_testing failed after 21.011 seconds: Failed to open TCP connection to primary.example.com:8140 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. - connect(2) for
