Check the host and see what happens on a full manual update. I have had
issues with ansible when the yum command was hanging on a host do to a
local issue with updating. Single packages were fine, but a full host
update failed. I had to resolve the full update issue on the host.
On Wed, Aug 9, 2023 at 3:14 PM Nicolas Goudry <nico...@kubolabs.io> wrote:
> I’m trying to perform a full system update with the `yum` module but
> ansible just hangs for a little bit more than an hour before failing.
>
> Here is the command I’m using:
>
> ansible all -u node-user -b --become-user=root -i exec/inventory -m yum -a
> 'name=* state=latest' -vvvv --limit=worker1
>
> Here is the output (redacted):
>
> ansible [core 2.12.5]
> config file = /home/nicolas/test-upgrade-os/ansible.cfg
> configured module search path =
> ['/home/nicolas/.ansible/plugins/modules',
> '/usr/share/ansible/plugins/modules']
> ansible python module location =
> /home/nicolas/test-upgrade-os/config/venv/lib64/python3.8/site-packages/ansible
> ansible collection location =
> /home/nicolas/.ansible/collections:/usr/share/ansible/collections
> executable location = ./config/venv/bin/ansible
> python version = 3.8.16 (default, Jun 25 2023, 05:53:51) [GCC 8.5.0
> 20210514 (Red Hat 8.5.0-18)]
> jinja version = 3.1.2
> libyaml = True
> Using /home/nicolas/test-upgrade-os/ansible.cfg as config file
> setting up inventory plugins
> host_list declined parsing /home/nicolas/test-upgrade-os/exec/inventory as
> it did not pass its verify_file() method
> script declined parsing /home/nicolas/test-upgrade-os/exec/inventory as it
> did not pass its verify_file() method
> auto declined parsing /home/nicolas/test-upgrade-os/exec/inventory as it
> did not pass its verify_file() method
> Parsed /home/nicolas/test-upgrade-os/exec/inventory inventory source with
> ini plugin
> Loading callback plugin minimal of type stdout, v2.0 from
> /home/nicolas/test-upgrade-os/config/venv/lib64/python3.8/site-packages/ansible/plugins/callback/minimal.py
> Skipping callback 'default', as we already have a stdout callback.
> Skipping callback 'minimal', as we already have a stdout callback.
> Skipping callback 'oneline', as we already have a stdout callback.
> META: ran handlers
> <10.10.0.101> ESTABLISH SSH CONNECTION FOR USER: node-user
> <10.10.0.101> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="node-user"' -o ConnectTimeout=10 -q
> -o UserKnownHostsFile=ssh/known_hosts -i ssh/node-user -o 'ProxyCommand=ssh
> -q -o UserKnownHostsFile=ssh/known_hosts -i ssh/bastion-user -W %h:%p -p22
> bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> 10.10.0.101 '/bin/sh -c '"'"'echo ~node-user && sleep 0'"'"''
> <10.10.0.101> (0, b'/home/node-user\n', b'')
> <10.10.0.101> ESTABLISH SSH CONNECTION FOR USER: node-user
> <10.10.0.101> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="node-user"' -o ConnectTimeout=10 -q
> -o UserKnownHostsFile=ssh/known_hosts -i ssh/node-user -o 'ProxyCommand=ssh
> -q -o UserKnownHostsFile=ssh/known_hosts -i ssh/bastion-user -W %h:%p -p22
> bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> 10.10.0.101 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo
> /home/node-user/.ansible/tmp `"&& mkdir "` echo
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576
> `" && echo ansible-tmp-1691583637.8116903-3768362-148267575047576="` echo
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576
> `" ) && sleep 0'"'"''
> <10.10.0.101> (0,
> b'ansible-tmp-1691583637.8116903-3768362-148267575047576=/home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576\n',
> b'')
> <worker1> Attempting python interpreter discovery
> <10.10.0.101> ESTABLISH SSH CONNECTION FOR USER: node-user
> <10.10.0.101> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="node-user"' -o ConnectTimeout=10 -q
> -o UserKnownHostsFile=ssh/known_hosts -i ssh/node-user -o 'ProxyCommand=ssh
> -q -o UserKnownHostsFile=ssh/known_hosts -i ssh/bastion-user -W %h:%p -p22
> bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> 10.10.0.101 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v
> '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v
> '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
> <10.10.0.101> (0,
> b'PLATFORM\nLinux\nFOUND\n/usr/libexec/platform-python\nENDFOUND\n', b'')
> <10.10.0.101> ESTABLISH SSH CONNECTION FOR USER: node-user
> <10.10.0.101> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="node-user"' -o ConnectTimeout=10 -q
> -o UserKnownHostsFile=ssh/known_hosts -i ssh/node-user -o 'ProxyCommand=ssh
> -q -o UserKnownHostsFile=ssh/known_hosts -i ssh/bastion-user -W %h:%p -p22
> bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> 10.10.0.101 '/bin/sh -c '"'"'/usr/libexec/platform-python && sleep 0'"'"''
> <10.10.0.101> (0, b'{"platform_dist_result": ["centos", "8.5", "Green
> Obsidian"], "osrelease_content": "NAME=\\"Rocky Linux\\"\\nVERSION=\\"8.5
> (Green Obsidian)\\"\\nID=\\"rocky\\"\\nID_LIKE=\\"rhel centos
> fedora\\"\\nVERSION_ID=\\"8.5\\"\\nPLATFORM_ID=\\"platform:el8\\"\\nPRETTY_NAME=\\"Rocky
> Linux 8.5 (Green
> Obsidian)\\"\\nANSI_COLOR=\\"0;32\\"\\nCPE_NAME=\\"cpe:/o:rocky:rocky:8:GA\\"\\nHOME_URL=\\"
> https://rockylinux.org/\\"\\nBUG_REPORT_URL=\\";
> https://bugs.rockylinux.org/\\"\\nROCKY_SUPPORT_PRODUCT=\\"Rocky
> Linux\\"\\nROCKY_SUPPORT_PRODUCT_VERSION=\\"8\\"\\n"}\n', b'')
> Using module file
> /home/nicolas/test-upgrade-os/config/venv/lib64/python3.8/site-packages/ansible/modules/setup.py
> <10.10.0.101> PUT
> /home/nicolas/test-upgrade-os/config/ansible/tmp/ansible-local-3768356wtqis0tq/tmpy4qpsqz0
> TO
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/AnsiballZ_setup.py
> <10.10.0.101> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="k
> ubonode"' -o ConnectTimeout=10 -q -o UserKnownHostsFile=ssh/known_hosts -i
> ssh/node-user -o 'ProxyCommand=ssh -q -o UserKnownHostsFile=ssh/known_hosts
> -i ssh/bastion-user -W %h:%p -p22 bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> '[10.10.0.101]'
> <10.10.0.101> (0, b'sftp> put
> /home/nicolas/test-upgrade-os/config/ansible/tmp/ansible-local-3768356wtqis0tq/tmpy4qpsqz0
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/AnsiballZ_setup.py\n',
> b'')
> <10.10.0.101> ESTABLISH SSH CONNECTION FOR USER: node-user
> <10.10.0.101> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="node-user"' -o ConnectTimeout=10 -q
> -o UserKnownHostsFile=ssh/known_hosts -i ssh/node-user -o 'ProxyCommand=ssh
> -q -o UserKnownHostsFile=ssh/known_hosts -i ssh/bastion-user -W %h:%p -p22
> bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> 10.10.0.101 '/bin/sh -c '"'"'chmod u+x
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/AnsiballZ_setup.py
> && sleep 0'"'"''
> <10.10.0.101> (0, b'', b'')
> <10.10.0.101> ESTABLISH SSH CONNECTION FOR USER: node-user
> <10.10.0.101> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="node-user"' -o ConnectTimeout=10 -q
> -o UserKnownHostsFile=ssh/known_hosts -i ssh/node-user -o 'ProxyCommand=ssh
> -q -o UserKnownHostsFile=ssh/known_hosts -i ssh/bastion-user -W %h:%p -p22
> bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> -tt 10.10.0.101 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c
> '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ztvxikfxzuzwogfymzcnlpfaroxhooqg ;
> /usr/libexec/platform-python
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/AnsiballZ_setup.py'"'"'"'"'"'"'"'"'
> && sleep 0'"'"''
> Escalation succeeded
> <10.10.0.101> (0, b'\r\n{"ansible_facts": {"ansible_pkg_mgr": "dnf"},
> "invocation": {"module_args": {"filter": ["ansible_pkg_mgr"],
> "gather_subset": ["!all"], "gather_timeout": 10, "fact_path":
> "/etc/ansible/facts.d"}}}\r\n', b'')
> Running ansible.legacy.dnf as the backend for the yum action plugin
> Using module file
> /home/nicolas/test-upgrade-os/config/venv/lib64/python3.8/site-packages/ansible/modules/dnf.py
> <10.10.0.101> PUT
> /home/nicolas/test-upgrade-os/config/ansible/tmp/ansible-local-3768356wtqis0tq/tmpomw666d5
> TO
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/AnsiballZ_dnf.py
> <10.10.0.101> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="node-user"' -o ConnectTimeout=10 -q
> -o UserKnownHostsFile=ssh/known_hosts -i ssh/node-user -o 'ProxyCommand=ssh
> -q -o UserKnownHostsFile=ssh/known_hosts -i ssh/bastion-user -W %h:%p -p22
> bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> '[10.10.0.101]'
> <10.10.0.101> (0, b'sftp> put
> /home/nicolas/test-upgrade-os/config/ansible/tmp/ansible-local-3768356wtqis0tq/tmpomw666d5
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/AnsiballZ_dnf.py\n',
> b
> '')
> <10.10.0.101> ESTABLISH SSH CONNECTION FOR USER: node-user
> <10.10.0.101> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="node-user"' -o ConnectTimeout=10 -q
> -o UserKnownHostsFile=ssh/known_hosts -i ssh/node-user -o 'ProxyCommand=ssh
> -q -o UserKnownHostsFile=ssh/known_hosts -i ssh/bastion-user -W %h:%p -p22
> bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> 10.10.0.101 '/bin/sh -c '"'"'chmod u+x
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/AnsiballZ_dnf.py
> && sleep 0'"'"''
> <10.10.0.101> (0, b'', b'')
> <10.10.0.101> ESTABLISH SSH CONNECTION FOR USER: node-user
> <10.10.0.101> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
> -o PasswordAuthentication=no -o 'User="node-user"' -o ConnectTimeout=10 -q
> -o UserKnownHostsFile=ssh/known_hosts -i ssh/node-user -o 'ProxyCommand=ssh
> -q -o UserKnownHostsFile=ssh/known_hosts -i ssh/bastion-user -W %h:%p -p22
> bastion-user@W.X.Y.Z' -o
> 'ControlPath="/home/nicolas/test-upgrade-os/config/ansible/cp/09896940d7"'
> -tt 10.10.0.101 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c
> '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-gjdfwphkqonajiudmalgairdspobkjad ;
> /usr/libexec/platform-python
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/AnsiballZ_dnf.py'"'"'"'"'"'"'"'"'
> && sleep 0'"'"''
> Escalation succeeded
>
> Before running ansible, I ssh'ed in the node and ran:
>
> watch "ps -aux | grep ansible"
>
> While ansible was performing the yum update, I saw the process
> /usr/libexec/platform-python
> /home/node-user/.ansible/tmp/ansible-tmp-1691583637.8116903-3768362-148267575047576/AnsiballZ_dnf.py
> was
> running for about 10-15 minutes and when it had disappear, ansible kept
> running for more than an hour before failing with the following error:
>
> worker1 | UNREACHABLE! => {
> "changed": false,
> "msg": "Failed to connect to the host via ssh: ",
> "unreachable": true
> }
>
> I tried using the dnf and package modules, which gave the exact same
> results.
>
> I tried updating a single package (tar) and it worked with yum, dnf and
> package modules.
>
> I’m running ansible on a Rocky Linux 8 machine with python 3.8.16. The
> worker1 machine is also using Rocky Linux 8 and the output of
> /usr/libexec/platform-python
> --version is Python 3.6.8.
>
> Should I file an issue in the ansible github repo for this matter ? Or am
> I doing something wrong ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-project+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/fa63bd90-69d7-494f-a494-0743de6c314an%40googlegroups.com
> <https://groups.google.com/d/msgid/ansible-project/fa63bd90-69d7-494f-a494-0743de6c314an%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAEcFzYxJrf03OFgOCQQxKCdBQ11nkgqa1zqHOdGv%3DhMzqnB4-g%40mail.gmail.com.
