[Yahoo-eng-team] [Bug 1253896] Re: Attempts to verify guests are running via SSH fails. SSH connection to guest does not work.
> However stable/havana failures appear to be all related to the neutron, and the reason has to be searched in the fact that the improvements for neutron have not been backported. Frankly I'm not sure they could be considered backportable at all as there are some large patches. To make it clear, I'll mark Neutron/Havana Won't Fix and add Tempest/Havana task. So what do we do now with stable/havana gate jobs? Which tests or whole jobs should be disabled for Havana? ** Also affects: tempest/havana Importance: Critical Assignee: Darragh O'Reilly (darragh-oreilly) Status: In Progress ** Changed in: tempest/havana Status: In Progress => Confirmed ** Changed in: tempest/havana Assignee: Darragh O'Reilly (darragh-oreilly) => (unassigned) ** Changed in: tempest/havana Milestone: icehouse-1 => havana-stable ** Also affects: neutron/havana Importance: Undecided Status: New ** Changed in: neutron/havana Status: New => Won't Fix ** Changed in: neutron/havana Importance: Undecided => Critical -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1253896 Title: Attempts to verify guests are running via SSH fails. SSH connection to guest does not work. Status in OpenStack Neutron (virtual network service): In Progress Status in neutron havana series: Won't Fix Status in OpenStack Compute (Nova): Triaged Status in Tempest: Confirmed Status in tempest havana series: Confirmed Bug description: An example of this can be found at http://logs.openstack.org/74/57774/2/gate/gate-tempest-devstack-vm- full/e592961/console.html. This test failing seems to cause the tearDownClass failure and the process exit code failure. Judging by the logs below, the VM is coming up, and the test is connecting to the SSH server (dropbear) running in the VM, but the authentication is failing. It appears that authentication is attempted several times before paramiko gives up causing the test to fail. I think this indicates there isn't a network or compute problem, instead is possible the client doesn't have the correct key or the authorized keys aren't configured properly on the server side. But these are just guesses, I haven't been able to get any concrete data that would support these theories. 2013-11-22 05:36:33.980 | 2013-11-22 05:32:17,029 Adding to shared resources of TestMinimumBasicScenario 2013-11-22 05:36:33.980 | 2013-11-22 05:32:34,226 starting thread (client mode): 0x52e7e50L 2013-11-22 05:36:33.980 | 2013-11-22 05:32:34,232 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,237 kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] server encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] client mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] server mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,238 Ciphers agreed: local=aes128-ctr, remote=aes128-ctr 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,238 using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,433 Switch to new keys ... 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,434 Adding ssh-rsa host key for 172.24.4.227: 189c16acb93fe44ae975e1c653f1856c 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,434 Trying SSH key 9a9afe52a9485c15495a59b94ebca6b6 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,437 userauth is OK 2013-11-22 05:36:33.982 | 2013-11-22 05:32:35,104 Authentication (publickey) failed. 2013-11-22 05:36:33.982 | 2013-11-22 05:32:36,693 starting thread (client mode): 0x52f9190L 2013-11-22 05:36:33.982 | 2013-11-22 05:32:36,697 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] server encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] client mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] server mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] client compress:['none'] server compress:['none'] clien
[Yahoo-eng-team] [Bug 1253896] Re: Attempts to verify guests are running via SSH fails. SSH connection to guest does not work.
** No longer affects: tempest/havana -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1253896 Title: Attempts to verify guests are running via SSH fails. SSH connection to guest does not work. Status in OpenStack Neutron (virtual network service): Fix Released Status in neutron havana series: Won't Fix Status in OpenStack Compute (Nova): Triaged Status in Tempest: Confirmed Bug description: An example of this can be found at http://logs.openstack.org/74/57774/2/gate/gate-tempest-devstack-vm- full/e592961/console.html. This test failing seems to cause the tearDownClass failure and the process exit code failure. Judging by the logs below, the VM is coming up, and the test is connecting to the SSH server (dropbear) running in the VM, but the authentication is failing. It appears that authentication is attempted several times before paramiko gives up causing the test to fail. I think this indicates there isn't a network or compute problem, instead is possible the client doesn't have the correct key or the authorized keys aren't configured properly on the server side. But these are just guesses, I haven't been able to get any concrete data that would support these theories. 2013-11-22 05:36:33.980 | 2013-11-22 05:32:17,029 Adding to shared resources of TestMinimumBasicScenario 2013-11-22 05:36:33.980 | 2013-11-22 05:32:34,226 starting thread (client mode): 0x52e7e50L 2013-11-22 05:36:33.980 | 2013-11-22 05:32:34,232 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,237 kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] server encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] client mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] server mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,238 Ciphers agreed: local=aes128-ctr, remote=aes128-ctr 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,238 using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,433 Switch to new keys ... 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,434 Adding ssh-rsa host key for 172.24.4.227: 189c16acb93fe44ae975e1c653f1856c 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,434 Trying SSH key 9a9afe52a9485c15495a59b94ebca6b6 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,437 userauth is OK 2013-11-22 05:36:33.982 | 2013-11-22 05:32:35,104 Authentication (publickey) failed. 2013-11-22 05:36:33.982 | 2013-11-22 05:32:36,693 starting thread (client mode): 0x52f9190L 2013-11-22 05:36:33.982 | 2013-11-22 05:32:36,697 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] server encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] client mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] server mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 Ciphers agreed: local=aes128-ctr, remote=aes128-ctr 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,903 Switch to new keys ... 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,904 Trying SSH key 9a9afe52a9485c15495a59b94ebca6b6 2013-11-22 05:36:33.984 | 2013-11-22 05:32:36,906 userauth is OK 2013-11-22 05:36:33.984 | 2013-11-22 05:32:37,438 Authentication (publickey) failed. 2013-11-22 05:36:33.984 | 2013-11-22 05:32:39,035 starting thread (client mode): 0x24c62d0L 2013-11-22 05:36:33.984 | 2013-11-22 05:32:39,043 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.991 | 2013-11-22 05:32:39,045 kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh
[Yahoo-eng-team] [Bug 1253896] Re: Attempts to verify guests are running via SSH fails. SSH connection to guest does not work.
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1253896 Title: Attempts to verify guests are running via SSH fails. SSH connection to guest does not work. Status in OpenStack Neutron (virtual network service): Fix Released Status in neutron havana series: Won't Fix Status in OpenStack Compute (Nova): Triaged Status in Tempest: Confirmed Status in tempest havana series: Confirmed Bug description: An example of this can be found at http://logs.openstack.org/74/57774/2/gate/gate-tempest-devstack-vm- full/e592961/console.html. This test failing seems to cause the tearDownClass failure and the process exit code failure. Judging by the logs below, the VM is coming up, and the test is connecting to the SSH server (dropbear) running in the VM, but the authentication is failing. It appears that authentication is attempted several times before paramiko gives up causing the test to fail. I think this indicates there isn't a network or compute problem, instead is possible the client doesn't have the correct key or the authorized keys aren't configured properly on the server side. But these are just guesses, I haven't been able to get any concrete data that would support these theories. 2013-11-22 05:36:33.980 | 2013-11-22 05:32:17,029 Adding to shared resources of TestMinimumBasicScenario 2013-11-22 05:36:33.980 | 2013-11-22 05:32:34,226 starting thread (client mode): 0x52e7e50L 2013-11-22 05:36:33.980 | 2013-11-22 05:32:34,232 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,237 kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] server encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] client mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] server mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,238 Ciphers agreed: local=aes128-ctr, remote=aes128-ctr 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,238 using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,433 Switch to new keys ... 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,434 Adding ssh-rsa host key for 172.24.4.227: 189c16acb93fe44ae975e1c653f1856c 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,434 Trying SSH key 9a9afe52a9485c15495a59b94ebca6b6 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,437 userauth is OK 2013-11-22 05:36:33.982 | 2013-11-22 05:32:35,104 Authentication (publickey) failed. 2013-11-22 05:36:33.982 | 2013-11-22 05:32:36,693 starting thread (client mode): 0x52f9190L 2013-11-22 05:36:33.982 | 2013-11-22 05:32:36,697 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] server encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] client mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] server mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 Ciphers agreed: local=aes128-ctr, remote=aes128-ctr 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,903 Switch to new keys ... 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,904 Trying SSH key 9a9afe52a9485c15495a59b94ebca6b6 2013-11-22 05:36:33.984 | 2013-11-22 05:32:36,906 userauth is OK 2013-11-22 05:36:33.984 | 2013-11-22 05:32:37,438 Authentication (publickey) failed. 2013-11-22 05:36:33.984 | 2013-11-22 05:32:39,035 starting thread (client mode): 0x24c62d0L 2013-11-22 05:36:33.984 | 2013-11-22 05:32:39,043 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.991 | 2013-11-22 05:32:39,045 kex algos:['d
[Yahoo-eng-team] [Bug 1253896] Re: Attempts to verify guests are running via SSH fails. SSH connection to guest does not work.
https://review.openstack.org/#/c/273042/ - seems to have largely fixed this from the Nova side. Final major issue was a reuse of non fully deallocated fixed ips. ** Changed in: nova Assignee: (unassigned) => Sean Dague (sdague) ** Changed in: nova Status: Confirmed => Fix Released ** Changed in: tempest Status: Incomplete => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1253896 Title: Attempts to verify guests are running via SSH fails. SSH connection to guest does not work. Status in neutron: Fix Released Status in neutron havana series: Won't Fix Status in OpenStack Compute (nova): Fix Released Status in tempest: Invalid Bug description: An example of this can be found at http://logs.openstack.org/74/57774/2/gate/gate-tempest-devstack-vm- full/e592961/console.html. This test failing seems to cause the tearDownClass failure and the process exit code failure. Judging by the logs below, the VM is coming up, and the test is connecting to the SSH server (dropbear) running in the VM, but the authentication is failing. It appears that authentication is attempted several times before paramiko gives up causing the test to fail. I think this indicates there isn't a network or compute problem, instead is possible the client doesn't have the correct key or the authorized keys aren't configured properly on the server side. But these are just guesses, I haven't been able to get any concrete data that would support these theories. 2013-11-22 05:36:33.980 | 2013-11-22 05:32:17,029 Adding to shared resources of TestMinimumBasicScenario 2013-11-22 05:36:33.980 | 2013-11-22 05:32:34,226 starting thread (client mode): 0x52e7e50L 2013-11-22 05:36:33.980 | 2013-11-22 05:32:34,232 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,237 kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] server encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] client mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] server mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,238 Ciphers agreed: local=aes128-ctr, remote=aes128-ctr 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,238 using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none 2013-11-22 05:36:33.981 | 2013-11-22 05:32:34,433 Switch to new keys ... 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,434 Adding ssh-rsa host key for 172.24.4.227: 189c16acb93fe44ae975e1c653f1856c 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,434 Trying SSH key 9a9afe52a9485c15495a59b94ebca6b6 2013-11-22 05:36:33.982 | 2013-11-22 05:32:34,437 userauth is OK 2013-11-22 05:36:33.982 | 2013-11-22 05:32:35,104 Authentication (publickey) failed. 2013-11-22 05:36:33.982 | 2013-11-22 05:32:36,693 starting thread (client mode): 0x52f9190L 2013-11-22 05:36:33.982 | 2013-11-22 05:32:36,697 Connected (version 2.0, client dropbear_2012.55) 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] server encrypt:['aes128-ctr', '3des-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish-cbc', 'twofish128-cbc'] client mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] server mac:['hmac-sha1-96', 'hmac-sha1', 'hmac-md5'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 Ciphers agreed: local=aes128-ctr, remote=aes128-ctr 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,699 using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,903 Switch to new keys ... 2013-11-22 05:36:33.983 | 2013-11-22 05:32:36,904 Trying SSH key 9a9afe52a9485c15495a59b94ebca6b6 2013-11-22 05:36:33.984 | 2013-11-22 05:32:36,906 userauth is OK 2013-11-22 05:36:33.984 | 2013-11-22 05:32:37,438 Authentication (publickey) failed. 2013-11-22 05:36:33.984 | 2013-11-22 05:32:39,035
