[Openstack-operators] [kolla][nova] Upgrade leaves nova_compute at earlier version
I am using kolla-ansible 6.0.0 with openstack_release set to master in globals.yml in a production environment. I am trying to fix an oslo_messaging.rpc.client.RemoteError when undertaking server add volume (https://bugs.launchpad.net/nova/+bug/1773393) and appear to have tracked it down to an inconsistency of container versions. kollo-ansible upgrade runs to completion without error but leaves the running nova_compute containers (and possibly others) at an earlier version. The version running is CONTAINER ID IMAGECOMMAND CREATED STATUS PORTS NAMES 944f620a445c 7dc2d4695962 "kolla_start" 4 weeks ago Up 4 hours nova_compute whereas docker images -a shows REPOSITORY TAG IMAGE ID CREATED SIZE kolla/ubuntu-source-nova-compute master f2df8187f14e 15 hours ago 1.29GB kolla/ubuntu-source-nova-compute 582561ac010f 39 hours ago 1.29GB kolla/ubuntu-source-nova-compute 7dc2d4695962 3 months ago 1.22GB This implies f2df8187f14e is the one I should be using. The image 582561ac010f was after I tried to switch to queens from master but without success due to a bootstrap_cinder problem: Error during database migration: "Database schema file with version 122 doesn't exist." I tried to investigate this but without any obvious resolution. All compute nodes show the same issue. As per the notes on https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html I have checked and virt_type is set to kvm in nova.conf and so I cannot see why the upgrade shouldnt have been successful. How do I get kolla to use the latest version pulled? Given I have running instances I am a little nervous of doing a deploy or reconfigure. Thanks for help. Dave ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [kolla][nova] Upgrade leaves nova_compute at earlier version
Hi, could you share a your globals file without secrets, a docker ps -a on all compute hosts and images too. If you are able to get n upgrade log will be helpful too. By the way, using master is not really recommended, many changes from other projects and kolla may break the deployment. Regards On Thu, Jun 14, 2018, 9:00 PM Dave Williams wrote: > I am using kolla-ansible 6.0.0 with openstack_release set to master in > globals.yml in a production environment. > > I am trying to fix an oslo_messaging.rpc.client.RemoteError when > undertaking server add volume > (https://bugs.launchpad.net/nova/+bug/1773393) and appear to have > tracked it down to an inconsistency of container versions. > > kollo-ansible upgrade runs to completion without error but leaves the > running nova_compute containers (and possibly others) at an earlier > version. > > The version running is > CONTAINER ID IMAGECOMMAND CREATED STATUS PORTS NAMES > 944f620a445c 7dc2d4695962 "kolla_start" 4 weeks ago Up 4 hours nova_compute > > whereas docker images -a shows > REPOSITORY TAG IMAGE ID CREATED SIZE > kolla/ubuntu-source-nova-compute master f2df8187f14e 15 hours ago 1.29GB > kolla/ubuntu-source-nova-compute 582561ac010f 39 hours ago 1.29GB > kolla/ubuntu-source-nova-compute 7dc2d4695962 3 months ago 1.22GB > > This implies f2df8187f14e is the one I should be using. > The image 582561ac010f was after I tried to switch to queens from master > but without success due to a bootstrap_cinder problem: > Error during database migration: > "Database schema file with version 122 doesn't exist." > I tried to investigate this but without any obvious resolution. > > All compute nodes show the same issue. > > As per the notes on > https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html > I have checked and virt_type is set to kvm in nova.conf and so I cannot > see why the upgrade shouldnt have been successful. > > How do I get kolla to use the latest version pulled? > > Given I have running instances I am a little nervous of doing a deploy or > reconfigure. > > Thanks for help. > > Dave > > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [kolla][nova] Upgrade leaves nova_compute at earlier version]
Sure: Apologies for a longer post than I would like. globals.yml with all commented out (default) values removed: kolla_base_distro: "ubuntu" kolla_install_type: "source" openstack_release: "master" kolla_internal_vip_address: "10.10.66.138" kolla_internal_fqdn: "openstackint.uklab.local" kolla_external_vip_address: "10.10.66.134" kolla_external_fqdn: "openstackext.uklab.local" kolla_enable_tls_external: "yes" kolla_external_fqdn_cert: "{{ node_config_directory }}/certificates/haproxy.pem" enable_central_logging: "yes" enable_ceph: "yes" enable_cinder: "yes" enable_cinder_backend_netapp_iscsi: "no" enable_neutron_provider_networks: "yes" ceph_pool_type: "replicated" ceph_pool_pg_num: 128 ceph_pool_pgp_num: 128 cinder_backup_driver: "ceph" tempest_image_id: tempest_flavor_ref_id: tempest_public_network_id: tempest_floating_network_name: inventory file again only section with changes (rest default): [control] mclaren ansible_connection=local network_interface=enp5s0 neutron_external_interface=enp3s0 api_interface=enp5s0 [network] mclaren ansible_connection=local network_interface=enp5s0 neutron_external_interface=enp3s0 api_interface=enp5s0 [inner-compute] [external-compute] mclaren ansible_connection=local network_interface=enp5s0 neutron_external_interface=enp3s0 api_interface=enp5s0 aston network_interface=enp3s0 neutron_external_interface=enp5s0 corvette2 network_interface=enp3s0 neutron_external_interface=enp5s0 lambo2 network_interface=enp3s0 neutron_external_interface=enp5s0 porsche network_interface=enp3s0 neutron_external_interface=enp5s0 ferrari network_interface=enp3s0 neutron_external_interface=enp5s0 lotus network_interface=enp3s0 neutron_external_interface=enp5s0 maseratinetwork_interface=eth0 neutron_external_interface=eth1 chevi2 network_interface=enp4s0f1 neutron_external_interface=enp4s0f0 audinetwork_interface=em1 neutron_external_interface=em2 saabnetwork_interface=em1 neutron_external_interface=em2 radical network_interface=eth0 neutron_external_interface=eth1 [compute:children] inner-compute external-compute [storage] mclaren ansible_connection=local network_interface=enp5s0 neutron_external_interface=enp3s0 api_interface=enp5s0 lambo2 network_interface=enp3s0 neutron_external_interface=enp5s0 porsche network_interface=enp3s0 neutron_external_interface=enp5s0 ferrari network_interface=enp3s0 neutron_external_interface=enp5s0 lotus network_interface=enp3s0 neutron_external_interface=enp5s0 maseratinetwork_interface=eth0 neutron_external_interface=eth1 chevi2 network_interface=enp4s0f1 neutron_external_interface=enp4s0f0 audinetwork_interface=em1 neutron_external_interface=em2 saabnetwork_interface=em1 neutron_external_interface=em2 radical network_interface=eth0 neutron_external_interface=eth1 [monitoring] mclaren ansible_connection=local network_interface=enp5s0 neutron_external_interface=enp3s0 api_interface=enp5s0 radical network_interface=eth0 neutron_external_interface=eth1 aston network_interface=enp3s0 neutron_external_interface=enp5s0 [deployment] mclaren ansible_connection=local network_interface=enp5s0 neutron_external_interface=enp3s0 api_interface=enp5s0 docker ps -a (sorry for lack of line wrap) SERVER aston CONTAINER IDIMAGE COMMAND CREATED STATUS PORTS NAMES 24e87677d14c0a1d3587ae5c"kolla_start" 9 days ago Up 9 days neutron_openvswitch_agent 22e7e820eb96760e48f2527b"kolla_start" 9 days ago Up 9 days openvswitch_vswitchd e6cb63995d02c07ab7bd1910"kolla_start" 9 days ago Up 9 days openvswitch_db 71ffd7f32eaac9af2493c9e1"kolla_start" 9 days ago Up 9 days nova_libvirt 8870b6ebaf61e9d2082aa62f"kolla_start" 9 days ago Up 9 days nova_ssh 042510be738972e8b021c493"kolla_start" 9 days ago Up 9 days cron 83e50da14764ac0c8b1aeef2"kolla_start" 9 days ago Up 9 days kolla_toolbox 5e8efb880a44afa0d8636841"kolla_start" 9 days ago Up 9 days fluentd 76a7b24d339f7dc2d4695962"kolla_start" 6 weeks ago Up 9 days nova_compute SERVER chevi2 CONTAINER IDIMAGE COMMAND CREATED STATUS PORTS NAMES 98c58e80ed71kolla/ubuntu-source-ceph