Re: [Openstack] rdo havana to icehouse: instances stuck in 'resized or migrated'
On 05/20/2014 07:06 PM, Julie Pichon wrote: On 20/05/14 17:08, Dimitri Maziuk wrote: On 05/20/2014 03:59 AM, Julie Pichon wrote: On 19/05/14 18:14, Dimitri Maziuk wrote: On 05/19/2014 11:15 AM, Julie Pichon wrote: I had a chat with a Nova developer who pointed me to the following patch at https://review.openstack.org/#/c/84755/ , recently merged in Havana and included in the latest RDO Havana packages. Resize specifically is one of the actions affected by this bug, you might want to check that you're running the latest packages on your Havana node(s) and see if this might help to resolve the problem? [root@irukandji ~]# yum up ... No Packages marked for Update So -- no, that's not it, unless the patch hasn't made it into the rpms yet. Could you provide the version numbers for the openstack-nova packages? Icehouse node: [root@squid ~]# rpm -q -a | grep openstack openstack-selinux-0.1.3-2.el6ost.noarch openstack-nova-api-2014.1-2.el6.noarch openstack-utils-2014.1-1.el6.noarch openstack-nova-compute-2014.1-2.el6.noarch openstack-nova-network-2014.1-2.el6.noarch openstack-nova-common-2014.1-2.el6.noarch Havana node: [root@irukandji ~]# rpm -q -a | grep openstack openstack-nova-common-2013.2.3-1.el6.noarch openstack-nova-network-2013.2.3-1.el6.noarch openstack-nova-api-2013.2.3-1.el6.noarch openstack-selinux-0.1.3-2.el6ost.noarch openstack-nova-compute-2013.2.3-1.el6.noarch openstack-utils-2013.2-2.el6.noarch Both nodes are [root@squid ~]# cat /etc/redhat-release CentOS release 6.5 (Final) Thanks for the information. I'm adding my colleague Nikola to this thread, he's familiar with Nova and should be better able to help. Julie Hi Dimitri, So for this kind of upgrade to work, you will need to set an RPC version cap so that all your icehouse computes know they are talking to possibly lower version nodes and downgrade their messages. This can be done (as described in much more detail on [1]) using the [upgrade_levels] section of your nova.conf file. Please let me know if this works and if you need further assistance. Thanks, Nikola Đipanov, SSE - OpenStack @ Red Hat [1] http://openstack.redhat.com/Upgrading_RDO_To_Icehouse ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] rdo havana to icehouse: instances stuck in 'resized or migrated'
On 19/05/14 18:14, Dimitri Maziuk wrote: On 05/19/2014 11:15 AM, Julie Pichon wrote: I had a chat with a Nova developer who pointed me to the following patch at https://review.openstack.org/#/c/84755/ , recently merged in Havana and included in the latest RDO Havana packages. Resize specifically is one of the actions affected by this bug, you might want to check that you're running the latest packages on your Havana node(s) and see if this might help to resolve the problem? [root@irukandji ~]# yum up ... No Packages marked for Update So -- no, that's not it, unless the patch hasn't made it into the rpms yet. Could you provide the version numbers for the openstack-nova packages? Thanks, Julie ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] rdo havana to icehouse: instances stuck in 'resized or migrated'
On 05/20/2014 03:59 AM, Julie Pichon wrote: On 19/05/14 18:14, Dimitri Maziuk wrote: On 05/19/2014 11:15 AM, Julie Pichon wrote: I had a chat with a Nova developer who pointed me to the following patch at https://review.openstack.org/#/c/84755/ , recently merged in Havana and included in the latest RDO Havana packages. Resize specifically is one of the actions affected by this bug, you might want to check that you're running the latest packages on your Havana node(s) and see if this might help to resolve the problem? [root@irukandji ~]# yum up ... No Packages marked for Update So -- no, that's not it, unless the patch hasn't made it into the rpms yet. Could you provide the version numbers for the openstack-nova packages? Icehouse node: [root@squid ~]# rpm -q -a | grep openstack openstack-selinux-0.1.3-2.el6ost.noarch openstack-nova-api-2014.1-2.el6.noarch openstack-utils-2014.1-1.el6.noarch openstack-nova-compute-2014.1-2.el6.noarch openstack-nova-network-2014.1-2.el6.noarch openstack-nova-common-2014.1-2.el6.noarch Havana node: [root@irukandji ~]# rpm -q -a | grep openstack openstack-nova-common-2013.2.3-1.el6.noarch openstack-nova-network-2013.2.3-1.el6.noarch openstack-nova-api-2013.2.3-1.el6.noarch openstack-selinux-0.1.3-2.el6ost.noarch openstack-nova-compute-2013.2.3-1.el6.noarch openstack-utils-2013.2-2.el6.noarch Both nodes are [root@squid ~]# cat /etc/redhat-release CentOS release 6.5 (Final) -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] rdo havana to icehouse: instances stuck in 'resized or migrated'
On 20/05/14 17:08, Dimitri Maziuk wrote: On 05/20/2014 03:59 AM, Julie Pichon wrote: On 19/05/14 18:14, Dimitri Maziuk wrote: On 05/19/2014 11:15 AM, Julie Pichon wrote: I had a chat with a Nova developer who pointed me to the following patch at https://review.openstack.org/#/c/84755/ , recently merged in Havana and included in the latest RDO Havana packages. Resize specifically is one of the actions affected by this bug, you might want to check that you're running the latest packages on your Havana node(s) and see if this might help to resolve the problem? [root@irukandji ~]# yum up ... No Packages marked for Update So -- no, that's not it, unless the patch hasn't made it into the rpms yet. Could you provide the version numbers for the openstack-nova packages? Icehouse node: [root@squid ~]# rpm -q -a | grep openstack openstack-selinux-0.1.3-2.el6ost.noarch openstack-nova-api-2014.1-2.el6.noarch openstack-utils-2014.1-1.el6.noarch openstack-nova-compute-2014.1-2.el6.noarch openstack-nova-network-2014.1-2.el6.noarch openstack-nova-common-2014.1-2.el6.noarch Havana node: [root@irukandji ~]# rpm -q -a | grep openstack openstack-nova-common-2013.2.3-1.el6.noarch openstack-nova-network-2013.2.3-1.el6.noarch openstack-nova-api-2013.2.3-1.el6.noarch openstack-selinux-0.1.3-2.el6ost.noarch openstack-nova-compute-2013.2.3-1.el6.noarch openstack-utils-2013.2-2.el6.noarch Both nodes are [root@squid ~]# cat /etc/redhat-release CentOS release 6.5 (Final) Thanks for the information. I'm adding my colleague Nikola to this thread, he's familiar with Nova and should be better able to help. Julie ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] rdo havana to icehouse: instances stuck in 'resized or migrated'
On 16/05/14 22:45, Dimitri Maziuk wrote: Hi all. Upgrading centos 6/rdo from havana to icehouse: I've updated all services on the controller as per the fine manual, then updated one of the 3 compute nodes. Now I'm trying to migrate the (shut off) instances from havana compute node to the icehouse compute node and they get stuck in resize_migrated. On the first one I reset-state --active, renamed /var/lib/nova/instances/uuid_resize and was able to issue a 'hard reboot' from the dashboard. Now I have the 2nd one stuck in the same state. I restarted compute with debug on on the target (icehouse) compute node, there's nothing interesting there so the problem is somewhere else. In fact, it looks like all it is is something isn't popping up the confirm resize button in the dashboard -- /var/lib/nova/instances is nfs-mounted so there's no actual data resizing/migration necessary. Hi Dimitri, From a dashboard perspective, the Confirm resize button is only displayed when the instance status gets to the final VERIFY_RESIZE stage. Out of curiosity, does the confirm resize command works on the stuck instances from the CLI? (nova resize-confirm server_id) I'm not familiar enough with Nova to be able to provide additional information on why the instances might be getting stuck, unfortunately, hopefully someone else can offer more insightful advice. Julie I really don't want to go through the reset-state/rename dir dance for every instance I have. Any suggestions? Anyone done the havana-to-icehouse dance with 2 compute nodes? ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] rdo havana to icehouse: instances stuck in 'resized or migrated'
On 19/05/14 11:48, Julie Pichon wrote: On 16/05/14 22:45, Dimitri Maziuk wrote: Hi all. Upgrading centos 6/rdo from havana to icehouse: I've updated all services on the controller as per the fine manual, then updated one of the 3 compute nodes. Now I'm trying to migrate the (shut off) instances from havana compute node to the icehouse compute node and they get stuck in resize_migrated. On the first one I reset-state --active, renamed /var/lib/nova/instances/uuid_resize and was able to issue a 'hard reboot' from the dashboard. Now I have the 2nd one stuck in the same state. I restarted compute with debug on on the target (icehouse) compute node, there's nothing interesting there so the problem is somewhere else. In fact, it looks like all it is is something isn't popping up the confirm resize button in the dashboard -- /var/lib/nova/instances is nfs-mounted so there's no actual data resizing/migration necessary. Hi Dimitri, From a dashboard perspective, the Confirm resize button is only displayed when the instance status gets to the final VERIFY_RESIZE stage. Out of curiosity, does the confirm resize command works on the stuck instances from the CLI? (nova resize-confirm server_id) I'm not familiar enough with Nova to be able to provide additional information on why the instances might be getting stuck, unfortunately, hopefully someone else can offer more insightful advice. I had a chat with a Nova developer who pointed me to the following patch at https://review.openstack.org/#/c/84755/ , recently merged in Havana and included in the latest RDO Havana packages. Resize specifically is one of the actions affected by this bug, you might want to check that you're running the latest packages on your Havana node(s) and see if this might help to resolve the problem? Julie Julie I really don't want to go through the reset-state/rename dir dance for every instance I have. Any suggestions? Anyone done the havana-to-icehouse dance with 2 compute nodes? ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] rdo havana to icehouse: instances stuck in 'resized or migrated'
Hi all. Upgrading centos 6/rdo from havana to icehouse: I've updated all services on the controller as per the fine manual, then updated one of the 3 compute nodes. Now I'm trying to migrate the (shut off) instances from havana compute node to the icehouse compute node and they get stuck in resize_migrated. On the first one I reset-state --active, renamed /var/lib/nova/instances/uuid_resize and was able to issue a 'hard reboot' from the dashboard. Now I have the 2nd one stuck in the same state. I restarted compute with debug on on the target (icehouse) compute node, there's nothing interesting there so the problem is somewhere else. In fact, it looks like all it is is something isn't popping up the confirm resize button in the dashboard -- /var/lib/nova/instances is nfs-mounted so there's no actual data resizing/migration necessary. I really don't want to go through the reset-state/rename dir dance for every instance I have. Any suggestions? Anyone done the havana-to-icehouse dance with 2 compute nodes? -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
