So as commented on the patch - I really think that we need to make sure that whatever gets created, also gets cleaned up on errors - while the patch https://review.openstack.org/166695 has some good ideas.
What I also noticed (when I was testing this some time ago) is that what really happens here is that the rpc client does not time out - the reason you see the failure is likely because the Nova API times out the request. This might actually be an issue with oslo.messaging rabbitmq driver (which would never time out a request) or the fact that we assume it would. ** Also affects: oslo.messaging Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1377161 Title: If volume-attach API is failed, Block Device Mapping record will remain Status in Cinder: Invalid Status in OpenStack Compute (Nova): In Progress Status in Messaging API for OpenStack: New Status in Python client library for Cinder: Invalid Bug description: I executed volume-attach API(nova V2 API) when RabbitMQ was down. As result of above API execution, volume-attach API was failed and volume's status is still available. But, block device mapping record remains on nova DB. This condition is inconsistency. And, remained block device mapping record maybe cause some problems. (I'm researching now.) I used openstack juno-3. ------------------------------------------------------------------------------------------------------------------ * Before executing volume-attach API: $ nova list +--------------------------------------+--------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------+--------+------------+-------------+--------------------+ | 0b529526-4c8d-4650-8295-b7155a977ba7 | testVM | ACTIVE | - | Running | private=10.0.0.104 | +--------------------------------------+--------+--------+------------+-------------+--------------------+ $ cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | e93478bf-ee37-430f-93df-b3cf26540212 | available | None | 1 | None | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ devstack@ubuntu-14-04-01-64-juno3-01:~$ mysql> select * from block_device_mapping where instance_uuid = '0b529526-4c8d-4650-8295-b7155a977ba7'; +---------------------+---------------------+------------+-----+-------------+-----------------------+-------------+-----------+-------------+-----------+-----------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+--------------------------------------+ | created_at | updated_at | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id | volume_size | no_device | connection_info | instance_uuid | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id | +---------------------+---------------------+------------+-----+-------------+-----------------------+-------------+-----------+-------------+-----------+-----------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+--------------------------------------+ | 2014-10-02 18:36:08 | 2014-10-02 18:36:10 | NULL | 145 | /dev/vda | 1 | NULL | NULL | NULL | NULL | NULL | 0b529526-4c8d-4650-8295-b7155a977ba7 | 0 | image | local | NULL | disk | NULL | 0 | c1d264fd-c559-446e-9b94-934ba8249ae1 | +---------------------+---------------------+------------+-----+-------------+-----------------------+-------------+-----------+-------------+-----------+-----------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+--------------------------------------+ 1 row in set (0.00 sec) * After executing volume-attach API: $ nova list --all-t +--------------------------------------+--------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------+--------+------------+-------------+--------------------+ | 0b529526-4c8d-4650-8295-b7155a977ba7 | testVM | ACTIVE | - | Running | private=10.0.0.104 | +--------------------------------------+--------+--------+------------+-------------+--------------------+ $ cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | e93478bf-ee37-430f-93df-b3cf26540212 | available | None | 1 | None | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ mysql> select * from block_device_mapping where instance_uuid = '0b529526-4c8d-4650-8295-b7155a977ba7'; +---------------------+---------------------+------------+-----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+--------------------------------------+ | created_at | updated_at | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id | volume_size | no_device | connection_info | instance_uuid | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id | +---------------------+---------------------+------------+-----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+--------------------------------------+ | 2014-10-02 18:36:08 | 2014-10-02 18:36:10 | NULL | 145 | /dev/vda | 1 | NULL | NULL | NULL | NULL | NULL | 0b529526-4c8d-4650-8295-b7155a977ba7 | 0 | image | local | NULL | disk | NULL | 0 | c1d264fd-c559-446e-9b94-934ba8249ae1 | | 2014-10-02 18:45:28 | NULL | NULL | 146 | /dev/vdb | 0 | NULL | 0b529526-4c8d-4650-8295-b7155a977ba7 | NULL | NULL | NULL | 0b529526-4c8d-4650-8295-b7155a977ba7 | 0 | volume | volume | NULL | NULL | NULL | NULL | NULL | +---------------------+---------------------+------------+-----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+--------------------------------------+ 2 rows in set (0.00 sec) ------------------------------------------------------------------------------------------------------------------ To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1377161/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp