** Changed in: glance-store Status: Fix Committed => Fix Released
-- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1401778 Title: 500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled Status in Glance: Triaged Status in glance_store: Fix Released Bug description: When we try to upload image with multi filesystem store enabled and 'filesystem_store_metadata_file' (containing list of all directories configured for multi filesystem store) is provided then glance throws 'HTTPInternalServerError (HTTP 500)' error. - Glance Configuration: 1. /etc/glance/glance-api.conf [DEFAULT] show_multiple_locations = True filesystem_store_metadata_file = /etc/glance/metadata.json [glance_store] filesystem_store_datadirs = /var/lib/glance/images1/:1 filesystem_store_datadirs = /var/lib/glance/images2/:2 2. /etc/glance/metadata.json [ { "id": "f0781415-cf81-47cd-8860-b83f9c2a415c", "mountpoint": "/var/lib/glance/images1/" }, { "id": "5d2dd1db-8684-46bb-880f-b94a1942cfd2", "mountpoint": "/var/lib/glance/images2/" } ] 3. 'df -ha' command result: openstack@openstack:/opt/stack$ df -ha Filesystem Size Used Avail Use% Mounted on proc 0 0 0 - /proc sysfs 0 0 0 - /sys tmpfs 799M 600K 798M 1% /run /dev/sda1 236M 41M 183M 19% /boot nfsd 0 0 0 - /proc/fs/nfsd 10.69.4.173:/export/images2 443G 23G 398G 6% /var/lib/glance/images2 10.69.4.172:/export/images1 447G 8.7G 415G 3% /var/lib/glance/images1 openstack@openstack:/opt/stack$ - Steps to reproduce: 1. Create image: glance --os-image-api-version 2 image-create --name Test123 --disk-format raw --container-format ami 2. Upload image data: openstack@openstack-150:~$ glance --os-image-api-version 2 image-upload 47d39050-cc7e-498a-a800-4faf80a72c93 < /home/openstack/workbook/test.py HTTPInternalServerError (HTTP 500) openstack@openstack-150:~$ - glance-api.log : 2014-12-11 22:16:59.586 3495 ERROR glance_store.backend [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] The storage driver <glance_store._drivers.filesystem.Store object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db- 8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type 2014-12-11 22:16:59.591 3495 ERROR glance.api.v2.image_data [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] Failed to upload image data due to internal error 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data Traceback (most recent call last): 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/v2/image_data.py", line 74, in upload 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data image.set_data(data, size) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/domain/proxy.py", line 160, in set_data 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.base.set_data(data, size) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 252, in set_data 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.notifier.error('image.upload', msg) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 201, in set_data 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/policy.py", line 176, in set_data 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return self.image.set_data(*args, **kwargs) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/quota/__init__.py", line 296, in set_data 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size=size) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/location.py", line 364, in set_data 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data context=self.context) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 357, in add_to_backend 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return store_add_to_backend(image_id, data, size, store, context) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 338, in store_add_to_backend 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data raise exceptions.BackendException(msg) 2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data BackendException: The storage driver <glance_store._drivers.filesystem.Store object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type When we enable 'filesystem_store_metadata_file' file option in glance-api.conf then it adds this json data as metadata for all glance images. This data is then used by NOVA to directly copy image using 'cp' command. (when glance image directories are shared between nova-compute and glance) For direct download in nova.conf we need to enable 'allowed_direct_url_schemes' option. Also nova needs 'id' and 'moutpoint' keys to be present in glance-image metadata for direct image copy. If there is only one glance image directory (ie- one nfs mountpoint) then we do not get this '500 internal server error'. But if there are multiple image data directories configured and in metadata JSON file we give list of dicts containing all mountpoints then we get above error. IMO we need to validate glance metadata JSON file (for 'id' and 'mountpoint' keys) before adding it as image-metadata and if there are multiple mountpoints (directories) configured for glance then we need to add appropriate metadata to glance-image instead of adding all image metadata which is provided in JSON file. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1401778/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp