[Yahoo-eng-team] [Bug 1808456] Re: ceph backend reporting meaningless error when no space left
** Changed in: glance-store
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1808456
Title:
ceph backend reporting meaningless error when no space left
Status in Glance:
Invalid
Status in glance_store:
Fix Released
Bug description:
When uploading image, but there's no space left in ceph(rbd) backend,
client(such as glanceclient) will receive a meaningless error:
500 Internal Server Error: The server has either erred or is incapable of
performing the requested operation. (HTTP 500)
steps to reproduce:
-
1.Prepare ceph backend for glance, make the free space small enough, e.g.
10MB.
To be simple, you also can modify ceph's code(function resize), to let it
raise errno.ENOSPC. I did this way.
2.uploading image: glance image-create --name img2-ceph --visibility public
--disk-format raw --container-format bare --progress --backend rbd --file
/opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
logs
-
stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility
public --disk-format raw --container-format bare --progress --backend rbd
--file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
>
/usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
-> backend = None
(Pdb) c
[=>] 100%
+--+--+
| Property | Value|
+--+--+
| checksum | None |
| container_format | bare |
| created_at | 2018-12-14T02:08:36Z |
| disk_format | raw |
| id | 8c2e48f0-aafc-4744-95b6-fe0b6fbfe975 |
| min_disk | 0|
| min_ram | 0|
| name | img2-ceph|
| os_hash_algo | None |
| os_hash_value| None |
| os_hidden| False|
| owner| 3242a198f7044fcd9b756866ec296391 |
| protected| False|
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2018-12-14T02:08:36Z |
| virtual_size | Not available|
| visibility | public |
+--+--+
500 Internal Server Error: The server has either erred or is incapable of
performing the requested operation. (HTTP 500)
expected
-
The correct message should be something related "Storage Full", rbd.py should
raise glance_store.StorageFull, and this exception will be caught by
notifier.py.
Some code snippet in notifier.py:
except glance_store.StorageFull as e:
msg = (_("Image storage media is full: %s") %
encodeutils.exception_to_unicode(e))
_send_notification(notify_error, 'image.upload', msg)
raise webob.exc.HTTPRequestEntityTooLarge(explanation=msg)
After doing this, the expected behavior will be:
stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility
public --disk-format raw --container-format bare --progress --backend rbd
--file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
>
/usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
-> backend = None
(Pdb) c
[=>] 100%
+--+--+
| Property | Value|
+--+--+
| checksum | None |
| container_format | bare |
| created_at | 2018-12-14T01:41:36Z |
| disk_format | raw |
| id | 8aefa92d-bd9c-4726-95ae-d8f698d7bc82 |
| min_disk | 0|
| min_ram | 0|
| name | img2-ceph|
| os_hash_algo | None |
| os_hash_value| None |
| os_hidden| False
[Yahoo-eng-team] [Bug 1808456] Re: ceph backend reporting meaningless error when no space left
** Also affects: glance-store
Importance: Undecided
Status: New
** Changed in: glance
Status: New => Invalid
** Changed in: glance-store
Importance: Undecided => Low
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1808456
Title:
ceph backend reporting meaningless error when no space left
Status in Glance:
Invalid
Status in glance_store:
New
Bug description:
When uploading image, but there's no space left in ceph(rbd) backend,
client(such as glanceclient) will receive a meaningless error:
500 Internal Server Error: The server has either erred or is incapable of
performing the requested operation. (HTTP 500)
steps to reproduce:
-
1.Prepare ceph backend for glance, make the free space small enough, e.g.
10MB.
To be simple, you also can modify ceph's code(function resize), to let it
raise errno.ENOSPC. I did this way.
2.uploading image: glance image-create --name img2-ceph --visibility public
--disk-format raw --container-format bare --progress --backend rbd --file
/opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
logs
-
stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility
public --disk-format raw --container-format bare --progress --backend rbd
--file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
>
/usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
-> backend = None
(Pdb) c
[=>] 100%
+--+--+
| Property | Value|
+--+--+
| checksum | None |
| container_format | bare |
| created_at | 2018-12-14T02:08:36Z |
| disk_format | raw |
| id | 8c2e48f0-aafc-4744-95b6-fe0b6fbfe975 |
| min_disk | 0|
| min_ram | 0|
| name | img2-ceph|
| os_hash_algo | None |
| os_hash_value| None |
| os_hidden| False|
| owner| 3242a198f7044fcd9b756866ec296391 |
| protected| False|
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2018-12-14T02:08:36Z |
| virtual_size | Not available|
| visibility | public |
+--+--+
500 Internal Server Error: The server has either erred or is incapable of
performing the requested operation. (HTTP 500)
expected
-
The correct message should be something related "Storage Full", rbd.py should
raise glance_store.StorageFull, and this exception will be caught by
notifier.py.
Some code snippet in notifier.py:
except glance_store.StorageFull as e:
msg = (_("Image storage media is full: %s") %
encodeutils.exception_to_unicode(e))
_send_notification(notify_error, 'image.upload', msg)
raise webob.exc.HTTPRequestEntityTooLarge(explanation=msg)
After doing this, the expected behavior will be:
stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility
public --disk-format raw --container-format bare --progress --backend rbd
--file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
>
/usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
-> backend = None
(Pdb) c
[=>] 100%
+--+--+
| Property | Value|
+--+--+
| checksum | None |
| container_format | bare |
| created_at | 2018-12-14T01:41:36Z |
| disk_format | raw |
| id | 8aefa92d-bd9c-4726-95ae-d8f698d7bc82 |
| min_disk | 0|
| min_ram | 0|
| name | img2-ceph|
| os_hash_algo | None
