Public bug reported: full glance log: https://paste.ubuntu.com/p/fVrj5vfc7m/
Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.577 55 DEBUG glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] creating image fdae789d-10ce-40f5-9c50-c8b206b81531 with order 23 and size 0 add /openstack/venvs/glance-22.0.1/lib/python3.8/site-packages/glance_store/_drivers/rbd.py:555 Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.578 55 WARNING glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Since image size is zero we will be doing resize-before-write which will be slower than normal Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.642 55 DEBUG glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] resizing image to 8192.0 KiB _resize_on_write /openstack/venvs/glance-22.0.1/lib/python3.8/site-packages/glance_store/_drivers/rbd.py:505 Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.761 55 ERROR glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Failed to store image fdae789d-10ce-40f5-9c50-c8b206b81531 Store Exception RBD incomplete write (Wrote only 8388608 out of 8394566 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.809 55 DEBUG glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Snap Operating Exception [errno 2] RBD image not found (error unprotecting snapshot b'fdae789d-10ce-40f5-9c50-c8b206b81531'@b'snap') Snapshot does not exist. _delete_image /openstack/venvs/glance-22.0.1/lib/python3.8/site-packages/glance_store/_drivers/rbd.py:464 Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.984 55 ERROR glance.api.v2.image_data [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Failed to upload image data due to internal error: rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) Feb 20 15:18:36 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:36.037 55 ERROR glance.common.wsgi [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Caught error: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) ** Affects: glance Importance: Undecided Status: New ** Attachment added: "glance debug log" https://bugs.launchpad.net/bugs/1916482/+attachment/5465930/+files/bug ** Summary changed: - Snapshots fail to be written to ceph/rbd backend since upgrade to Victoria release. Ceph v Nautilus + rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) since Victoria Upgrade, ceph v nautilus -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1916482 Title: rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) since Victoria Upgrade, ceph v nautilus Status in Glance: New Bug description: full glance log: https://paste.ubuntu.com/p/fVrj5vfc7m/ Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.577 55 DEBUG glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] creating image fdae789d-10ce-40f5-9c50-c8b206b81531 with order 23 and size 0 add /openstack/venvs/glance-22.0.1/lib/python3.8/site-packages/glance_store/_drivers/rbd.py:555 Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.578 55 WARNING glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Since image size is zero we will be doing resize-before-write which will be slower than normal Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.642 55 DEBUG glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] resizing image to 8192.0 KiB _resize_on_write /openstack/venvs/glance-22.0.1/lib/python3.8/site-packages/glance_store/_drivers/rbd.py:505 Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.761 55 ERROR glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Failed to store image fdae789d-10ce-40f5-9c50-c8b206b81531 Store Exception RBD incomplete write (Wrote only 8388608 out of 8394566 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.809 55 DEBUG glance_store._drivers.rbd [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Snap Operating Exception [errno 2] RBD image not found (error unprotecting snapshot b'fdae789d-10ce-40f5-9c50-c8b206b81531'@b'snap') Snapshot does not exist. _delete_image /openstack/venvs/glance-22.0.1/lib/python3.8/site-packages/glance_store/_drivers/rbd.py:464 Feb 20 15:18:35 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:35.984 55 ERROR glance.api.v2.image_data [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Failed to upload image data due to internal error: rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) Feb 20 15:18:36 xxx-glance-container-071516d6 glance-wsgi-api[55]: 2021-02-20 15:18:36.037 55 ERROR glance.common.wsgi [req-663e7646-b12e-4e4c-ad18-d4b23644a504 23fd1e8e87af40b88418b118085104cf 946f0e543169462596dbaaf7504f7a4a - default default] Caught error: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1916482/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp