** Description changed:
[Impact]
Deleting a share on Manila's CephFS driver with the NFS-Ganesha protocol
helper asks NFS-Ganesha over DBus to remove the share's export.
If that export no longer exists, the RemoveExport call fails and Manila
raises the error instead of completing the deletion.
The share remains in error_deleting, and an administrator has to reset its
state to work around it.
[Test Plan]
- Use a Jammy Manila deployment with a CephFS NFS backend, jammy-proposed
enabled, and access to the NFS-Ganesha host.
+ Use a Jammy Manila deployment with a CephFS NFS backend, and access to the
NFS-Ganesha host.
Create a share, repeat manila show until its status is available, then allow
access from a valid client IP:
- manila create nfs 1 --name testshare
- manila show testshare
- manila access-allow testshare ip <client-ip>
+ manila create nfs 1 --name testshare
+ manila show testshare
+ manila access-allow testshare ip <client-ip>
Find the share's export ID in the Manila logs and remove the runtime
export on the Ganesha host, replacing 1001 with that ID:
- sudo dbus-send --print-reply --system --dest=org.ganesha.nfsd \
- /org/ganesha/nfsd/ExportMgr \
- org.ganesha.nfsd.exportmgr.RemoveExport uint16:1001
+ sudo dbus-send --print-reply --system --dest=org.ganesha.nfsd \
+ /org/ganesha/nfsd/ExportMgr \
+ org.ganesha.nfsd.exportmgr.RemoveExport uint16:1001
Delete the share and check its status after deletion finishes:
- manila delete testshare
- manila list
+ manila delete testshare
+ manila list
Before the fix, testshare remains in error_deleting.
- Install the fixed packages from jammy-proposed and restart the Manila share
service:
+ Install the fixed packages and restart the Manila share service:
- sudo apt-get update
- sudo apt-get install --only-upgrade -t jammy-proposed manila-share
python3-manila
- sudo systemctl restart manila-share
+ sudo apt-get update
+ sudo apt-get install --only-upgrade manila-share python3-manila
+ sudo systemctl restart manila-share
Repeat the steps above with a new share named testshare2; its deletion should
complete and it should disappear from manila list.
As a regression check, create testshare3, wait until manila show reports it
available, allow access and delete it without removing its export:
- manila create nfs 1 --name testshare3
- manila show testshare3
- manila access-allow testshare3 ip <client-ip>
- manila delete testshare3
- manila list
+ manila create nfs 1 --name testshare3
+ manila show testshare3
+ manila access-allow testshare3 ip <client-ip>
+ manila delete testshare3
+ manila list
testshare3 should disappear from manila list.
On the Ganesha host, run RemoveExport for testshare3's export, replacing 1001
with its export ID from the Manila logs:
- sudo dbus-send --print-reply --system --dest=org.ganesha.nfsd \
- /org/ganesha/nfsd/ExportMgr \
- org.ganesha.nfsd.exportmgr.RemoveExport uint16:1001
+ sudo dbus-send --print-reply --system --dest=org.ganesha.nfsd \
+ /org/ganesha/nfsd/ExportMgr \
+ org.ganesha.nfsd.exportmgr.RemoveExport uint16:1001
The call should fail because Manila already removed that export.
[Where problems could occur]
The change affects export removal in Manila's NFS-Ganesha helper, which now
logs and ignores errors while reading or removing an export instead of failing
share deletion.
If a genuine DBus or Ganesha failure prevents the runtime export from being
removed, the share is deleted while that export remains on the Ganesha server,
with the error visible only in the Manila logs.
[Original Description]
User report:
-
- We have some manila shares which are stuck in 'error_deleting' state and
cannot be removed unless their status is reset to 'available'. One thing in
common with these shares is that the instances associated with all these shares
(share_instance_id) do not exist in the environment. Even so, the
'access_rules_status' is still 'active'.
+ We have some manila shares which are stuck in 'error_deleting' state and
cannot be removed unless their status is reset to 'available'. One thing in
common with these shares is that the instances associated with all these shares
(share_instance_id) do not exist in the environment. Even so, the
'access_rules_status' is still 'active'.
After the status is reset to active, the 'access_rules_status' gets set to
'error' state after which we are able to delete it.
For example, share ID: 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c
$ manila show 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c
+---------------------------------------+---------------------------------------------------------------------------+
| Property | Value
|
+---------------------------------------+---------------------------------------------------------------------------+
| id |
67d12f8b-5bb8-4486-83ce-b3a1b4030a9c |
| size | 5
|
| availability_zone | nova
|
| created_at | 2022-12-06T12:31:40.000000
|
| status | error_deleting
|
| name |
pvc-ce7985c3-c3d0-493b-aa70-37253922b057 |
| description |
provisioned-by=manila.csi.openstack.org |
| project_id | 7b4c0133359d44a4ba633f7db8357524
|
| snapshot_id | None
|
| share_network_id | None
|
| share_proto | NFS
|
| metadata |
{'manila.csi.openstack.org/cluster': 'cicd-f58nc'} |
| share_type |
f6b11e95-1072-44a6-8d89-b660993a505e |
| is_public | False
|
| snapshot_support | True
|
| task_state | None
|
| share_type_name | ceph
|
| access_rules_status | active
|
| replication_type | None
|
| has_replicas | False
|
| user_id |
0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 |
| create_share_from_snapshot_support | False
|
| revert_to_snapshot_support | False
|
| share_group_id | None
|
| source_share_group_snapshot_member_id | None
|
| mount_snapshot_support | False
|
| share_server_id | None
|
| host | hostgroup@cephfs#cephfs
|
| export_locations |
|
| | id =
a6b373a8-4c77-42e1-ad03-7dbbab48782a |
| | path =
172.16.33.4:/volumes/_nogroup/a257d35b-d254-4dff-9dac-657bcb1625fc |
| | preferred = False
|
| | share_instance_id =
a257d35b-d254-4dff-9dac-657bcb1625fc |
| | is_admin_only = False
|
+---------------------------------------+---------------------------------------------------------------------------+
$ manila reset-state --state available
67d12f8b-5bb8-4486-83ce-b3a1b4030a9c
$ manila show 67d12f8b-5bb8-4486-83ce-b3a1b4030a9c
+---------------------------------------+---------------------------------------------------------------------------+
| Property | Value
|
+---------------------------------------+---------------------------------------------------------------------------+
| id |
67d12f8b-5bb8-4486-83ce-b3a1b4030a9c |
| size | 5
|
| availability_zone | nova
|
| created_at | 2022-12-06T12:31:40.000000
|
| status | available
|
| name |
pvc-ce7985c3-c3d0-493b-aa70-37253922b057 |
| description |
provisioned-by=manila.csi.openstack.org |
| project_id | 7b4c0133359d44a4ba633f7db8357524
|
| snapshot_id | None
|
| share_network_id | None
|
| share_proto | NFS
|
| metadata |
{'manila.csi.openstack.org/cluster': 'cicd-f58nc'} |
| share_type |
f6b11e95-1072-44a6-8d89-b660993a505e |
| is_public | False
|
| snapshot_support | True
|
| task_state | None
|
| share_type_name | ceph
|
| access_rules_status | error
|
| replication_type | None
|
| has_replicas | False
|
| user_id |
0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9 |
| create_share_from_snapshot_support | False
|
| revert_to_snapshot_support | False
|
| share_group_id | None
|
| source_share_group_snapshot_member_id | None
|
| mount_snapshot_support | False
|
| share_server_id | None
|
| host | hostgroup@cephfs#cephfs
|
| export_locations |
|
| | id =
a6b373a8-4c77-42e1-ad03-7dbbab48782a |
| | path =
172.16.33.4:/volumes/_nogroup/a257d35b-d254-4dff-9dac-657bcb1625fc |
| | preferred = False
|
| | share_instance_id =
a257d35b-d254-4dff-9dac-657bcb1625fc |
| | is_admin_only = False
|
+---------------------------------------+---------------------------------------------------------------------------+
-
Some of the logs indicate that manila is not able to update the rules of
the shares which we are trying to delete where the share_instance_id for
that share does not exist in the environment.
For example:
Share ID: 5b5d2a38-2a46-4d81-8690-ba9cc94629a6
Request ID: req-d7e38b71-85db-444c-8689-f0d6298ffbce
~~~
manila-api.log.2.gz:2023-01-30 10:19:00.712 21 INFO manila.api.openstack.wsgi
[req-d7e38b71-85db-444c-8689-f0d6298ffbce
0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9
7b4c0133359d44a4ba633f7db8357524 - - -] DELETE
https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13786/v2/7b4c0133359d44a4ba633f7db8357524/shares/5b5d2a38-2a46-4d81-8690-ba9cc94629a6
manila-api.log.2.gz:2023-01-30 10:19:00.713 21 INFO manila.api.v1.shares
[req-d7e38b71-85db-444c-8689-f0d6298ffbce
0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9
7b4c0133359d44a4ba633f7db8357524 - - -] Delete share with id:
5b5d2a38-2a46-4d81-8690-ba9cc94629a6
manila-share.log:2023-01-30 10:19:01.742 42 DEBUG manila.share.access
[req-d7e38b71-85db-444c-8689-f0d6298ffbce
0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9
7b4c0133359d44a4ba633f7db8357524 - - -] Updating access rules for share
instance cf864d1b-8d5f-4eea-96a0-454ad90ab8b3 belonging to share
5b5d2a38-2a46-4d81-8690-ba9cc94629a6. update_access_rules
/usr/lib/python3.6/site-packages/manila/share/access.py:281
manila-share.log:2023-01-30 10:19:02.120 42 DEBUG
oslo_concurrency.processutils [req-d7e38b71-85db-444c-8689-f0d6298ffbce
0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9
7b4c0133359d44a4ba633f7db8357524 - - -] Running cmd (subprocess): dbus-send
--print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr
org.ganesha.nfsd.exportmgr.RemoveExport uint16:11406 execute
/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:372
manila-share.log:2023-01-30 10:19:02.149 42 DEBUG
oslo_concurrency.processutils [req-d7e38b71-85db-444c-8689-f0d6298ffbce
0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9
7b4c0133359d44a4ba633f7db8357524 - - -] CMD "dbus-send --print-reply --system
--dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr
org.ganesha.nfsd.exportmgr.RemoveExport uint16:11406" returned: 1 in 0.029s
execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409
manila-share.log:2023-01-30 10:19:02.149 42 DEBUG
oslo_concurrency.processutils [req-d7e38b71-85db-444c-8689-f0d6298ffbce
0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9
7b4c0133359d44a4ba633f7db8357524 - - -] 'dbus-send --print-reply --system
--dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr
org.ganesha.nfsd.exportmgr.RemoveExport uint16:11406' failed. Not Retrying.
execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:457
manila-share.log:2023-01-30 10:19:02.150 42 ERROR
manila.share.drivers.ganesha.manager [req-d7e38b71-85db-444c-8689-f0d6298ffbce
0a3daf14f6aecb825dbf414fb7eac3504d538f3ad0e53a2d0df84ce2e4ac3af9
7b4c0133359d44a4ba633f7db8357524 - - -] Error while executing management
command on Ganesha node <no name>: dbus call exportmgr.RemoveExport.:
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
running command.
~~~
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2035572
Title:
Unable to delete CephFS manila share if nfs-ganesha export rule
doesn't exist
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2035572/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
