Wracking my brains over this one, would appreciate any pointers...
Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu
Bionic. The first compute node is an NFS server for
/var/lib/nova/instances, and the other compute nodes mount that as NFS
clients.
Problem: Sometimes, when launching an instance which is scheduled to one of
the client nodes, nova-compute (in imagebackend.py) gets Permission Denied
(errno 13) when calling utime to touch the timestamp on the instance file.
Through various bits of debugging and hackery, I've established that:
- it looks like the problem never occurs when this is the call that
bootstraps the privsep setup; but it does occur quite frequently on later
calls
- when the problem occurs, retrying doesn't help (5 times, with 0.5s in
between)
- the instance file does exist, and is owned by root with read/write
permission for root
- the privsep helper is running as root
- the privsep helper receives and executes the request - so it's not a
problem with communication between nova-compute and the helper
- root is uid 0 on both NFS server and client
- NFS setup does not have the root_squash option
- there is some AppArmor setup, on both client and server, and I haven't
yet worked out whether that might be relevant.
Any ideas?
Many thanks,
Neil
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
