On Wed, Dec 17, 2014 at 1:25 AM, Shyam <srang...@redhat.com> wrote: > > This mail intends to present the lock migration across subvolumes problem > and seek solutions/thoughts around the same, so any feedback/corrections > are appreciated. > > # Current state of file locks post file migration during rebalance > Currently when a file is migrated during rebalance, its lock information > is not transferred over from the old subvol to the new subvol, that the > file now resides on. > > As further lock requests, post migration of the file, would now be sent to > the new subvol, any potential lock conflicts would not be detected, until > the locks are migrated over. > > The term locks above can refer to the POSIX locks aquired using the FOP lk > by consumers of the volume, or to the gluster internal(?) inode/dentry > locks. For now we limit the discussion to the POSIX locks supported by the > FOP lk. > > # Other areas in gluster that migrate locks > Current scheme of migrating locks in gluster on graph switches, trigger an > fd migration process that migrates the lock information from the old fd to > the new fd. This is driven by the gluster client stack, protocol layer > (FUSE, gfapi). > > This is done using the (set/get)xattr call with the attr name, > "trusted.glusterfs.lockinfo". Which in turn fetches the required key for > the old fd, and migrates the lock from this old fd to new fd. IOW, there is > very little information transferred as the locks are migrated across fds on > the same subvolume and not across subvolumes. > > Additionally locks that are in the blocked state, do not seem to be > migrated (at least the function to do so in FUSE is empty > (fuse_handle_blocked_locks), need to run a test case to confirm), or > responded to with an error. > > # High level solution requirements when migrating locks across subvols > 1) Block/deny new lock acquisitions on the new subvol, till locks are > migrated > - So that new locks that have overlapping ranges to the older ones are > not granted > - Potentially return EINTR on such requests? > 2) Ensure all _acquired_ locks from all clients are migrated first > - So that if and when placing blocked lock requests, these really do > block for previous reasons and are not granted now > 3) Migrate blocked locks post acquired locks are migrated (in any order?) > - OR, send back EINTR for the blocked locks > > (When we have upcalls/delegations added as features, those would have > similar requirements for migration across subvolumes) > > # Potential processes that could migrate the locks and issues thereof > 1) The rebalance process, that migrates the file can help with migrating > the locks, which would not involve any clients to the gluster volume > > Issues: > - Lock information is fd specific, when migrating these locks, the > clients need not have detected that the file is migrated, and hence opened > an fd against the new subvol, which when missing, would make this form of > migration a little more interesting > - Lock information also has client connection specific pointer > (client_t) that needs to be reassigned on the new subvol > - Other subvol specific information, maintained in the lock, that needs > to be migrated over will suffer the same limitations/solutions >
The tricky thing here is that rebalance process has no control over when 1. fd will be opened on dst-node, since clients open fd on dst-node on-demand based on the I/O happening through them. 2. client establishes connection on dst-node (client might've been cut off from dst-node). Unless we've a global mapping (like a client can always be identified using same uuid irrespective of the brick we are looking) this seems like a difficult thing to achieve. > Benefits: > - Can lock out/block newer lock requests effectively > - Need not _wait_ till all clients have registered that the file is > under migration and/or migrated their locks > > 2) DHT xlator in each client could be held responsible to migrate its > locks to the new subvolume > > Issues: > - Somehow need to let every client know that locks need to be migrated > (upcall infrastructure?) > - What if some client is not reachable at the given time? > - Have to wait till all clients replay the locks > > Benefits: > - Hmmm... Nothing really, if we could do it by the rebalance process > itself the solution maybe better. > > # Overall thoughts > - We could/should return EINTR for blocked locks, in the case of a graph > switch, and the case of a file migration, this would relieve the design of > that particular complexity, and is a legal error to return from a > flock/fcntl operation > > - If we can extract and map out all relevant lock information across > subvolumes, then having rebalance do this work seems like a good fit. > Additionally this could serve as a good way to migrate upcall requests and > state as well > > Thoughts? > > Shyam > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-devel > -- Raghavendra G
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel