Re: [PATCH 0/5] Add support for object properties
On Tue, Jan 07, 2014 at 11:51:08AM +, Filipe David Manana wrote: There seems to be a lot of details for this feature, and many other projects (such as those you point on the wiki) would benefit from properties. It might be hard to get all requirements at once, so I think it might be better to start simple, but in a way that doesn't prevent adding the more advanced requirements later. Absolutely, that's why I'm trying to gather all potential future requirements and usecases now so we don't become surpristed later. Although it's detailed, there are groups that are covered by saying eg. this is per-device, stored here, accessed trhough this xattr. If the proposal is able to give such answers, then it's what I'm expecting at this phase. As for the xattrs namespace: yes, we can capture the btrfs. namespace. hfsplus does this, it captures the osx. namespace. See my reply to Hugo about this on this thread: http://comments.gmane.org/gmane.comp.file-systems.btrfs/29972 JFYI, there's a discussion on fsdevel about replacing the existing file attributes ioctl with something else, xattr's are considered as the UI/storage, so it would need some coordination to be consistent with whatever gets decided. My work so far, for the kernel side, was all about inode (and subvolume) properties. Haven't considered properties for devices so far. I hope it's clear that I'm not asking about implementing all the properties from the beginning. I's fine that you're focused on inode props, get the things working so we have something tangible to play with. If the device props are not forgottent to add to some high-level xattr namespace, I'm satisfied. By using xattrs, while simplifying the storage and not touching/extending the on-disk format, it will always have some impact on performance - they take leaf space, hence btrees will be larger than before. Not all of the props need to end up as a xattr, we can still use the existing inode-flags field that's used for the file attributes, there are 11 bits occupied, plenty of room to add more where it would make sense. The rest shall be in xattr items. I think we need to clearly define what are the initial requirements, start with a simple approach, always thinking on how to extend/refine it later without the need for breaking backwards compatibility later of course. Well, I think once the xattr naming scheme is set it won't be changed or refined. I'm expecting only specific properties appended. Eg. if you propose btrfs.compression as an inode property, then it's a no-go to move it to btrfs.fs.compression once a whole-filesystem gains persistent option, and refine again when somebody like me comes and says that any *.compression should really be a napespace that groups all compression-related options. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Add support for object properties
On Sat, Nov 23, 2013 at 12:52 AM, David Sterba dste...@suse.cz wrote: On Tue, Nov 12, 2013 at 01:41:41PM +, Filipe David Borba Manana wrote: This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices). Currently the command group looks like this: btrfs prop set [-t type] /path/to/object name value btrfs prop get [-t type] /path/to/object [name] (omitting name dumps all) btrfs prop list [-t type] /path/to/object (lists properties with description) The type is used to explicitly specify what type of object you mean. This is necessary in case the object+property combination is ambiguous. For example '/path/to/fs/root' could mean the root subvolume, the directory inode or the filesystem itself. Normally, btrfs-progs will try to detect the type automatically. The generic commandline UI still looks ok to me, storing properties as xattr’s is also ok (provided that we can capture the “btrfs.” xattr namespace). I’ll dump my thoughts and questions about the rest. 1) Where are stored properties that are not directly attached to an inode? Ie. whole-filesystem and device. How can I access props for a subvolume that is not currently reachable in the directory tree? A fs or device props must be accessible any time, eg. no matter which subvolume is currently mounted. This should be probably a special case where the property can be queried from any inode but will be internally routed to eg. toplevel subvolume that will store the respective xattrs. 2) if a property’s object can be ambiguous, how is that distinguished in the xattrs? We don’t have a list of props yet, so I’m trying to use one that hopefully makes some sense. The example here can be persistent-mount-options that are attached to fs and a subvolume. The fs-wide props will apply when a different subvolume is explicitly mounted. Assuming that the xattrs are stored with the toplevel subvolume, the fs-wide and per-subvolume property must have a differnt xattr name (another option is to store fs-wide elsewhere). So the question is, if we should encode the property object into the xattr name directly. Eg.: btrfs.fs.persistent_mount btrfs.subvol.persistent_mount or if the fs-wide uses a reserved naming scheme that would appear as xattr named btrfs.persistent_mount but the value would differ if queried with ‘-t fs or ‘-t subvolume’. 3) property precedence, interaction with mount options The precedence should follow the rule of the least surprise, ie. if I set eg. a compression of a file to zlib, set the subvolume compression type to ‘none’ and have fs-wide mount the filesystem with compress=lzo, I’m expecting that the file will use ‘zlib’. The generic rule says that mount options (that have a corresponding property) take precedence. There may be exceptions. What if there are no specific mount options, but the subvolume has eg. compression set? Should new files inherit that automatically or only if explicitly marked as such? The background concern is how far do I need to look and whether this could be a potential performance problem. The lookup chain can be long: inode - dir - subvol - fs, plus mount options along the path where applicable. If the xattr values are cached in structs that are accessed anyway (containing subvol, fs_info) and updated when property xattr changes, this could work. 4) behaviour of tools that see the btrfs-specific props What would cp or rsync do when copying the xattrs to a different filesystem? They may simply ignore it and just try to copy the user. namespace, I haven’t done any research here. 5) properties to consider, naming, xattr name namespaces, inheritable props Here’s a list of properties that I’ve collected so far: filesystem-wide - persistent mount options - default raid allocation profile - default compression parameters - label - scrub ioprio subvolume - persistent mount options - default raid allocation profile - default compression parameters - writable device - writable/ro/dead - hot-spare - preferred for metadata - preferred data reads - preferred data writes - no new allocations if possible - speed profile, io priorities - allocation group - from project idea Chunk allocation groups - ssd cache, L2ARC - scrub ioprio file, directory - compression parameters - raid allocation profile details on compression parameters: - compression algo - compression level - desired compression ratio target - file compressibility hint - from project idea “Compression updates” - compression container type - dtto The props should be basically self explanatory. This is not a complete or finalized list, feel free to suggest what you think should/not be here. The number of
Re: [PATCH 0/5] Add support for object properties
Hi David, On 2013-11-23 01:52, David Sterba wrote: On Tue, Nov 12, 2013 at 01:41:41PM +, Filipe David Borba Manana wrote: This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices). Currently the command group looks like this: btrfs prop set [-t type] /path/to/object name value btrfs prop get [-t type] /path/to/object [name] (omitting name dumps all) btrfs prop list [-t type] /path/to/object (lists properties with description) The type is used to explicitly specify what type of object you mean. This is necessary in case the object+property combination is ambiguous. For example '/path/to/fs/root' could mean the root subvolume, the directory inode or the filesystem itself. Normally, btrfs-progs will try to detect the type automatically. The generic commandline UI still looks ok to me, storing properties as xattr’s is also ok (provided that we can capture the “btrfs.” xattr namespace). I’ll dump my thoughts and questions about the rest. 1) Where are stored properties that are not directly attached to an inode? Ie. whole-filesystem and device. How can I access props for a subvolume that is not currently reachable in the directory tree? A fs or device props must be accessible any time, eg. no matter which subvolume is currently mounted. This should be probably a special case where the property can be queried from any inode but will be internally routed to eg. toplevel subvolume that will store the respective xattrs. I think that we should divided how access the properties and where the properties are stored. - Storing the *inode* properties in xattrs makes sense: it is a clean interface, and the infrastructure is capable to hold all the informations. It could be discussed also if we need to use 1 property - 1 xattr or use one xattr to hold more properties (this could alleviate some performance problem) - Storing the *subvolume* and the *filesystem* properties in an xattr associated to subvolue or '/' inode could be done. When it is accessed this subvolume or '/' inode (during the mount and/or inode traversing) all the information could be read from the xattr and stored in the btrfs_fs_info struct and btrfs_root struct (which are easily accessible from the inode, avoiding performance issues) - For the *device* properties, we could think to use other trees like the device tree to store the information but still using the *xattr interfaces to access them. 2) if a property’s object can be ambiguous, how is that distinguished in the xattrs? We don’t have a list of props yet, so I’m trying to use one that hopefully makes some sense. The example here can be persistent-mount-options that are attached to fs and a subvolume. The fs-wide props will apply when a different subvolume is explicitly mounted. Assuming that the xattrs are stored with the toplevel subvolume, the fs-wide and per-subvolume property must have a differnt xattr name (another option is to store fs-wide elsewhere). So the question is, if we should encode the property object into the xattr name directly. Eg.: btrfs.fs.persistent_mount btrfs.subvol.persistent_mount or if the fs-wide uses a reserved naming scheme that would appear as xattr named btrfs.persistent_mount but the value would differ if queried with ‘-t fs or ‘-t subvolume’. 3) property precedence, interaction with mount options The precedence should follow the rule of the least surprise, ie. if I set eg. a compression of a file to zlib, set the subvolume compression type to ‘none’ and have fs-wide mount the filesystem with compress=lzo, I’m expecting that the file will use ‘zlib’. The generic rule says that mount options (that have a corresponding property) take precedence. There may be exceptions. As general rule I suggest the following priority list: fs props, subvol props, dir props, inode props, mount opts It should be possible to override via mount option all options (to force some behaviour). However I have some doubts about: 1) in case of nested subvolumes, should the inner subvolumes inherit the properties of the outer subvolumes? I think no, because it is not so obvious the hierarchy when a subvolume is mounted (or moved). 2) there are properties that are inode related. Other no. For example does make sense to have inode-setting about compression, raid profile, datasum/cow ... when the data is shared between different inode/subvolume which could have different setup? Which should be the least surprise behaviour ? Or we should intend these properties only as hints for the new file/data/chunk ? Anyway what it would do a balance in these case ? I am inclined to think that the storage policy (raid profile, compression, cow...) should be per
Re: [PATCH 0/5] Add support for object properties
On Tue, Nov 12, 2013 at 01:41:41PM +, Filipe David Borba Manana wrote: This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices). Currently the command group looks like this: btrfs prop set [-t type] /path/to/object name value btrfs prop get [-t type] /path/to/object [name] (omitting name dumps all) btrfs prop list [-t type] /path/to/object (lists properties with description) The type is used to explicitly specify what type of object you mean. This is necessary in case the object+property combination is ambiguous. For example '/path/to/fs/root' could mean the root subvolume, the directory inode or the filesystem itself. Normally, btrfs-progs will try to detect the type automatically. The generic commandline UI still looks ok to me, storing properties as xattr’s is also ok (provided that we can capture the “btrfs.” xattr namespace). I’ll dump my thoughts and questions about the rest. 1) Where are stored properties that are not directly attached to an inode? Ie. whole-filesystem and device. How can I access props for a subvolume that is not currently reachable in the directory tree? A fs or device props must be accessible any time, eg. no matter which subvolume is currently mounted. This should be probably a special case where the property can be queried from any inode but will be internally routed to eg. toplevel subvolume that will store the respective xattrs. 2) if a property’s object can be ambiguous, how is that distinguished in the xattrs? We don’t have a list of props yet, so I’m trying to use one that hopefully makes some sense. The example here can be persistent-mount-options that are attached to fs and a subvolume. The fs-wide props will apply when a different subvolume is explicitly mounted. Assuming that the xattrs are stored with the toplevel subvolume, the fs-wide and per-subvolume property must have a differnt xattr name (another option is to store fs-wide elsewhere). So the question is, if we should encode the property object into the xattr name directly. Eg.: btrfs.fs.persistent_mount btrfs.subvol.persistent_mount or if the fs-wide uses a reserved naming scheme that would appear as xattr named btrfs.persistent_mount but the value would differ if queried with ‘-t fs or ‘-t subvolume’. 3) property precedence, interaction with mount options The precedence should follow the rule of the least surprise, ie. if I set eg. a compression of a file to zlib, set the subvolume compression type to ‘none’ and have fs-wide mount the filesystem with compress=lzo, I’m expecting that the file will use ‘zlib’. The generic rule says that mount options (that have a corresponding property) take precedence. There may be exceptions. What if there are no specific mount options, but the subvolume has eg. compression set? Should new files inherit that automatically or only if explicitly marked as such? The background concern is how far do I need to look and whether this could be a potential performance problem. The lookup chain can be long: inode - dir - subvol - fs, plus mount options along the path where applicable. If the xattr values are cached in structs that are accessed anyway (containing subvol, fs_info) and updated when property xattr changes, this could work. 4) behaviour of tools that see the btrfs-specific props What would cp or rsync do when copying the xattrs to a different filesystem? They may simply ignore it and just try to copy the user. namespace, I haven’t done any research here. 5) properties to consider, naming, xattr name namespaces, inheritable props Here’s a list of properties that I’ve collected so far: filesystem-wide - persistent mount options - default raid allocation profile - default compression parameters - label - scrub ioprio subvolume - persistent mount options - default raid allocation profile - default compression parameters - writable device - writable/ro/dead - hot-spare - preferred for metadata - preferred data reads - preferred data writes - no new allocations if possible - speed profile, io priorities - allocation group - from project idea Chunk allocation groups - ssd cache, L2ARC - scrub ioprio file, directory - compression parameters - raid allocation profile details on compression parameters: - compression algo - compression level - desired compression ratio target - file compressibility hint - from project idea “Compression updates” - compression container type - dtto The props should be basically self explanatory. This is not a complete or finalized list, feel free to suggest what you think should/not be here. The number of compression parameters suggest that a namespace would be appropriate. This could establish a common practice for any set of properties that are not attached to
Re: [PATCH 0/5] Add support for object properties
On Tue, Nov 12, 2013 at 01:41:41PM +, Filipe David Borba Manana wrote: This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices). Thank you very much for working on that. I've added patches 2-5 to the unstable part of integration branch. More comments later. david -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/5] Add support for object properties
This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices). I preserved most of Alexander's work, rebasing only his main patch against latest integration branch, adding some missing static qualifiers, malloc failure checks and fixing some checkpatch.pl warnings. On top of his work, I added a few fixes and enhancements and added support for 1 inode property, named compression, which requires the corresponding kernel patch. Once there's agreement on this feature and patchset, I'll send my xfstests patch that tests this feature. Alexander's original cover letter: This patchset introduces the btrfs property subgroup. It is the result of a discussion we had on IRC. I tried to make the properties interface as generic and extensible as possible. Comments are welcome. Currently the command group looks like this: btrfs prop set [-t type] /path/to/object name value btrfs prop get [-t type] /path/to/object [name] (omitting name dumps all) btrfs prop list [-t type] /path/to/object (lists properties with description) The type is used to explicitly specify what type of object you mean. This is necessary in case the object+property combination is ambiguous. For example '/path/to/fs/root' could mean the root subvolume, the directory inode or the filesystem itself. Normally, btrfs-progs will try to detect the type automatically. David suggested that it should also be possible to specify objects by their id/uuid/fsid. I like that idea, but would be happy if someone else could take over that part :) For now, I've implemented two properties: 1. read-only. Usable on subvolumes to toggle the read-only flags. 2. label. I looked through btrfs to find good examples of things that could be moved to the new properties interface and the filesystem label looked like a good one. There are for sure more, but that is something for later (and maybe for someone else). I would suggest to move everthing that makes sense over to the props interface and mark the old interfaces as deprecated. Comments on this are welcome. Patch version history: v1 Initial version. v2 - Removed the filesystem prefix and implemented it as new command group - Switched from the name[=value] form to the set/get name [value] form. - Removed patches Btrfs-progs: make filesystem_cmd_group non const and Btrfs-progs: move skip_prefix and prefixcmp to utils.c. They are not needed anymore due to the 'btrfs prop list' command. - Udjusted the subvol flags patch to be compatible to the Btrfs: use _IOR for BTRFS_IOC_SUBVOL_GETFLAGS patch. - Using -t type instead of type: prefix now. - Changes are based on feedback from Ilya and David. Alex. Thanks. Alexander Block (1): Btrfs-progs: introduce btrfs property subgroup Filipe David Borba Manana (4): Btrfs-progs: let get_label return the label instead of printing it Btrfs-progs: fix detection of root objects in cmds-property.c Btrfs-progs: add type root to label property Btrfs-progs: add support for the compression property Makefile |4 +- btrfs.c |1 + cmds-filesystem.c | 14 +- cmds-property.c | 467 + commands.h|2 + ioctl.h | 12 ++ props.c | 184 + props.h | 43 + utils.c | 15 +- utils.h |2 +- 10 files changed, 729 insertions(+), 15 deletions(-) create mode 100644 cmds-property.c create mode 100644 props.c create mode 100644 props.h -- 1.7.9.5 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
