Re: [OpenAFS-devel] Re: AFS ... or equivalent ...
On Mon, Feb 04, 2008 at 12:58:29AM -0500, Kyle Moffett wrote: > On Jan 16, 2008 1:48 PM, Jeffrey Hutzelman <[EMAIL PROTECTED]> wrote: > > The "let's just slurp everything into the main distribution so we don't > > have to worry about stable interfaces" approach is really poor. It > > encourages bad engineering practice among people maintaining the main > > distribution, discourages innovation and extension by others, and generally > > doesn't scale. It's far better to either attempt to maintain stable > > external interfaces to the VFS and VM subsystems, or else admit that you > > don't have the resources to do so given the relatively small number of > > external users, in which case you almost certainly also don't have the > > resources to keep on top of updates to something like OpenAFS. > > The Linux Kernel presents a very strong counter-argument-by-example. > The amount of patches merged per released version has been linearly > increasing over the last several years; the 2.6.23 => 2.6.24 patch was > 49MB uncompressed, with a 5.7MB changelog. Of that, a significant > portion were VFS changes which touched most filesystems. The various > filesystem-related changes alone between 2.6.23 and 2.6.24 were > 2.9MB. So, there are reasons why many of us prefer FreeBSD to Linux. jerry For reference, the *entire* OpenAFS diff between 2.4.6 and > 2.5.30 is all of 8.2MB. The Linux Kernel changes include partial > support for having per-process views of a single filesystem > (Specifically /proc, so /proc/net can have differing contents between > network namespaces). Other features which Linux supports that > virtually no other OS does is multiple filesystem namespaces, where > the mount-tree is selectively independent or shared between > namespaces. > ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OpenAFS-devel] Re: AFS ... or equivalent ...
On Jan 16, 2008 1:48 PM, Jeffrey Hutzelman <[EMAIL PROTECTED]> wrote: > The "let's just slurp everything into the main distribution so we don't > have to worry about stable interfaces" approach is really poor. It > encourages bad engineering practice among people maintaining the main > distribution, discourages innovation and extension by others, and generally > doesn't scale. It's far better to either attempt to maintain stable > external interfaces to the VFS and VM subsystems, or else admit that you > don't have the resources to do so given the relatively small number of > external users, in which case you almost certainly also don't have the > resources to keep on top of updates to something like OpenAFS. The Linux Kernel presents a very strong counter-argument-by-example. The amount of patches merged per released version has been linearly increasing over the last several years; the 2.6.23 => 2.6.24 patch was 49MB uncompressed, with a 5.7MB changelog. Of that, a significant portion were VFS changes which touched most filesystems. The various filesystem-related changes alone between 2.6.23 and 2.6.24 were 2.9MB. For reference, the *entire* OpenAFS diff between 2.4.6 and 2.5.30 is all of 8.2MB. The Linux Kernel changes include partial support for having per-process views of a single filesystem (Specifically /proc, so /proc/net can have differing contents between network namespaces). Other features which Linux supports that virtually no other OS does is multiple filesystem namespaces, where the mount-tree is selectively independent or shared between namespaces. I realize that some people are probably already aware of most of that, but I thought it should be mentioned that "slurp everything into the main distribution" actually scales very well with respect to the Linux kernel. It means that the people who are making changes (to the VFS, for example) have to go around and fix *all* the filesystems, and in addition when a bug gets fixed in one filesystem then most of the others get checked for that same bug. OpenAFS also does not benchmark very well under Linux against most of the other networked filesystems (even ones using encryption), as it does not support the fine-grained locking that Linux does. Unfortunately it isn't practical for Linux to reuse existing OpenAFS code as the licenses are partially incompatible. > In the long run, I'm guessing that the OpenAFS cache manager evolves more > quickly than FreeBSD's VFS interface, which makes pulling the CM into the > kernel tree a losing battle. If you disagree, by all means fork that part > of AFS (or get someone else to do so) and see what happens (AFS's > user/kernel and RPC interfaces are both fairly stable, so forking just the > kernel parts should be mostly feasible). As it so happens this is exactly what is happening right now in the Linux Kernel. David Howells (original author of the Linux "keyring" subsystem) has been writing a generic userspace+kernelspace FS-Cache system which can use either a block device or a mounted filesystem as storage. It presently supports NFS and the minimal in-kernel AFS client and is planned to be mostly merged into 2.6.25. The benefits of being able to share innovations in caching between AFS, NFS, and other networked filesystems is quite significant. My apologies for anything in this email that may be construed as offensive; the intent is as an honest technical discussion of development methods and practices. Cheers, Kyle Moffett ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
VFS KPI was Re: [OpenAFS-devel] Re: AFS ... or equivalent ...
On Wed, 16 Jan 2008, Robert Watson wrote: [good stuff snipped] Right now we maintain a relatively stable VM/VFS KPI withing a major release (i.e, FreeBSD 6.0 -> 6.1 -> 6.2 -> 6.3), but see fairly significant changes between major releases (5.x -> 6.x -> 7.x, etc). I expect to see further changes in VFS for 8.x (and some of the locking-related ones have already started going in). This is loosely related to both the OpenAFS thread and the Mac OS X ZFS port thread, so I thought I'd ask... Has anyone considered trying to bring the FreeBSD VFS KPI (and others, for that matter) closed to the Darwin/Mac OS X ones? The Apple folks made quite dramatic changes to their VFS when going from Panther (very FreeBSD like) to Tiger, but seemed to have stabilized, at least for Leopard. It just seems that using the Mac OS X KPIs might leverage some work being done on both sides? (I don't know if there is an OpenAFS port to Mac OS X or interest in one, but I would think there would be a use for one, if it existed?) Although I'm far from an expert on the Mac OS X VFS (when I ported to it, I just cribbed the code and it worked:-), it seems that they pretty well got rid of the concept of a vnode-lock. If the underlying file system isn't SMP safe, it can put a lock on the subsystem at the VFS call. (I think it optionally does a global lock or a uses an smp lock in the vnode, but don't quote me on this. My code currently runs with the thread-safe flag false in the vfs_conf structure entry, which enables the automagic locking.) Just a thought, rick ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: VFS KPI was Re: [OpenAFS-devel] Re: AFS ... or equivalent ...
Rick Macklem wrote: On Wed, 16 Jan 2008, Robert Watson wrote: [good stuff snipped] Right now we maintain a relatively stable VM/VFS KPI withing a major release (i.e, FreeBSD 6.0 -> 6.1 -> 6.2 -> 6.3), but see fairly significant changes between major releases (5.x -> 6.x -> 7.x, etc). I expect to see further changes in VFS for 8.x (and some of the locking-related ones have already started going in). This is loosely related to both the OpenAFS thread and the Mac OS X ZFS port thread, so I thought I'd ask... Has anyone considered trying to bring the FreeBSD VFS KPI (and others, for that matter) closed to the Darwin/Mac OS X ones? The Apple folks made quite dramatic changes to their VFS when going from Panther (very FreeBSD like) to Tiger, but seemed to have stabilized, at least for Leopard. It just seems that using the Mac OS X KPIs might leverage some work being done on both sides? (I don't know if there is an OpenAFS port to Mac OS X or interest in one, but I would think there would be a use for one, if it existed?) Although I'm far from an expert on the Mac OS X VFS (when I ported to it, I just cribbed the code and it worked:-), it seems that they pretty well got rid of the concept of a vnode-lock. If the underlying file system isn't SMP safe, it can put a lock on the subsystem at the VFS call. (I think it optionally does a global lock or a uses an smp lock in the vnode, but don't quote me on this. My code currently runs with the thread-safe flag false in the vfs_conf structure entry, which enables the automagic locking.) Both Solaris and OSX seem to have found the path out of the VFS locking woods, and it would indeed be really nice if FreeBSD could follow suit. You're not the first to suggest the vnode locking move out of VFS and into the filesystems. I think that the work it would take to adapt the existing filesystems to this design would be far less than the ongoing work by everyone to fight the old design (both in FreeBSD proper and in companies that do their own custom filesystems in FreeBSD), but it does come at a cost of making things like nullfs much harder, if not nearly impossible. I wish I had time to work on something like this, but I encourage others to look into it and experiment. Scott ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OpenAFS-devel] Re: AFS ... or equivalent ...
On Wed, 16 Jan 2008, Robert Watson wrote: > On Wed, 16 Jan 2008, Jeffrey Hutzelman wrote: > > > The "let's just slurp everything into the main distribution so we don't have > > to worry about stable interfaces" approach is really poor. It encourages > > bad engineering practice among people maintaining the main distribution, > > discourages innovation and extension by others, and generally doesn't scale. > > It's far better to either attempt to maintain stable external interfaces to > > the VFS and VM subsystems, or else admit that you don't have the resources > > to do so given the relatively small number of external users, in which case > > you almost certainly also don't have the resources to keep on top of updates > > to something like OpenAFS. > > Right now we maintain a relatively stable VM/VFS KPI withing a major release > (i.e, FreeBSD 6.0 -> 6.1 -> 6.2 -> 6.3), but see fairly significant changes > between major releases (5.x -> 6.x -> 7.x, etc). I expect to see further > changes in VFS for 8.x (and some of the locking-related ones have already > started going in). Yup; that's a reasonable process. > The historic problem for Arla has been that instead of tracking these VFS > changes as they are made, they had to catch up every once in a while. Normally > that "every once in a while" has been at the point where a FreeBSD branch is > coming to the end of support rather than when it is new and shiny. Yes, that's a problem you're likely to run into unless you have a community of developers who are interested in keeping current versions working for their own use. For example, we tend to have relatively little trouble getting people to spend time making OpenAFS work on Linux or Solaris (sometimes we have trouble _getting_ it to work, but that's a different story). > In the case of > Arla, there's a quite logical path: if we import the nnpfs kernel module (but > not cache manager), then it will track FreeBSD development and almost > certainly work with little or no trouble on new major releases, as sweeps to > various KPIs will happen "for free". Yes. In fact, I think NetBSD has already done that. > So let's turn the question around: to get the OpenAFS client up and running on > FreeBSD, do you have any technical requirements not yet met by FreeBSD I don't think we know the answer to that... > , or is > it really about finding someone willing to spend some time doing the bulk of > the technical work and track bugs for a while? because this _is_ a significant part of the problem. So for starters, I think we're looking for someone who has some familiarity with OpenAFS and/or with FreeBSD's VFS layer, or thinks they can fake it, and who has cycles they're interested in spending on this. I'm sure such a person would be welcome on the openafs-devel list. -- Jeff ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OpenAFS-devel] Re: AFS ... or equivalent ...
On Wed, 16 Jan 2008, Jeffrey Hutzelman wrote: --On Monday, January 14, 2008 02:23:47 PM + Robert Watson <[EMAIL PROTECTED]> wrote: I'd like very much to get at least the kernel parts of an AFS client into the base system. That may well be realistic for arla, though I believe there was a period for a while where the kernel/arlad interface was evolving to support features like chunking. I pay only superficial attention to arla-drinkers, so I don't know what the status of any of that is; for that, you'd have to ask someone who is actively involved in arla development (I believe there are some such people on this list). It is unlikely ever to happen for OpenAFS, in which virtually all of the cache manager code is in-kernel and most of it is cross-platform. Trying to pull the OpenAFS cache manager into the FreeBSD kernel would be equivalent to forking OpenAFS; what you'd get would work and would keep up with FreeBSD, but it would be unlikely to keep up with OpenAFS. I chatted with Darrick for a while on IM yesterday (or was it the day before) to try and get a better understanding of the OpenAFS parts, and now that I know a little more, agree. My primary experience until now has been with Arla, which has a very stable interface between its relatively static kernel module and the userspace cache manager, so the main on-going engineering for the kernel module is tracking changes in the FreeBSD VFS rather than tracking Arla changes. The "let's just slurp everything into the main distribution so we don't have to worry about stable interfaces" approach is really poor. It encourages bad engineering practice among people maintaining the main distribution, discourages innovation and extension by others, and generally doesn't scale. It's far better to either attempt to maintain stable external interfaces to the VFS and VM subsystems, or else admit that you don't have the resources to do so given the relatively small number of external users, in which case you almost certainly also don't have the resources to keep on top of updates to something like OpenAFS. Right now we maintain a relatively stable VM/VFS KPI withing a major release (i.e, FreeBSD 6.0 -> 6.1 -> 6.2 -> 6.3), but see fairly significant changes between major releases (5.x -> 6.x -> 7.x, etc). I expect to see further changes in VFS for 8.x (and some of the locking-related ones have already started going in). The historic problem for Arla has been that instead of tracking these VFS changes as they are made, they had to catch up every once in a while. Normally that "every once in a while" has been at the point where a FreeBSD branch is coming to the end of support rather than when it is new and shiny. The result has been that Arla is pretty hard to use with FreeBSD as you either have to run a relatively old version of FreeBSD, or update the Arla kernel parts yourself (neither exciting prospects). In particular, if you are a FreeBSD kernel developer, you will never be running Arla as you are almost certainly running something on the development HEAD and not an aging branch. This leads to a bit of a chicken-and-egg problem, in which FreeBSD developers never use AFS, and this almost certainly an obstacle to it getting much use in the wider FreeBSD community. If there's sufficient interest in the AFS community to create and maintain a port of OpenAFS to FreeBSD, I think that would be wonderful. However, in light of the fact that it hasn't really happened to date, I've been trying to think of ways to help support that community a bit better. In the case of Arla, there's a quite logical path: if we import the nnpfs kernel module (but not cache manager), then it will track FreeBSD development and almost certainly work with little or no trouble on new major releases, as sweeps to various KPIs will happen "for free". If that doesn't work with OpenAFS due to structural differences from Arla, that's a shame (because it is easy in the case of Arla), but life. So let's turn the question around: to get the OpenAFS client up and running on FreeBSD, do you have any technical requirements not yet met by FreeBSD, or is it really about finding someone willing to spend some time doing the bulk of the technical work and track bugs for a while? Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OpenAFS-devel] Re: AFS ... or equivalent ...
--On Monday, January 14, 2008 02:23:47 PM + Robert Watson <[EMAIL PROTECTED]> wrote: I'd like very much to get at least the kernel parts of an AFS client into the base system. That may well be realistic for arla, though I believe there was a period for a while where the kernel/arlad interface was evolving to support features like chunking. I pay only superficial attention to arla-drinkers, so I don't know what the status of any of that is; for that, you'd have to ask someone who is actively involved in arla development (I believe there are some such people on this list). It is unlikely ever to happen for OpenAFS, in which virtually all of the cache manager code is in-kernel and most of it is cross-platform. Trying to pull the OpenAFS cache manager into the FreeBSD kernel would be equivalent to forking OpenAFS; what you'd get would work and would keep up with FreeBSD, but it would be unlikely to keep up with OpenAFS. The "let's just slurp everything into the main distribution so we don't have to worry about stable interfaces" approach is really poor. It encourages bad engineering practice among people maintaining the main distribution, discourages innovation and extension by others, and generally doesn't scale. It's far better to either attempt to maintain stable external interfaces to the VFS and VM subsystems, or else admit that you don't have the resources to do so given the relatively small number of external users, in which case you almost certainly also don't have the resources to keep on top of updates to something like OpenAFS. In the long run, I'm guessing that the OpenAFS cache manager evolves more quickly than FreeBSD's VFS interface, which makes pulling the CM into the kernel tree a losing battle. If you disagree, by all means fork that part of AFS (or get someone else to do so) and see what happens (AFS's user/kernel and RPC interfaces are both fairly stable, so forking just the kernel parts should be mostly feasible). -- Jeff ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
