Re: huge filesystems

2005-03-19 Thread Eric W. Biederman
Andreas Dilger <[EMAIL PROTECTED]> writes: > On Mar 14, 2005 21:37 -0700, jmerkey wrote: > > 1. Scaling issues with readdir() with huge numbers of files (not even > > huge really. 87000 files in a dir takes a while > > for readdir() to return results). I average 2-3 million files per > >

Re: huge filesystems

2005-03-19 Thread Eric W. Biederman
Andreas Dilger [EMAIL PROTECTED] writes: On Mar 14, 2005 21:37 -0700, jmerkey wrote: 1. Scaling issues with readdir() with huge numbers of files (not even huge really. 87000 files in a dir takes a while for readdir() to return results). I average 2-3 million files per directory on

Re: huge filesystems

2005-03-15 Thread Barry K. Nathan
On Mon, Mar 14, 2005 at 11:41:37AM -0500, Andreas Dilger wrote: > > What about the "LBD" patches - what limits are involved there, and have > > they been rolled into a Linus kernel, or one or more vendor kernels? > > These are part of stock 2.6 kernels. The caveat here is that there have > been

Re: huge filesystems

2005-03-15 Thread Barry K. Nathan
On Mon, Mar 14, 2005 at 11:41:37AM -0500, Andreas Dilger wrote: What about the LBD patches - what limits are involved there, and have they been rolled into a Linus kernel, or one or more vendor kernels? These are part of stock 2.6 kernels. The caveat here is that there have been some

Re: huge filesystems

2005-03-14 Thread Andreas Dilger
On Mar 14, 2005 21:37 -0700, jmerkey wrote: > 1. Scaling issues with readdir() with huge numbers of files (not even > huge really. 87000 files in a dir takes a while > for readdir() to return results). I average 2-3 million files per > directory on 2.6.9. It can take a up to a minute for >

Re: huge filesystems

2005-03-14 Thread jmerkey
Andrew Morton wrote: jmerkey <[EMAIL PROTECTED]> wrote: >I don't recall you reporting any of them. How can we expect to fix >anything if we aren't told about it? > > > I report them when I can't get around them myself. I've been able to get around most of them. Jeff, that's all take and

Re: huge filesystems

2005-03-14 Thread Andrew Morton
jmerkey <[EMAIL PROTECTED]> wrote: > > >I don't recall you reporting any of them. How can we expect to fix > >anything if we aren't told about it? > > > > > > > I report them when I can't get around them myself. I've been able to get > around most of them. Jeff, that's all take and no

Re: huge filesystems

2005-03-14 Thread jmerkey
Andrew Morton wrote: jmerkey <[EMAIL PROTECTED]> wrote: I am running the DSFS file system as a 7 TB file system on 2.6.9. On a 32-bit CPU? Yep. There are a host of problems with the current VFS, I don't recall you reporting any of them. How can we expect to fix anything if we

Re: huge filesystems

2005-03-14 Thread Andrew Morton
jmerkey <[EMAIL PROTECTED]> wrote: > > I am running the DSFS file system as a 7 TB file system on 2.6.9. On a 32-bit CPU? > There are a host of problems with the current VFS, I don't recall you reporting any of them. How can we expect to fix anything if we aren't told about it? > ad I have

Re: huge filesystems

2005-03-14 Thread jmerkey
I am running the DSFS file system as a 7 TB file system on 2.6.9. There are a host of problems with the current VFS, ad I have gotten around most of them by **NOT** using the linux page cache interface. The VFS I am using creates a virtual represeation of the files and it's own cache. You need

Re: huge filesystems

2005-03-14 Thread Andreas Dilger
On Mar 09, 2005 10:53 -0800, Dan Stromberg wrote: > The group I work in has been experimenting with GFS and Lustre, and I did > some NBD/ENBD experimentation on my own, described at > http://dcs.nac.uci.edu/~strombrg/nbd.html > > My question is, what is the current status of h

Re: huge filesystems

2005-03-14 Thread Andreas Dilger
On Mar 09, 2005 10:53 -0800, Dan Stromberg wrote: The group I work in has been experimenting with GFS and Lustre, and I did some NBD/ENBD experimentation on my own, described at http://dcs.nac.uci.edu/~strombrg/nbd.html My question is, what is the current status of huge filesystems - IE

Re: huge filesystems

2005-03-14 Thread jmerkey
I am running the DSFS file system as a 7 TB file system on 2.6.9. There are a host of problems with the current VFS, ad I have gotten around most of them by **NOT** using the linux page cache interface. The VFS I am using creates a virtual represeation of the files and it's own cache. You need

Re: huge filesystems

2005-03-14 Thread Andrew Morton
jmerkey [EMAIL PROTECTED] wrote: I am running the DSFS file system as a 7 TB file system on 2.6.9. On a 32-bit CPU? There are a host of problems with the current VFS, I don't recall you reporting any of them. How can we expect to fix anything if we aren't told about it? ad I have gotten

Re: huge filesystems

2005-03-14 Thread jmerkey
Andrew Morton wrote: jmerkey [EMAIL PROTECTED] wrote: I am running the DSFS file system as a 7 TB file system on 2.6.9. On a 32-bit CPU? Yep. There are a host of problems with the current VFS, I don't recall you reporting any of them. How can we expect to fix anything if we

Re: huge filesystems

2005-03-14 Thread Andrew Morton
jmerkey [EMAIL PROTECTED] wrote: I don't recall you reporting any of them. How can we expect to fix anything if we aren't told about it? I report them when I can't get around them myself. I've been able to get around most of them. Jeff, that's all take and no give. Please

Re: huge filesystems

2005-03-14 Thread jmerkey
Andrew Morton wrote: jmerkey [EMAIL PROTECTED] wrote: I don't recall you reporting any of them. How can we expect to fix anything if we aren't told about it? I report them when I can't get around them myself. I've been able to get around most of them. Jeff, that's all take and no

Re: huge filesystems

2005-03-14 Thread Andreas Dilger
On Mar 14, 2005 21:37 -0700, jmerkey wrote: 1. Scaling issues with readdir() with huge numbers of files (not even huge really. 87000 files in a dir takes a while for readdir() to return results). I average 2-3 million files per directory on 2.6.9. It can take a up to a minute for readdir()

Re: huge filesystems

2005-03-09 Thread Chris Wedgwood
On Wed, Mar 09, 2005 at 10:53:48AM -0800, Dan Stromberg wrote: > My question is, what is the current status of huge filesystems - IE, > filesystems that exceed 2 terabytes, and hopefully also exceeding 16 > terabytes? people can and do have >2T filesystems now. some people on

Re: huge filesystems

2005-03-09 Thread Miklos Szeredi
> The group I work in has been experimenting with GFS and Lustre, and I did > some NBD/ENBD experimentation on my own, described at > http://dcs.nac.uci.edu/~strombrg/nbd.html > > My question is, what is the current status of huge filesystems - IE, > filesystems that

huge filesystems

2005-03-09 Thread Dan Stromberg
The group I work in has been experimenting with GFS and Lustre, and I did some NBD/ENBD experimentation on my own, described at http://dcs.nac.uci.edu/~strombrg/nbd.html My question is, what is the current status of huge filesystems - IE, filesystems that exceed 2 terabytes, and hopefully also

huge filesystems

2005-03-09 Thread Dan Stromberg
The group I work in has been experimenting with GFS and Lustre, and I did some NBD/ENBD experimentation on my own, described at http://dcs.nac.uci.edu/~strombrg/nbd.html My question is, what is the current status of huge filesystems - IE, filesystems that exceed 2 terabytes, and hopefully also

Re: huge filesystems

2005-03-09 Thread Miklos Szeredi
The group I work in has been experimenting with GFS and Lustre, and I did some NBD/ENBD experimentation on my own, described at http://dcs.nac.uci.edu/~strombrg/nbd.html My question is, what is the current status of huge filesystems - IE, filesystems that exceed 2 terabytes, and hopefully

Re: huge filesystems

2005-03-09 Thread Chris Wedgwood
On Wed, Mar 09, 2005 at 10:53:48AM -0800, Dan Stromberg wrote: My question is, what is the current status of huge filesystems - IE, filesystems that exceed 2 terabytes, and hopefully also exceeding 16 terabytes? people can and do have 2T filesystems now. some people on x86 have hit the 16TB