Re: nfs+xfs - was Re: SL6.3 2.6.32-358.2.1.el6.x86_64 kernel panic
On Tue, Mar 19, 2013 at 5:51 AM, Valery Mitsyn v...@mammoth.jinr.ru wrote: On Mon, 18 Mar 2013, Nico Kadel-Garcia wrote: On Mon, Mar 18, 2013 at 5:57 PM, Paul Robert Marino prmari...@gmail.com wrote: As for JFS its been a long time since I tested it but I had the reverse issue. Oh and I know the issue you ran into with xfs its rare but has been known to happen I've hit it once my self on a laptop its a journal problem, and fsck isn't the tool to use. There is a specific xfs repair tool to fix the journal or can rebuild it from the backup inodes Then are you agreed that it's too likely to occur for high reliability filesystems, and only more suitable for high flowthrough data whose provenance is not so critical? Certainly not! Here at JINR we have more than 50 servers with about 2PB used space serviced by XFS. All data are critical for LHC experiments. Quite a few servers run for about a 5 years till now. Just a few files were lost due to damn 3-ware destroyed own DCB. The latest incident with xfs+nfs is not the xfs problem too. That's precisely why I ask. My xfs experience is apparently out of date, compared to your more recent experience.
nfs+xfs - was Re: SL6.3 2.6.32-358.2.1.el6.x86_64 kernel panic
On Sun, 17 Mar 2013, Nico Kadel-Garcia wrote: Also, *why* are you mixing xfs and nfs services in the same envirnment? And what kind of NFS and XFS servers are you using? Out of curiosity, why not ? In theory the choice of disk filesystem and network file sharing protocol should be independent. How different is the practice ? -- Dr. Andrew C. Aitchison Computer Officer, DPMMS, Cambridge a.c.aitchi...@dpmms.cam.ac.uk http://www.dpmms.cam.ac.uk/~werdna
Re: nfs+xfs - was Re: SL6.3 2.6.32-358.2.1.el6.x86_64 kernel panic
On Mon, Mar 18, 2013 at 2:59 AM, Dr Andrew C Aitchison a.c.aitchi...@dpmms.cam.ac.uk wrote: On Sun, 17 Mar 2013, Nico Kadel-Garcia wrote: Also, *why* are you mixing xfs and nfs services in the same envirnment? And what kind of NFS and XFS servers are you using? Out of curiosity, why not ? In theory the choice of disk filesystem and network file sharing protocol should be independent. How different is the practice ? I had some bad, bad experience with XFS and haven't used it since. It completely destabilized my bulk storage environment: things may have changed. I've deliberately and effectively kept my file systems below the 16 TB range and worked well with ext4. I've occasionally used larger scale commercial storage serves such as NetApp's for larger NFS environments since then.
Re: nfs+xfs - was Re: SL6.3 2.6.32-358.2.1.el6.x86_64 kernel panic
On 18 Mar 2013, at 08:37, Steven Haigh wrote: On 03/18/2013 06:34 PM, Nico Kadel-Garcia wrote: On Mon, Mar 18, 2013 at 2:59 AM, Dr Andrew C Aitchison a.c.aitchi...@dpmms.cam.ac.uk wrote: On Sun, 17 Mar 2013, Nico Kadel-Garcia wrote: Also, *why* are you mixing xfs and nfs services in the same envirnment? And what kind of NFS and XFS servers are you using? Out of curiosity, why not ? In theory the choice of disk filesystem and network file sharing protocol should be independent. How different is the practice ? I had some bad, bad experience with XFS and haven't used it since. It completely destabilized my bulk storage environment: things may have changed. I've deliberately and effectively kept my file systems below the 16 TB range and worked well with ext4. I've occasionally used larger scale commercial storage serves such as NetApp's for larger NFS environments since then. I use XFS on a small RAID6 array (its 2Tb - not huge), and I mount it via NFS to other systems. I haven't had a kernel crash as yet. We use XFS for some heavily loaded buffer storage systems, and we haven't had an issue - but no NFS there. We also have an NFS server using XFS (mostly because of the project quota feature we needed on some shares) and that's also working fine with NFSv3, serving about 200 clients; NFSv4 performance on XFS is disappointing compared to ext3 but we are not in an hurry to migrate. Cheers, Sergio -- Sergio Ballestrero - http://physics.uj.ac.za/psiwiki/Ballestrero University of Johannesburg, Physics Department ATLAS TDAQ sysadmin team - Office:75282 OnCall:164851
Re: nfs+xfs - was Re: SL6.3 2.6.32-358.2.1.el6.x86_64 kernel panic
I've used XFS for over a decade now. Its the most reliable crash resistant filesystem I've ever used according to all my tests and experience. But I have had a few bad patches on older versions of RHEL (before RedHat started supporting it) where it didn't work well, but historicity its worked perfectly on every non reheat based distro.By the way I know why the performance goes down on NFS4 its mostly due to the fact that it supports xattribs natively and ext3 does not unless you explicitly turn it on when you mount the file system.By the way I currently have several production servers running gluster on top of XFS serving both gluster native and NFS 3 clients and in several clusters it works perfectly for me.Oh and the earlier confusion Netapps are BSD UNIX boxes and just like Unix Linux can serve NFS volumes Netapp didn't invent NFS nor are they even the best implementation. Give me a linux box with a san or a good raid controller any day they are faster and in the case of a SAN the are cheaper to scale-- Sent from my HP Pre3On Mar 18, 2013 3:45 AM, Sergio Ballestrero sergio.ballestr...@cern.ch wrote: On 18 Mar 2013, at 08:37, Steven Haigh wrote:On 03/18/2013 06:34 PM, Nico Kadel-Garcia wrote:On Mon, Mar 18, 2013 at 2:59 AM, Dr Andrew C Aitchisona.c.aitchi...@dpmms.cam.ac.uk wrote:On Sun, 17 Mar 2013, Nico Kadel-Garcia wrote:Also, *why* are you mixing xfs and nfs services in the sameenvirnment? And what kind of NFS and XFS servers are you using?Out of curiosity, why not ?In theory the choice of disk filesystem and network file sharingprotocol should be independent.How different is the practice ?I had some bad, bad experience with XFS and haven't used it since. Itcompletely destabilized my bulk storage environment: things may havechanged.I've deliberately and effectively kept my file systems below the 16 TBrange and worked well with ext4. I've occasionally used larger scalecommercial storage serves such as NetApp's for larger NFS environmentssince then.I use XFS on a small RAID6 array (its 2Tb - not huge), and I mount it via NFS to other systems. I haven't had a kernel crash as yet.We use XFS for some heavily loaded "buffer storage" systems, and we haven't had an issue - but no NFS there.We also have an NFS server using XFS (mostly because of the "project quota" feature we needed on some shares) and that's also working fine with NFSv3, serving about 200 clients; NFSv4 performance on XFS is disappointing compared to ext3 but we are not in an hurry to migrate.Cheers, Sergio --Sergio Ballestrero -http://physics.uj.ac.za/psiwiki/BallestreroUniversity of Johannesburg, Physics DepartmentATLAS TDAQ sysadmin team - Office:75282 OnCall:164851
Re: nfs+xfs - was Re: SL6.3 2.6.32-358.2.1.el6.x86_64 kernel panic
Hi Paul Robert Marino! On 2013.03.18 at 08:55:39 -0400, Paul Robert Marino wrote next: I've used XFS for over a decade now. Its the most reliable crash resistant filesystem I've ever used according to all my tests and experience. But I have This might be true, but it's not the case for all. I've experienced very bad corruptions on xfs myself, resulting in lots of non-accessible fake files (random size, attributes etc) with random filenames including non-printable characters - and there was no way to remove them, fsck refused to fix them, too. Filesystem was in total mess and producing various errors - it's fortunate that I was able to copy all real data without corruption from it, though. Since then I try not to approach xfs without serious reason. I'd rather use JFS for huge filesystem which I've been using for many years until ext4 appeared.. But for fs 16 Tb jfs is still best option, I believe (far more stable in my experience compared to xfs, though might be not as fast). For several reasons most people don't consider JFS but I used it on tons of servers for filesystems 1 Tb (ext3 was a bad choice for huge filesystems for various reasons) and never had a single issue with it. At most, after multiple power failures during heavy write access I had errors which remounted it into R/O mode and fsck always fixed it. By the way I know why the performance goes down on NFS4 its mostly due to the fact that it supports xattribs natively and ext3 does not unless you explicitly turn it on when you mount the file system. I don't really understand your implication: xfs is slower *due* to xattr support? So if I will mount ext4 with user_xattr option, NFS4 from it will become slower? How come? -- Vladimir
Re: nfs+xfs - was Re: SL6.3 2.6.32-358.2.1.el6.x86_64 kernel panic
Its mostly due to the uid and gid name mapping to names instead of numbers introduced in NFS 4 by default if possible a backup is saved as an extended attribute and can also compound the atime update speed issue.As for JFS its been a long time since I tested it but I had the reverse issue.Oh and I know the issue you ran into with xfs its rare but has been known to happen I've hit it once my self on a laptop its a journal problem, and fsck isn't the tool to use.There is a specific xfs repair tool to fix the journal or can rebuild it from the backup inodes-- Sent from my HP Pre3On Mar 18, 2013 10:53 AM, Vladimir Mosgalin mosga...@vm10124.spb.edu wrote: Hi Paul Robert Marino! On 2013.03.18 at 08:55:39 -0400, Paul Robert Marino wrote next: I've used XFS for over a decade now. Its the most reliable crash resistant filesystem I've ever used according to all my tests and experience. But I have This might be true, but it's not the case for all. I've experienced very bad corruptions on xfs myself, resulting in lots of non-accessible fake files (random size, attributes etc) with random filenames including non-printable characters - and there was no way to remove them, fsck refused to fix them, too. Filesystem was in total mess and producing various errors - it's fortunate that I was able to copy all real data without corruption from it, though. Since then I try not to approach xfs without serious reason. I'd rather use JFS for huge filesystem which I've been using for many years until ext4 appeared.. But for fs 16 Tb jfs is still best option, I believe (far more stable in my experience compared to xfs, though might be not as fast). For several reasons most people don't consider JFS but I used it on tons of servers for filesystems 1 Tb (ext3 was a bad choice for huge filesystems for various reasons) and never had a single issue with it. At most, after multiple power failures during heavy write access I had errors which remounted it into R/O mode and fsck always fixed it. By the way I know why the performance goes down on NFS4 its mostly due to the fact that it supports xattribs natively and ext3 does not unless you explicitly turn it on when you mount the file system. I don't really understand your implication: xfs is slower *due* to xattr support? So if I will mount ext4 with user_xattr option, NFS4 from it will become slower? How come? -- Vladimir
