[CentOS] Difficulty configuring RDMA in CentOS
Hi We are trying to configure RDMA for an infiniband connection between our data server (running CentOS 6.8) and our compute nodes (running CentOS 6.6). We have been trying to follow the instructions in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_the_base_rdma_subsystemedit however we are getting conflicting information on whether the RDMA is correctly configured. Some of what we have done and some of the data are below. Can you suggest what other tests and what other data we should get to debug this problem? /etc/rdma/mlx4.conf to set the port types properly for RoCE/IBoE usage. Edit /etc/modprobe.d/mlx4.conf to instruct the driver on which packet priority is configured for the “no-drop” service on the Ethernet switches the cards are plugged into. vim /etc/rdma/mlx4.conf You can find the right pci device to use for any given card by loading # the mlx4_core module, then going to /sys/bus/pci/drivers/mlx4_core and # seeing what possible PCI devices are listed there.The possible values # for ports are: ib, eth, and auto.However, not all cards support all # types, so if you get messages from the kernel that your selected port # type isn't supported, there's nothing this script can do about it. [root@mseas-data2 mlx4_core]# ls /sys/bus/pci/drivers/mlx4_core :81:00.0bindmodulenew_idremove_idueventunbind mstconfig -d 81:00.0 query Device #1: -- Device type:ConnectX3Pro PCI device:81:00.0 Configurations:Current SRIOV_EN1 NUM_OF_VFS8 LINK_TYPE_P13 LINK_TYPE_P23 LOG_BAR_SIZE3 BOOT_PKEY_P10 BOOT_PKEY_P20 BOOT_OPTION_ROM_EN_P11 BOOT_VLAN_EN_P10 BOOT_RETRY_CNT_P10 LEGACY_BOOT_PROTOCOL_P11 BOOT_VLAN_P11 BOOT_OPTION_ROM_EN_P21 BOOT_VLAN_EN_P20 BOOT_RETRY_CNT_P20 LEGACY_BOOT_PROTOCOL_P21 BOOT_VLAN_P21 Needed packages: Most if not all packages appear to be installed on server [root@mseas-data2 ~]# service rdma status Low level hardware support loaded: mlx4_ib Upper layer protocol modules: ib_ipoib User space access modules: rdma_ucm ib_ucm ib_uverbs ib_umad Connection management modules: rdma_cm ib_cm iw_cm Configured IPoIB interfaces: none Currently active IPoIB interfaces: ib0 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] group write permissions not being respected
Trying the gluster client seems to fix the problem. On 09/02/2016 12:57 AM, Chris Murphy wrote: On Thu, Sep 1, 2016, 8:11 AM Pat Haley <pha...@mit.edu> wrote: For the enforcing=0, is that referring to SELinux? If so, we are not running SELinux. OK so neither that nor chcon nor context mount option apply. It's something else. On 08/31/2016 11:38 PM, Chris Murphy wrote: Try booting with enforcing=0 and if that fixes it, you need to find out what security label is needed for gluster. For the enforcing=0, is that referring to SELinux? If so, we are not running SELinux. - Chances are it's easiest to use -o context= mount option on the brick, but if the brick is not exclusive to gluster you'll need chcon -R. - We aren't sure exactly what you mean by this second paragraph, can you expand on this? Are these two exclusive options exclusive? We aren't sure what you what you mean by "exclusive to gluster" - If that's not it, maybe try the gluster client instead of using NFS. See if you get a different result that narrows down what's going on. My vague recollection is for Samba, without the correct SELinux label, I could neither read nor write. Chris Murphy ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] group write permissions not being respected
For the enforcing=0, is that referring to SELinux? If so, we are not running SELinux. On 08/31/2016 11:38 PM, Chris Murphy wrote: Try booting with enforcing=0 and if that fixes it, you need to find out what security label is needed for gluster. For the enforcing=0, is that referring to SELinux? If so, we are not running SELinux. - Chances are it's easiest to use -o context= mount option on the brick, but if the brick is not exclusive to gluster you'll need chcon -R. - We aren't sure exactly what you mean by this second paragraph, can you expand on this? Are these two exclusive options exclusive? We aren't sure what you what you mean by "exclusive to gluster" - If that's not it, maybe try the gluster client instead of using NFS. See if you get a different result that narrows down what's going on. My vague recollection is for Samba, without the correct SELinux label, I could neither read nor write. Chris Murphy ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] group write permissions not being respected
For example the directory /gdata/bibliography/Work/GroupBib/trunk/ can be written in by user phaley but not by other users who are member of the group mseasweb. The directory has permissions [root@mseas ~]# ls -lh /gdata/bibliography/Work/GroupBib total 12K drwxrwsr-x 4 phaley mseasweb 4.0K Aug 30 12:31 trunk The parent directory (/gdata/bibliography/Work/GroupBib) has permissions [root@mseas ~]# ls -lh /gdata/bibliography/Work/ total 8.0K drwxrwsr-x 6 phaley mseasweb 4.0K Aug 30 14:01 GroupBib On 08/31/2016 02:04 PM, m.r...@5-cent.us wrote: Stupid question, and note I missed most of the earlier posts in this thread: what are the permissions on the directory that this directory are in? mark ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] group write permissions not being respected
So far, those look the same client: [root@mseas FixOwn]# getfacl /gdata/bibliography/Work/GroupBib/trunk/ getfacl: Removing leading '/' from absolute path names # file: gdata/bibliography/Work/GroupBib/trunk/ # owner: phaley # group: mseasweb # flags: -s- user::rwx group::rwx other::r-x server: [root@mseas-data2 ~]# getfacl /gdata/bibliography/Work/GroupBib/trunk/ getfacl: Removing leading '/' from absolute path names # file: gdata/bibliography/Work/GroupBib/trunk/ # owner: phaley # group: mseasweb # flags: -s- user::rwx group::rwx other::r-x On 08/31/2016 12:50 PM, Gordon Messmer wrote: On 08/30/2016 03:01 PM, Pat Haley wrote: the owner of a directory can still write to that directory but any other member of the associated group cannot, even though the directory clearly has group write permissions set Use "getfacl" on both the client and server side to view the complete permission set. What do those look like? ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
[CentOS] group write permissions not being respected
Hi We have just migrated our data to a new file server (more space, old server was showing its age). We have a volume for collaborative use, based on group membership. In our new server, the group write permissions are not being respected (e.g. the owner of a directory can still write to that directory but any other member of the associated group cannot, even though the directory clearly has group write permissions set). This is occurring regardless of how many groups the user is a member of (i.e. users that are members of fewer then 16 groups are still affected). the relevant fstab line from the server looks like localhost:/data-volume /gdataglusterfs defaults 0 0 and for a client: mseas-data2:/gdata /gdata nfs defaults0 0 Any help would be greatly appreciated. Thanks -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS 6: files now owned by nobody:nobody
The setting the Domain = line in /etc/idmapd.conf option solved our problem Thanks On 08/29/2016 07:23 PM, Frank Cox wrote: On Mon, 29 Aug 2016 18:59:31 -0400 Pat Haley wrote: We noticed that all the files were owned by nobody Here are my notes for dealing with this issue: If all users come up as nobody on a nfs mount: Add nfs server name to the Domain = line in /etc/idmapd.conf on both the server and the clients, i.e. Domain = nameof.server /sbin/service rpcidmapd restart /sbin/service nfslock restart /sbin/service nfs restart Also, the complete hostname as specified (nameof.server) must be in /etc/hosts on the nfs clients as well as the server --- try /usr/sbin/nfsidmap -c on the client. Since I put this into my /etc/rc.local , I don't have the problem any longer. --- 1) /etc/idmapd.conf # Set domain to the domain name shared by your NFS servers. Domain: mycompany.com Set local-realms to the name of the nfs servers you'll be using. THIS WASN'T MENTIONED ELSEWHERE. Local-Realms: nfs1.mycompany.com,nfs2.mycompany.com # make the above changes on all the servers in question. 2) /etc/hosts: list with all the NFS servers you specified in local-realms above. This way DNS errors don't make your servers get hung 1.2.3.4nfs1.mycompany.com 1.2.3.5nfs2.mycompany.com 3) Make sure you synchronize your /etc/passwd files so that the account IDs match up or you'll get very strange results. 4) Reboot EVERYTHING. Restarting services was not enough. For documentation's sake, I restarted rpcidmapd, nfslock, and nfs, but didn't get the correct permissions until reboot. It doesn't seem important to run the nfs service on the clients. 5) Client mount: # CLI /bin/mount -t nfs servername:/path/to/share /local/mount/point # /etc/fstab servername.com:/path/to/share/local/mount/mount nfs ro,nolock0 0 # mount -a -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS 6: files now owned by nobody:nobody
I believe NFSv4. On the machine that contains the physical disks (is that the "server" or the "NSF mount"?) the relevant line from /etc/fstab seems to be UUID=bde58f42-4ac4-4763-b0a8-f83723f0e2a0 /home ext4defaults 1 2 while on my front-end machine its mseas-data2:/home /home nfs defaults0 0 where mseas-data2 is the name of the machine that contains the physical disks. Note that it isn't just root that's becoming "nobody" but all the users Thanks On 08/29/2016 07:14 PM, John R Pierce wrote: On 8/29/2016 3:59 PM, Pat Haley wrote: We are running a cluster under CentOS 6.6. We recently attached a new NAS device, running CentOS 6.8 and rsync'd our user file system to it. We noticed that all the files were owned by nobody (with nobody as the group). We copied over the /etc/passwd and /etc/group files from our front-end server to our NAS server. If we log in to the NAS server we see the files owned by their correct owners. However, doing an ls from the front-end server or any of the compute nodes still shows the files owned by "nobody". We rebooted one of the compute nodes but it still sees the files owned by nobody. a CentOS server isn't really a 'NAS device', as NAS implies an appliance storage device. this is NFS? NFSv3, or NFSv4? what NFS options are on the server and on the NFS mount?quite often NFS servers force root to nobody. -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
[CentOS] CentOS 6: files now owned by nobody:nobody
Hi, We are running a cluster under CentOS 6.6. We recently attached a new NAS device, running CentOS 6.8 and rsync'd our user file system to it. We noticed that all the files were owned by nobody (with nobody as the group). We copied over the /etc/passwd and /etc/group files from our front-end server to our NAS server. If we log in to the NAS server we see the files owned by their correct owners. However, doing an ls from the front-end server or any of the compute nodes still shows the files owned by "nobody". We rebooted one of the compute nodes but it still sees the files owned by nobody. Any advice would be greatly appreciated. Thanks -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Recommendations for Infiniband with CentOS 6.7
We have a new install of CentOS 6.7 with infiniband support installed. We can see the card in hardware and we can see the mlx4 drivers loaded in the kernel but cannot see the card as an ethernet interface, using ifconfig -a. Can you recommend an install procedure to see this as an ethernet interface? Thanks On 05/25/2016 07:32 AM, Fabian Arrotin wrote: On 25/05/16 03:08, Pat Haley wrote: Hi All, We looking for suggestions on dealing with mellanox drivers in CentOS 6.7 We tried installing mellanox drivers (MLNX_OFED_LINUX-3.2-2.0.0.0-rhel6.7-x86_64) on a Quanta Cirrascale server running Centos 6.7 - 2.6.32-573.22.1.el6.x86_64. When we rebooted the machine after installing the drivers, it went into a kernel panic for every installed kernel except for Centos 6.7 2.6.32-573.22.1.el6.x86_64.debug. After we uninstalled the drivers, the machine failed to boot for any installed kernel. Any suggestions on how to proceed would be greatly appreciated. Thanks Well, we (CentOS) are using a gluster setup on top of Infiniband, but we're just using the default mlx4_ib kernel module that is included with the kernel shipped with 6.7 (/lib/modules/2.6.32-573.22.1.el6.x86_64/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko) so nothing to be done at the kernel/initrd level. Is there a reason why you needed a different version ? PS : the IB HBA model we have in those servers is the following one : 81:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0) ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
[CentOS] Recommendations for Infiniband with CentOS 6.7
Hi All, We looking for suggestions on dealing with mellanox drivers in CentOS 6.7 We tried installing mellanox drivers (MLNX_OFED_LINUX-3.2-2.0.0.0-rhel6.7-x86_64) on a Quanta Cirrascale server running Centos 6.7 - 2.6.32-573.22.1.el6.x86_64. When we rebooted the machine after installing the drivers, it went into a kernel panic for every installed kernel except for Centos 6.7 2.6.32-573.22.1.el6.x86_64.debug. After we uninstalled the drivers, the machine failed to boot for any installed kernel. Any suggestions on how to proceed would be greatly appreciated. Thanks -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
[CentOS] Hard drives being renamed
Hi, We are running Centos 6.7 - 2.6.32-573.22.1.el6.x86_64 on a Quanta Cirrascale, up to date with patches. We have had a couple of instances in which the hard drives have become renamed after reboot (e.g. drive sda is renamed to sdc after reboot). One time this occurred when we rebooted following the installation of a 10GB NIC card, another time after we tried to install mellanox drivers (MLNX_OFED_LINUX-3.2-2.0.0.0-rhel6.7-x86_64). Currently we are unable to boot because that drive has been renamed. Our questions are * Has anyone else experienced similar issues? * What can we do to prevent such renaming in the future -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] NFS not recognizing available file space
Hi James, My system did not recognize the delaylog option, but when I mounted with nobarrier,inode64 things worked and I was able to write to the array! Thanks! Pat change mount to nobarrier,inode64,delaylog - Original Message - | | Hi: | | On Tue, Feb 4, 2014 at 1:03 PM, Pat Haley pha...@mit.edu wrote: | | Hi, | | I have a server running under CentOS 5.8 and I appear to be in a | situation in which the NFS file server is not recognizing the | available | space on a particular disk (actually a hardware RAID-6 of 13 2Tb | disks). | If I try to write to the disk I get the following error message | | [root@nas-0-1 mseas-data-0-1]# touch dum | touch: cannot touch `dum': No space left on device | | However, if I check the available space, there seems to | be plenty | | [root@nas-0-1 mseas-data-0-1]# df -h . | FilesystemSize Used Avail Use% Mounted on | /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1 | | | Maybe you're hitting the allocation of reserved blocks for root? | With your disk usage of 97% I'd think that could be the case. | | You didn't say what file system you're using for that 21TB array, | so we | (this list) won't be of too much help without knowing that. | | xfs file system. The fstab line for this array is: | | /dev/sdb1 /mseas-data-0-1 xfs defaults |1 0 | | | tune2fs [0] is your friend | | if I read the man pages correctly tune2fs will not work for xfs. | From xfs_info I get the following for mseas-data-0-1 | | [root@nas-0-1 mseas-data-0-1]# xfs_info . | meta-data=/dev/sdb1 isize=256agcount=32, | agsize=167846667 blks | = sectsz=512 attr=1 | data = bsize=4096 blocks=5371093344, | imaxpct=25 | = sunit=0 swidth=0 blks, | unwritten=1 | naming =version 2 bsize=4096 | log =internal bsize=4096 blocks=32768, version=1 | = sectsz=512 sunit=0 blks, | lazy-count=0 | realtime =none extsz=4096 blocks=0, rtextents=0 | | Unfortunately, I don't know how to interpret this or if | it is giving relevant information to the question at hand | | - use it to determine if there are reserved blocks | - use it to adjust the settings | | [0] | https://wiki.archlinux.org/index.php/ext4#Remove_reserved_blocks | | | [root@nas-0-1 mseas-data-0-1]# df -i . | FilesystemInodes IUsed IFree IUse% Mounted on | /dev/sdb13290047552 4391552 32856560001% | /mseas-data-0-1 | | I don't know if the following is relevant but the disk in question | is served as one of 3 bricks in a gluster namespace. | | Based on the test with touch, which is happening directly | at the NFS level, this seems to be an NFS rather than gluster | issue. I couldn't find any file in /var/log which had a | time that corresponded to the failed touch test and I didn't | see anything in dmesg. We have tried rebooting this system. | What else should we look at and/or try to resolve or debug | this issue? | | | If you have a non-root shell account on that box, can you write to | that | array from the NFS host? | ( Take NFS out of the equation. ) | | Unfortunately we only have a root account on that box. | | | | Thanks. | | Pat | | -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- | Pat Haley Email: pha...@mit.edu | Center for Ocean Engineering Phone: (617) 253-6824 | Dept. of Mechanical EngineeringFax:(617) 253-8125 | MIT, Room 5-213http://web.mit.edu/phaley/www/ | 77 Massachusetts Avenue | Cambridge, MA 02139-4301 | ___ | CentOS mailing list | CentOS@centos.org | http://lists.centos.org/mailman/listinfo/centos | | | | | | | -- | | -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- | Pat Haley Email: pha...@mit.edu | Center for Ocean Engineering Phone: (617) 253-6824 | Dept. of Mechanical EngineeringFax:(617) 253-8125 | MIT, Room 5-213http://web.mit.edu/phaley/www/ | 77 Massachusetts Avenue | Cambridge, MA 02139-4301 | ___ | CentOS mailing list | CentOS@centos.org | http://lists.centos.org/mailman/listinfo/centos | -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA
[CentOS] NFS not recognizing available file space
Hi, I have a server running under CentOS 5.8 and I appear to be in a situation in which the NFS file server is not recognizing the available space on a particular disk (actually a hardware RAID-6 of 13 2Tb disks). If I try to write to the disk I get the following error message [root@nas-0-1 mseas-data-0-1]# touch dum touch: cannot touch `dum': No space left on device However, if I check the available space, there seems to be plenty [root@nas-0-1 mseas-data-0-1]# df -h . FilesystemSize Used Avail Use% Mounted on /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1 [root@nas-0-1 mseas-data-0-1]# df -i . FilesystemInodes IUsed IFree IUse% Mounted on /dev/sdb13290047552 4391552 32856560001% /mseas-data-0-1 I don't know if the following is relevant but the disk in question is served as one of 3 bricks in a gluster namespace. Based on the test with touch, which is happening directly at the NFS level, this seems to be an NFS rather than gluster issue. I couldn't find any file in /var/log which had a time that corresponded to the failed touch test and I didn't see anything in dmesg. We have tried rebooting this system. What else should we look at and/or try to resolve or debug this issue? Thanks. Pat -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] NFS not recognizing available file space
Hi: On Tue, Feb 4, 2014 at 1:03 PM, Pat Haley pha...@mit.edu wrote: Hi, I have a server running under CentOS 5.8 and I appear to be in a situation in which the NFS file server is not recognizing the available space on a particular disk (actually a hardware RAID-6 of 13 2Tb disks). If I try to write to the disk I get the following error message [root@nas-0-1 mseas-data-0-1]# touch dum touch: cannot touch `dum': No space left on device However, if I check the available space, there seems to be plenty [root@nas-0-1 mseas-data-0-1]# df -h . FilesystemSize Used Avail Use% Mounted on /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1 Maybe you're hitting the allocation of reserved blocks for root? With your disk usage of 97% I'd think that could be the case. You didn't say what file system you're using for that 21TB array, so we (this list) won't be of too much help without knowing that. xfs file system. The fstab line for this array is: /dev/sdb1 /mseas-data-0-1 xfs defaults1 0 tune2fs [0] is your friend if I read the man pages correctly tune2fs will not work for xfs. From xfs_info I get the following for mseas-data-0-1 [root@nas-0-1 mseas-data-0-1]# xfs_info . meta-data=/dev/sdb1 isize=256agcount=32, agsize=167846667 blks = sectsz=512 attr=1 data = bsize=4096 blocks=5371093344, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=32768, version=1 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 Unfortunately, I don't know how to interpret this or if it is giving relevant information to the question at hand - use it to determine if there are reserved blocks - use it to adjust the settings [0] https://wiki.archlinux.org/index.php/ext4#Remove_reserved_blocks [root@nas-0-1 mseas-data-0-1]# df -i . FilesystemInodes IUsed IFree IUse% Mounted on /dev/sdb13290047552 4391552 32856560001% /mseas-data-0-1 I don't know if the following is relevant but the disk in question is served as one of 3 bricks in a gluster namespace. Based on the test with touch, which is happening directly at the NFS level, this seems to be an NFS rather than gluster issue. I couldn't find any file in /var/log which had a time that corresponded to the failed touch test and I didn't see anything in dmesg. We have tried rebooting this system. What else should we look at and/or try to resolve or debug this issue? If you have a non-root shell account on that box, can you write to that array from the NFS host? ( Take NFS out of the equation. ) Unfortunately we only have a root account on that box. Thanks. Pat -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Issue mounting /home area from NAS server
Hi, Just before the Thanksgiving break, we enabled quotas on the /home areas on the mseas-data server (running CentOS 5.8), using the following line in the updated /etc/fstab /dev/mapper/the_raid-lv_home /home ext3 defaults,usrquota,grpquota 1 0 Following the Thanksgiving reboot of mseas-data we have been experiencing problems with svn on mseas (our front-end machine, running CentOS 6.2). After consulting with the svn mailing list, it appeared to be a communication issue between mseas and mseas-data. We therefore rebooted mseas. Now mseas is not mounting /home. We get the following error [root@mseas etc]# mount mseas-data:/home mount.nfs: Unknown error 521 We have tried this mounting with both of the following lines in fstab and got the same error mseas-data:/home/home nfs defaults0 0 mseas-data:/home/home nfs defaults,usrquota,grpquota0 0 (the latter was to mimic the mounting of /home on mseas-data) The only thing we've found out on the web so far was advice on editting /etc/exports to have an insecure option for the export of /home, which we already had. Any thoughts on how to resolve or debug this issue would be greatly appreciated. Thank you. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Issue mounting /home area from NAS server
Hi, A quick follow-up. I tried reverting back to the set ups on both machines before we enabled quotas. Rebooted both machines. Still cannot mount /home from the client (mseas). Thanks. Hi, Just before the Thanksgiving break, we enabled quotas on the /home areas on the mseas-data server (running CentOS 5.8), using the following line in the updated /etc/fstab /dev/mapper/the_raid-lv_home /home ext3 defaults,usrquota,grpquota 1 0 Following the Thanksgiving reboot of mseas-data we have been experiencing problems with svn on mseas (our front-end machine, running CentOS 6.2). After consulting with the svn mailing list, it appeared to be a communication issue between mseas and mseas-data. We therefore rebooted mseas. Now mseas is not mounting /home. We get the following error [root@mseas etc]# mount mseas-data:/home mount.nfs: Unknown error 521 We have tried this mounting with both of the following lines in fstab and got the same error mseas-data:/home/home nfs defaults0 0 mseas-data:/home/home nfs defaults,usrquota,grpquota0 0 (the latter was to mimic the mounting of /home on mseas-data) The only thing we've found out on the web so far was advice on editting /etc/exports to have an insecure option for the export of /home, which we already had. Any thoughts on how to resolve or debug this issue would be greatly appreciated. Thank you. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Issue mounting /home area from NAS server
Hi Mike, I think v3 but I'm not sure (how do I check this?). At any rate, I tried your suggestion both ways [root@mseas etc]# mount -o vers=3 mseas-data:/home mount.nfs: Unknown error 521 [root@mseas etc]# mount -o vers=4 mseas-data:/home mount.nfs: mounting mseas-data:/home failed, reason given by server: No such file or directory What should I look at next? Thanks. Pat Hi, A quick follow-up. I tried reverting back to the set ups on both machines before we enabled quotas. Rebooted both machines. Still cannot mount /home from the client (mseas). Thanks. Hi, Just before the Thanksgiving break, we enabled quotas on the /home areas on the mseas-data server (running CentOS 5.8), using the following line in the updated /etc/fstab /dev/mapper/the_raid-lv_home /home ext3 defaults,usrquota,grpquota 1 0 Following the Thanksgiving reboot of mseas-data we have been experiencing problems with svn on mseas (our front-end machine, running CentOS 6.2). After consulting with the svn mailing list, it appeared to be a communication issue between mseas and mseas-data. We therefore rebooted mseas. Now mseas is not mounting /home. We get the following error [root@mseas etc]# mount mseas-data:/home mount.nfs: Unknown error 521 We have tried this mounting with both of the following lines in fstab and got the same error mseas-data:/home/home nfs defaults0 0 mseas-data:/home/home nfs defaults,usrquota,grpquota0 0 (the latter was to mimic the mounting of /home on mseas-data) The only thing we've found out on the web so far was advice on editting /etc/exports to have an insecure option for the export of /home, which we already had. Any thoughts on how to resolve or debug this issue would be greatly appreciated. By which NFS version are you exporting from the mseas-data server...v3 or v4? If v3, you may have to add vers=3 to the options section of your fstab. Try manually mounting with -o vers=3 in the command line to see if this fixes things. -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Issue mounting /home area from NAS server
Hi Mike, A colleague of mine did some hunting of his own and found that if he restarted nfs on the server (mseas-data) then the client was able to mount the disk. Thank you very much for all of your time and help. Pat Hi Mike, I think v3 but I'm not sure (how do I check this?). At any rate, I tried your suggestion both ways [root@mseas etc]# mount -o vers=3 mseas-data:/home mount.nfs: Unknown error 521 [root@mseas etc]# mount -o vers=4 mseas-data:/home mount.nfs: mounting mseas-data:/home failed, reason given by server: No such file or directory What should I look at next? You appear to be missing the target mountpoint in your manual commands. It should probably look more like: mount -o vers=3 mseas-data:/home /home On the other hand, a Google search for mount.nfs: Unknown error 521 turns up some interesting information: https://www.google.com/search?q=mount.nfs+unknown+error+521ie=utf-8oe=utf-8aq=trls=org.mozilla:en-US:officialclient=firefox-a One of the items is a bug report from Fedora in which it turns out that the /etc/exports file has some sort of corruption (probably invalid characters) in it that was causing the NFS server system to not properly export the filesystem. You may try, on the mseas-data system, to run exportfs to see what the system thinks it's exporting. -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] iptables settings for X11 forwarding in CentOS 6.2
Hi, We recently installed CentOS 6.2 on our cluster. During the installation/debugging of various secondary software, we had disabled iptables. When we re-enabled them, we found that the front-end would no longer X11 forward (although it does so when the iptables are off). What do we need to set in the iptables to permit X11 forwarding? Currently we're using iptables -P INPUT DROP iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT iptables -A INPUT -m limit --limit 15/minute -j LOG --log-level 7 --log-prefix Dropped by firewall: iptables -A INPUT -i eth1 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 8080 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6000 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6001 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6002 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6003 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6004 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6005 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p udp --dport 177 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p udp --dport 6000 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -i eth1 -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] iptables settings for X11 forwarding in CentOS 6.2
Hi, Actually we're talking about both SSH and XDMCP X11 forwarding. Both seem to be currently disabled by the iptables. We'll try out what you suggest and get back with the results. Thanks. Pat On Fri, Mar 29, 2013 at 11:34 AM, Pat Haley pha...@mit.edu wrote: Hi, We recently installed CentOS 6.2 on our cluster. During the installation/debugging of various secondary software, we had disabled iptables. When we re-enabled them, we found that the front-end would no longer X11 forward (although it does so when the iptables are off). What do we need to set in the iptables to permit X11 forwarding? Currently we're using [Based on the port numbers below] You're talking about XDMCP and not SSH X11 forwarding -- correct? I bumped into this [0] but don't have any XDMCP setups to test with. You have most of the recommended ports allowed given your rules. Might help: If you are using Gnome open up TCP ports 16001 and TCP 35091 in both directions. [0] [0] http://www.starnet.com/xwin32kb/What_ports_need_to_be_opened_for_XDMCP [1] http://www.tldp.org/HOWTO/html_single/XDMCP-HOWTO/#PREP Consider running tcpdump on the proper interface with the firewall disabled for a moment to get an idea of what happens when things work. iptables -P INPUT DROP iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT iptables -A INPUT -m limit --limit 15/minute -j LOG --log-level 7 --log-prefix Dropped by firewall: iptables -A INPUT -i eth1 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 8080 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6000 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6001 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6002 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6003 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6004 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport 6005 -m state --state NEW,ESTABLISHED -j ACCEPT You can simplify your rule by specifying a port range instead of individual rules: iptables -A INPUT -i eth1 -p tcp --dport 6000:6005 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p udp --dport 177 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p udp --dport 6000 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -i eth1 -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -i eth1 -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
