Re: [lustre-discuss] MDS/MGS has a block storage device mounted and it does not have any permissions (no read , no write, no execute)
Thanks Andreas. Given below are the output of the commands you asked to run:. > [root@lustre-mds-server-1 opc]# > • Assuming if the above is not an issue, after setting up OSS/OST and > Client node, When my client tries to mount, I get the below error: > [root@lustre-client-1 opc]# mount -t lustre 10.0.2.4@tcp:/lustrewt > /mnt > mount.lustre: mount 10.0.2.4@tcp:/lustrewt at /mnt failed: > Input/output error Is the MGS running? > [root@lustre-client-1 opc]# Andreas: Can you do "lctl ping" from the client to the MGS node? Most commonly this happens because the client still has a firewall configured, or it is defined to have "127.0.0.1" as the local node address. Pinkesh response: [root@lustre-client-1 opc]# lctl ping 10.0.2.6@tcp 12345-0@lo 12345-10.0.2.6@tcp So there is a "lo" mentioned here, could that be a problem? I also ran the mount command on client node to capture logs on both the client node and MDS node. (ran command at 18.11 time) [root@lustre-client-1 opc]# mount -t lustre 10.0.2.6@tcp:/lustrewt /mnt mount.lustre: mount 10.0.2.6@tcp:/lustrewt at /mnt failed: Input/output error Is the MGS running? [root@lustre-client-1 opc]# [root@lustre-mds-server-1 opc]# tail -f /var/log/messages Feb 6 18:11:38 lustre-mds-server-1 kernel: Lustre: MGS: Connection restored to 88e1c321-1eaa-6914-5a37-4fff2063b526 (at 10.0.0.2@tcp) Feb 6 18:11:38 lustre-mds-server-1 kernel: Lustre: Skipped 1 previous similar message Feb 6 18:11:45 lustre-mds-server-1 kernel: Lustre: MGS: Received new LWP connection from 10.0.0.2@tcp, removing former export from same NID Feb 6 18:11:45 lustre-mds-server-1 kernel: Lustre: MGS: Connection restored to 88e1c321-1eaa-6914-5a37-4fff2063b526 (at 10.0.0.2@tcp) [root@lustre-client-1 opc]# less /var/log/messages Feb 6 18:10:01 lustre-client-1 systemd: Removed slice User Slice of root. Feb 6 18:10:01 lustre-client-1 systemd: Stopping User Slice of root. Feb 6 18:11:45 lustre-client-1 kernel: Lustre: 10376:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549476698/real 1549476698] req@9259bb42a100 x1624614953288736/t0(0) o503->MGC10.0.2.6@tcp@10.0.2.6@tcp:26/25 lens 272/8416 e 0 to 1 dl 1549476705 ref 2 fl Rpc:X/0/ rc 0/-1 Feb 6 18:11:45 lustre-client-1 kernel: Lustre: 10376:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 6 18:11:45 lustre-client-1 kernel: LustreError: 166-1: MGC10.0.2.6@tcp: Connection to MGS (at 10.0.2.6@tcp) was lost; in progress operations using this service will fail Feb 6 18:11:45 lustre-client-1 kernel: LustreError: 15c-8: MGC10.0.2.6@tcp: The configuration from log 'lustrewt-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. Feb 6 18:11:45 lustre-client-1 kernel: Lustre: MGC10.0.2.6@tcp: Connection restored to MGC10.0.2.6@tcp_0 (at 10.0.2.6@tcp) Feb 6 18:11:45 lustre-client-1 kernel: Lustre: Unmounted lustrewt-client Feb 6 18:11:45 lustre-client-1 kernel: LustreError: 10376:0:(obd_mount.c:1582:lustre_fill_super()) Unable to mount (-5) Thanks, Pinkesh Valdria OCI – Big Data Principal Solutions Architect m: +1-206-234-4314 pinkesh.vald...@oracle.com -Original Message- From: Andreas Dilger Sent: Wednesday, February 6, 2019 2:28 AM To: Pinkesh Valdria Cc: lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] MDS/MGS has a block storage device mounted and it does not have any permissions (no read , no write, no execute) On Feb 5, 2019, at 15:39, Pinkesh Valdria wrote: > > Hello All, > > I am new to Lustre. I started by using the docs on this page to deploy > Lustre on Virtual machines running CentOS 7.x (CentOS-7-2018.08.15-0). > Included below are the content of the scripts I used and the error I get. > I have not done any setup for “o2ib0(ib0)” and lnet is using tcp. All the > nodes are on the same network & subnet and cannot communicate on my protocol > and port #. > > Thanks for your help. I am completely blocked and looking for ideas. > (already did google search ☹). > > I have 2 questions: > • The MDT mounted on MDS has no permissions (no read , no write, no > execute), even for root user on MDS/MGS node. Is that expected? . See > “MGS/MDS node setup” section for more details on what I did. > [root@lustre-mds-server-1 opc]# mount -t lustre /dev/sdb /mnt/mdt > > [root@lustre-mds-server-1 opc]# ll /mnt total 0 d-. 1 root > root 0 Jan 1 1970 mdt The mountpoint on the MDS is just there for "df" to work and to manage the block device. It does not provide access to filesystem. You need to do a client mount for that (typically on another node, but it also works on the MDS), like: mount -t lustre lustre-mds-server-1:/lustrewt /mnt/lustrewt or similar. > [root@lustre-mds-server-1 opc]# > •
Re: [lustre-discuss] MDS/MGS has a block storage device mounted and it does not have any permissions (no read , no write, no execute)
On Feb 5, 2019, at 15:39, Pinkesh Valdria wrote: > > Hello All, > > I am new to Lustre. I started by using the docs on this page to deploy > Lustre on Virtual machines running CentOS 7.x (CentOS-7-2018.08.15-0). > Included below are the content of the scripts I used and the error I get. > I have not done any setup for “o2ib0(ib0)” and lnet is using tcp. All the > nodes are on the same network & subnet and cannot communicate on my protocol > and port #. > > Thanks for your help. I am completely blocked and looking for ideas. > (already did google search ☹). > > I have 2 questions: > • The MDT mounted on MDS has no permissions (no read , no write, no > execute), even for root user on MDS/MGS node. Is that expected? . See > “MGS/MDS node setup” section for more details on what I did. > [root@lustre-mds-server-1 opc]# mount -t lustre /dev/sdb /mnt/mdt > > [root@lustre-mds-server-1 opc]# ll /mnt > total 0 > d-. 1 root root 0 Jan 1 1970 mdt The mountpoint on the MDS is just there for "df" to work and to manage the block device. It does not provide access to filesystem. You need to do a client mount for that (typically on another node, but it also works on the MDS), like: mount -t lustre lustre-mds-server-1:/lustrewt /mnt/lustrewt or similar. > [root@lustre-mds-server-1 opc]# > • Assuming if the above is not an issue, after setting up OSS/OST and > Client node, When my client tries to mount, I get the below error: > [root@lustre-client-1 opc]# mount -t lustre 10.0.2.4@tcp:/lustrewt /mnt > mount.lustre: mount 10.0.2.4@tcp:/lustrewt at /mnt failed: Input/output error > Is the MGS running? > [root@lustre-client-1 opc]# Can you do "lctl ping" from the client to the MGS node? Most commonly this happens because the client still has a firewall configured, or it is defined to have "127.0.0.1" as the local node address. > dmesg shows the below error on the client node: > [root@lustre-client-1 opc]# dmesg > [35639.535862] Lustre: 11730:0:(client.c:2114:ptlrpc_expire_one_request()) > @@@ Request sent has timed out for slow reply: [sent 1549386846/real > 1549386846] req@9259bb518c00 x1624614953288208/t0(0) > o250->MGC10.0.2.4@tcp@10.0.2.4@tcp:26/25 lens 520/544 e 0 to 1 dl 1549386851 > ref 1 fl Rpc:XN/0/ rc 0/-1 > [35640.535877] LustreError: 7718:0:(mgc_request.c:251:do_config_log_add()) > MGC10.0.2.4@tcp: failed processing log, type 1: rc = -5 > [35669.535028] Lustre: 11730:0:(client.c:2114:ptlrpc_expire_one_request()) > @@@ Request sent has timed out for slow reply: [sent 1549386871/real > 1549386871] req@9259bb428f00 x1624614953288256/t0(0) > o250->MGC10.0.2.4@tcp@10.0.2.4@tcp:26/25 lens 520/544 e 0 to 1 dl 1549386881 > ref 1 fl Rpc:XN/0/ rc 0/-1 > [35670.546671] LustreError: 15c-8: MGC10.0.2.4@tcp: The configuration from > log 'lustrewt-client' failed (-5). This may be the result of communication > errors between this node and the MGS, a bad configuration, or other errors. > See the syslog for more information. > [35670.557472] Lustre: Unmounted lustrewt-client > [35670.560432] LustreError: 7718:0:(obd_mount.c:1582:lustre_fill_super()) > Unable to mount (-5) > [root@lustre-client-1 opc]# Nothing here except that the client can't communicate with the MGS. There might be some more useful messages earlier on in the logs. > I have firewall turned off on all nodes (client, mds/mgs, oss), selinux is > disabled/setenforce=0 . I can telnet to the MDS/MGS node from client > machine. > > > Given below is the setup I have on different nodes: > > MGS/MDS node setup > #!/bin/bash > service firewalld stop > chkconfig firewalld off > > cat > /etc/yum.repos.d/lustre.repo << EOF > [hpddLustreserver] > name=CentOS- - Lustre > baseurl=https://downloads.whamcloud.com/public/lustre/latest-release/el7.6.1810/server/ > gpgcheck=0 > > [e2fsprogs] > name=CentOS- - Ldiskfs > baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/ > gpgcheck=0 > > [hpddLustreclient] > name=CentOS- - Lustre > baseurl=https://downloads.whamcloud.com/public/lustre/latest-release/el7.6.1810/client/ > gpgcheck=0 > EOF > > sudo yum install lustre-tests -y > > cp /etc/selinux/config /etc/selinux/config.backup > sed 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config > > setenforce 0 > > echo "complete. rebooting now" > reboot > > > > After reboot is complete, I login to the MGS/MDS node as root and run the > following steps: > > The node has a block storage device attached: /dev/sdb > Run the below command: > pvcreate -y /dev/sdb > mkfs.xfs -f /dev/sdb The "mkfs.xfs" command here is useless. Lustre uses only ext4 or ZFS for the MDT and OSTs, and reformats the filesystem with mkfs.lustre in any case. > [root@lustre-mds-server-1 opc]# setenforce 0 > [root@lustre-mds-server-1 opc]# mkfs.lustre --fsname=lustrewt --index=0 --mgs > --mdt /dev/sdb >Permanent disk data: > Target:
