Hi Raj, [root@pg-gpu01 ~]# lustre_rmmod
[root@pg-gpu01 ~]# modprobe -v lustre insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/net/lustre/libcfs.ko insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/lvfs.ko insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/net/lustre/lnet.ko networks=o2ib(ib0) insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/obdclass.ko insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/ptlrpc.ko insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/fid.ko insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/mdc.ko insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/osc.ko insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/lov.ko insmod /lib/modules/2.6.32-642.6.2.el6.x86_64/weak-updates/kernel/fs/lustre/lustre.ko dmesg: LNet: HW CPU cores: 24, npartitions: 4 alg: No test for crc32 (crc32-table) alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-pclmul) Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-573.el6.x86_64 LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180] But no luck, [root@pg-gpu01 ~]# lctl ping 172.23.55.211@o2ib failed to ping 172.23.55.211@o2ib: Input/output error [root@pg-gpu01 ~]# mount /home mount.lustre: mount 172.23.55.211@o2ib:172.23.55.212@o2ib:/pghome01 at /home failed: Input/output error Is the MGS running? On Mon, Apr 24, 2017 at 7:53 PM, Raj <[email protected]> wrote: > Yes, this is strange. Normally, I have seen that credits mismatch results > this scenario but it doesn't look like this is the case. > > You wouldn't want to put mgs into capture debug messages as there will be > a lot of data. > > I guess you already tried removing the lustre drivers and adding it again > ? > lustre_rmmod > modprobe -v lustre > > And check dmesg for any errors... > > > On Mon, Apr 24, 2017 at 12:43 PM Strikwerda, Ger <[email protected]> > wrote: > >> Hi Raj, >> >> When i do a lctl ping on a MGS server i do not see any logs at all. Also >> not when i do a sucessfull ping from a working node. Is there a way to >> verbose the Lustre logging to see more detail on the LNET level? >> >> It is very strange that a rebooted node is able to lctl ping compute >> nodes, but fails to lctl ping metadata and storage nodes. >> >> >> >> >> On Mon, Apr 24, 2017 at 7:35 PM, Raj <[email protected]> wrote: >> >>> Ger, >>> It looks like default configuration of lustre. >>> >>> Do you see any error message on the MGS side while you are doing lctl >>> ping from the rebooted clients? >>> On Mon, Apr 24, 2017 at 12:27 PM Strikwerda, Ger < >>> [email protected]> wrote: >>> >>>> Hi Eli, >>>> >>>> Nothing can be mounted on the Lustre filesystems so the output is: >>>> >>>> [root@pg-gpu01 ~]# lfs df /home/ger/ >>>> [root@pg-gpu01 ~]# >>>> >>>> Empty.. >>>> >>>> >>>> >>>> On Mon, Apr 24, 2017 at 7:24 PM, E.S. Rosenberg <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Mon, Apr 24, 2017 at 8:19 PM, Strikwerda, Ger < >>>>> [email protected]> wrote: >>>>> >>>>>> Hallo Eli, >>>>>> >>>>>> Logfile/syslog on the client-side: >>>>>> >>>>>> Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32- >>>>>> 573.el6.x86_64 >>>>>> LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180] >>>>>> LNetError: 2878:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) >>>>>> 172.23.55.211@o2ib rejected: consumer defined fatal error >>>>>> >>>>> >>>>> lctl df /path/to/some/file >>>>> >>>>> gives nothing useful? (the second one will dump *a lot*) >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Apr 24, 2017 at 7:16 PM, E.S. Rosenberg < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Apr 24, 2017 at 8:13 PM, Strikwerda, Ger < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Raj (and others), >>>>>>>> >>>>>>>> In which file should i state the credits/peer_credits stuff? >>>>>>>> >>>>>>>> Perhaps relevant config-files: >>>>>>>> >>>>>>>> [root@pg-gpu01 ~]# cd /etc/modprobe.d/ >>>>>>>> >>>>>>>> [root@pg-gpu01 modprobe.d]# ls >>>>>>>> anaconda.conf blacklist-kvm.conf dist-alsa.conf >>>>>>>> dist-oss.conf ib_ipoib.conf lustre.conf openfwwf.conf >>>>>>>> blacklist.conf blacklist-nouveau.conf dist.conf >>>>>>>> freeipmi-modalias.conf ib_sdp.conf mlnx.conf truescale.conf >>>>>>>> >>>>>>>> [root@pg-gpu01 modprobe.d]# cat ./ib_ipoib.conf >>>>>>>> alias netdev-ib* ib_ipoib >>>>>>>> >>>>>>>> [root@pg-gpu01 modprobe.d]# cat ./mlnx.conf >>>>>>>> # Module parameters for MLNX_OFED kernel modules >>>>>>>> >>>>>>>> [root@pg-gpu01 modprobe.d]# cat ./lustre.conf >>>>>>>> options lnet networks=o2ib(ib0) >>>>>>>> >>>>>>>> Are there more Lustre/LNET options that could help in this >>>>>>>> situation? >>>>>>>> >>>>>>> >>>>>>> What about the logfiles? >>>>>>> Any error messages in syslog? lctl debug options? >>>>>>> Veel geluk, >>>>>>> Eli >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Apr 24, 2017 at 7:02 PM, Raj <[email protected]> wrote: >>>>>>>> >>>>>>>>> May be worth checking your lnet credits and peer_credits in >>>>>>>>> /etc/modprobe.d ? >>>>>>>>> You can compare between working hosts and non working hosts. >>>>>>>>> Thanks >>>>>>>>> _Raj >>>>>>>>> >>>>>>>>> On Mon, Apr 24, 2017 at 10:10 AM Strikwerda, Ger < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Rick, >>>>>>>>>> >>>>>>>>>> Even without iptables rules and loading the correct modules >>>>>>>>>> afterwards, we get the same results: >>>>>>>>>> >>>>>>>>>> [root@pg-gpu01 sysconfig]# iptables --list >>>>>>>>>> Chain INPUT (policy ACCEPT) >>>>>>>>>> target prot opt source destination >>>>>>>>>> >>>>>>>>>> Chain FORWARD (policy ACCEPT) >>>>>>>>>> target prot opt source destination >>>>>>>>>> >>>>>>>>>> Chain OUTPUT (policy ACCEPT) >>>>>>>>>> target prot opt source destination >>>>>>>>>> >>>>>>>>>> Chain LOGDROP (0 references) >>>>>>>>>> target prot opt source destination >>>>>>>>>> LOG all -- anywhere anywhere LOG >>>>>>>>>> level warning >>>>>>>>>> DROP all -- anywhere anywhere >>>>>>>>>> >>>>>>>>>> [root@pg-gpu01 sysconfig]# modprobe lnet >>>>>>>>>> >>>>>>>>>> [root@pg-gpu01 sysconfig]# modprobe lustre >>>>>>>>>> >>>>>>>>>> [root@pg-gpu01 sysconfig]# lctl ping 172.23.55.211@o2ib >>>>>>>>>> >>>>>>>>>> failed to ping 172.23.55.211@o2ib: Input/output error >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Apr 24, 2017 at 4:59 PM, Mohr Jr, Richard Frank (Rick >>>>>>>>>> Mohr) <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> This might be a long shot, but have you checked for possible >>>>>>>>>>> firewall rules that might be causing the issue? I’m wondering if >>>>>>>>>>> there is >>>>>>>>>>> a chance that some rules were added after the nodes were up to >>>>>>>>>>> allow Lustre >>>>>>>>>>> access, and when a node got rebooted, it lost the rules. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Rick Mohr >>>>>>>>>>> Senior HPC System Administrator >>>>>>>>>>> National Institute for Computational Sciences >>>>>>>>>>> http://www.nics.tennessee.edu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > On Apr 24, 2017, at 10:19 AM, Strikwerda, Ger < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> > >>>>>>>>>>> > Hi Russell, >>>>>>>>>>> > >>>>>>>>>>> > Thanks for the IB subnet clues: >>>>>>>>>>> > >>>>>>>>>>> > [root@pg-gpu01 ~]# ibv_devinfo >>>>>>>>>>> > hca_id: mlx4_0 >>>>>>>>>>> > transport: InfiniBand (0) >>>>>>>>>>> > fw_ver: 2.32.5100 >>>>>>>>>>> > node_guid: f452:1403:00f5:4620 >>>>>>>>>>> > sys_image_guid: f452:1403:00f5:4623 >>>>>>>>>>> > vendor_id: 0x02c9 >>>>>>>>>>> > vendor_part_id: 4099 >>>>>>>>>>> > hw_ver: 0x1 >>>>>>>>>>> > board_id: MT_1100120019 >>>>>>>>>>> > phys_port_cnt: 1 >>>>>>>>>>> > port: 1 >>>>>>>>>>> > state: PORT_ACTIVE (4) >>>>>>>>>>> > max_mtu: 4096 (5) >>>>>>>>>>> > active_mtu: 4096 (5) >>>>>>>>>>> > sm_lid: 1 >>>>>>>>>>> > port_lid: 185 >>>>>>>>>>> > port_lmc: 0x00 >>>>>>>>>>> > link_layer: InfiniBand >>>>>>>>>>> > >>>>>>>>>>> > [root@pg-gpu01 ~]# sminfo >>>>>>>>>>> > sminfo: sm lid 1 sm guid 0xf452140300f62320, activity count >>>>>>>>>>> 80878098 priority 0 state 3 SMINFO_MASTER >>>>>>>>>>> > >>>>>>>>>>> > Looks like the rebooted node is able to connect/contact IB/IB >>>>>>>>>>> subnetmanager >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > On Mon, Apr 24, 2017 at 4:14 PM, Russell Dekema < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> > At first glance, this sounds like your Infiniband subnet >>>>>>>>>>> manager may >>>>>>>>>>> > be down or malfunctioning. In this case, nodes which were >>>>>>>>>>> already up >>>>>>>>>>> > when the subnet manager was working will continue to be able to >>>>>>>>>>> > communicate over IB, but nodes which reboot after the SM goes >>>>>>>>>>> down >>>>>>>>>>> > will not. >>>>>>>>>>> > >>>>>>>>>>> > You can test this theory by running the 'ibv_devinfo' command >>>>>>>>>>> on one >>>>>>>>>>> > of your rebooted nodes. If the relevant IB port is in state >>>>>>>>>>> PORT_INIT, >>>>>>>>>>> > this confirms there is a problem with your subnet manager. >>>>>>>>>>> > >>>>>>>>>>> > Sincerely, >>>>>>>>>>> > Rusty Dekema >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > On Mon, Apr 24, 2017 at 9:57 AM, Strikwerda, Ger >>>>>>>>>>> > <[email protected]> wrote: >>>>>>>>>>> > > Hi everybody, >>>>>>>>>>> > > >>>>>>>>>>> > > Here at the university of Groningen we are now experiencing >>>>>>>>>>> a strange Lustre >>>>>>>>>>> > > error. If a client reboots, it fails to mount the Lustre >>>>>>>>>>> storage. The client >>>>>>>>>>> > > is not able to reach the MSG service. The storage and nodes >>>>>>>>>>> are >>>>>>>>>>> > > communicating over IB and unitil now without any problems. >>>>>>>>>>> It looks like an >>>>>>>>>>> > > issue inside LNET. Clients cannot LNET ping/connect the >>>>>>>>>>> metadata and or >>>>>>>>>>> > > storage. But the clients are able to LNET ping each other. >>>>>>>>>>> Clients which not >>>>>>>>>>> > > have been rebooted, are working fine and have their mounts >>>>>>>>>>> on our Lustre >>>>>>>>>>> > > filesystem. >>>>>>>>>>> > > >>>>>>>>>>> > > Lustre client log: >>>>>>>>>>> > > >>>>>>>>>>> > > Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32- >>>>>>>>>>> 573.el6.x86_64 >>>>>>>>>>> > > LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180] >>>>>>>>>>> > > >>>>>>>>>>> > > LustreError: 15c-8: MGC172.23.55.211@o2ib: The >>>>>>>>>>> configuration from log >>>>>>>>>>> > > 'pgdata01-client' failed (-5). This may be the result of >>>>>>>>>>> communication >>>>>>>>>>> > > errors between this node and the MGS, a bad configuration, >>>>>>>>>>> or other errors. >>>>>>>>>>> > > See the syslog for more information. >>>>>>>>>>> > > LustreError: 3812:0:(llite_lib.c:1046:ll_fill_super()) >>>>>>>>>>> Unable to process >>>>>>>>>>> > > log: -5 >>>>>>>>>>> > > Lustre: Unmounted pgdata01-client >>>>>>>>>>> > > LustreError: 3812:0:(obd_mount.c:1325:lustre_fill_super()) >>>>>>>>>>> Unable to mount >>>>>>>>>>> > > (-5) >>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) >>>>>>>>>>> 172.23.55.212@o2ib >>>>>>>>>>> > > rejected: consumer defined fatal error >>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) >>>>>>>>>>> Skipped 1 previous >>>>>>>>>>> > > similar message >>>>>>>>>>> > > Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request()) >>>>>>>>>>> @@@ Request sent >>>>>>>>>>> > > has failed due to network error: [sent 1492789626/real >>>>>>>>>>> 1492789626] >>>>>>>>>>> > > req@ffff88105af2cc00 x1565303228072004/t0(0) >>>>>>>>>>> > > o250->MGC172.23.55.211@[email protected]@o2ib:26/25 lens >>>>>>>>>>> 400/544 e 0 to 1 >>>>>>>>>>> > > dl 1492789631 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 >>>>>>>>>>> > > Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request()) >>>>>>>>>>> Skipped 1 >>>>>>>>>>> > > previous similar message >>>>>>>>>>> > > LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req()) >>>>>>>>>>> @@@ send limit >>>>>>>>>>> > > expired req@ffff882041ffc000 x1565303228071996/t0(0) >>>>>>>>>>> > > o101->MGC172.23.55.211@[email protected]@o2ib:26/25 lens >>>>>>>>>>> 328/344 e 0 to 0 >>>>>>>>>>> > > dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 >>>>>>>>>>> > > LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req()) >>>>>>>>>>> Skipped 2 >>>>>>>>>>> > > previous similar messages >>>>>>>>>>> > > LustreError: 15c-8: MGC172.23.55.211@o2ib: The >>>>>>>>>>> configuration from log >>>>>>>>>>> > > 'pghome01-client' failed (-5). This may be the result of >>>>>>>>>>> communication >>>>>>>>>>> > > errors between this node and the MGS, a bad configuration, >>>>>>>>>>> or other errors. >>>>>>>>>>> > > See the syslog for more information. >>>>>>>>>>> > > LustreError: 3826:0:(llite_lib.c:1046:ll_fill_super()) >>>>>>>>>>> Unable to process >>>>>>>>>>> > > log: -5 >>>>>>>>>>> > > >>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) >>>>>>>>>>> 172.23.55.212@o2ib >>>>>>>>>>> > > rejected: consumer defined fatal error >>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) >>>>>>>>>>> Skipped 1 previous >>>>>>>>>>> > > similar message >>>>>>>>>>> > > LNet: 3755:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from >>>>>>>>>>> > > 172.23.55.211@o2ib failed: 5 >>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) >>>>>>>>>>> 172.23.55.211@o2ib >>>>>>>>>>> > > rejected: consumer defined fatal error >>>>>>>>>>> > > LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) >>>>>>>>>>> Skipped 1 previous >>>>>>>>>>> > > similar message >>>>>>>>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed()) >>>>>>>>>>> Deleting >>>>>>>>>>> > > messages for 172.23.55.211@o2ib: connection failed >>>>>>>>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed()) >>>>>>>>>>> Deleting >>>>>>>>>>> > > messages for 172.23.55.212@o2ib: connection failed >>>>>>>>>>> > > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from >>>>>>>>>>> > > 172.23.55.212@o2ib failed: 5 >>>>>>>>>>> > > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) >>>>>>>>>>> Skipped 17 previous >>>>>>>>>>> > > similar messages >>>>>>>>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed()) >>>>>>>>>>> Deleting >>>>>>>>>>> > > messages for 172.23.55.211@o2ib: connection failed >>>>>>>>>>> > > LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from >>>>>>>>>>> > > 172.23.55.212@o2ib failed: 5 >>>>>>>>>>> > > LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed()) >>>>>>>>>>> Deleting >>>>>>>>>>> > > messages for 172.23.55.212@o2ib: connection failed >>>>>>>>>>> > > >>>>>>>>>>> > > LNET ping of a metadata-node: >>>>>>>>>>> > > >>>>>>>>>>> > > [root@pg-gpu01 ~]# lctl ping 172.23.55.211@o2ib >>>>>>>>>>> > > failed to ping 172.23.55.211@o2ib: Input/output error >>>>>>>>>>> > > >>>>>>>>>>> > > LNET ping of the number 2 metadata-node: >>>>>>>>>>> > > >>>>>>>>>>> > > [root@pg-gpu01 ~]# lctl ping 172.23.55.212@o2ib >>>>>>>>>>> > > failed to ping 172.23.55.212@o2ib: Input/output error >>>>>>>>>>> > > >>>>>>>>>>> > > LNET ping of a random compute-node: >>>>>>>>>>> > > >>>>>>>>>>> > > [root@pg-gpu01 ~]# lctl ping 172.23.52.5@o2ib >>>>>>>>>>> > > 12345-0@lo >>>>>>>>>>> > > 12345-172.23.52.5@o2ib >>>>>>>>>>> > > >>>>>>>>>>> > > LNET to OST01: >>>>>>>>>>> > > >>>>>>>>>>> > > [root@pg-gpu01 ~]# lctl ping 172.23.55.201@o2ib >>>>>>>>>>> > > failed to ping 172.23.55.201@o2ib: Input/output error >>>>>>>>>>> > > >>>>>>>>>>> > > LNET to OST02: >>>>>>>>>>> > > >>>>>>>>>>> > > [root@pg-gpu01 ~]# lctl ping 172.23.55.202@o2ib >>>>>>>>>>> > > failed to ping 172.23.55.202@o2ib: Input/output error >>>>>>>>>>> > > >>>>>>>>>>> > > 'normal' pings (on ip level) works fine: >>>>>>>>>>> > > >>>>>>>>>>> > > [root@pg-gpu01 ~]# ping 172.23.55.201 >>>>>>>>>>> > > PING 172.23.55.201 (172.23.55.201) 56(84) bytes of data. >>>>>>>>>>> > > 64 bytes from 172.23.55.201: icmp_seq=1 ttl=64 time=0.741 ms >>>>>>>>>>> > > >>>>>>>>>>> > > [root@pg-gpu01 ~]# ping 172.23.55.202 >>>>>>>>>>> > > PING 172.23.55.202 (172.23.55.202) 56(84) bytes of data. >>>>>>>>>>> > > 64 bytes from 172.23.55.202: icmp_seq=1 ttl=64 time=0.704 ms >>>>>>>>>>> > > >>>>>>>>>>> > > lctl on a rebooted node: >>>>>>>>>>> > > >>>>>>>>>>> > > [root@pg-gpu01 ~]# lctl dl >>>>>>>>>>> > > >>>>>>>>>>> > > lctl on a not rebooted node: >>>>>>>>>>> > > >>>>>>>>>>> > > [root@pg-node005 ~]# lctl dl >>>>>>>>>>> > > 0 UP mgc MGC172.23.55.211@o2ib >>>>>>>>>>> > > 94bd1c8a-512f-b920-9a4e-a6aced3d386d >>>>>>>>>>> 5 >>>>>>>>>>> > > 1 UP lov pgtemp01-clilov-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 4 >>>>>>>>>>> > > 2 UP lmv pgtemp01-clilmv-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 4 >>>>>>>>>>> > > 3 UP mdc pgtemp01-MDT0000-mdc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 4 UP osc pgtemp01-OST0001-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 5 UP osc pgtemp01-OST0003-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 6 UP osc pgtemp01-OST0005-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 7 UP osc pgtemp01-OST0007-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 8 UP osc pgtemp01-OST0009-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 9 UP osc pgtemp01-OST000b-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 10 UP osc pgtemp01-OST000d-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 11 UP osc pgtemp01-OST000f-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 12 UP osc pgtemp01-OST0011-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 13 UP osc pgtemp01-OST0002-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 14 UP osc pgtemp01-OST0004-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 15 UP osc pgtemp01-OST0006-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 16 UP osc pgtemp01-OST0008-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 17 UP osc pgtemp01-OST000a-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 18 UP osc pgtemp01-OST000c-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 19 UP osc pgtemp01-OST000e-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 20 UP osc pgtemp01-OST0010-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 21 UP osc pgtemp01-OST0012-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 22 UP osc pgtemp01-OST0013-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 23 UP osc pgtemp01-OST0015-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 24 UP osc pgtemp01-OST0017-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 25 UP osc pgtemp01-OST0014-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 26 UP osc pgtemp01-OST0016-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 27 UP osc pgtemp01-OST0018-osc-ffff88206906d400 >>>>>>>>>>> > > 281c441f-8aa3-ab56-8812-e459d308f47c 5 >>>>>>>>>>> > > 28 UP lov pgdata01-clilov-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 4 >>>>>>>>>>> > > 29 UP lmv pgdata01-clilmv-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 4 >>>>>>>>>>> > > 30 UP mdc pgdata01-MDT0000-mdc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 31 UP osc pgdata01-OST0001-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 32 UP osc pgdata01-OST0003-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 33 UP osc pgdata01-OST0005-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 34 UP osc pgdata01-OST0007-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 35 UP osc pgdata01-OST0009-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 36 UP osc pgdata01-OST000b-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 37 UP osc pgdata01-OST000d-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 38 UP osc pgdata01-OST000f-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 39 UP osc pgdata01-OST0002-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 40 UP osc pgdata01-OST0004-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 41 UP osc pgdata01-OST0006-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 42 UP osc pgdata01-OST0008-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 43 UP osc pgdata01-OST000a-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 44 UP osc pgdata01-OST000c-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 45 UP osc pgdata01-OST000e-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 46 UP osc pgdata01-OST0010-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 47 UP osc pgdata01-OST0013-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 48 UP osc pgdata01-OST0015-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 49 UP osc pgdata01-OST0017-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 50 UP osc pgdata01-OST0014-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 51 UP osc pgdata01-OST0016-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 52 UP osc pgdata01-OST0018-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 53 UP osc pgdata01-OST0019-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 54 UP osc pgdata01-OST001a-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 55 UP osc pgdata01-OST001b-osc-ffff88204bab6400 >>>>>>>>>>> > > 996b1742-82eb-281c-c322-e244672d5225 5 >>>>>>>>>>> > > 56 UP lov pghome01-clilov-ffff88204bb50000 >>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4 >>>>>>>>>>> > > 57 UP lmv pghome01-clilmv-ffff88204bb50000 >>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4 >>>>>>>>>>> > > 58 UP mdc pghome01-MDT0000-mdc-ffff88204bb50000 >>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5 >>>>>>>>>>> > > 59 UP osc pghome01-OST0011-osc-ffff88204bb50000 >>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5 >>>>>>>>>>> > > 60 UP osc pghome01-OST0012-osc-ffff88204bb50000 >>>>>>>>>>> > > 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5 >>>>>>>>>>> > > >>>>>>>>>>> > > Please help, any clues/advice/hints/tips are appricated >>>>>>>>>>> > > >>>>>>>>>>> > > -- >>>>>>>>>>> > > >>>>>>>>>>> > > Vriendelijke groet, >>>>>>>>>>> > > >>>>>>>>>>> > > Ger Strikwerda >>>>>>>>>>> > > Chef Special >>>>>>>>>>> > > Rijksuniversiteit Groningen >>>>>>>>>>> > > Centrum voor Informatie Technologie >>>>>>>>>>> > > Unit Pragmatisch Systeembeheer >>>>>>>>>>> > > >>>>>>>>>>> > > Smitsborg >>>>>>>>>>> > > Nettelbosje 1 >>>>>>>>>>> > > 9747 AJ Groningen >>>>>>>>>>> > > Tel. 050 363 9276 >>>>>>>>>>> > > >>>>>>>>>>> > > "God is hard, God is fair >>>>>>>>>>> > > some men he gave brains, others he gave hair" >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > _______________________________________________ >>>>>>>>>>> > > lustre-discuss mailing list >>>>>>>>>>> > > [email protected] >>>>>>>>>>> > > http://lists.lustre.org/listinfo.cgi/lustre-discuss- >>>>>>>>>>> lustre.org >>>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > -- >>>>>>>>>>> > Vriendelijke groet, >>>>>>>>>>> > >>>>>>>>>>> > Ger Strikwerda >>>>>>>>>>> > >>>>>>>>>>> > Chef Special >>>>>>>>>>> > Rijksuniversiteit Groningen >>>>>>>>>>> > Centrum voor Informatie Technologie >>>>>>>>>>> > Unit Pragmatisch Systeembeheer >>>>>>>>>>> > >>>>>>>>>>> > Smitsborg >>>>>>>>>>> > Nettelbosje 1 >>>>>>>>>>> > 9747 AJ Groningen >>>>>>>>>>> > Tel. 050 363 9276 >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > "God is hard, God is fair >>>>>>>>>>> > some men he gave brains, others he gave hair" >>>>>>>>>>> > _______________________________________________ >>>>>>>>>>> > lustre-discuss mailing list >>>>>>>>>>> > [email protected] >>>>>>>>>>> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Vriendelijke groet, >>>>>>>>>> >>>>>>>>>> Ger StrikwerdaChef Special >>>>>>>>>> Rijksuniversiteit Groningen >>>>>>>>>> Centrum voor Informatie Technologie >>>>>>>>>> Unit Pragmatisch Systeembeheer >>>>>>>>>> >>>>>>>>>> Smitsborg >>>>>>>>>> Nettelbosje 1 >>>>>>>>>> 9747 AJ Groningen >>>>>>>>>> Tel. 050 363 9276 >>>>>>>>>> "God is hard, God is fair >>>>>>>>>> some men he gave brains, others he gave hair" >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> lustre-discuss mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Vriendelijke groet, >>>>>>>> >>>>>>>> Ger StrikwerdaChef Special >>>>>>>> Rijksuniversiteit Groningen >>>>>>>> Centrum voor Informatie Technologie >>>>>>>> Unit Pragmatisch Systeembeheer >>>>>>>> >>>>>>>> Smitsborg >>>>>>>> Nettelbosje 1 >>>>>>>> 9747 AJ Groningen >>>>>>>> Tel. 050 363 9276 >>>>>>>> "God is hard, God is fair >>>>>>>> some men he gave brains, others he gave hair" >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> lustre-discuss mailing list >>>>>>>> [email protected] >>>>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Vriendelijke groet, >>>>>> >>>>>> Ger StrikwerdaChef Special >>>>>> Rijksuniversiteit Groningen >>>>>> Centrum voor Informatie Technologie >>>>>> Unit Pragmatisch Systeembeheer >>>>>> >>>>>> Smitsborg >>>>>> Nettelbosje 1 >>>>>> 9747 AJ Groningen >>>>>> Tel. 050 363 9276 >>>>>> "God is hard, God is fair >>>>>> some men he gave brains, others he gave hair" >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Vriendelijke groet, >>>> >>>> Ger StrikwerdaChef Special >>>> Rijksuniversiteit Groningen >>>> Centrum voor Informatie Technologie >>>> Unit Pragmatisch Systeembeheer >>>> >>>> Smitsborg >>>> Nettelbosje 1 >>>> 9747 AJ Groningen >>>> Tel. 050 363 9276 >>>> "God is hard, God is fair >>>> some men he gave brains, others he gave hair" >>>> >>>> >> >> >> -- >> >> Vriendelijke groet, >> >> Ger StrikwerdaChef Special >> Rijksuniversiteit Groningen >> Centrum voor Informatie Technologie >> Unit Pragmatisch Systeembeheer >> >> Smitsborg >> Nettelbosje 1 >> 9747 AJ Groningen >> Tel. 050 363 9276 >> "God is hard, God is fair >> some men he gave brains, others he gave hair" >> >> -- Vriendelijke groet, Ger StrikwerdaChef Special Rijksuniversiteit Groningen Centrum voor Informatie Technologie Unit Pragmatisch Systeembeheer Smitsborg Nettelbosje 1 9747 AJ Groningen Tel. 050 363 9276 "God is hard, God is fair some men he gave brains, others he gave hair"
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
