Glad I could help.
Cheers,
Quentin
Le 29/11/2018 à 13:18, Lorand Szentannai a écrit :
Hi,
I was about to say I'm sure and then realized that I've had two lustreapi
libraries laying around on the system, one being leftover from a previous
lustre client rpm.
Obviously robinhood was picking up the wrong one.
-------------------------------------------------------------------------------
lorand@robinhood:~ $ rpmbuild -tb --with lustre
rpmbuild/SOURCES/robinhood-3.1.4.tar.gz >& compile-$(date +%F).log
lorand@robinhood:~ $ grep liblustreapi compile-2018-11-29.log
Requires: /bin/bash /bin/sh ld-linux-x86-64.so.2()(64bit)
ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) libc.so.6()(64bit)
libc.so.6(GLIBC_2.10)(64bit) libc.so.6(GLIBC_2.14)(64bit)
libc.so.6(GLIBC_2.17)(64bit) libc.so.6(GLIBC_2.2.5)(64bit)
libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.4)(64bit)
libcrypto.so.10()(64bit) libdl.so.2()(64bit) libdl.so.2(GLIBC_2.2.5)(64bit)
libglib-2.0.so.0()(64bit) libgthread-2.0.so.0()(64bit)
libjemalloc.so.1()(64bit) liblustreapi.so()(64bit) libm.so.6()(64bit)
libmysqlclient.so.18()(64bit) libmysqlclient.so.18(libmysqlclient_18)(64bit)
libpthread.so.0()(64bit) libpthread.so.0(GLIBC_2.2.5)(64bit)
libpthread.so.0(GLIBC_2.3.2)(64bit) librbh_mod_alerter.so.0()(64bit)
librbh_mod_backup.so.0()(64bit) librbh_mod_basic.so.0()(64bit)
librbh_mod_checker.so.0()(64bit) librbh_mod_common.so.0()(64bit)
librbh_mod_lhsm.so.0()(64bit) librbh_mod_modeguard.so.0()(64bit)
librbh_mod_test.so.0()(64bit) libssl.so.10()(64bit) libz.so.1()(64bit)
rtld(GNU_HASH)
Requires: libc.so.6()(64bit) libc.so.6(GLIBC_2.14)(64bit)
libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit)
libglib-2.0.so.0()(64bit) libgthread-2.0.so.0()(64bit)
libjemalloc.so.1()(64bit) liblustreapi.so()(64bit) libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit) librt.so.1()(64bit) libz.so.1()(64bit)
rtld(GNU_HASH)
lorand@robinhood:~ $ rpm -qa |egrep "lustre|robinhood"
lustre-client-debuginfo-2.10.5-1.el7.x86_64
kmod-lustre-client-tests-2.10.5-1.el7.x86_64
robinhood-tools-3.1.4-1.lustre.el7.x86_64
lustre-iokit-2.10.5-1.el7.x86_64
robinhood-tests-3.1.4-1.x86_64
kmod-lustre-client-2.10.5-1.el7.x86_64
lustre-client-tests-2.10.5-1.el7.x86_64
robinhood-debuginfo-3.1.4-1.el7.x86_64
lustre-client-2.10.5-1.el7.x86_64
robinhood-lustre-3.1.4-1.lustre.el7.x86_64
robinhood-adm-3.1.4-1.x86_64
lorand@robinhood:~ $ ldd $(which robinhood)
linux-vdso.so.1 => (0x00007ffc591f0000)
libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18
(0x00007f82f3278000)
libm.so.6 => /lib64/libm.so.6 (0x00007f82f2f76000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f82f2d72000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007f82f2b00000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f82f269f000)
liblustreapi.so => /lib/liblustreapi.so (0x00007f82f2485000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f82f2269000)
libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007f82f2067000)
libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f82f1d53000)
libjemalloc.so.1 => /lib64/libjemalloc.so.1 (0x00007f82f1b1f000)
libz.so.1 => /lib64/libz.so.1 (0x00007f82f1909000)
libc.so.6 => /lib64/libc.so.6 (0x00007f82f153c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f82f3778000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f82f1235000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f82f0fe8000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f82f0d00000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f82f0afc000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f82f08c9000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f82f0667000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f82f0451000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f82f0243000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f82f003f000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f82efe26000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f82efbff000)
lorand@robinhood:~ $ md5sum /lib/liblustreapi.so
c58640c0bfe0c7551e917db122f2f42d /lib/liblustreapi.so
lorand@robinhood:~ $ md5sum /usr/lib64/liblustreapi.so
1e649f4b0a541a8f62c1bf0c0402ae7f /usr/lib64/liblustreapi.so
lorand@robinhood:~ $ rpm -qf /usr/lib64/liblustreapi.so
lustre-client-2.10.5-1.el7.x86_64
lorand@robinhood:~ $ ls -la /lib/liblustreapi.so
-rw-r--r-- 1 root root 412152 Sep 18 2017 /lib/liblustreapi.so
[root@robinhood x86_64]# mv /lib/liblustreapi.so{,.dis}
-------------------------------------------------------------------------------
Thank you!
Lorand
On 29 Nov 2018, at 11:46, [email protected] wrote:
Hi again,
I am pretty sure this is it though. I remember having to deal with it while
working on lustre.
I initially installed robinhood from rpms, but now fetched the sources and
freshly compiled, built new rpms.
And are you _sure_ lustre 2.10 was installed on the node you compiled/built
robinhood's rpms?
Quentin
Le 29/11/2018 à 10:51, Lorand Szentannai a écrit :
Hi Quentin,
I initially installed robinhood from rpms, but now fetched the sources and
freshly compiled, built new rpms.
Unfortunately the behavior is still the same.
Thanks,
Lorand
On 28 Nov 2018, at 15:46, [email protected] wrote:
Hi Lorand,
I seem to remember this is caused by an update of lustre's changelog API.
Did compile robinhood against the right version of liblustreapi? (there is no
need for ioctl in Lustre 2.10's API)
Quentin
Le 28/11/2018 à 15:04, Lorand Szentannai a écrit :
Hi Thomas,
I actually did that already but forgot to add the strace lines to the initial
email. But it is still unclear to me why open function is called with the
O_RDONLY flag.
[pid 2348817] open("/cluster", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 12
[pid 2348817] ioctl(12, _IOC(_IOC_WRITE, 0x66, 0x94, 0x08), 0x7ffef3f20240) =
-1 ENOTTY (Inappropriate ioctl for device)
# grep lustre /proc/mounts
10.3.5.18@o2ib,10.4.5.18@o2ib1:10.3.5.19@o2ib,10.4.5.19@o2ib1:/scratch /cluster
lustre rw,flock,lazystatfs 0 0
Thanks,
Lorand Szentannai
On 28 Nov 2018, at 12:47, LEIBOVICI Thomas <[email protected]> wrote:
Hello,
If you run "strace -f robinhood --readlog" you should see the ioctl that
causes this error, and its arguments.
Thomas
On 11/28/18 07:36, Lorand Szentannai wrote:
Hi,
We've recently upgraded to Lustre 2.10.5 and to Robinhood to 3.1.4.
Processing changelogs fails now with "ERROR -25 opening changelog for MDT
'scratch-MDT0000': Inappropriate ioctl for device".
Could you please advice on how to solve this? Note that currently MGS and MDS
are running on two separate servers.
----------------------------------------------------------------------------------
ChangeLog {
MDT {
mdt_name = "MDT0000";
reader_id = "cl4"; # yes, it is cl4
}
force_polling = ON ;
polling_interval = 60s ;
}
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
# lfs changelog scratch-MDT0000 |head
8211574766 10OPEN 22:24:39.830737089 2018.11.27 0x43 t=[0x200033cc4:0x4e9d:0x0]
8211574767 13TRUNC 22:24:39.831280872 2018.11.27 0xe t=[0x200033cc4:0x4e9d:0x0]
8211574768 11CLOSE 22:24:39.831774075 2018.11.27 0x43 t=[0x200033cc4:0x4e9d:0x0]
8211574769 10OPEN 22:24:39.831950980 2018.11.27 0x43 t=[0x200033cc4:0x4e9e:0x0]
8211574770 13TRUNC 22:24:39.832289331 2018.11.27 0xe t=[0x200033cc4:0x4e9e:0x0]
8211574771 11CLOSE 22:24:39.832765269 2018.11.27 0x43 t=[0x200033cc4:0x4e9e:0x0]
8211574772 10OPEN 22:24:39.832942039 2018.11.27 0x43 t=[0x200033cc4:0x4e9f:0x0]
8211574773 13TRUNC 22:24:39.833241628 2018.11.27 0xe t=[0x200033cc4:0x4e9f:0x0]
8211574774 11CLOSE 22:24:39.833541406 2018.11.27 0x43 t=[0x200033cc4:0x4e9f:0x0]
8211574775 10OPEN 22:24:39.833672445 2018.11.27 0x43 t=[0x200033cc4:0x4ea0:0x0]
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
# robinhood --readlog
2018/11/28 07:20:54 [2070664/1] Main | EntryProcessor successfully initialized
2018/11/28 07:20:54 [2070664/1] ChangeLog | mdt[0] = MDT0000
2018/11/28 07:20:54 [2070664/1] ListMgr | Logged on to database
'robinhood_scratch' successfully
2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SET SESSION TRANSACTION
ISOLATION LEVEL READ COMMITTED
2018/11/28 07:20:54 [2070664/3] ListMgr | Logged on to database
'robinhood_scratch' successfully
2018/11/28 07:20:54 [2070664/3] ListMgr | SQL query: SET SESSION TRANSACTION
ISOLATION LEVEL READ COMMITTED
2018/11/28 07:20:54 [2070664/5] ListMgr | Logged on to database
'robinhood_scratch' successfully
2018/11/28 07:20:54 [2070664/5] ListMgr | SQL query: SET SESSION TRANSACTION
ISOLATION LEVEL READ COMMITTED
2018/11/28 07:20:54 [2070664/6] ListMgr | Logged on to database
'robinhood_scratch' successfully
2018/11/28 07:20:54 [2070664/6] ListMgr | SQL query: SET SESSION TRANSACTION
ISOLATION LEVEL READ COMMITTED
2018/11/28 07:20:54 [2070664/4] ListMgr | Logged on to database
'robinhood_scratch' successfully
2018/11/28 07:20:54 [2070664/4] ListMgr | SQL query: SET SESSION TRANSACTION
ISOLATION LEVEL READ COMMITTED
2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SELECT value FROM VARS
WHERE varname='CL_LastCommit_MDT0000'
2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SELECT value FROM VARS
WHERE varname='ChangelogLastCommit_MDT0000'
2018/11/28 07:20:54 [2070664/1] ChangeLog | Opening chglog for scratch-MDT0000
(start_rec=0)
2018/11/28 07:20:54 [2070664/1] ChangeLog | ERROR -25 opening changelog for MDT
'scratch-MDT0000': Inappropriate ioctl for device
2018/11/28 07:20:54 [2070664/1] Main | Error 25 initializing ChangeLog Reader
----------------------------------------------------------------------------------
Thanks,
Lorand Szentannai
_______________________________________________
robinhood-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/robinhood-support
_______________________________________________
robinhood-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/robinhood-support
_______________________________________________
robinhood-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/robinhood-support
