Hi,
I was about to say I'm sure and then realized that I've had two lustreapi
libraries laying around on the system, one being leftover from a previous
lustre client rpm.
Obviously robinhood was picking up the wrong one.
-------------------------------------------------------------------------------
lorand@robinhood:~ $ rpmbuild -tb --with lustre
rpmbuild/SOURCES/robinhood-3.1.4.tar.gz >& compile-$(date +%F).log
lorand@robinhood:~ $ grep liblustreapi compile-2018-11-29.log
Requires: /bin/bash /bin/sh ld-linux-x86-64.so.2()(64bit)
ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) libc.so.6()(64bit)
libc.so.6(GLIBC_2.10)(64bit) libc.so.6(GLIBC_2.14)(64bit)
libc.so.6(GLIBC_2.17)(64bit) libc.so.6(GLIBC_2.2.5)(64bit)
libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.4)(64bit)
libcrypto.so.10()(64bit) libdl.so.2()(64bit) libdl.so.2(GLIBC_2.2.5)(64bit)
libglib-2.0.so.0()(64bit) libgthread-2.0.so.0()(64bit)
libjemalloc.so.1()(64bit) liblustreapi.so()(64bit) libm.so.6()(64bit)
libmysqlclient.so.18()(64bit) libmysqlclient.so.18(libmysqlclient_18)(64bit)
libpthread.so.0()(64bit) libpthread.so.0(GLIBC_2.2.5)(64bit)
libpthread.so.0(GLIBC_2.3.2)(64bit) librbh_mod_alerter.so.0()(64bit)
librbh_mod_backup.so.0()(64bit) librbh_mod_basic.so.0()(64bit)
librbh_mod_checker.so.0()(64bit) librbh_mod_common.so.0()(64bit)
librbh_mod_lhsm.so.0()(64bit) librbh_mod_modeguard.so.0()(64bit)
librbh_mod_test.so.0()(64bit) libssl.so.10()(64bit) libz.so.1()(64bit)
rtld(GNU_HASH)
Requires: libc.so.6()(64bit) libc.so.6(GLIBC_2.14)(64bit)
libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit)
libglib-2.0.so.0()(64bit) libgthread-2.0.so.0()(64bit)
libjemalloc.so.1()(64bit) liblustreapi.so()(64bit) libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit) librt.so.1()(64bit) libz.so.1()(64bit)
rtld(GNU_HASH)
lorand@robinhood:~ $ rpm -qa |egrep "lustre|robinhood"
lustre-client-debuginfo-2.10.5-1.el7.x86_64
kmod-lustre-client-tests-2.10.5-1.el7.x86_64
robinhood-tools-3.1.4-1.lustre.el7.x86_64
lustre-iokit-2.10.5-1.el7.x86_64
robinhood-tests-3.1.4-1.x86_64
kmod-lustre-client-2.10.5-1.el7.x86_64
lustre-client-tests-2.10.5-1.el7.x86_64
robinhood-debuginfo-3.1.4-1.el7.x86_64
lustre-client-2.10.5-1.el7.x86_64
robinhood-lustre-3.1.4-1.lustre.el7.x86_64
robinhood-adm-3.1.4-1.x86_64
lorand@robinhood:~ $ ldd $(which robinhood)
linux-vdso.so.1 => (0x00007ffc591f0000)
libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18
(0x00007f82f3278000)
libm.so.6 => /lib64/libm.so.6 (0x00007f82f2f76000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f82f2d72000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007f82f2b00000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f82f269f000)
liblustreapi.so => /lib/liblustreapi.so (0x00007f82f2485000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f82f2269000)
libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007f82f2067000)
libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f82f1d53000)
libjemalloc.so.1 => /lib64/libjemalloc.so.1 (0x00007f82f1b1f000)
libz.so.1 => /lib64/libz.so.1 (0x00007f82f1909000)
libc.so.6 => /lib64/libc.so.6 (0x00007f82f153c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f82f3778000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f82f1235000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f82f0fe8000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f82f0d00000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f82f0afc000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f82f08c9000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f82f0667000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f82f0451000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f82f0243000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f82f003f000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f82efe26000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f82efbff000)
lorand@robinhood:~ $ md5sum /lib/liblustreapi.so
c58640c0bfe0c7551e917db122f2f42d /lib/liblustreapi.so
lorand@robinhood:~ $ md5sum /usr/lib64/liblustreapi.so
1e649f4b0a541a8f62c1bf0c0402ae7f /usr/lib64/liblustreapi.so
lorand@robinhood:~ $ rpm -qf /usr/lib64/liblustreapi.so
lustre-client-2.10.5-1.el7.x86_64
lorand@robinhood:~ $ ls -la /lib/liblustreapi.so
-rw-r--r-- 1 root root 412152 Sep 18 2017 /lib/liblustreapi.so
[root@robinhood x86_64]# mv /lib/liblustreapi.so{,.dis}
-------------------------------------------------------------------------------
Thank you!
Lorand
> On 29 Nov 2018, at 11:46, [email protected] wrote:
>
> Hi again,
>
> I am pretty sure this is it though. I remember having to deal with it while
> working on lustre.
>
> > I initially installed robinhood from rpms, but now fetched the sources and
> > freshly compiled, built new rpms.
>
> And are you _sure_ lustre 2.10 was installed on the node you compiled/built
> robinhood's rpms?
>
> Quentin
>
> Le 29/11/2018 à 10:51, Lorand Szentannai a écrit :
>> Hi Quentin,
>>
>> I initially installed robinhood from rpms, but now fetched the sources and
>> freshly compiled, built new rpms.
>> Unfortunately the behavior is still the same.
>>
>>
>> Thanks,
>> Lorand
>>
>>> On 28 Nov 2018, at 15:46, [email protected] wrote:
>>>
>>> Hi Lorand,
>>>
>>> I seem to remember this is caused by an update of lustre's changelog API.
>>>
>>> Did compile robinhood against the right version of liblustreapi? (there is
>>> no need for ioctl in Lustre 2.10's API)
>>>
>>> Quentin
>>>
>>> Le 28/11/2018 à 15:04, Lorand Szentannai a écrit :
>>>> Hi Thomas,
>>>>
>>>> I actually did that already but forgot to add the strace lines to the
>>>> initial email. But it is still unclear to me why open function is called
>>>> with the O_RDONLY flag.
>>>>
>>>> [pid 2348817] open("/cluster", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 12
>>>> [pid 2348817] ioctl(12, _IOC(_IOC_WRITE, 0x66, 0x94, 0x08),
>>>> 0x7ffef3f20240) = -1 ENOTTY (Inappropriate ioctl for device)
>>>>
>>>> # grep lustre /proc/mounts
>>>> 10.3.5.18@o2ib,10.4.5.18@o2ib1:10.3.5.19@o2ib,10.4.5.19@o2ib1:/scratch
>>>> /cluster lustre rw,flock,lazystatfs 0 0
>>>>
>>>>
>>>> Thanks,
>>>> Lorand Szentannai
>>>>
>>>>> On 28 Nov 2018, at 12:47, LEIBOVICI Thomas <[email protected]>
>>>>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> If you run "strace -f robinhood --readlog" you should see the ioctl that
>>>>> causes this error, and its arguments.
>>>>>
>>>>> Thomas
>>>>>
>>>>> On 11/28/18 07:36, Lorand Szentannai wrote:
>>>>>> Hi,
>>>>>>
>>>>>> We've recently upgraded to Lustre 2.10.5 and to Robinhood to 3.1.4.
>>>>>> Processing changelogs fails now with "ERROR -25 opening changelog for
>>>>>> MDT 'scratch-MDT0000': Inappropriate ioctl for device".
>>>>>>
>>>>>> Could you please advice on how to solve this? Note that currently MGS
>>>>>> and MDS are running on two separate servers.
>>>>>>
>>>>>> ----------------------------------------------------------------------------------
>>>>>> ChangeLog {
>>>>>> MDT {
>>>>>> mdt_name = "MDT0000";
>>>>>> reader_id = "cl4"; # yes, it is cl4
>>>>>> }
>>>>>> force_polling = ON ;
>>>>>> polling_interval = 60s ;
>>>>>> }
>>>>>> ----------------------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------------------
>>>>>> # lfs changelog scratch-MDT0000 |head
>>>>>>
>>>>>> 8211574766 10OPEN 22:24:39.830737089 2018.11.27 0x43
>>>>>> t=[0x200033cc4:0x4e9d:0x0]
>>>>>> 8211574767 13TRUNC 22:24:39.831280872 2018.11.27 0xe
>>>>>> t=[0x200033cc4:0x4e9d:0x0]
>>>>>> 8211574768 11CLOSE 22:24:39.831774075 2018.11.27 0x43
>>>>>> t=[0x200033cc4:0x4e9d:0x0]
>>>>>> 8211574769 10OPEN 22:24:39.831950980 2018.11.27 0x43
>>>>>> t=[0x200033cc4:0x4e9e:0x0]
>>>>>> 8211574770 13TRUNC 22:24:39.832289331 2018.11.27 0xe
>>>>>> t=[0x200033cc4:0x4e9e:0x0]
>>>>>> 8211574771 11CLOSE 22:24:39.832765269 2018.11.27 0x43
>>>>>> t=[0x200033cc4:0x4e9e:0x0]
>>>>>> 8211574772 10OPEN 22:24:39.832942039 2018.11.27 0x43
>>>>>> t=[0x200033cc4:0x4e9f:0x0]
>>>>>> 8211574773 13TRUNC 22:24:39.833241628 2018.11.27 0xe
>>>>>> t=[0x200033cc4:0x4e9f:0x0]
>>>>>> 8211574774 11CLOSE 22:24:39.833541406 2018.11.27 0x43
>>>>>> t=[0x200033cc4:0x4e9f:0x0]
>>>>>> 8211574775 10OPEN 22:24:39.833672445 2018.11.27 0x43
>>>>>> t=[0x200033cc4:0x4ea0:0x0]
>>>>>> ----------------------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------------------
>>>>>> # robinhood --readlog
>>>>>>
>>>>>> 2018/11/28 07:20:54 [2070664/1] Main | EntryProcessor successfully
>>>>>> initialized
>>>>>> 2018/11/28 07:20:54 [2070664/1] ChangeLog | mdt[0] = MDT0000
>>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | Logged on to database
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SET SESSION
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/3] ListMgr | Logged on to database
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/3] ListMgr | SQL query: SET SESSION
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/5] ListMgr | Logged on to database
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/5] ListMgr | SQL query: SET SESSION
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/6] ListMgr | Logged on to database
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/6] ListMgr | SQL query: SET SESSION
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/4] ListMgr | Logged on to database
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/4] ListMgr | SQL query: SET SESSION
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SELECT value FROM
>>>>>> VARS WHERE varname='CL_LastCommit_MDT0000'
>>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SELECT value FROM
>>>>>> VARS WHERE varname='ChangelogLastCommit_MDT0000'
>>>>>> 2018/11/28 07:20:54 [2070664/1] ChangeLog | Opening chglog for
>>>>>> scratch-MDT0000 (start_rec=0)
>>>>>> 2018/11/28 07:20:54 [2070664/1] ChangeLog | ERROR -25 opening changelog
>>>>>> for MDT 'scratch-MDT0000': Inappropriate ioctl for device
>>>>>> 2018/11/28 07:20:54 [2070664/1] Main | Error 25 initializing ChangeLog
>>>>>> Reader
>>>>>> ----------------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Lorand Szentannai
>>>>>>
>>>>>> _______________________________________________
>>>>>> robinhood-support mailing list
>>>>>> [email protected]
>>>>>> https://lists.sourceforge.net/lists/listinfo/robinhood-support
>>>>
>>>> _______________________________________________
>>>> robinhood-support mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/robinhood-support
>>>>
>>
>
_______________________________________________
robinhood-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/robinhood-support
