Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Thanks for your detailed response. I freshly installed a CentOS 7.6 and run install-deps.sh and do_cmake.sh this time, and it works this time. Maybe the problem was caused by dirty environment. Best, Can Zhang On Fri, Apr 19, 2019 at 6:28 PM Brad Hubbard wrote: > > OK. So this works for me with master commit > bdaac2d619d603f53a16c07f9d7bd47751137c4c on Centos 7.5.1804. > > I cloned the repo and ran './install-deps.sh' and './do_cmake.sh > -DWITH_FIO=ON' then 'make all'. > > # find ./lib -iname '*.so*' | xargs nm -AD 2>&1 | grep > _ZTIN13PriorityCache8PriCacheE > ./lib/libfio_ceph_objectstore.so:018f72d0 V > _ZTIN13PriorityCache8PriCacheE > > # LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > conf: Path to a ceph configuration file > oi_attr_len : Set OI(aka '_') attribute to specified length > snapset_attr_len: Set 'snapset' attribute to specified length > _fastinfo_omap_len : Set '_fastinfo' OMAP attribute to specified length > pglog_simulation: Enables PG Log simulation behavior > pglog_omap_len : Set pglog omap entry to specified length > pglog_dup_omap_len : Set duplicate pglog omap entry to specified length > single_pool_mode: Enables the mode when all jobs run against > the same pool > preallocate_files : Enables/disables file preallocation (touch > and resize) on init > > So my result above matches your result on ubuntu but not on centos. It > looks to me like we used to define in libceph-common but currently > it's defined in libfio_ceph_objectstore.so. For reasons that are > unclear you are seeing the old behaviour. Why this is and why it isn't > working as designed is not clear to me but I suspect if you clone the > repo again and build from scratch (maybe in a different directory if > you wish to keep debugging, see below) you should get a working > result. Could you try that as a test? > > If, on the other hand, you wish to keep debugging your current > environment I'd suggest looking at the output of the following command > as it may shed further light on the issue. > > # LD_DEBUG=all LD_LIBRARY_PATH=./lib ./bin/fio > --enghelp=libfio_ceph_objectstore.so > > 'LD_DEBUG=lib' may suffice but that's difficult to judge without > knowing what the problem is. I still suspect somehow you have > mis-matched libraries and, if that's the case, it's probably not worth > pursuing. If you can give me specific steps so I can reproduce this > from a freshly cloned tree I'd be happy to look further into it. > > Good luck. > > On Thu, Apr 18, 2019 at 7:00 PM Brad Hubbard wrote: > > > > Let me try to reproduce this on centos 7.5 with master and I'll let > > you know how I go. > > > > On Thu, Apr 18, 2019 at 3:59 PM Can Zhang wrote: > > > > > > Using the commands you provided, I actually find some differences: > > > > > > On my CentOS VM: > > > ``` > > > # sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > > _ZTIN13PriorityCache8PriCacheE > > > ./libceph-common.so:0221cc08 V _ZTIN13PriorityCache8PriCacheE > > > ./libceph-common.so.0:0221cc08 V _ZTIN13PriorityCache8PriCacheE > > > ./libfio_ceph_objectstore.so: U > > > _ZTIN13PriorityCache8PriCacheE > > > ``` > > > ``` > > > # ldd libfio_ceph_objectstore.so |grep common > > > libceph-common.so.0 => /root/ceph/build/lib/libceph-common.so.0 > > > (0x7fd13f3e7000) > > > ``` > > > On my Ubuntu VM: > > > ``` > > > $ sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > > _ZTIN13PriorityCache8PriCacheE > > > ./libfio_ceph_objectstore.so:019d13e0 V > > > _ZTIN13PriorityCache8PriCacheE > > > ``` > > > ``` > > > $ ldd libfio_ceph_objectstore.so |grep common > > > libceph-common.so.0 => > > > /home/can/work/ceph/build/lib/libceph-common.so.0 (0x7f024a89e000) > > > ``` > > > > > > Notice the "U" and "V" from nm results. > > > > > > > > > > > > > > > Best, > > > Can Zhang > > > > > > On Thu, Apr 18, 2019 at 9:36 AM Brad Hubbard wrote: > > > > > > > > Does it define _ZTIN13PriorityCache8PriCacheE ? If it does, and all is > > > > as you say, then it should not say that _ZTIN13PriorityCache8PriCacheE > > > > is undefined. Does ldd show that it is finding the libraries you think > > > > it is? Either it is finding a different version of that library
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Using the commands you provided, I actually find some differences: On my CentOS VM: ``` # sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep _ZTIN13PriorityCache8PriCacheE ./libceph-common.so:0221cc08 V _ZTIN13PriorityCache8PriCacheE ./libceph-common.so.0:0221cc08 V _ZTIN13PriorityCache8PriCacheE ./libfio_ceph_objectstore.so: U _ZTIN13PriorityCache8PriCacheE ``` ``` # ldd libfio_ceph_objectstore.so |grep common libceph-common.so.0 => /root/ceph/build/lib/libceph-common.so.0 (0x7fd13f3e7000) ``` On my Ubuntu VM: ``` $ sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep _ZTIN13PriorityCache8PriCacheE ./libfio_ceph_objectstore.so:019d13e0 V _ZTIN13PriorityCache8PriCacheE ``` ``` $ ldd libfio_ceph_objectstore.so |grep common libceph-common.so.0 => /home/can/work/ceph/build/lib/libceph-common.so.0 (0x7f024a89e000) ``` Notice the "U" and "V" from nm results. Best, Can Zhang On Thu, Apr 18, 2019 at 9:36 AM Brad Hubbard wrote: > > Does it define _ZTIN13PriorityCache8PriCacheE ? If it does, and all is > as you say, then it should not say that _ZTIN13PriorityCache8PriCacheE > is undefined. Does ldd show that it is finding the libraries you think > it is? Either it is finding a different version of that library > somewhere else or the version you have may not define that symbol. > > On Thu, Apr 18, 2019 at 11:12 AM Can Zhang wrote: > > > > It's already in LD_LIBRARY_PATH, under the same directory of > > libfio_ceph_objectstore.so > > > > > > $ ll lib/|grep libceph-common > > lrwxrwxrwx. 1 root root19 Apr 17 11:15 libceph-common.so -> > > libceph-common.so.0 > > -rwxr-xr-x. 1 root root 211853400 Apr 17 11:15 libceph-common.so.0 > > > > > > > > > > Best, > > Can Zhang > > > > On Thu, Apr 18, 2019 at 7:00 AM Brad Hubbard wrote: > > > > > > On Wed, Apr 17, 2019 at 1:37 PM Can Zhang wrote: > > > > > > > > Thanks for your suggestions. > > > > > > > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > > > > > > > ``` > > > > $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > > > > > > > fio: engine libfio_ceph_objectstore.so not loadable > > > > IO engine libfio_ceph_objectstore.so not found > > > > ``` > > > > > > > > I managed to print the dlopen error, it said: > > > > > > > > ``` > > > > dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > > > > _ZTIN13PriorityCache8PriCacheE > > > > > > $ c++filt _ZTIN13PriorityCache8PriCacheE > > > typeinfo for PriorityCache::PriCache > > > > > > $ sudo find /lib* /usr/lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > > _ZTIN13PriorityCache8PriCacheE > > > /usr/lib64/ceph/libceph-common.so:008edab0 V > > > _ZTIN13PriorityCache8PriCacheE > > > /usr/lib64/ceph/libceph-common.so.0:008edab0 V > > > _ZTIN13PriorityCache8PriCacheE > > > > > > It needs to be able to find libceph-common, put it in your path or > > > preload it. > > > > > > > ``` > > > > > > > > I found a not-so-relevant > > > > issue(https://tracker.ceph.com/issues/38360), the error seems to be > > > > caused by mixed versions. My build environment is CentOS 7.5.1804 with > > > > SCL devtoolset-7, and ceph is latest master branch. Does someone know > > > > about the symbol? > > > > > > > > > > > > Best, > > > > Can Zhang > > > > > > > > Best, > > > > Can Zhang > > > > > > > > > > > > On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > > > > > > > > > Besides already mentioned store_test.cc one can also use ceph > > > > > objectstore fio plugin > > > > > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > > > > > standalone BlueStore instance from FIO benchmarking tool. > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Igor > > > > > > > > > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > > > > > Hi, > > > > > > > > > > > > I'd like to run a standalone Bluestore instance so as to test and > > > > > > tune > > > > > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > Can Zhang > > > > > > > > > > > > ___ > > > > > > ceph-users mailing list > > > > > > ceph-users@lists.ceph.com > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > ___ > > > > ceph-users mailing list > > > > ceph-users@lists.ceph.com > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > -- > > > Cheers, > > > Brad > > > > -- > Cheers, > Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
It's already in LD_LIBRARY_PATH, under the same directory of libfio_ceph_objectstore.so $ ll lib/|grep libceph-common lrwxrwxrwx. 1 root root19 Apr 17 11:15 libceph-common.so -> libceph-common.so.0 -rwxr-xr-x. 1 root root 211853400 Apr 17 11:15 libceph-common.so.0 Best, Can Zhang On Thu, Apr 18, 2019 at 7:00 AM Brad Hubbard wrote: > > On Wed, Apr 17, 2019 at 1:37 PM Can Zhang wrote: > > > > Thanks for your suggestions. > > > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > > > ``` > > $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > > > fio: engine libfio_ceph_objectstore.so not loadable > > IO engine libfio_ceph_objectstore.so not found > > ``` > > > > I managed to print the dlopen error, it said: > > > > ``` > > dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > > _ZTIN13PriorityCache8PriCacheE > > $ c++filt _ZTIN13PriorityCache8PriCacheE > typeinfo for PriorityCache::PriCache > > $ sudo find /lib* /usr/lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > _ZTIN13PriorityCache8PriCacheE > /usr/lib64/ceph/libceph-common.so:008edab0 V > _ZTIN13PriorityCache8PriCacheE > /usr/lib64/ceph/libceph-common.so.0:008edab0 V > _ZTIN13PriorityCache8PriCacheE > > It needs to be able to find libceph-common, put it in your path or preload it. > > > ``` > > > > I found a not-so-relevant > > issue(https://tracker.ceph.com/issues/38360), the error seems to be > > caused by mixed versions. My build environment is CentOS 7.5.1804 with > > SCL devtoolset-7, and ceph is latest master branch. Does someone know > > about the symbol? > > > > > > Best, > > Can Zhang > > > > Best, > > Can Zhang > > > > > > On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > > > > > Besides already mentioned store_test.cc one can also use ceph > > > objectstore fio plugin > > > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > > > standalone BlueStore instance from FIO benchmarking tool. > > > > > > > > > Thanks, > > > > > > Igor > > > > > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > > > Hi, > > > > > > > > I'd like to run a standalone Bluestore instance so as to test and tune > > > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > > > > > > > > > Best, > > > > Can Zhang > > > > > > > > ___ > > > > ceph-users mailing list > > > > ceph-users@lists.ceph.com > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Cheers, > Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
I fully rebuilt libfio_ceph_objectstore file on my Ubuntu VM. Best, Can Zhang On Wed, Apr 17, 2019 at 10:39 PM Igor Fedotov wrote: > > Or try full rebuild? > > On 4/17/2019 5:37 PM, Igor Fedotov wrote: > > Could you please check if libfio_ceph_objectstore.so has been rebuilt > > with your last build? > > > > On 4/17/2019 6:37 AM, Can Zhang wrote: > >> Thanks for your suggestions. > >> > >> I tried to build libfio_ceph_objectstore.so, but it fails to load: > >> > >> ``` > >> $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > >> > >> fio: engine libfio_ceph_objectstore.so not loadable > >> IO engine libfio_ceph_objectstore.so not found > >> ``` > >> > >> I managed to print the dlopen error, it said: > >> > >> ``` > >> dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > >> _ZTIN13PriorityCache8PriCacheE > >> ``` > >> > >> I found a not-so-relevant > >> issue(https://tracker.ceph.com/issues/38360), the error seems to be > >> caused by mixed versions. My build environment is CentOS 7.5.1804 with > >> SCL devtoolset-7, and ceph is latest master branch. Does someone know > >> about the symbol? > >> > >> > >> Best, > >> Can Zhang > >> > >> Best, > >> Can Zhang > >> > >> > >> On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > >>> Besides already mentioned store_test.cc one can also use ceph > >>> objectstore fio plugin > >>> (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > >>> standalone BlueStore instance from FIO benchmarking tool. > >>> > >>> > >>> Thanks, > >>> > >>> Igor > >>> > >>> On 4/16/2019 7:58 AM, Can ZHANG wrote: > >>>> Hi, > >>>> > >>>> I'd like to run a standalone Bluestore instance so as to test and tune > >>>> its performance. Are there any tools about it, or any suggestions? > >>>> > >>>> > >>>> > >>>> Best, > >>>> Can Zhang > >>>> > >>>> ___ > >>>> ceph-users mailing list > >>>> ceph-users@lists.ceph.com > >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Then I tried to build libfio_ceph_objectstore.so on a Ubuntu 18.04 vm, it seems to be working now. Best, Can Zhang On Wed, Apr 17, 2019 at 11:37 AM Can Zhang wrote: > > Thanks for your suggestions. > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > ``` > $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > fio: engine libfio_ceph_objectstore.so not loadable > IO engine libfio_ceph_objectstore.so not found > ``` > > I managed to print the dlopen error, it said: > > ``` > dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > _ZTIN13PriorityCache8PriCacheE > ``` > > I found a not-so-relevant > issue(https://tracker.ceph.com/issues/38360), the error seems to be > caused by mixed versions. My build environment is CentOS 7.5.1804 with > SCL devtoolset-7, and ceph is latest master branch. Does someone know > about the symbol? > > > Best, > Can Zhang > > Best, > Can Zhang > > > On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > > > Besides already mentioned store_test.cc one can also use ceph > > objectstore fio plugin > > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > > standalone BlueStore instance from FIO benchmarking tool. > > > > > > Thanks, > > > > Igor > > > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > > Hi, > > > > > > I'd like to run a standalone Bluestore instance so as to test and tune > > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > > > > > Best, > > > Can Zhang > > > > > > ___ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Thanks for your suggestions. I tried to build libfio_ceph_objectstore.so, but it fails to load: ``` $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so fio: engine libfio_ceph_objectstore.so not loadable IO engine libfio_ceph_objectstore.so not found ``` I managed to print the dlopen error, it said: ``` dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: _ZTIN13PriorityCache8PriCacheE ``` I found a not-so-relevant issue(https://tracker.ceph.com/issues/38360), the error seems to be caused by mixed versions. My build environment is CentOS 7.5.1804 with SCL devtoolset-7, and ceph is latest master branch. Does someone know about the symbol? Best, Can Zhang Best, Can Zhang On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > Besides already mentioned store_test.cc one can also use ceph > objectstore fio plugin > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > standalone BlueStore instance from FIO benchmarking tool. > > > Thanks, > > Igor > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > Hi, > > > > I'd like to run a standalone Bluestore instance so as to test and tune > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > Best, > > Can Zhang > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Is it possible to run a standalone Bluestore instance?
Hi, I'd like to run a standalone Bluestore instance so as to test and tune its performance. Are there any tools about it, or any suggestions? Best, Can Zhang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
