Re: [ceph-users] how to build libradosstriper
Hi Guys, The striping seems to be slightly better than non-striping write, given that my storage is configured with 4OSS, and 48OSD. each OSD is 8+2 Raid 6 of 24TB capacity. But still the performance is around 100MB/sec. On a single haswell core, I'm able to get 1GB/sec with dd buffered IO. My questions are: 1. Does librados use direct IO? 2. Is there option to leverage page cache? 3. What is the strategy to get the peak bandwidth on a rados object store? 4. Can I use MPI for parallel IO? Best, Jialin NERSC On Wed, May 30, 2018 at 11:08 AM, Jialin Liu wrote: > Thanks Kefu. > > Best, > Jialin > NERSC > > On Tue, May 29, 2018 at 11:52 PM, kefu chai wrote: > >> On Wed, May 30, 2018 at 11:53 AM, Jialin Liu wrote: >> > Hi Brad, >> > >> > You are correct. the librados.so has the symbol but what I copied is a >> wrong >> > file. >> > Now I can test the striper api with the previous C example and this cpp >> > example: >> > http://mrbojangles3.github.io/ceph/systems/striping/alignmen >> t/2017/05/28/Ceph-Stripe/ >> > Both are working, but I haven't got any performance number yet. >> > ps. I also found that the stripe unit must be set before object_size >> > properly. >> > >> > Thanks much. >> > >> > Best, >> > Jialin >> > NERSC >> > >> > >> > On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard >> wrote: >> >> >> >> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu wrote: >> >> > Thanks Brad, >> >> > I run nm on those .so file, it prints 'no symbol' >> >> >> >> OK, well you need to link to a library that exports that symbol (has >> >> it defined in its Text section). I suspect you'll find it is defined >> >> in libceph-common.so so try linking to that explicitly. >> >> please note, libceph-common does not exist in jewel (i.e. v10.2.10) >> yet. it was introduce in luminous. and libceph-common is not supposed >> to be linked against by rados application directly. it is an internal >> library and is linked by librados, libradosstriper and other rados >> tools/daemons. >> >> >> >> >> > >> >> > Then with ldd librados.so, I don't see the libceph-common.so, >> instead: >> >> >> >> >> >> jialin@cori12: ldd librados.so >> >> >> linux-vdso.so.1 (0x2aacf000) >> >> >> libboost_thread-mt.so.1.53.0 >> >> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0 >> (0x2d24d000) >> >> >> libboost_system-mt.so.1.53.0 => >> >> >> /rados_install/lib/libboost_system-mt.so.1.53.0 >> (0x2d464000) >> >> >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000) >> >> >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000) >> >> >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000) >> >> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000) >> >> >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000) >> >> >> libplc4.so => /usr/lib64/libplc4.so (0x2e23) >> >> >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000) >> >> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000) >> >> >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89) >> >> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000) >> >> >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000) >> >> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000) >> >> >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000) >> >> >> libc.so.6 => /lib64/libc.so.6 (0x2f527000) >> >> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000) >> >> >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000) >> >> >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000) >> >> >> >> Things were obviously different in jewel but the concept remains the >> >> same. For each external symbol your binary requires you need to link >> >> to a library that exports that symbol so finding the library that >> >> exports the symbols in the error message should resolve the error. >> >> >> >> > >> >> > >> >> > I also found this thread: http://tracker.ceph.com/issues/14788 >> >> > which looks similar to the error I run into, and that thread >> mentioned >> >> > the >> >> > version between the headers file and the .so file, I think in my case >> >> > both >> >> > of them are v10.2.10, which should ok. >> >> > >> >> > Best, >> >> > Jialin, >> >> > >> >> > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard >> >> > wrote: >> >> >> >> >> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu >> wrote: >> >> >> > Hi, >> >> >> > I'm trying to use the libradosstriper api, but having some >> trouble in >> >> >> > linking to lradosstriper. I copied only the `required' libraries >> from >> >> >> > an >> >> >> > pre-installed ceph (10.2.10), and put them under my local >> directory >> >> >> > /rados_install/lib and rados_install/include, on a linux machine. >> >> >> > >> >> >> > /rados_install/lib: >> >> >> >> >> >> >> >> libboost_date_time.so.1.53.0 libboost_system.so.1.53.0 >> >> >> >> libboost_filesystem-mt.so.1.53.0 libboost_thread-mt.so.1.53.0 >> >> >> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0
Re: [ceph-users] how to build libradosstriper
Thanks Kefu. Best, Jialin NERSC On Tue, May 29, 2018 at 11:52 PM, kefu chai wrote: > On Wed, May 30, 2018 at 11:53 AM, Jialin Liu wrote: > > Hi Brad, > > > > You are correct. the librados.so has the symbol but what I copied is a > wrong > > file. > > Now I can test the striper api with the previous C example and this cpp > > example: > > http://mrbojangles3.github.io/ceph/systems/striping/ > alignment/2017/05/28/Ceph-Stripe/ > > Both are working, but I haven't got any performance number yet. > > ps. I also found that the stripe unit must be set before object_size > > properly. > > > > Thanks much. > > > > Best, > > Jialin > > NERSC > > > > > > On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard > wrote: > >> > >> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu wrote: > >> > Thanks Brad, > >> > I run nm on those .so file, it prints 'no symbol' > >> > >> OK, well you need to link to a library that exports that symbol (has > >> it defined in its Text section). I suspect you'll find it is defined > >> in libceph-common.so so try linking to that explicitly. > > please note, libceph-common does not exist in jewel (i.e. v10.2.10) > yet. it was introduce in luminous. and libceph-common is not supposed > to be linked against by rados application directly. it is an internal > library and is linked by librados, libradosstriper and other rados > tools/daemons. > > >> > >> > > >> > Then with ldd librados.so, I don't see the libceph-common.so, instead: > >> >> > >> >> jialin@cori12: ldd librados.so > >> >> linux-vdso.so.1 (0x2aacf000) > >> >> libboost_thread-mt.so.1.53.0 > >> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0 > (0x2d24d000) > >> >> libboost_system-mt.so.1.53.0 => > >> >> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000) > >> >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000) > >> >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000) > >> >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000) > >> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000) > >> >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000) > >> >> libplc4.so => /usr/lib64/libplc4.so (0x2e23) > >> >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000) > >> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000) > >> >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89) > >> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000) > >> >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000) > >> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000) > >> >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000) > >> >> libc.so.6 => /lib64/libc.so.6 (0x2f527000) > >> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000) > >> >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000) > >> >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000) > >> > >> Things were obviously different in jewel but the concept remains the > >> same. For each external symbol your binary requires you need to link > >> to a library that exports that symbol so finding the library that > >> exports the symbols in the error message should resolve the error. > >> > >> > > >> > > >> > I also found this thread: http://tracker.ceph.com/issues/14788 > >> > which looks similar to the error I run into, and that thread mentioned > >> > the > >> > version between the headers file and the .so file, I think in my case > >> > both > >> > of them are v10.2.10, which should ok. > >> > > >> > Best, > >> > Jialin, > >> > > >> > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard > >> > wrote: > >> >> > >> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu > wrote: > >> >> > Hi, > >> >> > I'm trying to use the libradosstriper api, but having some trouble > in > >> >> > linking to lradosstriper. I copied only the `required' libraries > from > >> >> > an > >> >> > pre-installed ceph (10.2.10), and put them under my local directory > >> >> > /rados_install/lib and rados_install/include, on a linux machine. > >> >> > > >> >> > /rados_install/lib: > >> >> >> > >> >> >> libboost_date_time.so.1.53.0 libboost_system.so.1.53.0 > >> >> >> libboost_filesystem-mt.so.1.53.0 libboost_thread-mt.so.1.53.0 > >> >> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0 > >> >> >> libboost_iostreams-mt.so.1.53.0 > >> >> >> libboost_iostreams.so.1.53.0 > >> >> >> libboost_random-mt.so.1.53.0 librados.so.2.0.0 > >> >> >> libboost_regex.so.1.53.0 librados_tp.so.2.0.0 > >> >> >> libboost_system-mt.so.1.53.0 libradosstriper.so.1.0.0 > >> >> >> libcephfs.so librados.so libradosstriper.so > >> >> > > >> >> > > >> >> > /rados_install/include: > >> >> >> > >> >> >> |-- rados > >> >> >> | |-- buffer.h -> ../buffer.h > >> >> >> | |-- buffer_fwd.h -> ../buffer_fwd.h > >> >> >> | |-- crc32c.h -> ../crc32c.h > >> >> >> | |-- inline_memory.h -> ../inline_memory.h > >> >> >> | |-- librados.h > >> >> >> | |-- librados.hpp > >> >> >> | |--
Re: [ceph-users] how to build libradosstriper
On Wed, May 30, 2018 at 11:53 AM, Jialin Liu wrote: > Hi Brad, > > You are correct. the librados.so has the symbol but what I copied is a wrong > file. > Now I can test the striper api with the previous C example and this cpp > example: > http://mrbojangles3.github.io/ceph/systems/striping/alignment/2017/05/28/Ceph-Stripe/ > Both are working, but I haven't got any performance number yet. > ps. I also found that the stripe unit must be set before object_size > properly. > > Thanks much. > > Best, > Jialin > NERSC > > > On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard wrote: >> >> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu wrote: >> > Thanks Brad, >> > I run nm on those .so file, it prints 'no symbol' >> >> OK, well you need to link to a library that exports that symbol (has >> it defined in its Text section). I suspect you'll find it is defined >> in libceph-common.so so try linking to that explicitly. please note, libceph-common does not exist in jewel (i.e. v10.2.10) yet. it was introduce in luminous. and libceph-common is not supposed to be linked against by rados application directly. it is an internal library and is linked by librados, libradosstriper and other rados tools/daemons. >> >> > >> > Then with ldd librados.so, I don't see the libceph-common.so, instead: >> >> >> >> jialin@cori12: ldd librados.so >> >> linux-vdso.so.1 (0x2aacf000) >> >> libboost_thread-mt.so.1.53.0 >> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0 (0x2d24d000) >> >> libboost_system-mt.so.1.53.0 => >> >> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000) >> >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000) >> >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000) >> >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000) >> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000) >> >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000) >> >> libplc4.so => /usr/lib64/libplc4.so (0x2e23) >> >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000) >> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000) >> >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89) >> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000) >> >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000) >> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000) >> >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000) >> >> libc.so.6 => /lib64/libc.so.6 (0x2f527000) >> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000) >> >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000) >> >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000) >> >> Things were obviously different in jewel but the concept remains the >> same. For each external symbol your binary requires you need to link >> to a library that exports that symbol so finding the library that >> exports the symbols in the error message should resolve the error. >> >> > >> > >> > I also found this thread: http://tracker.ceph.com/issues/14788 >> > which looks similar to the error I run into, and that thread mentioned >> > the >> > version between the headers file and the .so file, I think in my case >> > both >> > of them are v10.2.10, which should ok. >> > >> > Best, >> > Jialin, >> > >> > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard >> > wrote: >> >> >> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu wrote: >> >> > Hi, >> >> > I'm trying to use the libradosstriper api, but having some trouble in >> >> > linking to lradosstriper. I copied only the `required' libraries from >> >> > an >> >> > pre-installed ceph (10.2.10), and put them under my local directory >> >> > /rados_install/lib and rados_install/include, on a linux machine. >> >> > >> >> > /rados_install/lib: >> >> >> >> >> >> libboost_date_time.so.1.53.0 libboost_system.so.1.53.0 >> >> >> libboost_filesystem-mt.so.1.53.0 libboost_thread-mt.so.1.53.0 >> >> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0 >> >> >> libboost_iostreams-mt.so.1.53.0 >> >> >> libboost_iostreams.so.1.53.0 >> >> >> libboost_random-mt.so.1.53.0 librados.so.2.0.0 >> >> >> libboost_regex.so.1.53.0 librados_tp.so.2.0.0 >> >> >> libboost_system-mt.so.1.53.0 libradosstriper.so.1.0.0 >> >> >> libcephfs.so librados.so libradosstriper.so >> >> > >> >> > >> >> > /rados_install/include: >> >> >> >> >> >> |-- rados >> >> >> | |-- buffer.h -> ../buffer.h >> >> >> | |-- buffer_fwd.h -> ../buffer_fwd.h >> >> >> | |-- crc32c.h -> ../crc32c.h >> >> >> | |-- inline_memory.h -> ../inline_memory.h >> >> >> | |-- librados.h >> >> >> | |-- librados.hpp >> >> >> | |-- librgw.h >> >> >> | |-- memory.h -> ../memory.h >> >> >> | |-- objclass.h >> >> >> | |-- page.h -> ../page.h >> >> >> | |-- rados_types.h >> >> >> | |-- rados_types.hpp >> >> >> | `-- rgw_file.h >> >> >> `-- radosstriper >> >> >> |-- libradosstriper.h >> >> >> `-- libradosstriper.hpp >> >> > >> >> > >> >>
Re: [ceph-users] how to build libradosstriper
Hi Brad, You are correct. the librados.so has the symbol but what I copied is a wrong file. Now I can test the striper api with the previous C example and this cpp example: http://mrbojangles3.github.io/ceph/systems/striping/alignment/2017/05/28/Ceph-Stripe/ Both are working, but I haven't got any performance number yet. ps. I also found that the stripe unit must be set before object_size properly. Thanks much. Best, Jialin NERSC On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard wrote: > On Wed, May 30, 2018 at 11:52 AM, Jialin Liu wrote: > > Thanks Brad, > > I run nm on those .so file, it prints 'no symbol' > > OK, well you need to link to a library that exports that symbol (has > it defined in its Text section). I suspect you'll find it is defined > in libceph-common.so so try linking to that explicitly. > > > > > Then with ldd librados.so, I don't see the libceph-common.so, instead: > >> > >> jialin@cori12: ldd librados.so > >> linux-vdso.so.1 (0x2aacf000) > >> libboost_thread-mt.so.1.53.0 > >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0 (0x2d24d000) > >> libboost_system-mt.so.1.53.0 => > >> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000) > >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000) > >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000) > >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000) > >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000) > >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000) > >> libplc4.so => /usr/lib64/libplc4.so (0x2e23) > >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000) > >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000) > >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89) > >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000) > >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000) > >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000) > >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000) > >> libc.so.6 => /lib64/libc.so.6 (0x2f527000) > >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000) > >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000) > >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000) > > Things were obviously different in jewel but the concept remains the > same. For each external symbol your binary requires you need to link > to a library that exports that symbol so finding the library that > exports the symbols in the error message should resolve the error. > > > > > > > I also found this thread: http://tracker.ceph.com/issues/14788 > > which looks similar to the error I run into, and that thread mentioned > the > > version between the headers file and the .so file, I think in my case > both > > of them are v10.2.10, which should ok. > > > > Best, > > Jialin, > > > > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard > wrote: > >> > >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu wrote: > >> > Hi, > >> > I'm trying to use the libradosstriper api, but having some trouble in > >> > linking to lradosstriper. I copied only the `required' libraries from > an > >> > pre-installed ceph (10.2.10), and put them under my local directory > >> > /rados_install/lib and rados_install/include, on a linux machine. > >> > > >> > /rados_install/lib: > >> >> > >> >> libboost_date_time.so.1.53.0 libboost_system.so.1.53.0 > >> >> libboost_filesystem-mt.so.1.53.0 libboost_thread-mt.so.1.53.0 > >> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0 > >> >> libboost_iostreams-mt.so.1.53.0 > >> >> libboost_iostreams.so.1.53.0 > >> >> libboost_random-mt.so.1.53.0 librados.so.2.0.0 > >> >> libboost_regex.so.1.53.0 librados_tp.so.2.0.0 > >> >> libboost_system-mt.so.1.53.0 libradosstriper.so.1.0.0 > >> >> libcephfs.so librados.so libradosstriper.so > >> > > >> > > >> > /rados_install/include: > >> >> > >> >> |-- rados > >> >> | |-- buffer.h -> ../buffer.h > >> >> | |-- buffer_fwd.h -> ../buffer_fwd.h > >> >> | |-- crc32c.h -> ../crc32c.h > >> >> | |-- inline_memory.h -> ../inline_memory.h > >> >> | |-- librados.h > >> >> | |-- librados.hpp > >> >> | |-- librgw.h > >> >> | |-- memory.h -> ../memory.h > >> >> | |-- objclass.h > >> >> | |-- page.h -> ../page.h > >> >> | |-- rados_types.h > >> >> | |-- rados_types.hpp > >> >> | `-- rgw_file.h > >> >> `-- radosstriper > >> >> |-- libradosstriper.h > >> >> `-- libradosstriper.hpp > >> > > >> > > >> > The test code is simple, as attached, and I compiled it with dynamic > >> > linking: > >> >> > >> >> cc -L/rados_install/lib -I/rados_install/include -o librados_test > >> >> librados_test.c -lrados -lradosstriper > >> > > >> > > >> > But got the following message, can anyone plz advise? > >> > > >> >> > >> >> /rados_install/lib/libradosstriper.so: undefined reference to > >> >> `ceph::buffer::ptr::ptr(ceph::buffer::ptr&&)' > >> >> /rados_install/lib/libradosstriper.so: undefined
Re: [ceph-users] how to build libradosstriper
On Wed, May 30, 2018 at 11:52 AM, Jialin Liu wrote: > Thanks Brad, > I run nm on those .so file, it prints 'no symbol' OK, well you need to link to a library that exports that symbol (has it defined in its Text section). I suspect you'll find it is defined in libceph-common.so so try linking to that explicitly. > > Then with ldd librados.so, I don't see the libceph-common.so, instead: >> >> jialin@cori12: ldd librados.so >> linux-vdso.so.1 (0x2aacf000) >> libboost_thread-mt.so.1.53.0 >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0 (0x2d24d000) >> libboost_system-mt.so.1.53.0 => >> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000) >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000) >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000) >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000) >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000) >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000) >> libplc4.so => /usr/lib64/libplc4.so (0x2e23) >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000) >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89) >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000) >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000) >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000) >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000) >> libc.so.6 => /lib64/libc.so.6 (0x2f527000) >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000) >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000) >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000) Things were obviously different in jewel but the concept remains the same. For each external symbol your binary requires you need to link to a library that exports that symbol so finding the library that exports the symbols in the error message should resolve the error. > > > I also found this thread: http://tracker.ceph.com/issues/14788 > which looks similar to the error I run into, and that thread mentioned the > version between the headers file and the .so file, I think in my case both > of them are v10.2.10, which should ok. > > Best, > Jialin, > > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard wrote: >> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu wrote: >> > Hi, >> > I'm trying to use the libradosstriper api, but having some trouble in >> > linking to lradosstriper. I copied only the `required' libraries from an >> > pre-installed ceph (10.2.10), and put them under my local directory >> > /rados_install/lib and rados_install/include, on a linux machine. >> > >> > /rados_install/lib: >> >> >> >> libboost_date_time.so.1.53.0 libboost_system.so.1.53.0 >> >> libboost_filesystem-mt.so.1.53.0 libboost_thread-mt.so.1.53.0 >> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0 >> >> libboost_iostreams-mt.so.1.53.0 >> >> libboost_iostreams.so.1.53.0 >> >> libboost_random-mt.so.1.53.0 librados.so.2.0.0 >> >> libboost_regex.so.1.53.0 librados_tp.so.2.0.0 >> >> libboost_system-mt.so.1.53.0 libradosstriper.so.1.0.0 >> >> libcephfs.so librados.so libradosstriper.so >> > >> > >> > /rados_install/include: >> >> >> >> |-- rados >> >> | |-- buffer.h -> ../buffer.h >> >> | |-- buffer_fwd.h -> ../buffer_fwd.h >> >> | |-- crc32c.h -> ../crc32c.h >> >> | |-- inline_memory.h -> ../inline_memory.h >> >> | |-- librados.h >> >> | |-- librados.hpp >> >> | |-- librgw.h >> >> | |-- memory.h -> ../memory.h >> >> | |-- objclass.h >> >> | |-- page.h -> ../page.h >> >> | |-- rados_types.h >> >> | |-- rados_types.hpp >> >> | `-- rgw_file.h >> >> `-- radosstriper >> >> |-- libradosstriper.h >> >> `-- libradosstriper.hpp >> > >> > >> > The test code is simple, as attached, and I compiled it with dynamic >> > linking: >> >> >> >> cc -L/rados_install/lib -I/rados_install/include -o librados_test >> >> librados_test.c -lrados -lradosstriper >> > >> > >> > But got the following message, can anyone plz advise? >> > >> >> >> >> /rados_install/lib/libradosstriper.so: undefined reference to >> >> `ceph::buffer::ptr::ptr(ceph::buffer::ptr&&)' >> >> /rados_install/lib/libradosstriper.so: undefined reference to >> >> `ceph::buffer::ptr::operator=(ceph::buffer::ptr&&)' >> >> /rados_install/lib/libradosstriper.so: undefined reference to >> >> `librados::IoCtx::remove(std::string const&, int)' >> >> /rados_install/lib/libradosstriper.so: undefined reference to >> >> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned >> >> int)' >> >> /rados_install/lib/libradosstriper.so: undefined reference to >> >> `ceph::buffer::list::is_provided_buffer(char const*) const' >> >> /rados_install/lib/libradosstriper.so: undefined reference to >> >> `ceph::buffer::list::append(ceph::buffer::ptr&&)' >> >> /rados_install/lib/libradosstriper.so: undefined reference to >> >>
Re: [ceph-users] how to build libradosstriper
Thanks Brad, I run nm on those .so file, it prints 'no symbol' Then with ldd librados.so, I don't see the libceph-common.so, instead: > jialin@cori12: ldd librados.so > linux-vdso.so.1 (0x2aacf000) > libboost_thread-mt.so.1.53.0 > =>/rados_install/lib/libboost_thread-mt.so.1.53.0 (0x2d24d000) > libboost_system-mt.so.1.53.0 => > /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000) > libssl3.so => /usr/lib64/libssl3.so (0x2d668000) > libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000) > libnss3.so => /usr/lib64/libnss3.so (0x2dadb000) > libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000) > libplds4.so => /usr/lib64/libplds4.so (0x2e02c000) > libplc4.so => /usr/lib64/libplc4.so (0x2e23) > libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000) > libdl.so.2 => /lib64/libdl.so.2 (0x2e89) > libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000) > librt.so.1 => /lib64/librt.so.1 (0x2ec99000) > libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000) > libm.so.6 => /lib64/libm.so.6 (0x2f22a000) > libc.so.6 => /lib64/libc.so.6 (0x2f527000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000) > libz.so.1 => /lib64/libz.so.1 (0x2fadf000) > /lib64/ld-linux-x86-64.so.2 (0x2aaab000) I also found this thread: http://tracker.ceph.com/issues/14788 which looks similar to the error I run into, and that thread mentioned the version between the headers file and the .so file, I think in my case both of them are v10.2.10, which should ok. Best, Jialin, On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard wrote: > On Wed, May 30, 2018 at 10:42 AM, Jialin Liu wrote: > > Hi, > > I'm trying to use the libradosstriper api, but having some trouble in > > linking to lradosstriper. I copied only the `required' libraries from an > > pre-installed ceph (10.2.10), and put them under my local directory > > /rados_install/lib and rados_install/include, on a linux machine. > > > > /rados_install/lib: > >> > >> libboost_date_time.so.1.53.0 libboost_system.so.1.53.0 > >> libboost_filesystem-mt.so.1.53.0 libboost_thread-mt.so.1.53.0 > >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0 > >> libboost_iostreams-mt.so.1.53.0 > >> libboost_iostreams.so.1.53.0 > >> libboost_random-mt.so.1.53.0 librados.so.2.0.0 > >> libboost_regex.so.1.53.0 librados_tp.so.2.0.0 > >> libboost_system-mt.so.1.53.0 libradosstriper.so.1.0.0 > >> libcephfs.so librados.so libradosstriper.so > > > > > > /rados_install/include: > >> > >> |-- rados > >> | |-- buffer.h -> ../buffer.h > >> | |-- buffer_fwd.h -> ../buffer_fwd.h > >> | |-- crc32c.h -> ../crc32c.h > >> | |-- inline_memory.h -> ../inline_memory.h > >> | |-- librados.h > >> | |-- librados.hpp > >> | |-- librgw.h > >> | |-- memory.h -> ../memory.h > >> | |-- objclass.h > >> | |-- page.h -> ../page.h > >> | |-- rados_types.h > >> | |-- rados_types.hpp > >> | `-- rgw_file.h > >> `-- radosstriper > >> |-- libradosstriper.h > >> `-- libradosstriper.hpp > > > > > > The test code is simple, as attached, and I compiled it with dynamic > > linking: > >> > >> cc -L/rados_install/lib -I/rados_install/include -o librados_test > >> librados_test.c -lrados -lradosstriper > > > > > > But got the following message, can anyone plz advise? > > > >> > >> /rados_install/lib/libradosstriper.so: undefined reference to > >> `ceph::buffer::ptr::ptr(ceph::buffer::ptr&&)' > >> /rados_install/lib/libradosstriper.so: undefined reference to > >> `ceph::buffer::ptr::operator=(ceph::buffer::ptr&&)' > >> /rados_install/lib/libradosstriper.so: undefined reference to > >> `librados::IoCtx::remove(std::string const&, int)' > >> /rados_install/lib/libradosstriper.so: undefined reference to > >> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned > int)' > >> /rados_install/lib/libradosstriper.so: undefined reference to > >> `ceph::buffer::list::is_provided_buffer(char const*) const' > >> /rados_install/lib/libradosstriper.so: undefined reference to > >> `ceph::buffer::list::append(ceph::buffer::ptr&&)' > >> /rados_install/lib/libradosstriper.so: undefined reference to > >> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned > int, > >> std::_List_iterator, unsigned int)' > >> /rados_install/lib/libradosstriper.so: undefined reference to > >> `ceph::buffer::list::list(ceph::buffer::list&&)' > > Try doing the following in your lib directory. > > $ nm -A -C *.so|grep is_provided_buffer > libceph-common.so:00223df0 T > ceph::buffer::list::is_provided_buffer(char const*) const > librados.so:000bfef0 T > ceph::buffer::list::is_provided_buffer(char const*) const > libradosstriper.so: U > ceph::buffer::list::is_provided_buffer(char const*) const > > 'ldd librados.so' shows it links to
Re: [ceph-users] how to build libradosstriper
On Wed, May 30, 2018 at 10:42 AM, Jialin Liu wrote: > Hi, > I'm trying to use the libradosstriper api, but having some trouble in > linking to lradosstriper. I copied only the `required' libraries from an > pre-installed ceph (10.2.10), and put them under my local directory > /rados_install/lib and rados_install/include, on a linux machine. > > /rados_install/lib: >> >> libboost_date_time.so.1.53.0 libboost_system.so.1.53.0 >> libboost_filesystem-mt.so.1.53.0 libboost_thread-mt.so.1.53.0 >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0 >> libboost_iostreams-mt.so.1.53.0 >> libboost_iostreams.so.1.53.0 >> libboost_random-mt.so.1.53.0 librados.so.2.0.0 >> libboost_regex.so.1.53.0 librados_tp.so.2.0.0 >> libboost_system-mt.so.1.53.0 libradosstriper.so.1.0.0 >> libcephfs.so librados.so libradosstriper.so > > > /rados_install/include: >> >> |-- rados >> | |-- buffer.h -> ../buffer.h >> | |-- buffer_fwd.h -> ../buffer_fwd.h >> | |-- crc32c.h -> ../crc32c.h >> | |-- inline_memory.h -> ../inline_memory.h >> | |-- librados.h >> | |-- librados.hpp >> | |-- librgw.h >> | |-- memory.h -> ../memory.h >> | |-- objclass.h >> | |-- page.h -> ../page.h >> | |-- rados_types.h >> | |-- rados_types.hpp >> | `-- rgw_file.h >> `-- radosstriper >> |-- libradosstriper.h >> `-- libradosstriper.hpp > > > The test code is simple, as attached, and I compiled it with dynamic > linking: >> >> cc -L/rados_install/lib -I/rados_install/include -o librados_test >> librados_test.c -lrados -lradosstriper > > > But got the following message, can anyone plz advise? > >> >> /rados_install/lib/libradosstriper.so: undefined reference to >> `ceph::buffer::ptr::ptr(ceph::buffer::ptr&&)' >> /rados_install/lib/libradosstriper.so: undefined reference to >> `ceph::buffer::ptr::operator=(ceph::buffer::ptr&&)' >> /rados_install/lib/libradosstriper.so: undefined reference to >> `librados::IoCtx::remove(std::string const&, int)' >> /rados_install/lib/libradosstriper.so: undefined reference to >> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned int)' >> /rados_install/lib/libradosstriper.so: undefined reference to >> `ceph::buffer::list::is_provided_buffer(char const*) const' >> /rados_install/lib/libradosstriper.so: undefined reference to >> `ceph::buffer::list::append(ceph::buffer::ptr&&)' >> /rados_install/lib/libradosstriper.so: undefined reference to >> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned int, >> std::_List_iterator, unsigned int)' >> /rados_install/lib/libradosstriper.so: undefined reference to >> `ceph::buffer::list::list(ceph::buffer::list&&)' Try doing the following in your lib directory. $ nm -A -C *.so|grep is_provided_buffer libceph-common.so:00223df0 T ceph::buffer::list::is_provided_buffer(char const*) const librados.so:000bfef0 T ceph::buffer::list::is_provided_buffer(char const*) const libradosstriper.so: U ceph::buffer::list::is_provided_buffer(char const*) const 'ldd librados.so' shows it links to libceph-common.so so I suspect you may need that library. Don't forget that you need to tell the linker where to find these libraries at runtime so you will need to pass something like '-Wl,-rpath=/rados_install/lib' to the compiler so it finds the correct versions of the libraries (you can check which ones it will find with 'ldd librados_test'. HTH > > > Best, > Jialin > NERSC > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] how to build libradosstriper
Hi, I'm trying to use the libradosstriper api, but having some trouble in linking to lradosstriper. I copied only the `required' libraries from an pre-installed ceph (10.2.10), and put them under my local directory /rados_install/lib and rados_install/include, on a linux machine. */rados_install/lib:* > libboost_date_time.so.1.53.0 libboost_system.so.1.53.0 > libboost_filesystem-mt.so.1.53.0 libboost_thread-mt.so.1.53.0 > libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0 > libboost_iostreams-mt.so.1.53.0 > libboost_iostreams.so.1.53.0 > libboost_random-mt.so.1.53.0 librados.so.2.0.0 > libboost_regex.so.1.53.0 librados_tp.so.2.0.0 > libboost_system-mt.so.1.53.0 libradosstriper.so.1.0.0 > libcephfs.so librados.so libradosstriper.so */rados_install/include: * > |-- rados > | |-- buffer.h -> ../buffer.h > | |-- buffer_fwd.h -> ../buffer_fwd.h > | |-- crc32c.h -> ../crc32c.h > | |-- inline_memory.h -> ../inline_memory.h > | |-- librados.h > | |-- librados.hpp > | |-- librgw.h > | |-- memory.h -> ../memory.h > | |-- objclass.h > | |-- page.h -> ../page.h > | |-- rados_types.h > | |-- rados_types.hpp > | `-- rgw_file.h > `-- radosstriper > |-- libradosstriper.h > `-- libradosstriper.hpp The test code is simple, as attached, and I compiled it with dynamic linking: > cc -L/rados_install/lib -I/rados_install/include -o librados_test > librados_test.c -lrados -lradosstriper But got the following message, can anyone plz advise? > /rados_install/lib/libradosstriper.so: undefined reference to > `ceph::buffer::ptr::ptr(ceph::buffer::ptr&&)' > /rados_install/lib/libradosstriper.so: undefined reference to > `ceph::buffer::ptr::operator=(ceph::buffer::ptr&&)' > /rados_install/lib/libradosstriper.so: undefined reference to > `librados::IoCtx::remove(std::string const&, int)' > /rados_install/lib/libradosstriper.so: undefined reference to > `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned int)' > /rados_install/lib/libradosstriper.so: undefined reference to > `ceph::buffer::list::is_provided_buffer(char const*) const' > /rados_install/lib/libradosstriper.so: undefined reference to > `ceph::buffer::list::append(ceph::buffer::ptr&&)' > /rados_install/lib/libradosstriper.so: undefined reference to > `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned int, > std::_List_iterator, unsigned int)' > /rados_install/lib/libradosstriper.so: undefined reference to > `ceph::buffer::list::list(ceph::buffer::list&&)' Best, Jialin NERSC librados_test.c Description: Binary data ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
