Last I heard (read) was that the RDMA implementation is somewhat experimental. Search for "troubleshooting ceph rdma performance" on this mailing list for more info.
(Adding Roman in CC who has been working on this recently.) Mohamad On 12/18/18 11:42 AM, Michael Green wrote: > I don't know. > Ceph documentation on Mimic doesn't appear to go into too much details > on RDMA in general, but still it's mentioned in the Ceph docs here and > there. Some examples: > Change log - http://docs.ceph.com/docs/master/releases/mimic/ > Async messenger options > - http://docs.ceph.com/docs/master/rados/configuration/ms-ref/ > > I want to believe that the official docs wouldn't mention something > that's completely broken? > > There are multiple posts in this very mailing list from people trying > to make it work. > *-- > Michael Green > *Customer Support & Integration > Tel. +1 (518) 9862385 > gr...@e8storage.com <mailto:gr...@e8storage.com> > > E8 Storage has a new look, find out more > <https://e8storage.com/when-performance-matters-a-new-look-for-e8-storage/> > > > > > > > > > > >> On Dec 18, 2018, at 6:55 AM, Виталий Филиппов <vita...@yourcmc.ru >> <mailto:vita...@yourcmc.ru>> wrote: >> >> Is RDMA officially supported? I'm asking because I recently tried to >> use DPDK and it seems it's broken... i.e the code is there, but does >> not compile until I fix cmake scripts, and after fixing the build >> OSDs just get segfaults and die after processing something like 40-50 >> incoming packets. >> >> Maybe RDMA is in the same state? >> >> 13 декабря 2018 г. 2:42:23 GMT+03:00, Michael Green >> <gr...@e8storage.com <mailto:gr...@e8storage.com>> пишет: >> >> Sorry for bumping the thread. I refuse to believe there are no >> people on this list who have successfully enabled and run RDMA >> with Mimic. :) >> >> Mike >> >>> Hello collective wisdom, >>> >>> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) >>> mimic (stable) here. >>> >>> I have a working cluster here consisting of 3 monitor hosts, 64 >>> OSD processes across 4 osd hosts, plus 2 MDSs, plus 2 MGRs. All >>> of that is consumed by 10 client nodes. >>> >>> Every host in the cluster, including clients is >>> RHEL 7.5 >>> Mellanox OFED 4.4-2.0.7.0 >>> RoCE NICs are either MCX416A-CCAT or MCX414A-CCAT @ 50Gbit/sec >>> The NICs are all mlx5_0 port 1 >>> >>> ring and ib_send_bw work fine both ways on any two nodes in the >>> cluster. >>> >>> Full configuration of the cluster is pasted below, but RDMA >>> related parameters are configured as following: >>> >>> >>> ms_public_type = async+rdma >>> ms_cluster = async+rdma >>> # Exclude clients for now >>> ms_type = async+posix >>> >>> ms_async_rdma_device_name = mlx5_0 >>> ms_async_rdma_polling_us = 0 >>> ms_async_rdma_port_num=1 >>> >>> When I try to start MON, it immediately fails as below. Anybody >>> has seen this or could give any pointers what to/where to look next? >>> >>> >>> ------ceph-mon.rio.log--begin------ >>> 2018-12-12 22:35:30.011 7f515dc39140 0 set uid:gid to 167:167 >>> (ceph:ceph) >>> 2018-12-12 22:35:30.011 7f515dc39140 0 ceph version 13.2.2 >>> (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable), >>> process ceph-mon, pid 2129843 >>> 2018-12-12 22:35:30.011 7f515dc39140 0 pidfile_write: ignore >>> empty --pid-file >>> 2018-12-12 22:35:30.036 7f515dc39140 0 load: jerasure load: lrc >>> load: isa >>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >>> compression = kNoCompression >>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >>> level_compaction_dynamic_level_bytes = true >>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >>> write_buffer_size = 33554432 >>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >>> compression = kNoCompression >>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >>> level_compaction_dynamic_level_bytes = true >>> 2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option >>> write_buffer_size = 33554432 >>> 2018-12-12 22:35:30.147 7f51442ed700 2 Event(0x55d927e95700 >>> nevent=5000 time_id=1).set_owner idx=1 owner=139987012998912 >>> 2018-12-12 22:35:30.147 7f51442ed700 10 stack operator() starting >>> 2018-12-12 22:35:30.147 7f5143aec700 2 Event(0x55d927e95200 >>> nevent=5000 time_id=1).set_owner idx=0 owner=139987004606208 >>> 2018-12-12 22:35:30.147 7f5144aee700 2 Event(0x55d927e95c00 >>> nevent=5000 time_id=1).set_owner idx=2 owner=139987021391616 >>> 2018-12-12 22:35:30.147 7f5143aec700 10 stack operator() starting >>> 2018-12-12 22:35:30.147 7f5144aee700 10 stack operator() starting >>> 2018-12-12 22:35:30.147 7f515dc39140 0 starting mon.rio rank 0 >>> at public addr 192.168.1.58:6789/0 at bind addr >>> 192.168.1.58:6789/0 mon_data /var/lib/ceph/mon/ceph-rio fsid >>> 376540c8-a362-41cc-9a58-9c8ceca0e4ee >>> 2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind bind >>> 192.168.1.58:6789/0 >>> 2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind Network Stack >>> is not ready for bind yet - postponed >>> 2018-12-12 22:35:30.147 7f515dc39140 0 starting mon.rio rank 0 >>> at 192.168.1.58:6789/0 mon_data /var/lib/ceph/mon/ceph-rio fsid >>> 376540c8-a362-41cc-9a58-9c8ceca0e4ee >>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).mds >>> e84 new map >>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).mds >>> e84 print_map >>> e84 >>> enable_multiple, ever_enabled_multiple: 0,0 >>> compat: compat={},rocompat={},incompat={1=base v0.20,2=client >>> writeable ranges,3=default file layouts on dirs,4=dir inode in >>> separate object,5=mds uses versioned encoding,6=dirfrag is >>> stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} >>> legacy client fscid: -1 >>> >>> No filesystems configured >>> Standby daemons: >>> >>> 5906437: 192.168.1.152:6800/1077205146 'prince' mds.-1.0 >>> up:standby seq 2 >>> 6284118: 192.168.1.59:6800/1266235911 'salvador' mds.-1.0 >>> up:standby seq 2 >>> >>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd >>> e25894 crush map has features 288514051259236352, adjusting msgr >>> requires >>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd >>> e25894 crush map has features 288514051259236352, adjusting msgr >>> requires >>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd >>> e25894 crush map has features 1009089991638532096, adjusting >>> msgr requires >>> 2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd >>> e25894 crush map has features 288514051259236352, adjusting msgr >>> requires >>> 2018-12-12 22:35:30.149 7f515dc39140 10 -- - create_connect >>> 192.168.1.88:6800/1638, creating connection and registering >>> 2018-12-12 22:35:30.149 7f515dc39140 10 -- - >> >>> 192.168.1.88:6800/1638 conn(0x55d9281fbe00 :-1 s=STATE_NONE >>> pgs=0 cs=0 l=0)._connect csq=0 >>> 2018-12-12 22:35:30.149 7f515dc39140 10 -- - get_connection >>> mgr.5894115 192.168.1.88:6800/1638 new 0x55d9281fbe00 >>> 2018-12-12 22:35:30.150 7f515dc39140 1 -- - --> >>> 192.168.1.88:6800/1638 -- mgropen(unknown.rio) v3 -- >>> 0x55d92844e000 con 0 >>> 2018-12-12 22:35:30.151 7f515dc39140 1 -- - start start >>> 2018-12-12 22:35:30.151 7f515dc39140 1 -- - start start >>> 2018-12-12 22:35:30.151 7f515dc39140 10 -- - ready - >>> 2018-12-12 22:35:30.151 7f515dc39140 10 -- - bind bind >>> 192.168.1.58:6789/0 >>> 2018-12-12 22:35:30.151 7f515dc39140 10 Processor -- bind >>> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband binding_port >>> found active port 1 >>> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init receive >>> queue length is 4096 receive buffers >>> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init >>> assigning: 1024 send buffers >>> 2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init device >>> allow 4194303 completion entries >>> 2018-12-12 22:35:30.509 7f515dc39140 10 Processor -- bind bound >>> to 192.168.1.58:6789/0 >>> 2018-12-12 22:35:30.509 7f515dc39140 1 -- 192.168.1.58:6789/0 >>> learned_addr learned my addr 192.168.1.58:6789/0 >>> 2018-12-12 22:35:30.509 7f515dc39140 1 -- 192.168.1.58:6789/0 >>> _finish_bind bind my_inst.addr is 192.168.1.58:6789/0 >>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- - ready - >>> 2018-12-12 22:35:30.510 7f515dc39140 1 Processor -- start >>> 2018-12-12 22:35:30.510 7f515dc39140 0 mon.rio@-1(probing) e5 >>> my rank is now 0 (was -1) >>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 >>> shutdown_connections >>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 >>> _send_message--> mon.1 192.168.1.59:6789/0 -- mon_probe(probe >>> 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 >>> 0x55d928525680 >>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >>> create_connect 192.168.1.59:6789/0, creating connection and >>> registering >>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >>> >> 192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_NONE >>> pgs=0 cs=0 l=0)._connect csq=0 >>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 >>> --> 192.168.1.59:6789/0 -- mon_probe(probe >>> 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- >>> 0x55d928525680 con 0 >>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 >>> _send_message--> mon.2 192.168.1.65:6789/0 -- mon_probe(probe >>> 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 >>> 0x55d928525900 >>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >>> create_connect 192.168.1.65:6789/0, creating connection and >>> registering >>> 2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >>> >> 192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_NONE >>> pgs=0 cs=0 l=0)._connect csq=0 >>> 2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 >>> --> 192.168.1.65:6789/0 -- mon_probe(probe >>> 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- >>> 0x55d928525900 con 0 >>> 2018-12-12 22:35:30.513 7f5143aec700 10 NetHandler >>> generic_connect connect: (111) Connection refused >>> 2018-12-12 22:35:30.513 7f51442ed700 10 NetHandler >>> generic_connect connect: (111) Connection refused >>> 2018-12-12 22:35:30.513 7f5144aee700 10 Infiniband send_msg >>> sending: 0, 12894, 0, 0, fe800000000000007efe90fffe1e2524 >>> 2018-12-12 22:35:30.513 7f5143aec700 1 RDMAStack connect try >>> connecting failed. >>> 2018-12-12 22:35:30.513 7f51442ed700 1 RDMAStack connect try >>> connecting failed. >>> 2018-12-12 22:35:30.513 7f5144aee700 10 -- - >> >>> 192.168.1.88:6800/1638 conn(0x55d9281fbe00 :-1 >>> s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection >>> nonblock connect inprogress >>> 2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >>> >> 192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 >>> s=STATE_CONNECTING pgs=0 cs=0 l=0).fault waiting 0.200000 >>> 2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >>> >> 192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 >>> s=STATE_CONNECTING pgs=0 cs=0 l=0).fault waiting 0.200000 >>> 2018-12-12 22:35:30.513 7f5144aee700 10 -- - >> >>> 192.168.1.88:6800/1638 conn(0x55d9281fbe00 :-1 >>> s=STATE_CONNECTING_RE pgs=0 cs=0 l=0).handle_write >>> 2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >>> >> 192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 >>> s=STATE_CONNECTING pgs=0 cs=0 l=0).handle_write >>> 2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >>> >> 192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 >>> s=STATE_CONNECTING pgs=0 cs=0 l=0).handle_write >>> 2018-12-12 22:35:30.513 7f5144aee700 5 Infiniband recv_msg >>> recevd: 105, 0, 262144, 0, >>> 2018-12-12 22:35:30.513 7f5144aee700 -1 RDMAConnectedSocketImpl >>> activate failed to transition to RTR state: (113) No route to host >>> 2018-12-12 22:35:30.515 7f5144aee700 -1 >>> >>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc >>> <http://rdmaconnectedsocketimpl.cc/>: In function 'void >>> RDMAConnectedSocketImpl::handle_connection()' thread >>> 7f5144aee700 time 2018-12-12 22:35:30.514762 >>> >>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc >>> <http://rdmaconnectedsocketimpl.cc/>: 224: FAILED assert(!r) >>> >>> ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) >>> mimic (stable) >>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>> char const*)+0xff) [0x7f515506e6bf] >>> 2: (()+0x285887) [0x7f515506e887] >>> 3: (RDMAConnectedSocketImpl::handle_connection()+0x6f5) >>> [0x7f51551b9655] >>> 4: (EventCenter::process_events(unsigned int, >>> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >>> >*)+0x695) [0x7f51551a76f5] >>> 5: (()+0x3c15cc) [0x7f51551aa5cc] >>> 6: (()+0x6afaef) [0x7f5155498aef] >>> 7: (()+0x7e25) [0x7f5154396e25] >>> 8: (clone()+0x6d) [0x7f5150cb7bad] >>> NOTE: a copy of the executable, or `objdump -rdS <executable>` >>> is needed to interpret this. >>> ------ceph-mon.rio.log--end------ >>> >>> -----ceph.conf---begin----- >>> [client] >>> rbd_cache = False >>> rbd_cache_writethrough_until_flush = False >>> #admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok >>> #log file = /var/log/ceph/ >>> >>> [global] >>> ms_type = async+posix >>> # set RDMA messaging just for the public or cluster network >>> ms_public_type = async+rdma >>> ms_cluster = async+rdma >>> # >>> # set a device name according to IB or ROCE device used, e.g. >>> ms_async_rdma_device_name = mlx5_0 >>> # >>> # for better performance if using LUMINOUS 12.2.x release >>> ms_async_rdma_polling_us = 0 >>> ms_async_rdma_port_num=1 >>> >>> rgw_override_bucket_index_max_shards=1 >>> auth client required = none >>> auth cluster required = none >>> auth service required = none >>> auth supported = none >>> cephx require signatures = False >>> cephx sign messages = False >>> cluster network = 192.168.1.0/24 >>> debug asok = 0/0 >>> debug auth = 0/0 >>> debug bluefs = 0/0 >>> debug bluestore = 0/0 >>> debug buffer = 0/0 >>> debug client = 0/0 >>> debug context = 0/0 >>> debug crush = 0/0 >>> debug filer = 0/0 >>> debug filestore = 0/0 >>> debug finisher = 0/0 >>> #debug hadoop = 0/0 -- doesn't work in E8 setup >>> debug heartbeatmap = 0/0 >>> debug journal = 0/0 >>> debug journaler = 0/0 >>> debug lockdep = 0/0 >>> debug log = 0 >>> debug mds = 0/0 >>> debug mds_balancer = 0/0 >>> debug mds_locker = 0/0 >>> debug mds_log = 0/0 >>> debug mds_log_expire = 0/0 >>> debug mds_migrator = 0/0 >>> debug mon = 0/0 >>> debug monc = 0/0 >>> debug ms = 10/10 >>> #debug ms = 0/0 >>> debug objclass = 0/0 >>> debug objectcacher = 0/0 >>> debug objecter = 0/0 >>> debug optracker = 0/0 >>> debug osd = 0/0 >>> debug paxos = 0/0 >>> debug perfcounter = 0/0 >>> debug rados = 0/0 >>> debug rbd = 0/0 >>> debug rgw = 0/0 >>> debug rocksdb = 0/0 >>> debug throttle = 0/0 >>> debug timer = 0/0 >>> debug tp = 0/0 >>> #debug zs = 0/0 -- doesn't work in E8 setup >>> fsid = 376540c8-a362-41cc-9a58-9c8ceca0e4ee >>> mon_host = 192.168.1.58,192.168.1.65,192.168.1.59 >>> mon pg warn max per osd = 800 >>> mon_allow_pool_delete = True >>> mon_max_pg_per_osd = 800 >>> ms type = async >>> ms_crc_data = False >>> #ms_crc_header = False -- broken! >>> osd objectstore = bluestore >>> osd_pool_default_size = 2 >>> perf = True >>> public network = 192.168.1.0/24 >>> rocksdb_perf = True >>> # Parameter not present in Micron's config, but introduced by >>> ceph-deploy >>> mon_initial_members = rio >>> # The following param claims to reduce CPU usage; found it at >>> # http://docs.ceph.com/docs/mimic/rados/configuration/ms-ref/ >>> ms nocrc = true >>> #rbd_op_threads=4 >>> [mon] >>> mon_max_pool_pg_num = 166496 >>> mon_osd_max_split_count = 10000 >>> [osd] >>> osd_min_pg_log_entries = 10 >>> osd_max_pg_log_entries = 10 >>> osd_pg_log_dups_tracked = 10 >>> osd_pg_log_trim_min = 10 >>> bluestore_cache_kv_max = 96G >>> bluestore_cache_kv_ratio = 0.2 >>> bluestore_cache_meta_ratio = 0.8 >>> bluestore_cache_size_ssd = 7G >>> bluestore_csum_type = none >>> bluestore_extent_map_shard_max_size = 200 >>> bluestore_extent_map_shard_min_size = 50 >>> bluestore_extent_map_shard_target_size = 100 >>> >>> bluestore_rocksdb_options = >>> >>> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB >>> >>> [client.rgw.sm26] >>> rgw_frontends = "civetweb port=7480" >>> >>> #[osd.0] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.1] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.2] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.3] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.4] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.5] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.6] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.7] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.8] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.9] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.10] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.11] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.12] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.13] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.14] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> #[osd.15] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae >>> # >>> ## BONJOVI1 >>> # >>> #[osd.16] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.17] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.18] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.19] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.20] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.21] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.22] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.23] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.24] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.25] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.26] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.27] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.28] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.29] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.30] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> #[osd.31] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af >>> # >>> ## PRINCE >>> # >>> #[osd.32] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.33] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> #[osd.34] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.35] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.36] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.37] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.38] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.39] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.40] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.41] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.42] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.43] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.44] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.45] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.46] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> #[osd.47] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198 >>> # >>> # >>> ## RINGO >>> # >>> #[osd.48] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.49] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.50] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.51] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.52] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.53] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.54] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.55] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.56] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.57] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.58] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.59] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.60] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.61] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.62] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[osd.63] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e >>> # >>> #[mon.rio] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013a >>> # >>> #[mon.salvador] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013b >>> # >>> #[mon.medellin] >>> #ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0141 >>> >>> -----ceph.conf---end----- >>> >>> >>> *-- >>> * >> *Michael Green >> * >> >> >> >> >> >> >> >> >> >> >> >> -- >> With best regards, >> Vitaliy Filippov > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
