Hi David, I noticed the public interface of the server I am running the test from is heavily used so I will bond that one too
I doubt though that this explains the poor performance Thanks for your advice Steven On 22 January 2018 at 12:02, David Turner <drakonst...@gmail.com> wrote: > I'm not speaking to anything other than your configuration. > > "I am using 2 x 10 GB bonded ( BONDING_OPTS="mode=4 miimon=100 > xmit_hash_policy=1 lacp_rate=1") for cluster and 1 x 1GB for public" > It might not be a bad idea for you to forgo the public network on the 1Gb > interfaces and either put everything on one network or use VLANs on the > 10Gb connections. I lean more towards that in particular because your > public network doesn't have a bond on it. Just as a note, communication > between the OSDs and the MONs are all done on the public network. If that > interface goes down, then the OSDs are likely to be marked down/out from > your cluster. I'm a fan of VLANs, but if you don't have the equipment or > expertise to go that route, then just using the same subnet for public and > private is a decent way to go. > > On Mon, Jan 22, 2018 at 11:37 AM Steven Vacaroaia <ste...@gmail.com> > wrote: > >> I did test with rados bench ..here are the results >> >> rados bench -p ssdpool 300 -t 12 write --no-cleanup && rados bench -p >> ssdpool 300 -t 12 seq >> >> Total time run: 300.322608 >> Total writes made: 10632 >> Write size: 4194304 >> Object size: 4194304 >> Bandwidth (MB/sec): 141.608 >> Stddev Bandwidth: 74.1065 >> Max bandwidth (MB/sec): 264 >> Min bandwidth (MB/sec): 0 >> Average IOPS: 35 >> Stddev IOPS: 18 >> Max IOPS: 66 >> Min IOPS: 0 >> Average Latency(s): 0.33887 >> Stddev Latency(s): 0.701947 >> Max latency(s): 9.80161 >> Min latency(s): 0.015171 >> >> Total time run: 300.829945 >> Total reads made: 10070 >> Read size: 4194304 >> Object size: 4194304 >> Bandwidth (MB/sec): 133.896 >> Average IOPS: 33 >> Stddev IOPS: 14 >> Max IOPS: 68 >> Min IOPS: 3 >> Average Latency(s): 0.35791 >> Max latency(s): 4.68213 >> Min latency(s): 0.0107572 >> >> >> rados bench -p scbench256 300 -t 12 write --no-cleanup && rados bench -p >> scbench256 300 -t 12 seq >> >> Total time run: 300.747004 >> Total writes made: 10239 >> Write size: 4194304 >> Object size: 4194304 >> Bandwidth (MB/sec): 136.181 >> Stddev Bandwidth: 75.5 >> Max bandwidth (MB/sec): 272 >> Min bandwidth (MB/sec): 0 >> Average IOPS: 34 >> Stddev IOPS: 18 >> Max IOPS: 68 >> Min IOPS: 0 >> Average Latency(s): 0.352339 >> Stddev Latency(s): 0.72211 >> Max latency(s): 9.62304 >> Min latency(s): 0.00936316 >> hints = 1 >> >> >> Total time run: 300.610761 >> Total reads made: 7628 >> Read size: 4194304 >> Object size: 4194304 >> Bandwidth (MB/sec): 101.5 >> Average IOPS: 25 >> Stddev IOPS: 11 >> Max IOPS: 61 >> Min IOPS: 0 >> Average Latency(s): 0.472321 >> Max latency(s): 15.636 >> Min latency(s): 0.0188098 >> >> >> On 22 January 2018 at 11:34, Steven Vacaroaia <ste...@gmail.com> wrote: >> >>> sorry ..send the message too soon >>> Here is more info >>> Vendor Id : SEAGATE >>> Product Id : ST600MM0006 >>> State : Online >>> Disk Type : SAS,Hard Disk Device >>> Capacity : 558.375 GB >>> Power State : Active >>> >>> ( SSD is in slot 0) >>> >>> megacli -LDGetProp -Cache -LALL -a0 >>> >>> Adapter 0-VD 0(target id: 0): Cache Policy:WriteThrough, ReadAheadNone, >>> Direct, No Write Cache if bad BBU >>> Adapter 0-VD 1(target id: 1): Cache Policy:WriteBack, ReadAdaptive, >>> Direct, No Write Cache if bad BBU >>> Adapter 0-VD 2(target id: 2): Cache Policy:WriteBack, ReadAdaptive, >>> Direct, No Write Cache if bad BBU >>> Adapter 0-VD 3(target id: 3): Cache Policy:WriteBack, ReadAdaptive, >>> Direct, No Write Cache if bad BBU >>> Adapter 0-VD 4(target id: 4): Cache Policy:WriteBack, ReadAdaptive, >>> Direct, No Write Cache if bad BBU >>> Adapter 0-VD 5(target id: 5): Cache Policy:WriteBack, ReadAdaptive, >>> Direct, No Write Cache if bad BBU >>> >>> [root@osd01 ~]# megacli -LDGetProp -DskCache -LALL -a0 >>> >>> Adapter 0-VD 0(target id: 0): Disk Write Cache : Disabled >>> Adapter 0-VD 1(target id: 1): Disk Write Cache : Disk's Default >>> Adapter 0-VD 2(target id: 2): Disk Write Cache : Disk's Default >>> Adapter 0-VD 3(target id: 3): Disk Write Cache : Disk's Default >>> Adapter 0-VD 4(target id: 4): Disk Write Cache : Disk's Default >>> Adapter 0-VD 5(target id: 5): Disk Write Cache : Disk's Default >>> >>> >>> CPU >>> Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz >>> >>> Centos 7 kernel 3.10.0-693.11.6.el7.x86_64 >>> >>> sysctl -p >>> net.ipv4.tcp_sack = 0 >>> net.core.netdev_budget = 600 >>> net.ipv4.tcp_window_scaling = 1 >>> net.core.rmem_max = 16777216 >>> net.core.wmem_max = 16777216 >>> net.core.rmem_default = 16777216 >>> net.core.wmem_default = 16777216 >>> net.core.optmem_max = 40960 >>> net.ipv4.tcp_rmem = 4096 87380 16777216 >>> net.ipv4.tcp_wmem = 4096 65536 16777216 >>> net.ipv4.tcp_syncookies = 0 >>> net.core.somaxconn = 1024 >>> net.core.netdev_max_backlog = 20000 >>> net.ipv4.tcp_max_syn_backlog = 30000 >>> net.ipv4.tcp_max_tw_buckets = 2000000 >>> net.ipv4.tcp_tw_reuse = 1 >>> net.ipv4.tcp_slow_start_after_idle = 0 >>> net.ipv4.conf.all.send_redirects = 0 >>> net.ipv4.conf.all.accept_redirects = 0 >>> net.ipv4.conf.all.accept_source_route = 0 >>> vm.min_free_kbytes = 262144 >>> vm.swappiness = 0 >>> vm.vfs_cache_pressure = 100 >>> fs.suid_dumpable = 0 >>> kernel.core_uses_pid = 1 >>> kernel.msgmax = 65536 >>> kernel.msgmnb = 65536 >>> kernel.randomize_va_space = 1 >>> kernel.sysrq = 0 >>> kernel.pid_max = 4194304 >>> fs.file-max = 100000 >>> >>> >>> ceph.conf >>> >>> >>> public_network = 10.10.30.0/24 >>> cluster_network = 192.168.0.0/24 >>> >>> >>> osd_op_num_threads_per_shard = 2 >>> osd_op_num_shards = 25 >>> osd_pool_default_size = 2 >>> osd_pool_default_min_size = 1 # Allow writing 1 copy in a degraded state >>> osd_pool_default_pg_num = 256 >>> osd_pool_default_pgp_num = 256 >>> osd_crush_chooseleaf_type = 1 >>> osd_scrub_load_threshold = 0.01 >>> osd_scrub_min_interval = 137438953472 >>> osd_scrub_max_interval = 137438953472 >>> osd_deep_scrub_interval = 137438953472 >>> osd_max_scrubs = 16 >>> osd_op_threads = 8 >>> osd_max_backfills = 1 >>> osd_recovery_max_active = 1 >>> osd_recovery_op_priority = 1 >>> >>> >>> >>> >>> debug_lockdep = 0/0 >>> debug_context = 0/0 >>> debug_crush = 0/0 >>> debug_buffer = 0/0 >>> debug_timer = 0/0 >>> debug_filer = 0/0 >>> debug_objecter = 0/0 >>> debug_rados = 0/0 >>> debug_rbd = 0/0 >>> debug_journaler = 0/0 >>> debug_objectcatcher = 0/0 >>> debug_client = 0/0 >>> debug_osd = 0/0 >>> debug_optracker = 0/0 >>> debug_objclass = 0/0 >>> debug_filestore = 0/0 >>> debug_journal = 0/0 >>> debug_ms = 0/0 >>> debug_monc = 0/0 >>> debug_tp = 0/0 >>> debug_auth = 0/0 >>> debug_finisher = 0/0 >>> debug_heartbeatmap = 0/0 >>> debug_perfcounter = 0/0 >>> debug_asok = 0/0 >>> debug_throttle = 0/0 >>> debug_mon = 0/0 >>> debug_paxos = 0/0 >>> debug_rgw = 0/0 >>> >>> >>> [mon] >>> mon_allow_pool_delete = true >>> >>> [osd] >>> osd_heartbeat_grace = 20 >>> osd_heartbeat_interval = 5 >>> bluestore_block_db_size = 16106127360 <(610)%20612-7360> >>> bluestore_block_wal_size = 1073741824 >>> >>> [osd.6] >>> host = osd01 >>> osd_journal = /dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0- >>> 062c0ceff05d.1d58775a-5019-42ea-8149-a126f51a2501 >>> crush_location = root=ssds host=osd01-ssd >>> >>> [osd.7] >>> host = osd02 >>> osd_journal = /dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0- >>> 062c0ceff05d.683dc52d-5d69-4ff0-b5d9-b17056a55681 >>> crush_location = root=ssds host=osd02-ssd >>> >>> [osd.8] >>> host = osd04 >>> osd_journal = /dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0- >>> 062c0ceff05d.bd7c0088-b724-441e-9b88-9457305c541d >>> crush_location = root=ssds host=osd04-ssd >>> >>> >>> On 22 January 2018 at 11:29, Steven Vacaroaia <ste...@gmail.com> wrote: >>> >>>> Hi David, >>>> >>>> Yes, I meant no separate partitions for WAL and DB >>>> >>>> I am using 2 x 10 GB bonded ( BONDING_OPTS="mode=4 miimon=100 >>>> xmit_hash_policy=1 lacp_rate=1") for cluster and 1 x 1GB for public >>>> Disks are >>>> Vendor Id : TOSHIBA >>>> Product Id : PX05SMB040Y >>>> State : Online >>>> Disk Type : SAS,Solid State Device >>>> Capacity : 372.0 GB >>>> >>>> >>>> On 22 January 2018 at 11:24, David Turner <drakonst...@gmail.com> >>>> wrote: >>>> >>>>> Disk models, other hardware information including CPU, network >>>>> config? You say you're using Luminous, but then say journal on same >>>>> device. I'm assuming you mean that you just have the bluestore OSD >>>>> configured without a separate WAL or DB partition? Any more specifics you >>>>> can give will be helpful. >>>>> >>>>> On Mon, Jan 22, 2018 at 11:20 AM Steven Vacaroaia <ste...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'll appreciate if you can provide some guidance / suggestions >>>>>> regarding perfomance issues on a test cluster ( 3 x DELL R620, 1 >>>>>> Entreprise >>>>>> SSD, 3 x 600 GB ,Entreprise HDD, 8 cores, 64 GB RAM) >>>>>> >>>>>> I created 2 pools ( replication factor 2) one with only SSD and the >>>>>> other with only HDD >>>>>> ( journal on same disk for both) >>>>>> >>>>>> The perfomance is quite similar although I was expecting to be at >>>>>> least 5 times better >>>>>> No issues noticed using atop >>>>>> >>>>>> What should I check / tune ? >>>>>> >>>>>> Many thanks >>>>>> Steven >>>>>> >>>>>> >>>>>> >>>>>> HDD based pool ( journal on the same disk) >>>>>> >>>>>> ceph osd pool get scbench256 all >>>>>> >>>>>> size: 2 >>>>>> min_size: 1 >>>>>> crash_replay_interval: 0 >>>>>> pg_num: 256 >>>>>> pgp_num: 256 >>>>>> crush_rule: replicated_rule >>>>>> hashpspool: true >>>>>> nodelete: false >>>>>> nopgchange: false >>>>>> nosizechange: false >>>>>> write_fadvise_dontneed: false >>>>>> noscrub: false >>>>>> nodeep-scrub: false >>>>>> use_gmt_hitset: 1 >>>>>> auid: 0 >>>>>> fast_read: 0 >>>>>> >>>>>> >>>>>> rbd bench --io-type write image1 --pool=scbench256 >>>>>> bench type write io_size 4096 io_threads 16 bytes 1073741824 pattern >>>>>> sequential >>>>>> SEC OPS OPS/SEC BYTES/SEC >>>>>> 1 46816 46836.46 191842139.78 >>>>>> 2 90658 45339.11 185709011.80 >>>>>> 3 133671 44540.80 182439126.08 >>>>>> 4 177341 44340.36 181618100.14 >>>>>> 5 217300 43464.04 178028704.54 >>>>>> 6 259595 42555.85 174308767.05 >>>>>> elapsed: 6 ops: 262144 ops/sec: 42694.50 bytes/sec: >>>>>> 174876688.23 >>>>>> >>>>>> fio /home/cephuser/write_256.fio >>>>>> write-4M: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, >>>>>> iodepth=32 >>>>>> fio-2.2.8 >>>>>> Starting 1 process >>>>>> rbd engine: RBD version: 1.12.0 >>>>>> Jobs: 1 (f=1): [r(1)] [100.0% done] [66284KB/0KB/0KB /s] [16.6K/0/0 >>>>>> iops] [eta 00m:00s] >>>>>> >>>>>> >>>>>> fio /home/cephuser/write_256.fio >>>>>> write-4M: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, >>>>>> iodepth=32 >>>>>> fio-2.2.8 >>>>>> Starting 1 process >>>>>> rbd engine: RBD version: 1.12.0 >>>>>> Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/14464KB/0KB /s] [0/3616/0 >>>>>> iops] [eta 00m:00s] >>>>>> >>>>>> >>>>>> SSD based pool >>>>>> >>>>>> >>>>>> ceph osd pool get ssdpool all >>>>>> >>>>>> size: 2 >>>>>> min_size: 1 >>>>>> crash_replay_interval: 0 >>>>>> pg_num: 128 >>>>>> pgp_num: 128 >>>>>> crush_rule: ssdpool >>>>>> hashpspool: true >>>>>> nodelete: false >>>>>> nopgchange: false >>>>>> nosizechange: false >>>>>> write_fadvise_dontneed: false >>>>>> noscrub: false >>>>>> nodeep-scrub: false >>>>>> use_gmt_hitset: 1 >>>>>> auid: 0 >>>>>> fast_read: 0 >>>>>> >>>>>> rbd -p ssdpool create --size 52100 image2 >>>>>> >>>>>> rbd bench --io-type write image2 --pool=ssdpool >>>>>> bench type write io_size 4096 io_threads 16 bytes 1073741824 pattern >>>>>> sequential >>>>>> SEC OPS OPS/SEC BYTES/SEC >>>>>> 1 42412 41867.57 171489557.93 >>>>>> 2 78343 39180.86 160484805.88 >>>>>> 3 118082 39076.48 160057256.16 >>>>>> 4 155164 38683.98 158449572.38 >>>>>> 5 192825 38307.59 156907885.84 >>>>>> 6 230701 37716.95 154488608.16 >>>>>> elapsed: 7 ops: 262144 ops/sec: 36862.89 bytes/sec: >>>>>> 150990387.29 >>>>>> >>>>>> >>>>>> [root@osd01 ~]# fio /home/cephuser/write_256.fio >>>>>> write-4M: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, >>>>>> iodepth=32 >>>>>> fio-2.2.8 >>>>>> Starting 1 process >>>>>> rbd engine: RBD version: 1.12.0 >>>>>> Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/20224KB/0KB /s] [0/5056/0 >>>>>> iops] [eta 00m:00s] >>>>>> >>>>>> >>>>>> fio /home/cephuser/write_256.fio >>>>>> write-4M: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, >>>>>> iodepth=32 >>>>>> fio-2.2.8 >>>>>> Starting 1 process >>>>>> rbd engine: RBD version: 1.12.0 >>>>>> Jobs: 1 (f=1): [r(1)] [100.0% done] [76096KB/0KB/0KB /s] [19.3K/0/0 >>>>>> iops] [eta 00m:00s] >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> ceph-users@lists.ceph.com >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>> >>>>> >>>> >>> >>
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
