[ceph-users] multiple active connections to a single LUN
hi, install iscsi gateway as write http://docs.ceph.com/docs/master/rbd/iscsi-overview/ kernel version 4.16.1 ceph v12.2.10 Luminous early i use tgt iscsi target i can connect 3 esxi server to one LUN as a shared drive and if one esxi was dropped the machines were transferred to others from the same storage Now when I try to connect one LAN to 3 servers, I get an error on the ceph server Dec 29 07:24:29 ceph-node1 tcmu-runner: tcmu_notify_lock_lost: 209 rbd / main_pool.store: Async lock drop. Old state 1 Dec 29 07:24:30 ceph-node1 tcmu-runner: alua_implicit_transition: 566 rbd / main_pool.store: Starting lock acquisition operation. Dec 29 07:24:30 ceph-node1 tcmu-runner: tcmu_rbd_lock: 757 rbd / main_pool.store: Acquired exclusive lock. Dec 29 07:24:35 ceph-node1 tcmu-runner: alua_implicit_transition: 566 rbd / main_pool.store: Starting lock acquisition operation. such a scheme is possible in this configuration? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Falls cluster then one node switch off
Hello!
>>mon_osd_down_out_subtree_limit = host
Thanks! This realy help me!!!
>>So again, not a full duplication of the data, but a significant amount.
If on the host who was left alone, will step down at this time one OSD. ALL
data will available? Or part of data, pgs which are marked as 'active +
undersized + degraded' may be lost?
26.05.2016, 12:33, "Christian Balzer" <ch...@gol.com>:
> Hello,
>
> I've expanded the cache-tier in my test cluster from a single node
> to 2, increased the pool size from 1 to 2, then waited until all the data
> was rebalanced/duplicated and the cluster was healthy again.
>
> Then I stopped all OSDs on one of the 2 nodes and nothing other than
> degraded/undersized PGs happened (even after 10 minutes):
> ---
> pgmap v1529705: 1920 pgs: 1152 active+clean, 768 active+undersized+degraded;
> 214 GB data, 695 GB used, 4932 GB / 5665 GB avail; 26059/397156 objects
> degraded (6.561%)
> ---
>
> However I do have this setting in my ceph.conf:
> ---
> mon_osd_down_out_subtree_limit = host
> ---
>
> Which prevents the OSDs to be set out when host goes down.
> The idea here of course is to prevent data movement since it's likely
> faster to get the host back up than all the data moved.
>
> So the 4 OSDs on the "dead" node where still in and had a re-weight of 1.
>
> So next I did set them out and indeed I saw the same thing as you did:
> ---
> pgmap v1529737: 1920 pgs: 75 peering, 44
> active+recovery_wait+degraded+remapped, 1152 active+clean, 2
> active+recovering+degraded+remapped, 8 active+remapped, 608
> active+undersized+degraded, 31
> active+undersized+degraded+remapped+wait_backfill; 214 GB data, 672 GB used,
> 3279 GB / 3989 GB avail; 23132/392266 objects degraded (5.897%); 2657/392266
> objects misplaced (0.677%)
> ---
>
> However that stopped pretty quickly and the data usage on the still
> up/in OSDs only increased about 15%.
> Definitely NOT duplicating all the PGs/objects.
>
> So what I think is happening here (confirmation by the devs appreciated)
> is that due to the OSDs being OUT CRUSH goes and does its thing, thus
> moving some PGs around.
> So if PG 1.1 was primary on osd.0 and now the CRUSH algorithm thinks it
> should be primary on osd.0 what happens is that the data for that PG gets
> copied from osd.0 to osd.1, using up more space on osd.1 and NOT freeing up
> the space on osd.0.
>
> So again, not a full duplication of the data, but a significant amount.
>
> Thus my advice for you (and anybody reading this) would be:
> a) set "mon_osd_down_out_subtree_limit = host" (and monitor your systems
> of course).
> b) keep your space usage on dual node clusters/pools below 70%.
>
> Regards,
>
> Christian
>
> On Wed, 25 May 2016 14:51:24 +0900 Christian Balzer wrote:
>
>> Hello,
>>
>> Thanks for the update and I totally agree that it should try to do 2x
>> replication on the single storage node.
>>
>> I'll try to reproduce what you're seeing tomorrow on my test cluster,
>> need to move some data around first.
>>
>> Christian
>>
>> On Wed, 25 May 2016 08:58:54 +0700 Никитенко Виталий wrote:
>>
>> > I'm sorry it was not right map, that map right
>> >
>> > # begin crush map
>> > tunable choose_local_tries 0
>> > tunable choose_local_fallback_tries 0
>> > tunable choose_total_tries 50
>> > tunable chooseleaf_descend_once 1
>> > tunable straw_calc_version 1
>> >
>> > # devices
>> > device 0 osd.0
>> > device 1 osd.1
>> > device 2 osd.2
>> > device 3 osd.3
>> > device 4 osd.4
>> > device 5 osd.5
>> >
>> > # types
>> > type 0 osd
>> > type 1 host
>> > type 2 chassis
>> > type 3 rack
>> > type 4 row
>> > type 5 pdu
>> > type 6 pod
>> > type 7 room
>> > type 8 datacenter
>> > type 9 region
>> > type 10 root
>> >
>> > # buckets
>> > host ceph1-node {
>> > id -2 # do not change unnecessarily
>> > # weight 0.030
>> > alg straw
>> > hash 0 # rjenkins1
>> > item osd.0 weight 0.010
>> > item osd.1 weight 0.010
>> > item osd.2 weight 0.010
>> > }
>> > host ceph2-node {
>> > id -3 # do not change unnecessarily
>> > # weight 0.030
>> > alg straw
>> > hash 0 # rjenkins1
>> > item osd.3 weight 0.010
>> > item osd.4 weight 0.010
>> > item osd.5 weight 0.010
>> > }
>> > root defau
Re: [ceph-users] Falls cluster then one node switch off
1 pgs=0 cs=0 l=1
c=0x7fe7bc05dfe0).fault
min_size: 1
root@ceph1-node:~# ceph osd dump
pool 1 'hdd' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins
pg_num 512 pgp_num 512 last_change 53 flags hashpspool stripe_width 0
after remapped end
root@ceph1-node:~# ceph -s
2016-05-24 17:23:10.123542 7f2c001cf700 0 -- :/623268863 >>
192.168.241.12:6789/0 pipe(0x7f2bfc059cd0 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7f2bfc05dfc0).fault
cluster 808ee682-c121-4867-9fe4-a347d95bf3f0
health HEALTH_WARN
408 pgs degraded
262 pgs stuck degraded
640 pgs stuck unclean
262 pgs stuck undersized
408 pgs undersized
recovery 315/1098 objects degraded (28.689%)
recovery 234/1098 objects misplaced (21.311%)
1 mons down, quorum 0,2 ceph1-node,ceph-mon2
monmap e1: 3 mons at
{ceph-mon2=192.168.241.20:6789/0,ceph1-node=192.168.241.2:6789/0,ceph2-node=192.168.241.12:6789/0}
election epoch 18, quorum 0,2 ceph1-node,ceph-mon2
osdmap e63: 6 osds: 3 up, 3 in; 232 remapped pgs
pgmap v209: 640 pgs, 2 pools, 2118 MB data, 549 objects
3149 MB used, 27537 MB / 30686 MB avail
315/1098 objects degraded (28.689%)
234/1098 objects misplaced (21.311%)
408 active+undersized+degraded
232 active+remapped
Any idea why ceph making redundancies on local disks host?
24.05.2016, 12:53, "Christian Balzer" <ch...@gol.com>:
> Hello,
>
> On Tue, 24 May 2016 10:28:02 +0700 Никитенко Виталий wrote:
>
>> Hello!
>> I have a cluster of 2 nodes with 3 OSD each. The cluster full about 80%.
>
> According to your CRUSH map that's not quite true, namely ceph1-node2
> entry.
>
> And while that again according to your CRUSH map isn't in the default root
> I wonder WHERE it is and if it confuses Ceph into believing that there is
> actually a third node?
>
> "ceph osd tree" output may help, as well as removing ceph1-node2 from the
> picture.
>
>> df -H
>> /dev/sdc1 27G 24G 3.9G 86% /var/lib/ceph/osd/ceph-1
>> /dev/sdd1 27G 20G 6.9G 75% /var/lib/ceph/osd/ceph-2
>> /dev/sdb1 27G 24G 3.5G 88% /var/lib/ceph/osd/ceph-0
>>
>> When I switch off one server, then after 10 minutes begins remapped pgs
>
> [snip]
>> As a result, one disk overflow and the cluster falls. Why ceph remapped
>> pgs, it was supposed to simply mark all pgs as active+degraded, while
>> second node down?
>
> Yes, I agree, that shouldn't happen with a properly configured 2 node
> cluster.
>
>> ceph version 0.80.11
>
> Not aware of any bugs in there and in fact I did test a 2 node cluster
> with Firefly, but be aware that this version is EoL and no longer receiving
> updates.
>
>> root@ceph1-node:~# cat /etc/ceph/ceph.conf
>> [global]
>> fsid = b66c7daa-d6d8-46c7-9e61-15adbb749ed7
>> mon_initial_members = ceph1-node, ceph2-node, ceph-mon2
>> mon_host = 192.168.241.97,192.168.241.110,192.168.241.123
>> auth_cluster_required = cephx
>> auth_service_required = cephx
>> auth_client_required = cephx
>> filestore_xattr_use_omap = true
>> osd_pool_default_size = 2
>> osd_pool_default_min_size = 1
>
> Have you verified (ceph osd get size / min_size) that all your
> pools are actually set like this?
>
> Regards,
>
> Christian
>> mon_clock_drift_allowed = 2
>>
>> root@ceph1-node:~#cat crush-map.txt
>> # begin crush
>> map tunable choose_local_tries
>> 0 tunable choose_local_fallback_tries
>> 0 tunable choose_total_tries
>> 50 tunable chooseleaf_descend_once
>> 1 tunable straw_calc_version
>> 1
>> #
>> devices device 0
>> osd.0 device 1
>> osd.1 device 2
>> osd.2 device 3
>> osd.3 device 4
>> osd.4 device 5
>> osd.5
>> #
>> types type 0
>> osd type 1
>> host type 2
>> chassis type 3
>> rack type 4
>> row type 5
>> pdu type 6
>> pod type 7
>> room type 8
>> datacenter type 9
>> region type 10
>> root
>> #
>> buckets host ceph1-node
>> { id -2 # do not change
>> unnecessarily # weight
>> 0.060 alg
>> straw hash 0 #
>> rjenkins1 item osd.0 weight
>> 0.020 item osd.1 weight
>> 0.020 item osd.2 weight
>> 0.020 }
>> host ceph2-node
>> { id -3 # do not change unnecessarily
>> # weight 0.060
>> alg straw
>> hash 0 # rjenkins1
>> item osd.3 weight 0.020
>> item osd.4 weight 0.020
>> item osd.5 weight 0.020
>> }
>> root
[ceph-users] Falls cluster then one node switch off
Hello! I have a cluster of 2 nodes with 3 OSD each. The cluster full about 80%. df -H /dev/sdc127G 24G 3.9G 86% /var/lib/ceph/osd/ceph-1 /dev/sdd127G 20G 6.9G 75% /var/lib/ceph/osd/ceph-2 /dev/sdb127G 24G 3.5G 88% /var/lib/ceph/osd/ceph-0 When I switch off one server, then after 10 minutes begins remapped pgs ceph -w 2016-05-24 09:37:00.065134 mon.0 [INF] pgmap v4850: 704 pgs: 469 active+degraded, 186 active+remapped, 26 active+recovery_wait+remapped, 23 active+recovering+remapped; 56038 MB data, 65904 MB used, 10854 MB / 76759 MB avail; 11787/28038 objects degraded (42.039%); 6438 kB/s, 1 objects/s recovering 2016-05-24 09:37:03.889165 mon.0 [INF] pgmap v4851: 704 pgs: 469 active+degraded, 186 active+remapped, 26 active+recovery_wait+remapped, 23 active+recovering+remapped; 56038 MB data, 65928 MB used, 10830 MB / 76759 MB avail; 11786/28038 objects degraded (42.036%); 5753 kB/s, 1 objects/s recovering 2016-05-24 09:37:08.027062 osd.0 [WRN] OSD near full (90%) 2016-05-24 09:37:09.713479 mon.0 [INF] pgmap v4853: 704 pgs: 469 active+degraded, 186 active+remapped, 26 active+recovery_wait+remapped, 23 active+recovering+remapped; 56038 MB data, 65556 MB used, 11203 MB / 76759 MB avail; 11776/28038 objects degraded (42.000%); 7158 kB/s, 1 objects/s recovering 2016-05-24 09:37:16.232069 mon.0 [INF] pgmap v4854: 704 pgs: 469 active+degraded, 186 active+remapped, 26 active+recovery_wait+remapped, 23 active+recovering+remapped; 56038 MB data, 65626 MB used, 11133 MB / 76759 MB avail; 11773/28038 objects degraded (41.989%); 5273 kB/s, 1 objects/s recovering As a result, one disk overflow and the cluster falls. Why ceph remapped pgs, it was supposed to simply mark all pgs as active+degraded, while second node down? ceph version 0.80.11 root@ceph1-node:~# cat /etc/ceph/ceph.conf [global] fsid = b66c7daa-d6d8-46c7-9e61-15adbb749ed7 mon_initial_members = ceph1-node, ceph2-node, ceph-mon2 mon_host = 192.168.241.97,192.168.241.110,192.168.241.123 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd_pool_default_size = 2 osd_pool_default_min_size = 1 mon_clock_drift_allowed = 2 root@ceph1-node:~#cat crush-map.txt # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable straw_calc_version 1 # devices device 0 osd.0 device 1 osd.1
[ceph-users] active+clean+scrubbing+deep
Hi!
I have ceph version 0.94.1.
root@ceph-node1:~# ceph -s
cluster 3e0d58cd-d441-4d44-b49b-6cff08c20abf
health HEALTH_OK
monmap e2: 3 mons at
{ceph-mon=10.10.100.3:6789/0,ceph-node1=10.10.100.1:6789/0,ceph-node2=10.10.100.2:6789/0}
election epoch 428, quorum 0,1,2 ceph-node1,ceph-node2,ceph-mon
osdmap e978: 16 osds: 16 up, 16 in
pgmap v6735569: 2012 pgs, 8 pools, 2801 GB data, 703 kobjects
5617 GB used, 33399 GB / 39016 GB avail
2011 active+clean
1 active+clean+scrubbing+deep
client io 174 kB/s rd, 30641 kB/s wr, 80 op/s
root@ceph-node1:~# ceph pg dump | grep -i deep | cut -f 1
dumped all in format plain
pg_stat
19.b3
In log file i see
2015-05-14 03:23:51.556876 7fc708a37700 0 log_channel(cluster) log [INF] :
19.b3 deep-scrub starts
but no 19.b3 deep-scrub ok
then i do ceph pg deep-scrub 19.b3, nothing happens and in logs file no any
records about it.
What can i do to pg return in active + clean station?
is there any sense restart OSD or the entirely server where the OSD?
Thanks.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] active+clean+scrubbing+deep
Thank Irek, it really worked 02.06.2015, 15:58, "Irek Fasikhov" malm...@gmail.com:Hi. Restart the OSD. :)2015-06-02 11:55 GMT+03:00 Никитенко Виталий v1...@yandex.ru:Hi! I have ceph version 0.94.1. root@ceph-node1:~# ceph -s cluster 3e0d58cd-d441-4d44-b49b-6cff08c20abf health HEALTH_OK monmap e2: 3 mons at {ceph-mon=10.10.100.3:6789/0,ceph-node1=10.10.100.1:6789/0,ceph-node2=10.10.100.2:6789/0} election epoch 428, quorum 0,1,2 ceph-node1,ceph-node2,ceph-mon osdmap e978: 16 osds: 16 up, 16 in pgmap v6735569: 2012 pgs, 8 pools, 2801 GB data, 703 kobjects 5617 GB used, 33399 GB / 39016 GB avail 2011 active+clean 1 active+clean+scrubbing+deep client io 174 kB/s rd, 30641 kB/s wr, 80 op/s root@ceph-node1:~# ceph pg dump | grep -i deep | cut -f 1 dumped all in format plain pg_stat 19.b3 In log file i see 2015-05-14 03:23:51.556876 7fc708a37700 0 log_channel(cluster) log [INF] : 19.b3 deep-scrub starts but no "19.b3 deep-scrub ok" then i do "ceph pg deep-scrub 19.b3", nothing happens and in logs file no any records about it. What can i do to pg return in "active + clean" station? is there any sense restart OSD or the entirely server where the OSD? Thanks. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- С уважением, Фасихов Ирек НургаязовичМоб.: +79229045757___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] rbd loaded 100%
Hi! I have a server (ceph version 0.80.7, links 10Gb), there is set: 1 pool is write to 5 osd. I'm using the iscsi-target write to this pool (disk rbd3) some data from other server. And speed on network is near 150 Mbit / sec. In this case, iostat shows the usage rbd3 drive 100%, but drives on which there are 5 osd (sdc sdd sde sdf sdg) loaded in the region of 20% each. Who knows why this could be and what i can run the utility for the diagnosis? iostat -x 1 avg-cpu: %user %nice %system %iowait %steal %idle 1.800.001.460.710.00 96.03 Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.000.000.00 0.00 0.00 0.00 0.000.000.000.00 0.00 0.00 sdb 0.00 9.000.006.00 0.0068.0022.67 0.000.670.000.67 0.67 0.40 sdc 0.00 2.000.00 33.00 0.00 7756.00 470.06 2.76 83.760.00 83.76 5.45 18.00 sdd 0.00 0.000.00 59.00 0.00 9236.00 313.08 0.579.690.009.69 6.58 38.80 sde 0.00 0.000.00 29.00 0.00 5112.00 352.55 0.43 13.930.00 13.93 7.03 20.40 sdf 0.00 0.000.00 28.00 0.00 4612.00 329.43 0.269.140.009.14 6.57 18.40 sdg 0.00 0.000.00 24.00 0.00 4032.00 336.00 0.228.670.008.67 6.67 16.00 rbd0 0.00 0.000.000.00 0.00 0.00 0.00 0.000.000.000.00 0.00 0.00 rbd1 0.00 0.000.000.00 0.00 0.00 0.00 0.000.000.000.00 0.00 0.00 rbd2 0.00 0.000.000.00 0.00 0.00 0.00 0.000.000.000.00 0.00 0.00 rbd3 0.00 0.000.00 318.00 0.00 20045.00 126.07 7.28 28.290.00 28.29 3.13 99.60 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] iscsi and cache pool
Good day! I have server with Ubunu 14.04 and installed ceph firefly. Configured main_pool (2 osd) and ssd_pool (1 ssd osd). I want use ssd_pool as cache pool for main_pool ceph osd tier add main_pool ssd_pool ceph osd tier cache-mode ssd_pool writeback ceph osd tier set-overlay main_pool ssd_pool ceph osd pool set ssd_pool hit_set_type bloom ceph osd pool set ssd_pool hit_set_count 1 ceph osd pool set ssd_pool hit_set_period 600 ceph osd pool set ssd_pool target_max_bytes 1000 If use tgt as: tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 --bstype rbd --backing-store main_pool/store_main --bsopts conf=/etc/ceph/ceph.conf and then connected from iscsi initiator to this Lun1, i see that ssd_pool is used as cache (i see through iostat -x 1) but slow speed If use tgt as (or other sush as scst, iscsitarget): tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 -b /dev/rbd1 (where rbd1=main_pool/store_main) and then connected from iscsi initiator to this Lun1, i see that ssd_pool is not used, that write through to 2 osd Help me, anyone work this iscsi and cache pool? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] iscsi and cache pool
Thank you. try to do it 02.07.2014, 05:30, Gregory Farnum g...@inktank.com: Yeah, the features are new from January or something so you need a very new kernel to support it. There are no options to set. But in general I wouldn't use krbd if you can use librbd instead; it's easier to update and more featureful! -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Tue, Jul 1, 2014 at 5:44 PM, Никитенко Виталий v1...@yandex.ru wrote: Hi! There is some option in the kernel which must be included, or just upgrade to the latest version of the kernel? I use 3.13.0-24 Thanks 01.07.2014, 20:17, Gregory Farnum g...@inktank.com: It looks like you're using a kernel RBD mount in the second case? I imagine your kernel doesn't support caching pools and you'd need to upgrade for it to work. -Greg On Tuesday, July 1, 2014, Никитенко Виталий v1...@yandex.ru wrote: Good day! I have server with Ubunu 14.04 and installed ceph firefly. Configured main_pool (2 osd) and ssd_pool (1 ssd osd). I want use ssd_pool as cache pool for main_pool ceph osd tier add main_pool ssd_pool ceph osd tier cache-mode ssd_pool writeback ceph osd tier set-overlay main_pool ssd_pool ceph osd pool set ssd_pool hit_set_type bloom ceph osd pool set ssd_pool hit_set_count 1 ceph osd pool set ssd_pool hit_set_period 600 ceph osd pool set ssd_pool target_max_bytes 1000 If use tgt as: tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 --bstype rbd --backing-store main_pool/store_main --bsopts conf=/etc/ceph/ceph.conf and then connected from iscsi initiator to this Lun1, i see that ssd_pool is used as cache (i see through iostat -x 1) but slow speed If use tgt as (or other sush as scst, iscsitarget): tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 -b /dev/rbd1 (where rbd1=main_pool/store_main) and then connected from iscsi initiator to this Lun1, i see that ssd_pool is not used, that write through to 2 osd Help me, anyone work this iscsi and cache pool? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Software Engineer #42 @ http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Low write speed
Hi, Wido den Hollander Good day! Please help me solve the problem. There are the following scheme : Server ESXi with 1Gb NICs. it has local store store2Tb and two isci storage connected to the second server . The second server supermicro: two 1TB hdd (lsi 9261-8i with battery), 8 CPU cores, 32 GB RAM and 2 1Gb NICs . On /dev/sda installed ubuntu 12 and ceph-emperor. /dev/sdb disk placed under osd.0. How do you do journaling? When i create osd i see: NFO:ceph-disk:Will colocate journal with data on /dev/sdb Have you tried TGT instead? I tried tgt (with --bstype rbd) and result same. Have you also tried to run a rados benchmark? (rados bench) rados bench -p my_pool 300 write Total time run: 30.821284 Total writes made: 371 Write size: 4194304 Bandwidth (MB/sec): 48.149 Stddev Bandwidth: 38.1729 Max bandwidth (MB/sec): 116 Min bandwidth (MB/sec): 0 Average Latency:1.31857 Stddev Latency: 1.6014 Max latency:9.2685 Min latency:0.013897 Also, be aware that Ceph excels in it's parallel performance. You shouldn't look at the performance of a single LUN or RBD image that much, it's much more interesting to see the aggegrated performance of 10 or maybe 100 LUNs together. I dont understand how do it. I must create 10 LUN and 10 iscsi storage on esxi, and then test migrate 10 VM from local store to iscsi storage? Thanks! Vitaliy ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Low write speed
Hi, Ирек Нургаязович. Number of PGS is default pg_num 64 pgp_num 64 17.01.2014, 14:39, Ирек Фасихов malm...@gmail.com: Hi, Виталий. Whether a sufficient number of PGS? 2014/1/17 Никитенко Виталий v1...@yandex.ru Good day! Please help me solve the problem. There are the following scheme : Server ESXi with 1Gb NICs. it has local store store2Tb and two isci storage connected to the second server . The second server supermicro: two 1TB hdd (lsi 9261-8i with battery), 8 CPU cores, 32 GB RAM and 2 1Gb NICs . On /dev/sda installed ubuntu 12 and ceph-emperor. /dev/sdb disk placed under osd.0. What i do next: # rbd create esxi # rbd map esxi Get /dev/rbd1 which shared using iscsitarget # cat ietd.conf Target iqn.2014-01.ru.ceph: rados.iscsi.001 Lun 0 Path = / dev/rbd1, Type = blockio, ScsiId = f817ab Target iqn.2014-01.ru.ceph: rados.iscsi.002 Lun 1 Path = / opt/storlun0.bin, Type = fileio, ScsiId = lun1, ScsiSN = lun1 For test I also create iscsi storage on /dev/sda (Lun1). When migrating a virtual machine from store2Tb to Lun0 (ceph) the rate of migration of 400-450 Mbit/second. When migrating a VM from store2Tb to Lun1 (ubuntu file) then the rate of migration of 800-900 Mbit / second. From this I conclude that the rate is not limited by disk(controller) and not to the network. Tried osd format to ext4 and xfs and btrfs but same speed. For me, speed is very important , especially since the plan translate 10Gb network links. Thanks. Vitaliy ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- С уважением, Фасихов Ирек НургаязовичМоб.: +79229045757 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
