So nobody has any clue on this one ???
Should I go with this one to dev mailing list ?
> On 27 Aug 2017, at 01:49, Tomasz Kusmierz <tom.kusmi...@gmail.com> wrote:
>
> Hi,
> for purposes of experimenting I’m running a home cluster that consists of
> single node and 4 OSD (weights in crush map are true to actual hdd size). I
> prefer to test all new stuff on home equipment before getting egg in the face
> at work :)
> Anyway recently I’ve upgrade to Luminous, and replaced my ancient 8x 2TB
> drives with 2x 8TB drives (with hopes of getting more in near future). While
> doing that I’ve converted everything to bluestore. while still on 12.1.1
>
> Everything was running smooth and performance was good (for ceph).
>
> I’ve decided to upgrade recently to 12.1.2 and this is where everything
> started acting up. I’m aware that
> - single node cluster is not a cluster
> - in the end I might need more OSD (old joke right ?)
> - I need to switch from spinning rust to SSD
>
> Before upgrade my “cluster” was only switching to WRN only when I was pumping
> a lot of data into it and it would just come up with “slow requests” stuff.
> Now while completely static, not doing anything (no read, no write) OSD’s are
> committing suicide due to timeout, also before they will commit suicide I
> can’t actually access data from cluster, which make me think that while
> performing a scrub those are unaccessible. Bellow I’ll attach a log excerpt
> just please notice that it happens on deep scrub and normal scrub as well.
>
> After I’ve discovered that I’ve tried to play around with sysctl.conf and
> with ceph.conf ( up to this point sysctl.conf was stock, and ceph.conf was
> just adjusted to allow greater OSD full capacity and disable cephx to speed
> it up)
>
> also I’m running 3 pools on top of this cluster (all three have size = 2
> min_size = 2):
> cephfs_data pg=256 (99.99% of data used in cluster)
> cephfs_metadata pg=4 (0.01% of data used in cluster)
> rbd pg=8 but this pool contains no data and I’m considering removing it since
> in my use case I’ve got nothing for it.
>
> Please note that while this logs were produced cephFS was not even mounted :/
>
>
>
> FYI hardware is old and trusted hp proliant DL180 G6 with 2 xeons @2.2GHz
> giving 16 cores and 32GB or ECC ram and LSI in HBA mode (2x 6GB SAS)
>
>
>
> (
> As a side issue could somebody explain to my why with bluestore that was
> supposed to cure cancer write performance still sucks ? I know that filestore
> did suffer from writing everything multiple times to same drive, and I did
> experience this first hand when after exhausting journals it was just dead
> slow, but now while within same host in my current configuration it keeps
> choking [flaps 70MB/s -> 10 MB/s -> 70MB/s] and I never seen it even approach
> speed of single slowest drive. This server is not a speed daemon, I know, but
> when performing a simultaneous read / write for those drives I was getting
> around 760MB/s sequential R/W speed.
> Right now I’m struggling to comprehend where the bottleneck is while
> performing operations within same host ?! network should not be an issue
> (correct me if I’m wrong here), dumping a singular blob into pool should
> produce a nice long sequence of object placed into drives …
> I’m just puzzled why ceph will not exceed combined 40MB/s while still
> switching “cluster” into warning state due to “slow responses”
> 2017-08-24 20:49:34.457191 osd.8 osd.8 192.168.1.240:6814/3393 503 : cluster
> [WRN] slow request 63.878717 seconds old, received at 2017-08-24
> 20:48:30.578398: osd_op(client.994130.1:13659 1.9700016d
> 1:b68000e9:::100000ffeef.00000068:head [write 0~4194304 [1@-1]] snapc 1=[]
> ondisk+write+known_if_redirected e4306) currently waiting for active
> 2017-08-24 20:49:34.457195 osd.8 osd.8 192.168.1.240:6814/3393 504 : cluster
> [WRN] slow request 64.177858 seconds old, received at 2017-08-24
> 20:48:30.279257: osd_op(client.994130.1:13568 1.b95e13a4
> 1:25c87a9d:::100000ffeef.0000000d:head [write 0~4194304 [1@-1]] snapc 1=[]
> ondisk+write+known_if_redirected e4306) currently waiting for active
> 2017-08-24 20:49:34.457198 osd.8 osd.8 192.168.1.240:6814/3393 505 : cluster
> [WRN] slow request 64.002653 seconds old, received at 2017-08-24
> 20:48:30.454463: osd_op(client.994130.1:13626 1.b426420e
> 1:7042642d:::100000ffeef.00000047:head [write 0~4194304 [1@-1]] snapc 1=[]
> ondisk+write+known_if_redirected e4306) currently waiting for active
> 2017-08-24 20:49:34.457200 osd.8 osd.8 192.168.1.240:6814/3393 506 : cluster
> [WRN] slow request 63.873519 seconds old, received at 2017-08-24
> 20:48:30.583596: osd_op(client.994130.1:13661 1.31551a8
> 1:158aa8c0:::100000ffeef.0000006a:head [write 0~4194304 [1@-1]] snapc 1=[]
> ondisk+write+known_if_redirected e4306) currently waiting for active
> 2017-08-24 20:49:34.457206 osd.8 osd.8 192.168.1.240:6814/3393 507 : cluster
> [WRN] slow request 64.171785 seconds old, received at 2017-08-24
> 20:48:30.285330: osd_op(client.994130.1:13570 1.49610cc6
> 1:63308692:::100000ffeef.0000000f:head [write 0~4194304 [1@-1]] snapc 1=[]
> ondisk+write+known_if_redirected e4306) currently waiting for active
> )
>
>
> So some data on main issue:
>
>
> FAIL ON DEEP SCRUB
> 2017-08-26 01:50:04.421944 osd.0 osd.0 192.168.1.240:6814/12991 7 : cluster
> [INF] 6.5 scrub ok
> 2017-08-26 01:50:09.331095 osd.0 osd.0 192.168.1.240:6814/12991 8 : cluster
> [INF] 1.1c scrub starts
> 2017-08-26 01:51:03.339509 osd.0 osd.0 192.168.1.240:6814/12991 9 : cluster
> [INF] 1.1c scrub ok
> 2017-08-26 02:21:00.706695 osd.10 osd.10 192.168.1.240:6806/21564 7 : cluster
> [INF] 1.d1 scrub starts
> 2017-08-26 02:21:34.066183 osd.10 osd.10 192.168.1.240:6806/21564 8 : cluster
> [INF] 1.d1 scrub ok
> 2017-08-26 02:21:56.943046 osd.8 osd.8 192.168.1.240:6810/22002 7 : cluster
> [INF] 1.17 scrub starts
> 2017-08-26 02:23:06.341409 osd.8 osd.8 192.168.1.240:6810/22002 8 : cluster
> [INF] 1.17 scrub ok
> 2017-08-26 02:35:51.099649 osd.8 osd.8 192.168.1.240:6810/22002 9 : cluster
> [INF] 1.89 scrub starts
> 2017-08-26 02:36:42.605600 osd.8 osd.8 192.168.1.240:6810/22002 10 : cluster
> [INF] 1.89 scrub ok
> 2017-08-26 02:38:27.132698 osd.8 osd.8 192.168.1.240:6810/22002 11 : cluster
> [INF] 1.ce scrub starts
> 2017-08-26 02:38:49.820489 osd.8 osd.8 192.168.1.240:6810/22002 12 : cluster
> [INF] 1.ce scrub ok
> 2017-08-26 03:23:27.619669 osd.8 osd.8 192.168.1.240:6810/22002 13 : cluster
> [INF] 1.8c scrub starts
> 2017-08-26 03:23:49.679403 osd.8 osd.8 192.168.1.240:6810/22002 14 : cluster
> [INF] 1.8c scrub ok
> 2017-08-26 03:32:19.475812 osd.0 osd.0 192.168.1.240:6814/12991 10 : cluster
> [INF] 1.d4 deep-scrub starts
> 2017-08-26 03:38:46.708163 mon.0 mon.0 192.168.1.240:6789/0 1201 : cluster
> [INF] osd.0 failed (root=default,host=proxmox1) (connection refused reported
> by osd.8)
> 2017-08-26 03:38:46.759470 mon.0 mon.0 192.168.1.240:6789/0 1207 : cluster
> [WRN] Health check failed: 1 osds down (OSD_DOWN)
> 2017-08-26 03:38:49.820122 mon.0 mon.0 192.168.1.240:6789/0 1212 : cluster
> [WRN] Health check failed: Reduced data availability: 12 pgs inactive
> (PG_AVAILABILITY)
> 2017-08-26 03:38:49.820165 mon.0 mon.0 192.168.1.240:6789/0 1213 : cluster
> [WRN] Health check failed: Degraded data redundancy: 292260/3786364 objects
> degraded (7.719%), 38 pgs unclean, 38 pgs degraded (PG_DEGRADED)
> 2017-08-26 03:38:51.088934 mon.0 mon.0 192.168.1.240:6789/0 1214 : cluster
> [WRN] Health check update: Reduced data availability: 16 pgs inactive
> (PG_AVAILABILITY)
> 2017-08-26 03:38:51.088975 mon.0 mon.0 192.168.1.240:6789/0 1215 : cluster
> [WRN] Health check update: Degraded data redundancy: 392568/3786364 objects
> degraded (10.368%), 52 pgs unclean, 52 pgs degraded (PG_DEGRADED)
> 2017-08-26 03:38:53.090178 mon.0 mon.0 192.168.1.240:6789/0 1216 : cluster
> [WRN] Health check update: Reduced data availability: 29 pgs inactive
> (PG_AVAILABILITY)
> 2017-08-26 03:38:53.090216 mon.0 mon.0 192.168.1.240:6789/0 1217 : cluster
> [WRN] Health check update: Degraded data redundancy: 592033/3786364 objects
> degraded (15.636%), 82 pgs unclean, 82 pgs degraded (PG_DEGRADED)
> 2017-08-26 03:39:37.928816 mon.0 mon.0 192.168.1.240:6789/0 1220 : cluster
> [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
> 2017-08-26 03:39:37.941007 mon.0 mon.0 192.168.1.240:6789/0 1221 : cluster
> [INF] osd.0 192.168.1.240:6814/15727 boot
> 2017-08-26 03:39:39.949551 mon.0 mon.0 192.168.1.240:6789/0 1226 : cluster
> [WRN] Health check update: Degraded data redundancy: 436309/3786364 objects
> degraded (11.523%), 82 pgs unclean, 60 pgs degraded (PG_DEGRADED)
> 2017-08-26 03:39:41.974996 mon.0 mon.0 192.168.1.240:6789/0 1227 : cluster
> [WRN] Health check update: Degraded data redundancy: 379236/3786364 objects
> degraded (10.016%), 74 pgs unclean, 52 pgs degraded (PG_DEGRADED)
> 2017-08-26 03:39:43.120495 mon.0 mon.0 192.168.1.240:6789/0 1228 : cluster
> [WRN] Health check update: Degraded data redundancy: 22 pgs unclean
> (PG_DEGRADED)
> 2017-08-26 03:39:43.120534 mon.0 mon.0 192.168.1.240:6789/0 1229 : cluster
> [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability:
> 29 pgs inactive)
> 2017-08-26 03:39:45.121340 mon.0 mon.0 192.168.1.240:6789/0 1230 : cluster
> [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22
> pgs unclean)
> 2017-08-26 03:39:45.121386 mon.0 mon.0 192.168.1.240:6789/0 1231 : cluster
> [INF] Cluster is now healthy
> 2017-08-26 03:40:11.568499 osd.10 osd.10 192.168.1.240:6806/21564 9 : cluster
> [INF] 1.b5 scrub starts
> 2017-08-26 03:40:51.874519 osd.10 osd.10 192.168.1.240:6806/21564 10 :
> cluster [INF] 1.b5 scrub ok
> 2017-08-26 03:41:15.794026 osd.8 osd.8 192.168.1.240:6810/22002 15 : cluster
> [INF] 1.77 scrub starts
> 2017-08-26 03:42:19.561924 osd.8 osd.8 192.168.1.240:6810/22002 16 : cluster
> [INF] 1.77 scrub ok
> 2017-08-26 03:42:30.895351 osd.0 osd.0 192.168.1.240:6814/15727 1 : cluster
> [INF] 1.d4 deep-scrub starts
> 2017-08-26 03:42:30.842869 osd.8 osd.8 192.168.1.240:6810/22002 17 : cluster
> [INF] 1.12 scrub starts
> 2017-08-26 03:43:15.478366 osd.8 osd.8 192.168.1.240:6810/22002 18 : cluster
> [INF] 1.12 scrub ok
> 2017-08-26 03:47:17.962016 osd.0 osd.0 192.168.1.240:6814/15727 2 : cluster
> [INF] 1.d4 deep-scrub ok
> 2017-08-26 03:48:30.668792 osd.10 osd.10 192.168.1.240:6806/21564 11 :
> cluster [INF] 1.1 scrub starts
> 2017-08-26 03:49:05.879546 osd.10 osd.10 192.168.1.240:6806/21564 12 :
> cluster [INF] 1.1 scrub ok
> 2017-08-26 03:50:53.709500 osd.10 osd.10 192.168.1.240:6806/21564 13 :
> cluster [INF] 1.9d scrub starts
> 2017-08-26 03:52:13.278975 osd.10 osd.10 192.168.1.240:6806/21564 14 :
> cluster [INF] 1.9d scrub ok
> 2017-08-26 04:31:37.144944 osd.10 osd.10 192.168.1.240:6806/21564 15 :
> cluster [INF] 1.82 scrub starts
> 2017-08-26 04:32:35.917646 osd.10 osd.10 192.168.1.240:6806/21564 16 :
> cluster [INF] 1.82 scrub ok
> 2017-08-26 04:33:03.930435 osd.9 osd.9 192.168.1.240:6802/32411 36 : cluster
> [INF] 1.f4 scrub starts
> 2017-08-26 04:34:08.360134 osd.9 osd.9 192.168.1.240:6802/32411 37 : cluster
> [INF] 1.f4 scrub ok
>
>
>
> FAIL ON NORMAL SCRUB
> 2017-08-25 23:28:55.310602 osd.8 osd.8 192.168.1.240:6806/2820 29 : cluster
> [INF] 6.3 deep-scrub starts
> 2017-08-25 23:28:55.415144 osd.8 osd.8 192.168.1.240:6806/2820 30 : cluster
> [INF] 6.3 deep-scrub ok
> 2017-08-25 23:29:01.273979 osd.8 osd.8 192.168.1.240:6806/2820 31 : cluster
> [INF] 1.d2 scrub starts
> 2017-08-25 23:30:47.518484 osd.8 osd.8 192.168.1.240:6806/2820 32 : cluster
> [INF] 1.d2 scrub ok
> 2017-08-25 23:31:40.311045 osd.8 osd.8 192.168.1.240:6806/2820 33 : cluster
> [INF] 1.6e scrub starts
> 2017-08-25 23:32:22.150274 osd.8 osd.8 192.168.1.240:6806/2820 34 : cluster
> [INF] 1.6e scrub ok
> 2017-08-25 23:32:58.297062 osd.9 osd.9 192.168.1.240:6802/7091 32 : cluster
> [INF] 1.d5 scrub starts
> 2017-08-25 23:35:19.285841 osd.9 osd.9 192.168.1.240:6802/7091 33 : cluster
> [INF] 1.d5 scrub ok
> 2017-08-25 23:36:38.375447 osd.8 osd.8 192.168.1.240:6806/2820 35 : cluster
> [INF] 1.3 scrub starts
> 2017-08-25 23:37:25.012116 osd.8 osd.8 192.168.1.240:6806/2820 36 : cluster
> [INF] 1.3 scrub ok
> 2017-08-25 23:38:29.406144 osd.8 osd.8 192.168.1.240:6806/2820 37 : cluster
> [INF] 1.45 scrub starts
> 2017-08-25 23:38:53.020365 mon.0 mon.0 192.168.1.240:6789/0 831 : cluster
> [INF] osd.9 failed (root=default,host=proxmox1) (connection refused reported
> by osd.8)
> 2017-08-25 23:38:53.166364 mon.0 mon.0 192.168.1.240:6789/0 832 : cluster
> [WRN] Health check failed: 1 osds down (OSD_DOWN)
> 2017-08-25 23:38:56.200767 mon.0 mon.0 192.168.1.240:6789/0 837 : cluster
> [WRN] Health check failed: Degraded data redundancy: 100309/3786338 objects
> degraded (2.649%), 14 pgs unclean, 14 pgs degraded (PG_DEGRADED)
> 2017-08-25 23:38:58.155562 mon.0 mon.0 192.168.1.240:6789/0 838 : cluster
> [WRN] Health check failed: Reduced data availability: 1 pg inactive
> (PG_AVAILABILITY)
> 2017-08-25 23:38:58.155601 mon.0 mon.0 192.168.1.240:6789/0 839 : cluster
> [WRN] Health check update: Degraded data redundancy: 715775/3786338 objects
> degraded (18.904%), 101 pgs unclean, 102 pgs degraded (PG_DEGRADED)
> 2017-08-25 23:39:30.172451 mon.0 mon.0 192.168.1.240:6789/0 840 : cluster
> [WRN] Health check update: Degraded data redundancy: 715775/3786338 objects
> degraded (18.904%), 102 pgs unclean, 102 pgs degraded (PG_DEGRADED)
> 2017-08-25 23:39:47.851497 mon.0 mon.0 192.168.1.240:6789/0 843 : cluster
> [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
> 2017-08-25 23:39:47.864774 mon.0 mon.0 192.168.1.240:6789/0 844 : cluster
> [INF] osd.9 192.168.1.240:6802/32411 boot
> 2017-08-25 23:39:50.876761 mon.0 mon.0 192.168.1.240:6789/0 849 : cluster
> [WRN] Health check update: Degraded data redundancy: 672540/3786338 objects
> degraded (17.762%), 96 pgs unclean, 96 pgs degraded (PG_DEGRADED)
> 2017-08-25 23:39:52.184954 mon.0 mon.0 192.168.1.240:6789/0 850 : cluster
> [WRN] Health check update: Degraded data redundancy: 476349/3786338 objects
> degraded (12.581%), 69 pgs unclean, 69 pgs degraded (PG_DEGRADED)
> 2017-08-25 23:39:50.533429 osd.0 osd.0 192.168.1.240:6814/16223 13 : cluster
> [INF] 1.80 scrub starts
> 2017-08-25 23:39:55.056537 mon.0 mon.0 192.168.1.240:6789/0 851 : cluster
> [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability:
> 1 pg inactive)
> 2017-08-25 23:39:55.056574 mon.0 mon.0 192.168.1.240:6789/0 852 : cluster
> [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy:
> 476349/3786338 objects degraded (12.581%), 69 pgs unclean, 69 pgs degraded)
> 2017-08-25 23:39:55.056591 mon.0 mon.0 192.168.1.240:6789/0 853 : cluster
> [INF] Cluster is now healthy
> 2017-08-25 23:40:17.806395 osd.0 osd.0 192.168.1.240:6814/16223 14 : cluster
> [INF] 1.80 scrub ok
> 2017-08-25 23:40:19.775012 osd.9 osd.9 192.168.1.240:6802/32411 1 : cluster
> [INF] 1.5a scrub starts
> 2017-08-25 23:40:46.458847 osd.9 osd.9 192.168.1.240:6802/32411 2 : cluster
> [INF] 1.5a scrub ok
> 2017-08-25 23:40:53.807218 osd.9 osd.9 192.168.1.240:6802/32411 3 : cluster
> [INF] 1.56 scrub starts
> 2017-08-25 23:41:16.197304 osd.9 osd.9 192.168.1.240:6802/32411 4 : cluster
> [INF] 1.56 scrub ok
> 2017-08-25 23:41:24.814502 osd.9 osd.9 192.168.1.240:6802/32411 5 : cluster
> [INF] 1.92 deep-scrub starts
> 2017-08-25 23:51:35.881952 osd.9 osd.9 192.168.1.240:6802/32411 6 : cluster
> [INF] 1.92 deep-scrub ok
> 2017-08-25 23:52:54.476268 osd.10 osd.10 192.168.1.240:6810/4355 39 : cluster
> [INF] 1.f2 scrub starts
> 2017-08-25 23:53:21.208291 osd.10 osd.10 192.168.1.240:6810/4355 40 : cluster
> [INF] 1.f2 scrub ok
> 2017-08-25 23:53:47.475879 osd.10 osd.10 192.168.1.240:6810/4355 41 : cluster
> [INF] 1.c8 deep-scrub starts
> 2017-08-26 00:01:08.611371 osd.10 osd.10 192.168.1.240:6810/4355 42 : cluster
> [INF] 1.c8 deep-scrub ok
> 20
>
>
> root@proxmox1:/# ceph pg dump | egrep -v '^(0\.|1\.|2\.|3\.)' | egrep -v
> '(^pool\ (0|1|2|3))' | column -t
> dumped all
> version 9678
> stamp 2017-08-27 01:27:53.321763
> last_osdmap_epoch 0
> last_pg_scan 0
> full_ratio 0
> nearfull_ratio 0
> PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED
> UNFOUND BYTES LOG DISK_LOG STATE STATE_STAMP VERSION
> REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY
> LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
> 6.4 0 0 0 0 0
> 0 0 0 active+clean 2017-08-26
> 22:55:06.289033 0'0 4725:2848 [10,8] 10 [10,8]
> 10 0'0 2017-08-26 22:55:06.288961 0'0
> 2017-08-20 23:32:51.270895
> 6.5 0 0 0 0 0
> 0 0 0 active+clean 2017-08-26
> 23:03:07.062129 0'0 4726:2101 [0,10] 0 [0,10]
> 0 0'0 2017-08-26 01:50:04.421951 0'0
> 2017-08-22 14:26:19.915612
> 6.6 0 0 0 0 0
> 0 0 0 active+clean 2017-08-27
> 00:46:09.548107 0'0 4726:2344 [10,9] 10 [10,9]
> 10 0'0 2017-08-27 00:46:09.548029 0'0
> 2017-08-24 13:08:56.447183
> 6.7 0 0 0 0 0
> 0 0 0 active+clean 2017-08-26
> 22:52:44.635393 0'0 4725:1481 [10,8] 10 [10,8]
> 10 0'0 2017-08-25 22:02:26.297723 0'0
> 2017-08-23 15:55:58.299570
> 6.3 0 0 0 0 0
> 0 0 0 active+clean 2017-08-26
> 22:52:44.632667 0'0 4725:1971 [8,10] 8 [8,10]
> 8 0'0 2017-08-25 23:28:55.415148 0'0
> 2017-08-25 23:28:55.415148
> 5.0 18661 0 0 0 0
> 12583538 1563 1563 active+clean 2017-08-26
> 22:03:03.809158 4652'1197298 4725:1382436 [10,9] 10 [10,9]
> 10 4623'1197263 2017-08-26 19:49:19.819627 4270'1161119
> 2017-08-20 02:04:03.373813
> 6.2 0 0 0 0 0
> 0 0 0 active+clean 2017-08-26
> 22:52:45.677622 0'0 4725:1440 [9,8] 9 [9,8]
> 9 0'0 2017-08-26 20:58:34.722865 0'0
> 2017-08-26 20:58:34.722865
> 5.1 18878 0 0 0 0
> 12583048 1573 1573 active+clean 2017-08-26
> 23:03:07.062298 4640'959478 4726:1131301 [0,8] 0 [0,8]
> 0 4596'958844 2017-08-26 13:47:19.329350 4393'956123
> 2017-08-25 09:32:09.556396
> 6.1 0 0 0 0 0
> 0 0 0 active+clean 2017-08-26
> 22:52:44.736333 0'0 4725:1615 [8,9] 8 [8,9]
> 8 0'0 2017-08-26 01:28:24.476136 0'0
> 2017-08-22 16:20:13.243273
> 5.2 18472 0 0 0 0
> 32462655 1592 1592 active+clean 2017-08-26
> 22:52:44.634997 4652'952265 4725:1174014 [10,8] 10 [10,8]
> 10 4652'952265 2017-08-26 22:45:06.916647 4270'930889
> 2017-08-23 05:50:46.370503
> 6.0 0 0 0 0 0
> 0 0 0 active+clean 2017-08-26
> 23:03:07.061426 0'0 4726:2441 [10,0] 10 [10,0]
> 10 0'0 2017-08-26 21:59:03.746276 0'0
> 2017-08-23 02:26:18.206975
> 5.3 18512 0 0 0 0
> 10928869 1519 1519 active+clean 2017-08-26
> 23:03:07.062484 4639'984496 4726:1199339 [0,8] 0 [0,8]
> 0 4531'983789 2017-08-26 00:09:32.283691 4270'975964
> 2017-08-23 16:15:09.546043
> 5 74523 0 0 0 0
> 68558110 6247 6247
> 6 0 0 0 0 0
> 0 0 0
> 1 1821197 0 0 0 0
> 6962542387273 401319 401319
> sum 1895720 0 0 0 0
> 6962610945383 407566 407566
> OSD_STAT USED AVAIL TOTAL HB_PEERS
> PG_SUM PRIMARY_PG_SUM
> 8 4288G 3163G 7451G [0,9,10] 177
> 93
> 10 4240G 3211G 7451G [0,8,9] 175
> 93
> 0 1984G 809G 2794G [8,9,10] 82
> 37
> 9 2492G 1233G 3725G [0,8,10] 102
> 45
> sum 13005G 8418G 21424G
>
>
> root@proxmox1:~# ceph versions
> {
> "mon": {
> "ceph version 12.1.2 (cd7bc3b11cdbe6fa94324b7322fb2a4716a052a7)
> luminous (rc)": 1
> },
> "mgr": {
> "ceph version 12.1.2 (cd7bc3b11cdbe6fa94324b7322fb2a4716a052a7)
> luminous (rc)": 1
> },
> "osd": {
> "ceph version 12.1.2 (cd7bc3b11cdbe6fa94324b7322fb2a4716a052a7)
> luminous (rc)": 4
> },
> "mds": {
> "ceph version 12.1.2 (cd7bc3b11cdbe6fa94324b7322fb2a4716a052a7)
> luminous (rc)": 1
> },
> "overall": {
> "ceph version 12.1.2 (cd7bc3b11cdbe6fa94324b7322fb2a4716a052a7)
> luminous (rc)": 7
> }
> }
>
> Crush map:
> # begin crush map
> tunable choose_local_tries 0
> tunable choose_local_fallback_tries 0
> tunable choose_total_tries 50
> tunable chooseleaf_descend_once 1
> tunable chooseleaf_vary_r 1
> tunable chooseleaf_stable 1
> tunable straw_calc_version 1
> tunable allowed_bucket_algs 54
>
> # devices
> device 0 osd.0 class hdd
> device 1 device1
> device 2 device2
> device 3 device3
> device 4 device4
> device 5 device5
> device 6 device6
> device 7 device7
> device 8 osd.8 class hdd
> device 9 osd.9 class hdd
> device 10 osd.10 class hdd
>
> # types
> type 0 osd
> type 1 host
> type 2 chassis
> type 3 rack
> type 4 row
> type 5 pdu
> type 6 pod
> type 7 room
> type 8 datacenter
> type 9 region
> type 10 root
>
> # buckets
> host proxmox1 {
> id -2 # do not change unnecessarily
> id -3 class hdd # do not change unnecessarily
> # weight 20.922
> alg straw
> hash 0 # rjenkins1
> item osd.10 weight 7.277
> item osd.9 weight 3.639
> item osd.0 weight 2.729
> item osd.8 weight 7.277
> }
> root default {
> id -1 # do not change unnecessarily
> id -4 class hdd # do not change unnecessarily
> # weight 20.922
> alg straw
> hash 0 # rjenkins1
> item proxmox1 weight 20.922
> }
>
> # rules
> rule replicated_ruleset {
> id 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type osd
> step emit
> }
>
> # end crush map
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
