Hello, On Tue, 12 Jul 2016 11:01:30 +0200 Mateusz Skała wrote:
> Thank You for replay. Answers below. > > > -----Original Message----- > > From: Christian Balzer [mailto:ch...@gol.com] > > Sent: Tuesday, July 12, 2016 3:37 AM > > To: ceph-users@lists.ceph.com > > Cc: Mateusz Skała <mateusz.sk...@budikom.net> > > Subject: Re: [ceph-users] Cache Tier configuration > > > > > > Hello, > > > > On Mon, 11 Jul 2016 16:19:58 +0200 Mateusz Skała wrote: > > > > > Hello Cephers. > > > > > > Can someone help me in my cache tier configuration? I have 4 same SSD > > > drives 176GB (184196208K) in SSD pool, how to determine > > target_max_bytes? > > > > What exact SSD models are these? > > What version of Ceph? > > Intel DC S3610 (SSDSC2BX200G401), ceph version 9.2.1 > (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd) > Good, these are decent SSDs and at 3DWPD probably durable enough, too. You will want to monitor their wear-out level anyway, though. Remember, dead cache pool means unaccessible and/or lost data. Jewel has improved cache controls and a different, less aggressive default behavior, you may want to consider upgrading to it, especially if you don't want to become a cache tiering specialist. ^o^ Also Infernalis is no longer receiving updates. > > > I assume > > > that should be (4 drives* 188616916992 bytes )/ 3 replica = > > > 251489222656 bytes *85% (because of full disk warning) > > > > In theory correct, but you might want to consider (like with all pools) the > > impact of loosing a single SSD. > > In short, backfilling and then the remaining 3 getting full anyway. > > > > OK, so better to make lower max target bates than I have space? For example > 170GB? Then I will have 1 osd reserve. > Something like this, though failures with these SSDs are very unlikely. > > > It will be 213765839257 bytes ~200GB. I make this little bit lower > > > (160GB) and after some time whole cluster stops on full disk error. > > > One of SSD drives are full. I see that use of space at the osd is not > > > equal: > > > > > > 32 0.17099 1.00000 175G 127G 49514M 72.47 1.77 95 > > > > > > 42 0.17099 1.00000 175G 120G 56154M 68.78 1.68 90 > > > > > > 37 0.17099 1.00000 175G 136G 39670M 77.95 1.90 102 > > > > > > 47 0.17099 1.00000 175G 130G 46599M 74.09 1.80 97 > > > > > > > What's the exact error message? > > > > None of these are over 85 or 95%, how are they full? > > Osd.37 was full on 96%, after error (heath ERR, 1 full osd).Then I set > max_target_bytes on 100GB. Flushing reduced used space, now cluster is > working ok, but I want to clarify my configuration. > Don't get flushing (copying dirty objects to the backing pool) and eviction (deleting, really zero-ing, clean objects). Eviction is what frees up space, but it needs flushed (clean) objects to work with. > > > > > If the above is a snapshot of when Ceph thinks something is "full", it may > > be > > an indication that you've reached target_max_bytes and Ceph simply has no > > clean (flushed) objects ready to evict. > > Which means a configuration problem (all ratios, not the defaults, for this > > pool please) or your cache filling up faster than it can flush. > > > Above snapshot is at this time, when cluster Is working OK. Filling faster > than flushing is very possible, when the error become I have in config min > 'promote' set at 1, like this > > "osd_tier_default_cache_min_read_recency_for_promote": "1", > "osd_tier_default_cache_min_write_recency_for_promote": "1", > > Now I changed this to 3, and looks like is working, 3 days without near full > osd. > There are a number of other options to control things, especially with Jewel. Also setting your cache mode to readforward might be a good idea depending on your use case. > > Space is never equal with Ceph, you need a high enough number of PGs for > > starters and then some fine-tuning. > > > > After fiddling with the weights my cache-tier SSD OSDs are all very close to > > each other: > > --- > > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR > > 18 0.64999 1.00000 679G 543G 136G 79.96 4.35 > > 19 0.67000 1.00000 679G 540G 138G 79.61 4.33 > > 20 0.64999 1.00000 679G 534G 144G 78.70 4.28 > > 21 0.64999 1.00000 679G 536G 142G 79.03 4.30 > > 26 0.62999 1.00000 679G 540G 138G 79.57 4.33 > > 27 0.62000 1.00000 679G 538G 140G 79.30 4.32 > > 28 0.67000 1.00000 679G 539G 140G 79.35 4.32 > > 29 0.69499 1.00000 679G 536G 142G 78.96 4.30 > > --- > In Your snapshot used space is near equal, only 1% difference, I have near > 10% differences in used space. It depends on number of PG, or maybe weight? > As I wrote, both. 10% suggests that you probably already have enough PGs, time to fine-tune the weights, see the differences in my list above. > > > > > > > > > > > My setup: > > > > > > ceph --admin-daemon /var/run/ceph/ceph-osd.32.asok config show | grep > > > cache > > > > > > > > Nearly all of these are irrelevant, output of "ceph osd pool ls detail" > > please, at least for the cache pool. > > > ceph osd pool ls detail > pool 2 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash > rjenkins pg_num 2048 pgp_num 2048 last_change 68565 flags hashpspool > min_read_recency_for_promote 1 min_write_recency_for_promote 1 stripe_width 0 > removed_snaps [1~2,4~12,17~2e,46~ad,f9~2,fd~2,101~2] > pool 4 'ssd' replicated size 3 min_size 1 crush_ruleset 1 object_hash > rjenkins pg_num 128 pgp_num 128 last_change 68913 flags > hashpspool,incomplete_clones tier_of 5 cache_mode writeback target_bytes > 182536110080 hit_set bloom{false_positive_probability: 0.05, target_size: 0, > seed: 0} 600s x6 stripe_width 0 > removed_snaps [1~3,6~2,9~2,d~8,17~6,1f~10,33~8,3f~a,4d~2,55~22,79~2] > pool 5 'sata' replicated size 3 min_size 1 crush_ruleset 2 object_hash > rjenkins pg_num 128 pgp_num 128 last_change 68910 lfor 66807 flags hashpspool > tiers 4 read_tier 4 write_tier 4 stripe_width 0 > removed_snaps [1~3,6~2,9~2,d~8,17~6,1f~10,33~8,3f~a,4d~2,55~22,79~2] > I'd go for 256 PGs, how big (OSDs) is your "sata" pool? Christian > Cache tier on 'ssd' pool for 'sata' pool. > > > > > Have you read the documentation and my thread in this ML labeled "Cache > > tier operation clarifications"? > > I have read documentation and some Intel blog > (https://software.intel.com/en-us/blogs/2015/03/03/ceph-cache-tiering-introduction), > I will search now for Your post and read them. > > > > > > > > > Can someone help? Any ideas? It is normal that whole cluster stops at > > > disk full error on cache tier, I was thinking that only one of pools > > > can stops and other without cache tier should still work. > > > > > Once you activate a cache-tier it becomes for all intends and purposes the > > the pool it's caching for. > > So any problem with it will be fatal. > > OK. > > > > > Christian > > -- > > Christian Balzer Network/Systems Engineer > > ch...@gol.com Global OnLine Japan/Rakuten Communications > > http://www.gol.com/ > > Thank You for Your help. > Mateusz > > -- Christian Balzer Network/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
