Thank you, I will attempt to play around with these settings and see if I can achieve better read performance.
Appreciate your insights. Peter On Tue, May 3, 2016 at 3:00 PM, Nick Fisk <n...@fisk.me.uk> wrote: > > > > -----Original Message----- > > From: Peter Kerdisle [mailto:peter.kerdi...@gmail.com] > > Sent: 03 May 2016 12:15 > > To: n...@fisk.me.uk > > Cc: ceph-users@lists.ceph.com > > Subject: Re: [ceph-users] Erasure pool performance expectations > > > > Hey Nick, > > > > Thanks for taking the time to answer my questions. Some in-line comments. > > > > On Tue, May 3, 2016 at 10:51 AM, Nick Fisk <n...@fisk.me.uk> wrote: > > Hi Peter, > > > > > > > -----Original Message----- > > > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf > > Of > > > Peter Kerdisle > > > Sent: 02 May 2016 08:17 > > > To: ceph-users@lists.ceph.com > > > Subject: [ceph-users] Erasure pool performance expectations > > > > > > Hi guys, > > > > > > I am currently testing the performance of RBD using a cache pool and a > 4/2 > > > erasure profile pool. > > > > > > I have two SSD cache servers (2 SSDs for journals, 7 SSDs for data) > with > > > 2x10Gbit bonded each and a six OSD nodes with a 10Gbit public and > 10Gbit > > > cluster network for the erasure pool (10x3TB without separate journal). > > This > > > is all on Jewel. > > > > > > What I would like to know is if the performance I'm seeing is to be > > expected > > > and if there is some way to test this in a more qualifiable way. > > > > > > Everything works as expected if the files are present on the cache > pool, > > > however when things need to be retrieved from the cache pool I see > > > performance degradation. I'm trying to simulate real usage as much as > > > possible and trying to retrieve files from the RBD volume over FTP > from a > > > client server. What I'm seeing is that the FTP transfer will stall for > seconds > > at a > > > time and then get some more data which results in an average speed of > > > 200KB/s. From the cache this is closer to 10MB/s. Is this the expected > > > behaviour from a erasure coded tier with cache in front? > > > > Unfortunately yes. The whole Erasure/Cache thing only really works well > if > > the data in the EC tier is only accessed infrequently, otherwise the > overheads > > in cache promotion/flushing quickly brings the cluster down to its knees. > > However it looks as though you are mainly doing reads, which means you > can > > probably alter your cache settings to not promote so aggressively on > reads, > > as reads can be proxied through to the EC tier instead of promoting. This > > should reduce the amount of required cache promotions. > > > > You are correct that reads have a lower priority of being cached, only > when > > they are used very frequently should this be done in an ideal situation. > > > > > > Can you try setting min_read_recency_for promote to something higher? > > > > I looked into the setting before but I must admit it's exact purpose > eludes me > > still. Would it be correct to simplify it as > 'min_read_recency_for_promote > > determines the amount of times a piece would have to be read in a certain > > interval (set by hit_set_period) in order to promote it to the caching > tier' ? > > Yes that’s correct. Every hit_set_period (assuming there is IO going on) a > new hitset is created up until the hit_set_count limit. The recency defines > how many of the last x hitsets an object must have been accessed in. > > Tuning it is a bit of a dark art at the moment as you have to try and get > all the values to match your workload. For starters try something like > > Read recency = 2 or 3 > Hit_set_count =10 > Hit_set_period=60 > > Which will mean if an object is read more than 2 or 3 times in a row > within the last few minutes it will be promoted. There is no granularity > below a single hitset, so if an object gets hit a 1000 times in 1 minute > but then nothing for 5 minutes it will not cause a promotion. > > > > > > > Also can you check what your hit_set_period and hit_set_count is > currently > > set to. > > > > hit_set_count is set to 1 and hit_set_period to 1800. > > > > What would increasing the hit_set_count do exactly? > > > > > > > > > Right now I'm unsure how to scientifically test the performance > retrieving > > > files when there is a cache miss. If somebody could point me towards a > > > better way of doing that I would appreciate the help. > > > > > > An other thing is that I'm seeing a lot of messages popping up in > dmesg on > > > my client server on which the RBD volumes are mounted. (IPs removed) > > > > > > [685881.477383] libceph: osd50 :6800 socket closed (con state OPEN) > > > [685895.597733] libceph: osd54 :6808 socket closed (con state OPEN) > > > [685895.663971] libceph: osd54 :6808 socket closed (con state OPEN) > > > [685895.710424] libceph: osd54 :6808 socket closed (con state OPEN) > > > [685895.749417] libceph: osd54 :6808 socket closed (con state OPEN) > > > [685896.517778] libceph: osd54 :6808 socket closed (con state OPEN) > > > [685906.690445] libceph: osd74 :6824 socket closed (con state OPEN) > > > > > > Is this a symptom of something? > > > > This is just stale connections to the OSD's timing out after the idle > period and > > is nothing to worry about. > > > > Glad to hear that, I was fearing something might be wrong. > > > > Thanks again. > > > > Peter > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
