Re: [ceph-users] Modification Time of RBD Images
Jason, sorry for the typo of your email address in my last mail... On 29/03/2017, 00:36, Jason Dillaman wrote: While certainly that could be a feature that could be added to "rbd info", it will take a while for this feature to reach full use since it would rely on new versions of librbd / krbd. Additionally, access and modified timestamps would require sending out an update notification so that other clients notice the change. You would also want to highly throttle any updates to the modification timestamp -- rendering it a rough approximation of the true last modification time. Finally, a client might not have access to update an image when it opens it read-only --- rendering the last access time, again, as a rough approximation. IMHO, I think there are a lot of other, higher-priority backlog items for RBD (and supporting services) [1] -- but I've added it to the bottom of backlog. [1] https://trello.com/b/ugTc2QFH/ceph-backlog Yes, agree, let's focus on the other higher-priority backlog items now. Thanx On Fri, Mar 24, 2017 at 3:27 AM, Dongsheng Yang wrote: Hi jason, do you think this is a good feature for rbd? maybe we can implement a "rbd stat" command to show atime, mtime and ctime of an image. Yang On 03/23/2017 08:36 PM, Christoph Adomeit wrote: Hi, no i did not enable the journalling feature since we do not use mirroring. On Thu, Mar 23, 2017 at 08:10:05PM +0800, Dongsheng Yang wrote: Did you enable the journaling feature? On 03/23/2017 07:44 PM, Christoph Adomeit wrote: Hi Yang, I mean "any write" to this image. I am sure we have a lot of not-used-anymore rbd images in our pool and I am trying to identify them. The mtime would be a good hint to show which images might be unused. Christoph On Thu, Mar 23, 2017 at 07:32:49PM +0800, Dongsheng Yang wrote: Hi Christoph, On 03/23/2017 07:16 PM, Christoph Adomeit wrote: Hello List, i am wondering if there is meanwhile an easy method in ceph to find more information about rbd-images. For example I am interested in the modification time of an rbd image. Do you mean some metadata changing? such as resize? Or any write to this image? Thanx Yang I found some posts from 2015 that say we have to go over all the objects of an rbd image and find the newest mtime put this is not a preferred solution for me. It takes to much time and too many system resources. Any Ideas ? Thanks Christoph ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Questions on rbd-mirror
Jason, do you think it's good idea to introduce a rbd_config object to record some configurations of per-pool, such as default_features. That means, we can set some configurations differently in different pool. In this way, we can also handle the per-pool setting in rbd-mirror. Thanx Yang On 27/03/2017, 21:20, Jason Dillaman wrote: On Mon, Mar 27, 2017 at 4:00 AM, Dongsheng Yang wrote: Hi Fulvio, On 03/24/2017 07:19 PM, Fulvio Galeazzi wrote: Hallo, apologies for my (silly) questions, I did try to find some doc on rbd-mirror but was unable to, apart from a number of pages explaining how to install it. My environment is CenOS7 and Ceph 10.2.5. Can anyone help me understand a few minor things: - is there a cleaner way to configure the user which will be used for rbd-mirror, other than editing the ExecStart in file /usr/lib/systemd/system/ceph-rbd-mirror@.service ? For example some line in ceph.conf... looks like the username defaults to the cluster name, am I right? It should just be "ceph", no matter what the cluster name is, if I read the code correctly. The user id is passed in via the systemd instance name. For example, if you wanted to use the "mirror" user id to connect to the local cluster, you would run "systemctl enable ceph-rbd-mirror@mirror". - is it possible to throttle mirroring? Sure, it's a crazy thing to do for "cinder" pools, but may make sense for slowly changing ones, like a "glance" pool. The rbd core team is working on this. Jason, right? This is in our backlog of desired items for the rbd-mirror daemon. Having different settings for different pools was not in our original plan, but this is something that also came up during the Vault conference last week. I've added an additional backlog item to cover per-pool settings. - is it possible to set per-pool default features? I read about "rbd default features = ###" but this is a global setting. (Ok, I can still restrict pools to be mirrored with "ceph auth" for the user doing mirroring) "per-pool default features" sounds like a reasonable feature request. About the "ceph auth" for mirroring, I am working on a rbd acl design, will consider pool-level, namespace-level and image-level. Then I think we can do a permission check on this. Right now, the best way to achieve that is by using different configs / user ids for different services. For example, if OpenStack glance used "glance" and cinder user "cinder", the ceph.conf's "[client.glance]" section could have different default features as compared to a "[client.cinder]" section. Thanx Yang Thanks! Fulvio ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RBD image perf counters: usage, access
On 03/27/2017 04:06 PM, Masha Atakova wrote:
Hi Yang,
Hi Masha,
Thank you for your reply. This is very useful indeed that there are
many ImageCtx objects for one image.
But in my setting, I don't have any particular ceph client connected
to ceph (I could, but this is not the point). I'm trying to get
metrics for particular image while not performing anything with it
myself.
The perf counter you mentioned in your first mail, is just for one
particular image client, that means,
these perf counter will disappear as the client disconnected.
And I'm trying to get access to performance counters listed in the
ImageCtx class, they don't seem to be reported by the perf tool.
Do you mean get the perf counters via api? At first this counter is only
for a particular ImageCtx (connected client), then you can
read the counters by the perf dump command in my last mail I think.
If you want to get the performance counter for an image (no matter how
many ImageCtx, connected or disconnected),
maybe you need to wait this one:
http://pad.ceph.com/p/ceph-top
Yang
Thanks!
On 27/03/17 12:29, Dongsheng Yang wrote:
Hi Masha
you can get the counters by perf dump command on the asok file of
your client. such as that:
$ ceph --admin-daemon out/client.admin.9921.asok perf dump|grep rd
"rd": 656754,
"rd_bytes": 656754,
"rd_latency": {
"discard": 0,
"discard_bytes": 0,
"discard_latency": {
"omap_rd": 0,
But, note that, this is a counter of this one ImageCtx, but not the
counter for this image. There are
possible several ImageCtxes reading or writing on the same image.
Yang
On 03/27/2017 12:23 PM, Masha Atakova wrote:
Hi everyone,
I was going around trying to figure out how to get ceph metrics on a
more detailed level than daemons. Of course, I found and explored
API for watching rados objects, but I'm more interested in getting
metrics about RBD images. And while I could get list of objects for
particular image, and then watch all of them, it doesn't seem like
very efficient way to go about it.
I checked librbd API and there isn't anything helping with my goal.
So I went through the source code and found list of performance
counters for image which are incremented by other parts of ceph when
making corresponding operations:
https://github.com/ceph/ceph/blob/master/src/librbd/ImageCtx.cc#L364
I have 2 questions about it:
1) is there any workaround to use those counters right now? maybe
when compiling against ceph the code doing it. Looks like I need to
be able to access particular ImageCtx object (instead of creating my
own), and I just can't find appropriate class / part of the librbd
allowing me to do so.
2) are there any plans on making those counters accessible via API
like librbd or librados?
I see that these questions might be more appropriate for the devel
list, but:
- it seems to me that question of getting ceph metrics is more
interesting for those who use ceph
- I couldn't subscribe to it with an error provided below.
Thanks!
majord...@vger.kernel.org:
SMTP error from remote server for MAIL FROM command, host: vger.kernel.org
(209.132.180.67) reason: 553 5.7.1 Hello [74.208.4.201], for your MAIL FROM
address policy analysis reported: Your address is not
liked source for email
--- The header of the original message is following. ---
Received: from [192.168.1.10] ([223.206.146.181]) by mail.gmx.com (mrgmxus001
[74.208.5.15]) with ESMTPSA (Nemesis) id 0M92q3-1d0LS03yov-00CTwW for
; Mon, 27 Mar 2017 05:55:46 +0200
To:majord...@vger.kernel.org
From: Masha Atakova
Subject: subscribe ceph-devel
Message-ID:<174d9bc0-b50d-fc80-ede8-5ba9d472e...@mail.com>
Date: Mon, 27 Mar 2017 10:55:43 +0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
Thunderbird/45.7.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Provags-ID: V03:K0:Lau7llt7/MuJt+nRLjXIhY91IuCvCBJGtqDzxLgqkh2ERVkWeep
5CDyh9GHW7QSodn80xWCPOOD2kvvnr6YxrB5R9SZ1iloI9VO2YoTXAauDq4mtWh+abwUOiY
wQgj6YvUcLjfUinsh0t68Q9m3h3ufZIoKIeWhKFGbsRALqsvZjgWBVlaAR/V5Vt4O/wFJGG
YULQ6/t4oDSsBuy4agFdQ==
X-UI-Out-Filterresults: notjunk:1;V01:K0:xLdjozptxu8=:nO7vxZvAbidrXk7gcv7Wqc
Bjr14pXiTEv8gVIlRTZ78cNDEQthT557sAgBBRnJkDGXkP1efvEN2QqsZAzfa52Og4ysSFXub
BPSiDOI0wkzxQMu1QHqWzvURobFX9LxrctwYB3k9nrOtHFgJwm0eQWfV1QKg7i0ESzT244u2c
2xKpGGrhNUspJtEep97xjY3DyDvR3ApYx9x+RO9ZQAE0Is9AO0mBYqDR3NqrF1KzabJWuCA7I
yu1y9N0QILgr/WmUf74qxeh1k20n+7yYuYPzgIl9Cm2vyrVu2ONUTJMpN2p+iUit8hhUsTuYQ
/TNde22Q5OOCz+oGVhWq04J+CBP23VrEkent4kw2vhejDjQD/F2J4o2XkfkPt7ZqpMreGWBfB
jtpfz4jHyp+voLlldhw7+cKUGY4ux8dihtlaCm9N3FQ2qvQ9CTsFuLsTNHNe7uRx5oeZgBFFh
6t1OVBLlRR1wwSMDbx6vE5UTx47vbAtu5I/vyryQ1jVnzyQitjWE6iLMEC8faatMquOxJreoF
4ALLNVStuHEkaGC0zimjQ5YkiFe6
Re: [ceph-users] Questions on rbd-mirror
Hi Fulvio, On 03/24/2017 07:19 PM, Fulvio Galeazzi wrote: Hallo, apologies for my (silly) questions, I did try to find some doc on rbd-mirror but was unable to, apart from a number of pages explaining how to install it. My environment is CenOS7 and Ceph 10.2.5. Can anyone help me understand a few minor things: - is there a cleaner way to configure the user which will be used for rbd-mirror, other than editing the ExecStart in file /usr/lib/systemd/system/ceph-rbd-mirror@.service ? For example some line in ceph.conf... looks like the username defaults to the cluster name, am I right? It should just be "ceph", no matter what the cluster name is, if I read the code correctly. - is it possible to throttle mirroring? Sure, it's a crazy thing to do for "cinder" pools, but may make sense for slowly changing ones, like a "glance" pool. The rbd core team is working on this. Jason, right? - is it possible to set per-pool default features? I read about "rbd default features = ###" but this is a global setting. (Ok, I can still restrict pools to be mirrored with "ceph auth" for the user doing mirroring) "per-pool default features" sounds like a reasonable feature request. About the "ceph auth" for mirroring, I am working on a rbd acl design, will consider pool-level, namespace-level and image-level. Then I think we can do a permission check on this. Thanx Yang Thanks! Fulvio ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RBD image perf counters: usage, access
Hi Masha
you can get the counters by perf dump command on the asok file of
your client. such as that:
$ ceph --admin-daemon out/client.admin.9921.asok perf dump|grep rd
"rd": 656754,
"rd_bytes": 656754,
"rd_latency": {
"discard": 0,
"discard_bytes": 0,
"discard_latency": {
"omap_rd": 0,
But, note that, this is a counter of this one ImageCtx, but not the
counter for this image. There are
possible several ImageCtxes reading or writing on the same image.
Yang
On 03/27/2017 12:23 PM, Masha Atakova wrote:
Hi everyone,
I was going around trying to figure out how to get ceph metrics on a
more detailed level than daemons. Of course, I found and explored API
for watching rados objects, but I'm more interested in getting metrics
about RBD images. And while I could get list of objects for particular
image, and then watch all of them, it doesn't seem like very efficient
way to go about it.
I checked librbd API and there isn't anything helping with my goal.
So I went through the source code and found list of performance
counters for image which are incremented by other parts of ceph when
making corresponding operations:
https://github.com/ceph/ceph/blob/master/src/librbd/ImageCtx.cc#L364
I have 2 questions about it:
1) is there any workaround to use those counters right now? maybe when
compiling against ceph the code doing it. Looks like I need to be able
to access particular ImageCtx object (instead of creating my own), and
I just can't find appropriate class / part of the librbd allowing me
to do so.
2) are there any plans on making those counters accessible via API
like librbd or librados?
I see that these questions might be more appropriate for the devel
list, but:
- it seems to me that question of getting ceph metrics is more
interesting for those who use ceph
- I couldn't subscribe to it with an error provided below.
Thanks!
majord...@vger.kernel.org:
SMTP error from remote server for MAIL FROM command, host: vger.kernel.org
(209.132.180.67) reason: 553 5.7.1 Hello [74.208.4.201], for your MAIL FROM
address policy analysis reported: Your address is not
liked source for email
--- The header of the original message is following. ---
Received: from [192.168.1.10] ([223.206.146.181]) by mail.gmx.com (mrgmxus001
[74.208.5.15]) with ESMTPSA (Nemesis) id 0M92q3-1d0LS03yov-00CTwW for
; Mon, 27 Mar 2017 05:55:46 +0200
To:majord...@vger.kernel.org
From: Masha Atakova
Subject: subscribe ceph-devel
Message-ID:<174d9bc0-b50d-fc80-ede8-5ba9d472e...@mail.com>
Date: Mon, 27 Mar 2017 10:55:43 +0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
Thunderbird/45.7.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Provags-ID: V03:K0:Lau7llt7/MuJt+nRLjXIhY91IuCvCBJGtqDzxLgqkh2ERVkWeep
5CDyh9GHW7QSodn80xWCPOOD2kvvnr6YxrB5R9SZ1iloI9VO2YoTXAauDq4mtWh+abwUOiY
wQgj6YvUcLjfUinsh0t68Q9m3h3ufZIoKIeWhKFGbsRALqsvZjgWBVlaAR/V5Vt4O/wFJGG
YULQ6/t4oDSsBuy4agFdQ==
X-UI-Out-Filterresults: notjunk:1;V01:K0:xLdjozptxu8=:nO7vxZvAbidrXk7gcv7Wqc
Bjr14pXiTEv8gVIlRTZ78cNDEQthT557sAgBBRnJkDGXkP1efvEN2QqsZAzfa52Og4ysSFXub
BPSiDOI0wkzxQMu1QHqWzvURobFX9LxrctwYB3k9nrOtHFgJwm0eQWfV1QKg7i0ESzT244u2c
2xKpGGrhNUspJtEep97xjY3DyDvR3ApYx9x+RO9ZQAE0Is9AO0mBYqDR3NqrF1KzabJWuCA7I
yu1y9N0QILgr/WmUf74qxeh1k20n+7yYuYPzgIl9Cm2vyrVu2ONUTJMpN2p+iUit8hhUsTuYQ
/TNde22Q5OOCz+oGVhWq04J+CBP23VrEkent4kw2vhejDjQD/F2J4o2XkfkPt7ZqpMreGWBfB
jtpfz4jHyp+voLlldhw7+cKUGY4ux8dihtlaCm9N3FQ2qvQ9CTsFuLsTNHNe7uRx5oeZgBFFh
6t1OVBLlRR1wwSMDbx6vE5UTx47vbAtu5I/vyryQ1jVnzyQitjWE6iLMEC8faatMquOxJreoF
4ALLNVStuHEkaGC0zimjQ5YkiFe6nHqxwsaYU7Vcy0j9GXTkiakh6kwluOyLqy5Q1e1FHPfSG
/swFoOHGvb07bK81+G1OLT7nIIArC+NrsHGmsrycXpw9gvZGubLYoYSgRskhJ1F+QxCzspFK0
XOgA5Ko3M3djFYkMM0S+xHHyVIIpUr4qQXv1sKuWUY63wlalu3JLwWn7t8CBhC2R0s/3ec0WT
WD+iDs0hWe0INwfX+BNVWIuyzim7qKg8wbG95YWyAI9J9dyx7lv4VETd2Zf5raU1TgNFB/6OP
RQrUx3O
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Modification Time of RBD Images
Hi jason, do you think this is a good feature for rbd? maybe we can implement a "rbd stat" command to show atime, mtime and ctime of an image. Yang On 03/23/2017 08:36 PM, Christoph Adomeit wrote: Hi, no i did not enable the journalling feature since we do not use mirroring. On Thu, Mar 23, 2017 at 08:10:05PM +0800, Dongsheng Yang wrote: Did you enable the journaling feature? On 03/23/2017 07:44 PM, Christoph Adomeit wrote: Hi Yang, I mean "any write" to this image. I am sure we have a lot of not-used-anymore rbd images in our pool and I am trying to identify them. The mtime would be a good hint to show which images might be unused. Christoph On Thu, Mar 23, 2017 at 07:32:49PM +0800, Dongsheng Yang wrote: Hi Christoph, On 03/23/2017 07:16 PM, Christoph Adomeit wrote: Hello List, i am wondering if there is meanwhile an easy method in ceph to find more information about rbd-images. For example I am interested in the modification time of an rbd image. Do you mean some metadata changing? such as resize? Or any write to this image? Thanx Yang I found some posts from 2015 that say we have to go over all the objects of an rbd image and find the newest mtime put this is not a preferred solution for me. It takes to much time and too many system resources. Any Ideas ? Thanks Christoph ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Modification Time of RBD Images
Did you enable the journaling feature? On 03/23/2017 07:44 PM, Christoph Adomeit wrote: Hi Yang, I mean "any write" to this image. I am sure we have a lot of not-used-anymore rbd images in our pool and I am trying to identify them. The mtime would be a good hint to show which images might be unused. Christoph On Thu, Mar 23, 2017 at 07:32:49PM +0800, Dongsheng Yang wrote: Hi Christoph, On 03/23/2017 07:16 PM, Christoph Adomeit wrote: Hello List, i am wondering if there is meanwhile an easy method in ceph to find more information about rbd-images. For example I am interested in the modification time of an rbd image. Do you mean some metadata changing? such as resize? Or any write to this image? Thanx Yang I found some posts from 2015 that say we have to go over all the objects of an rbd image and find the newest mtime put this is not a preferred solution for me. It takes to much time and too many system resources. Any Ideas ? Thanks Christoph ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Modification Time of RBD Images
On 03/23/2017 07:32 PM, Dongsheng Yang wrote: Hi Christoph, On 03/23/2017 07:16 PM, Christoph Adomeit wrote: Hello List, i am wondering if there is meanwhile an easy method in ceph to find more information about rbd-images. For example I am interested in the modification time of an rbd image. Do you mean some metadata changing? such as resize? If you mean metadata changing, I think this command would be enough: $ rados -p rbd stat rbd_header.11e3238e1f29 rbd/rbd_header.11e3238e1f29 mtime 2017-03-23 19:31:52.00, size 0 Or any write to this image? But if you want this one, I am afraid it's not so handy currently. maybe going through the all data blocks of this image and sorting the mtime of them would be workable. Thanx Yang I found some posts from 2015 that say we have to go over all the objects of an rbd image and find the newest mtime put this is not a preferred solution for me. It takes to much time and too many system resources. Any Ideas ? Thanks Christoph ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Modification Time of RBD Images
Hi Christoph, On 03/23/2017 07:16 PM, Christoph Adomeit wrote: Hello List, i am wondering if there is meanwhile an easy method in ceph to find more information about rbd-images. For example I am interested in the modification time of an rbd image. Do you mean some metadata changing? such as resize? Or any write to this image? Thanx Yang I found some posts from 2015 that say we have to go over all the objects of an rbd image and find the newest mtime put this is not a preferred solution for me. It takes to much time and too many system resources. Any Ideas ? Thanks Christoph ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Fwd: rbd: show the snapshot tree
Resending. as the mailing list does not accept too large mail. I removed the attachments from the original message. Please refer to [1] for the pictures. [1]: https://github.com/ceph/ceph/pull/13870#issuecomment-287042199 Thanx Dongsheng Original Message Subject:rbd: show the snapshot tree Date: Fri, 17 Mar 2017 11:50:17 +0800 From: Dongsheng Yang To: 'Ceph Users' CC: jason , chenhanx...@gmail.com Hi guys, There is an idea about showing the snapshots of an image in a tree view, as what vmware is doing in screenshot attached(vmware.jpeg). So I think that's a good to implement a similar feature in rbd, as attached(rbd_snap_tree.jpeg). But, is that a hot requirement for us? as Jason mentioned, A rollback is a pretty large sledgehammer (i.e. not efficient) approach to switch between a hierarchy of historical configurations. Maybe clone is much better. If so, the hierarchy of snapshots is not a "tree", but a "line". Therefore, I am here to collect more opinions on this topic. What do you think about this feature? BTW, Jason provide another idea, show a tree view about the relationship between parents and children. Yea, that's another idea, but I think that's good. what do you think about this one? Thanx Dongsheng ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
BTW, is there any body using EnhanceIO? On 02/15/2017 05:51 PM, Dongsheng Yang wrote: thanx Nick, Gregory and Wido, So at least, we can say the cache tiering in Jewel is stable enough I think. I like cache tiering more than the others, but yes, there is a problem about cache tiering in flushing data between different nodes, which are not a problem in local caching solution. guys: Is there any plan to enhance cache tiering to solve such problem? Or as Nick asked, is that cache tiering fading away? Yang On 15/02/2017, 06:42, Nick Fisk wrote: -Original Message- From: Gregory Farnum [mailto:gfar...@redhat.com] Sent: 14 February 2017 21:05 To: Wido den Hollander Cc: Dongsheng Yang ; Nick Fisk ; Ceph Users Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander wrote: Op 14 februari 2017 om 11:14 schreef Nick Fisk : -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dongsheng Yang Sent: 14 February 2017 09:01 To: Sage Weil Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com Subject: [ceph-users] bcache vs flashcache vs cache tiering Hi Sage and all, We are going to use SSDs for cache in ceph. But I am not sure which one is the best solution, bcache? flashcache? or cache tier? I would vote for cache tier. Being able to manage it from within Ceph, instead of having to manage X number of bcache/flashcache instances, appeals to me more. Also last time I looked Flashcache seems unmaintained and bcache might be going that way with talk of this new bcachefs. Another point to consider is that Ceph has had a lot of work done on it to ensure data consistency; I don't ever want to be in a position where I'm trying to diagnose problems that might be being caused by another layer sitting in-between Ceph and the Disk. However, I know several people on here are using bcache and potentially getting better performance than with cache tiering, so hopefully someone will give their views. I am using Bcache on various systems and it performs really well. The caching layer in Ceph is slow. Promoting Objects is slow and it also involves additional RADOS lookups. Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use cases where they're effective are still pretty limited and I think in-node caching has a brighter future. We just don't like to maintain the global state that makes separate caching locations viable and unless you're doing something analogous to the supercomputing "burst buffers" (which some people are!), it's going to be hard to beat something that doesn't have to pay the cost of extra network hops/bandwidth. Cache tiers are also not a feature that all the vendors support in their downstream products, so it will probably see less ongoing investment than you'd expect from such a system. Should that be taken as an unofficial sign that the tiering support is likely to fade away? I think both approaches have different strengths and probably the difference between a tiering system and a caching one is what causes some of the problems. If something like bcache is going to be the preferred approach, then I think more work needs to be done around certifying it for use with Ceph and allowing its behavior to be more controlled by Ceph as well. I assume there are issues around backfilling and scrubbing polluting the cache? Maybe you would want to be able to pass hints down from Ceph, which could also allow per pool cache behavior?? -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
thanx Nick, Gregory and Wido, So at least, we can say the cache tiering in Jewel is stable enough I think. I like cache tiering more than the others, but yes, there is a problem about cache tiering in flushing data between different nodes, which are not a problem in local caching solution. guys: Is there any plan to enhance cache tiering to solve such problem? Or as Nick asked, is that cache tiering fading away? Yang On 15/02/2017, 06:42, Nick Fisk wrote: -Original Message- From: Gregory Farnum [mailto:gfar...@redhat.com] Sent: 14 February 2017 21:05 To: Wido den Hollander Cc: Dongsheng Yang ; Nick Fisk ; Ceph Users Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander wrote: Op 14 februari 2017 om 11:14 schreef Nick Fisk : -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dongsheng Yang Sent: 14 February 2017 09:01 To: Sage Weil Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com Subject: [ceph-users] bcache vs flashcache vs cache tiering Hi Sage and all, We are going to use SSDs for cache in ceph. But I am not sure which one is the best solution, bcache? flashcache? or cache tier? I would vote for cache tier. Being able to manage it from within Ceph, instead of having to manage X number of bcache/flashcache instances, appeals to me more. Also last time I looked Flashcache seems unmaintained and bcache might be going that way with talk of this new bcachefs. Another point to consider is that Ceph has had a lot of work done on it to ensure data consistency; I don't ever want to be in a position where I'm trying to diagnose problems that might be being caused by another layer sitting in-between Ceph and the Disk. However, I know several people on here are using bcache and potentially getting better performance than with cache tiering, so hopefully someone will give their views. I am using Bcache on various systems and it performs really well. The caching layer in Ceph is slow. Promoting Objects is slow and it also involves additional RADOS lookups. Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use cases where they're effective are still pretty limited and I think in-node caching has a brighter future. We just don't like to maintain the global state that makes separate caching locations viable and unless you're doing something analogous to the supercomputing "burst buffers" (which some people are!), it's going to be hard to beat something that doesn't have to pay the cost of extra network hops/bandwidth. Cache tiers are also not a feature that all the vendors support in their downstream products, so it will probably see less ongoing investment than you'd expect from such a system. Should that be taken as an unofficial sign that the tiering support is likely to fade away? I think both approaches have different strengths and probably the difference between a tiering system and a caching one is what causes some of the problems. If something like bcache is going to be the preferred approach, then I think more work needs to be done around certifying it for use with Ceph and allowing its behavior to be more controlled by Ceph as well. I assume there are issues around backfilling and scrubbing polluting the cache? Maybe you would want to be able to pass hints down from Ceph, which could also allow per pool cache behavior?? -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] bcache vs flashcache vs cache tiering
Hi Sage and all, We are going to use SSDs for cache in ceph. But I am not sure which one is the best solution, bcache? flashcache? or cache tier? I found there are some CAUTION in ceph.com about cache tiering. Is cache tiering is already production ready? especially for rbd. thanx in advance. Yang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
