Re: [ceph-users] CephFS and caching

[Gregory Farnum]

On Thu, Sep 10, 2015 at 1:07 PM, Kyle Hutson  wrote:
> A 'rados -p cachepool ls' takes about 3 hours - not exactly useful.
>
> I'm intrigued that you say a single read may not promote it into the cache.
> My understanding is that if you have an EC-backed pool the clients can't
> talk to them directly, which means they would necessarily be promoted to the
> cache pool so the client could read it. Is my understanding wrong?

You're not wrong (although you may be in the future!), but for
instance there is code somewhere (in Infernalis-to-be, I think?) that
will simply proxy the read through the cache OSD, without actually
reading up the data.

>
> I'm also wondering if it's possible to use RAM as a read-cache layer.
> Obviously, we don't want this for write-cache because of power outages,
> motherboard failures, etc., but it seems to make sense for a read-cache. Is
> that something that's being done, can be done, is going to be done, or has
> even been considered?

We don't do anything special to make use of RAM because the linux page
cache is generally more intelligent about it than we could be.
Sufficiently hot objects will be in RAM without us needing to do
anything special. ;)
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS and caching

[Ilya Dryomov]

On Wed, Sep 9, 2015 at 5:34 PM, Gregory Farnum  wrote:
> On Wed, Sep 9, 2015 at 3:27 PM, Kyle Hutson  wrote:
>> We are using Hammer - latest released version. How do I check if it's
>> getting promoted into the cache?
>
> Umm...that's a good question. You can run rados ls on the cache pool,
> but that's not exactly scalable; you can turn up logging and dig into
> them to see if redirects are happening, or watch the OSD operations
> happening via the admin socket. But I don't know if there's a good
> interface for users to just query the cache state of a single object.
> :/
>
>>
>> We're using the latest ceph kernel client. Where do I poke at readahead
>> settings there?
>
> Just the standard kernel readahead settings; I'm not actually familiar
> with how to configure those but I don't believe Ceph's are in any way
> special. What do you mean by "latest ceph kernel client"; are you
> running one of the developer testing kernels or something? I think
> Ilya might have mentioned some issues with readahead being
> artificially blocked, but that might have only been with RBD.

That's a system wide issue - it affects md folks and generally
everybody who can benefit from a large readahead window.  Notes from
the perf etherpad:

- kernel page cache readahead is capped at 2M since 3.15, that patch
was backported so older kernels (e.g. 3.14.21+) are affected as well
- Red Hat and Oracle seem to be shipping a patch that effectively
reverts this, but someone needs to check that
- a patch that gets rid of this limit is in the works, Linus seems to
be happy with it, expect it in 4.3 or 4.4?

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS and caching

[Kyle Hutson]

A 'rados -p cachepool ls' takes about 3 hours - not exactly useful.

I'm intrigued that you say a single read may not promote it into the cache.
My understanding is that if you have an EC-backed pool the clients can't
talk to them directly, which means they would necessarily be promoted to
the cache pool so the client could read it. Is my understanding wrong?

I'm also wondering if it's possible to use RAM as a read-cache layer.
Obviously, we don't want this for write-cache because of power outages,
motherboard failures, etc., but it seems to make sense for a read-cache. Is
that something that's being done, can be done, is going to be done, or has
even been considered?

On Wed, Sep 9, 2015 at 10:33 AM, Gregory Farnum  wrote:

> On Wed, Sep 9, 2015 at 4:26 PM, Kyle Hutson  wrote:
> >
> >
> > On Wed, Sep 9, 2015 at 9:34 AM, Gregory Farnum 
> wrote:
> >>
> >> On Wed, Sep 9, 2015 at 3:27 PM, Kyle Hutson  wrote:
> >> > We are using Hammer - latest released version. How do I check if it's
> >> > getting promoted into the cache?
> >>
> >> Umm...that's a good question. You can run rados ls on the cache pool,
> >> but that's not exactly scalable; you can turn up logging and dig into
> >> them to see if redirects are happening, or watch the OSD operations
> >> happening via the admin socket. But I don't know if there's a good
> >> interface for users to just query the cache state of a single object.
> >> :/
> >
> >
> > even using 'rados ls', I (naturally) get cephfs object names - is there a
> > way to see a filename -> objectname conversion ... or objectname ->
> filename
> > ?
>
> The object name is .. So you can
> look at the file inode and then see which of its objects are actually
> in the pool.
> -Greg
>
> >
> >>
> >> > We're using the latest ceph kernel client. Where do I poke at
> readahead
> >> > settings there?
> >>
> >> Just the standard kernel readahead settings; I'm not actually familiar
> >> with how to configure those but I don't believe Ceph's are in any way
> >> special. What do you mean by "latest ceph kernel client"; are you
> >> running one of the developer testing kernels or something?
> >
> >
> > No, just what comes with the latest stock kernel. Sorry for any
> confusion.
> >
> >>
> >> I think
> >> Ilya might have mentioned some issues with readahead being
> >> artificially blocked, but that might have only been with RBD.
> >>
> >> Oh, are the files you're using sparse? There was a bug with sparse
> >> files not filling in pages that just got patched yesterday or
> >> something.
> >
> >
> > No, these are not sparse files. Just really big.
> >
> >>
> >> >
> >> > On Tue, Sep 8, 2015 at 8:29 AM, Gregory Farnum 
> >> > wrote:
> >> >>
> >> >> On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson 
> >> >> wrote:
> >> >> > I was wondering if anybody could give me some insight as to how
> >> >> > CephFS
> >> >> > does
> >> >> > its caching - read-caching in particular.
> >> >> >
> >> >> > We are using CephFS with an EC pool on the backend with a
> replicated
> >> >> > cache
> >> >> > pool in front of it. We're seeing some very slow read times. Trying
> >> >> > to
> >> >> > compute an md5sum on a 15GB file twice in a row (so it should be in
> >> >> > cache)
> >> >> > takes the time from 23 minutes down to 17 minutes, but this is
> over a
> >> >> > 10Gbps
> >> >> > network and with a crap-ton of OSDs (over 300), so I would expect
> it
> >> >> > to
> >> >> > be
> >> >> > down in the 2-3 minute range.
> >> >>
> >> >> A single sequential read won't necessarily promote an object into the
> >> >> cache pool (although if you're using Hammer I think it will), so you
> >> >> want to check if it's actually getting promoted into the cache before
> >> >> assuming that's happened.
> >> >>
> >> >> >
> >> >> > I'm just trying to figure out what we can do to increase the
> >> >> > performance. I
> >> >> > have over 300 TB of live data that I have to be careful with,
> though,
> >> >> > so
> >> >> > I
> >> >> > have to have some level of caution.
> >> >> >
> >> >> > Is there some other caching we can do (client-side or server-side)
> >> >> > that
> >> >> > might give us a decent performance boost?
> >> >>
> >> >> Which client are you using for this testing? Have you looked at the
> >> >> readahead settings? That's usually the big one; if you're only asking
> >> >> for 4KB at once then stuff is going to be slow no matter what (a
> >> >> single IO takes at minimum about 2 milliseconds right now, although
> >> >> the RADOS team is working to improve that).
> >> >> -Greg
> >> >>
> >> >> >
> >> >> > ___
> >> >> > ceph-users mailing list
> >> >> > ceph-users@lists.ceph.com
> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >> >
> >> >
> >> >
> >
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com

Re: [ceph-users] CephFS and caching

[Kyle Hutson]

We are using Hammer - latest released version. How do I check if it's
getting promoted into the cache?

We're using the latest ceph kernel client. Where do I poke at readahead
settings there?

On Tue, Sep 8, 2015 at 8:29 AM, Gregory Farnum  wrote:

> On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson  wrote:
> > I was wondering if anybody could give me some insight as to how CephFS
> does
> > its caching - read-caching in particular.
> >
> > We are using CephFS with an EC pool on the backend with a replicated
> cache
> > pool in front of it. We're seeing some very slow read times. Trying to
> > compute an md5sum on a 15GB file twice in a row (so it should be in
> cache)
> > takes the time from 23 minutes down to 17 minutes, but this is over a
> 10Gbps
> > network and with a crap-ton of OSDs (over 300), so I would expect it to
> be
> > down in the 2-3 minute range.
>
> A single sequential read won't necessarily promote an object into the
> cache pool (although if you're using Hammer I think it will), so you
> want to check if it's actually getting promoted into the cache before
> assuming that's happened.
>
> >
> > I'm just trying to figure out what we can do to increase the
> performance. I
> > have over 300 TB of live data that I have to be careful with, though, so
> I
> > have to have some level of caution.
> >
> > Is there some other caching we can do (client-side or server-side) that
> > might give us a decent performance boost?
>
> Which client are you using for this testing? Have you looked at the
> readahead settings? That's usually the big one; if you're only asking
> for 4KB at once then stuff is going to be slow no matter what (a
> single IO takes at minimum about 2 milliseconds right now, although
> the RADOS team is working to improve that).
> -Greg
>
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS and caching

[Gregory Farnum]

On Wed, Sep 9, 2015 at 3:27 PM, Kyle Hutson  wrote:
> We are using Hammer - latest released version. How do I check if it's
> getting promoted into the cache?

Umm...that's a good question. You can run rados ls on the cache pool,
but that's not exactly scalable; you can turn up logging and dig into
them to see if redirects are happening, or watch the OSD operations
happening via the admin socket. But I don't know if there's a good
interface for users to just query the cache state of a single object.
:/

>
> We're using the latest ceph kernel client. Where do I poke at readahead
> settings there?

Just the standard kernel readahead settings; I'm not actually familiar
with how to configure those but I don't believe Ceph's are in any way
special. What do you mean by "latest ceph kernel client"; are you
running one of the developer testing kernels or something? I think
Ilya might have mentioned some issues with readahead being
artificially blocked, but that might have only been with RBD.

Oh, are the files you're using sparse? There was a bug with sparse
files not filling in pages that just got patched yesterday or
something.
-Greg

>
> On Tue, Sep 8, 2015 at 8:29 AM, Gregory Farnum  wrote:
>>
>> On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson  wrote:
>> > I was wondering if anybody could give me some insight as to how CephFS
>> > does
>> > its caching - read-caching in particular.
>> >
>> > We are using CephFS with an EC pool on the backend with a replicated
>> > cache
>> > pool in front of it. We're seeing some very slow read times. Trying to
>> > compute an md5sum on a 15GB file twice in a row (so it should be in
>> > cache)
>> > takes the time from 23 minutes down to 17 minutes, but this is over a
>> > 10Gbps
>> > network and with a crap-ton of OSDs (over 300), so I would expect it to
>> > be
>> > down in the 2-3 minute range.
>>
>> A single sequential read won't necessarily promote an object into the
>> cache pool (although if you're using Hammer I think it will), so you
>> want to check if it's actually getting promoted into the cache before
>> assuming that's happened.
>>
>> >
>> > I'm just trying to figure out what we can do to increase the
>> > performance. I
>> > have over 300 TB of live data that I have to be careful with, though, so
>> > I
>> > have to have some level of caution.
>> >
>> > Is there some other caching we can do (client-side or server-side) that
>> > might give us a decent performance boost?
>>
>> Which client are you using for this testing? Have you looked at the
>> readahead settings? That's usually the big one; if you're only asking
>> for 4KB at once then stuff is going to be slow no matter what (a
>> single IO takes at minimum about 2 milliseconds right now, although
>> the RADOS team is working to improve that).
>> -Greg
>>
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS and caching

[Kyle Hutson]

On Wed, Sep 9, 2015 at 9:34 AM, Gregory Farnum  wrote:

> On Wed, Sep 9, 2015 at 3:27 PM, Kyle Hutson  wrote:
> > We are using Hammer - latest released version. How do I check if it's
> > getting promoted into the cache?
>
> Umm...that's a good question. You can run rados ls on the cache pool,
> but that's not exactly scalable; you can turn up logging and dig into
> them to see if redirects are happening, or watch the OSD operations
> happening via the admin socket. But I don't know if there's a good
> interface for users to just query the cache state of a single object.
> :/
>

even using 'rados ls', I (naturally) get cephfs object names - is there a
way to see a filename -> objectname conversion ... or objectname ->
filename ?


> > We're using the latest ceph kernel client. Where do I poke at readahead
> > settings there?
>
> Just the standard kernel readahead settings; I'm not actually familiar
> with how to configure those but I don't believe Ceph's are in any way
> special. What do you mean by "latest ceph kernel client"; are you
> running one of the developer testing kernels or something?


No, just what comes with the latest stock kernel. Sorry for any confusion.


> I think
> Ilya might have mentioned some issues with readahead being
> artificially blocked, but that might have only been with RBD.
>
> Oh, are the files you're using sparse? There was a bug with sparse
> files not filling in pages that just got patched yesterday or
> something.
>

No, these are not sparse files. Just really big.


> >
> > On Tue, Sep 8, 2015 at 8:29 AM, Gregory Farnum 
> wrote:
> >>
> >> On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson 
> wrote:
> >> > I was wondering if anybody could give me some insight as to how CephFS
> >> > does
> >> > its caching - read-caching in particular.
> >> >
> >> > We are using CephFS with an EC pool on the backend with a replicated
> >> > cache
> >> > pool in front of it. We're seeing some very slow read times. Trying to
> >> > compute an md5sum on a 15GB file twice in a row (so it should be in
> >> > cache)
> >> > takes the time from 23 minutes down to 17 minutes, but this is over a
> >> > 10Gbps
> >> > network and with a crap-ton of OSDs (over 300), so I would expect it
> to
> >> > be
> >> > down in the 2-3 minute range.
> >>
> >> A single sequential read won't necessarily promote an object into the
> >> cache pool (although if you're using Hammer I think it will), so you
> >> want to check if it's actually getting promoted into the cache before
> >> assuming that's happened.
> >>
> >> >
> >> > I'm just trying to figure out what we can do to increase the
> >> > performance. I
> >> > have over 300 TB of live data that I have to be careful with, though,
> so
> >> > I
> >> > have to have some level of caution.
> >> >
> >> > Is there some other caching we can do (client-side or server-side)
> that
> >> > might give us a decent performance boost?
> >>
> >> Which client are you using for this testing? Have you looked at the
> >> readahead settings? That's usually the big one; if you're only asking
> >> for 4KB at once then stuff is going to be slow no matter what (a
> >> single IO takes at minimum about 2 milliseconds right now, although
> >> the RADOS team is working to improve that).
> >> -Greg
> >>
> >> >
> >> > ___
> >> > ceph-users mailing list
> >> > ceph-users@lists.ceph.com
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
> >
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS and caching

[Gregory Farnum]

On Wed, Sep 9, 2015 at 4:26 PM, Kyle Hutson  wrote:
>
>
> On Wed, Sep 9, 2015 at 9:34 AM, Gregory Farnum  wrote:
>>
>> On Wed, Sep 9, 2015 at 3:27 PM, Kyle Hutson  wrote:
>> > We are using Hammer - latest released version. How do I check if it's
>> > getting promoted into the cache?
>>
>> Umm...that's a good question. You can run rados ls on the cache pool,
>> but that's not exactly scalable; you can turn up logging and dig into
>> them to see if redirects are happening, or watch the OSD operations
>> happening via the admin socket. But I don't know if there's a good
>> interface for users to just query the cache state of a single object.
>> :/
>
>
> even using 'rados ls', I (naturally) get cephfs object names - is there a
> way to see a filename -> objectname conversion ... or objectname -> filename
> ?

The object name is .. So you can
look at the file inode and then see which of its objects are actually
in the pool.
-Greg

>
>>
>> > We're using the latest ceph kernel client. Where do I poke at readahead
>> > settings there?
>>
>> Just the standard kernel readahead settings; I'm not actually familiar
>> with how to configure those but I don't believe Ceph's are in any way
>> special. What do you mean by "latest ceph kernel client"; are you
>> running one of the developer testing kernels or something?
>
>
> No, just what comes with the latest stock kernel. Sorry for any confusion.
>
>>
>> I think
>> Ilya might have mentioned some issues with readahead being
>> artificially blocked, but that might have only been with RBD.
>>
>> Oh, are the files you're using sparse? There was a bug with sparse
>> files not filling in pages that just got patched yesterday or
>> something.
>
>
> No, these are not sparse files. Just really big.
>
>>
>> >
>> > On Tue, Sep 8, 2015 at 8:29 AM, Gregory Farnum 
>> > wrote:
>> >>
>> >> On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson 
>> >> wrote:
>> >> > I was wondering if anybody could give me some insight as to how
>> >> > CephFS
>> >> > does
>> >> > its caching - read-caching in particular.
>> >> >
>> >> > We are using CephFS with an EC pool on the backend with a replicated
>> >> > cache
>> >> > pool in front of it. We're seeing some very slow read times. Trying
>> >> > to
>> >> > compute an md5sum on a 15GB file twice in a row (so it should be in
>> >> > cache)
>> >> > takes the time from 23 minutes down to 17 minutes, but this is over a
>> >> > 10Gbps
>> >> > network and with a crap-ton of OSDs (over 300), so I would expect it
>> >> > to
>> >> > be
>> >> > down in the 2-3 minute range.
>> >>
>> >> A single sequential read won't necessarily promote an object into the
>> >> cache pool (although if you're using Hammer I think it will), so you
>> >> want to check if it's actually getting promoted into the cache before
>> >> assuming that's happened.
>> >>
>> >> >
>> >> > I'm just trying to figure out what we can do to increase the
>> >> > performance. I
>> >> > have over 300 TB of live data that I have to be careful with, though,
>> >> > so
>> >> > I
>> >> > have to have some level of caution.
>> >> >
>> >> > Is there some other caching we can do (client-side or server-side)
>> >> > that
>> >> > might give us a decent performance boost?
>> >>
>> >> Which client are you using for this testing? Have you looked at the
>> >> readahead settings? That's usually the big one; if you're only asking
>> >> for 4KB at once then stuff is going to be slow no matter what (a
>> >> single IO takes at minimum about 2 milliseconds right now, although
>> >> the RADOS team is working to improve that).
>> >> -Greg
>> >>
>> >> >
>> >> > ___
>> >> > ceph-users mailing list
>> >> > ceph-users@lists.ceph.com
>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >
>> >
>> >
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS and caching

[Gregory Farnum]

On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson  wrote:
> I was wondering if anybody could give me some insight as to how CephFS does
> its caching - read-caching in particular.
>
> We are using CephFS with an EC pool on the backend with a replicated cache
> pool in front of it. We're seeing some very slow read times. Trying to
> compute an md5sum on a 15GB file twice in a row (so it should be in cache)
> takes the time from 23 minutes down to 17 minutes, but this is over a 10Gbps
> network and with a crap-ton of OSDs (over 300), so I would expect it to be
> down in the 2-3 minute range.

A single sequential read won't necessarily promote an object into the
cache pool (although if you're using Hammer I think it will), so you
want to check if it's actually getting promoted into the cache before
assuming that's happened.

>
> I'm just trying to figure out what we can do to increase the performance. I
> have over 300 TB of live data that I have to be careful with, though, so I
> have to have some level of caution.
>
> Is there some other caching we can do (client-side or server-side) that
> might give us a decent performance boost?

Which client are you using for this testing? Have you looked at the
readahead settings? That's usually the big one; if you're only asking
for 4KB at once then stuff is going to be slow no matter what (a
single IO takes at minimum about 2 milliseconds right now, although
the RADOS team is working to improve that).
-Greg

>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS and caching

[Les]

Cephfs can use fscache. I am testing it at the moment.

Some lines from my deployment process:

sudo apt-get install linux-generic-lts-utopic cachefilesd
sudo reboot
sudo mkdir /mnt/cephfs
sudo mkdir /mnt/ceph_cache
sudo mkfs -t xfs /dev/md3 # A 100gb local raid partition
sudo bash -c "echo /dev/md/3 /mnt/ceph_cache xfs defaults,noatime 0 0 >> 
/etc/fstab"
sudo bash -c "echo M1:6789,M2:6789:/ /mnt/cephfs ceph 
name=fsuser,secretfile=/etc/ceph/fsuser.secret,noatime,fsc,_netdev 0 0 >> 
/etc/fstab"
sudo bash -c "echo REDACTED > /etc/ceph/fsuser.secret"
sudo chmod 400 /etc/ceph/fsuser.secret
sudo mount /mnt/ceph_cache/
sudo sed -i 's/#RUN=yes/RUN=yes/g' /etc/default/cachefilesd
sudo vim /etc/cachefilesd.conf #Change dir to /mnt/ceph_cache, tag to 
ceph_cache and comment everything else
sudo service cachefilesd start
sudo mount /mnt/cephfs

Cheers, Les

On 04.09.2015 00:58, Kyle Hutson wrote:
I was wondering if anybody could give me some insight as to how CephFS does its 
caching - read-caching in particular.

We are using CephFS with an EC pool on the backend with a replicated cache pool 
in front of it. We're seeing some very slow read times. Trying to compute an 
md5sum on a 15GB file twice in a row (so it should be in cache) takes the time 
from 23 minutes down to 17 minutes, but this is over a 10Gbps network and with 
a crap-ton of OSDs (over 300), so I would expect it to be down in the 2-3 
minute range.

I'm just trying to figure out what we can do to increase the performance. I 
have over 300 TB of live data that I have to be careful with, though, so I have 
to have some level of caution.

Is there some other caching we can do (client-side or server-side) that might 
give us a decent performance boost?



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CephFS and caching

[Kyle Hutson]

I was wondering if anybody could give me some insight as to how CephFS does
its caching - read-caching in particular.

We are using CephFS with an EC pool on the backend with a replicated cache
pool in front of it. We're seeing some very slow read times. Trying to
compute an md5sum on a 15GB file twice in a row (so it should be in cache)
takes the time from 23 minutes down to 17 minutes, but this is over a
10Gbps network and with a crap-ton of OSDs (over 300), so I would expect it
to be down in the 2-3 minute range.

I'm just trying to figure out what we can do to increase the performance. I
have over 300 TB of live data that I have to be careful with, though, so I
have to have some level of caution.

Is there some other caching we can do (client-side or server-side) that
might give us a decent performance boost?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com