[ceph-users] How to install Ceph on ARM?
Dear all, I wonder how to install Ceph on ARM processors. When I executed "$ ceph-deploy install [hosts]" on x86_64, ceph-deploy installed Ceph v10.2.9. However, when it is executed on ARM64, the installation is failed. $ ceph-deploy install ubuntu (...) [ubuntu][DEBUG ] add deb repo to /etc/apt/sources.list.d/ [ubuntu][INFO ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update [ubuntu][DEBUG ] Hit:1 http://ports.ubuntu.com/ubuntu-ports xenial InRelease [ubuntu][DEBUG ] Get:2 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease [102 kB] [ubuntu][DEBUG ] Get:3 https://download.ceph.com/debian-jewel xenial InRelease [8474 B] [ubuntu][DEBUG ] Get:4 https://download.ceph.com/debian-jewel xenial/main arm64 Packages [780 B] [ubuntu][DEBUG ] Get:5 http://ports.ubuntu.com/ubuntu-ports xenial-backports InRelease [102 kB] [ubuntu][DEBUG ] Get:6 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease [102 kB] [ubuntu][DEBUG ] Fetched 316 kB in 5s (60.4 kB/s) [ubuntu][DEBUG ] Reading package lists... [ubuntu][INFO ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph-osd ceph-mds ceph-mon radosgw [ubuntu][DEBUG ] Reading package lists... [ubuntu][DEBUG ] Building dependency tree... [ubuntu][DEBUG ] Reading state information... [ubuntu][WARNIN] E: Unable to locate package ceph-osd [ubuntu][WARNIN] E: Unable to locate package ceph-mon [ubuntu][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph-osd ceph-mds ceph-mon radosgw I found that there is no *.deb file of v10.2.9 for ARM64 in repos ( http://download.ceph.com/debian-jewel/pool/main/c/ceph/ ). But, I cannot know how to install Ceph v10.2.8 by ceph-deploy. May you tell me how to install a specific version of Ceph by ceph-deploy? or how to install Ceph on ARM processors? Thanks, Jae -- Jaemyoun Lee CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) E-mail: jaemy...@hanyang.ac.kr Website: http://cpslab.hanyang.ac.kr Personal site: http://jaemyoun.com --- This e-mail is intended only for the named recipient. Dissemination, distribution, forwarding, or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received it in error, please notify the sender by e-mail and completely delete it. Thank you for your cooperation. 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다. 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다. 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. ---___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to know if an object is stored in clients?
Thanks a lot! On Fri, Dec 30, 2016 at 6:32 PM Shinobu Kinjo <ski...@redhat.com> wrote: > You can track activity of acting set by using: > > # ceph daemon osd.${osd id} dump_ops_in_flight > > On Fri, Dec 30, 2016 at 3:59 PM, Jaemyoun Lee <jaemy...@hanyang.ac.kr> > wrote: > > Dear Wido, > Is there a command to check the ACK? Or, may you tell me a source code > function for the received ACK? > > Thanks, > Jae > > On Thu, Dec 29, 2016 at 6:56 PM Wido den Hollander <w...@42on.com> wrote: > > > > Op 28 december 2016 om 12:58 schreef Jaemyoun Lee < > jaemy...@hanyang.ac.kr>: > > > > > > Hello, > > > > I executed the RADOS tool to store an object as follows: > > ``` > > user@ClientA:~$ rados put -p=rbd objectA a.txt > > ``` > > > > I wonder how the client knows a completion of storing the object in some > > OSDs. > > > > When the primary OSD for a PG Acks to the client it knows that it is > stored on ALL replicas. > > RADOS/Ceph always writes synchronous. > > Wido > > > Thanks, > > Jae > > > > -- > > Jaemyoun Lee > > > > CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) > > E-mail : jaemy...@hanyang.ac.kr > > Website : http://cpslab.hanyang.ac.kr > > > > > > > --- > > 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 > 정보가 포함돼 있을 수 있습니다. > > 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 > 공개, 복사, 전송, 배포해서는 안 됩니다. > > 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. > > > > This e-mail is intended only for the named recipient. > > Dissemination, distribution, forwarding, or copying of this e-mail by > anyone other than the intended recipient is prohibited. > > If you have received it in error, please notify the sender by e-mail and > completely delete it. Thank you for your cooperation. > > > -------___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Jaemyoun Lee > > CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) > E-mail : jaemy...@hanyang.ac.kr > Website : http://cpslab.hanyang.ac.kr > > > 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 > 정보가 포함돼 있을 수 있습니다. > > 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, > 복사, 전송, 배포해서는 안 됩니다. > > 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. > > > This e-mail is intended only for the named recipient. > > Dissemination, distribution, forwarding, or copying of this e-mail by > anyone other than the intended recipient is prohibited. > > If you have received it in error, please notify the sender by e-mail and > completely delete it. Thank you for your cooperation. > > -- > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Jaemyoun Lee CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) E-mail : jaemy...@hanyang.ac.kr Website : http://cpslab.hanyang.ac.kr --- 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다. 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다. 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. This e-mail is intended only for the named recipient. Dissemination, distribution, forwarding, or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received it in error, please notify the sender by e-mail and completely delete it. Thank you for your cooperation. --- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to know if an object is stored in clients?
Dear Wido, Is there a command to check the ACK? Or, may you tell me a source code function for the received ACK? Thanks, Jae On Thu, Dec 29, 2016 at 6:56 PM Wido den Hollander <w...@42on.com> wrote: > > > Op 28 december 2016 om 12:58 schreef Jaemyoun Lee < > jaemy...@hanyang.ac.kr>: > > > > > > Hello, > > > > I executed the RADOS tool to store an object as follows: > > ``` > > user@ClientA:~$ rados put -p=rbd objectA a.txt > > ``` > > > > I wonder how the client knows a completion of storing the object in some > > OSDs. > > > > When the primary OSD for a PG Acks to the client it knows that it is > stored on ALL replicas. > > RADOS/Ceph always writes synchronous. > > Wido > > > Thanks, > > Jae > > > > -- > > Jaemyoun Lee > > > > CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) > > E-mail : jaemy...@hanyang.ac.kr > > Website : http://cpslab.hanyang.ac.kr > > > > > > > --- > > 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 > 정보가 포함돼 있을 수 있습니다. > > 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 > 공개, 복사, 전송, 배포해서는 안 됩니다. > > 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. > > > > This e-mail is intended only for the named recipient. > > Dissemination, distribution, forwarding, or copying of this e-mail by > anyone other than the intended recipient is prohibited. > > If you have received it in error, please notify the sender by e-mail and > completely delete it. Thank you for your cooperation. > > > -------___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- Jaemyoun Lee CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) E-mail : jaemy...@hanyang.ac.kr Website : http://cpslab.hanyang.ac.kr --- 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다. 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다. 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. This e-mail is intended only for the named recipient. Dissemination, distribution, forwarding, or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received it in error, please notify the sender by e-mail and completely delete it. Thank you for your cooperation. --- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] How to know if an object is stored in clients?
Hello, I executed the RADOS tool to store an object as follows: ``` user@ClientA:~$ rados put -p=rbd objectA a.txt ``` I wonder how the client knows a completion of storing the object in some OSDs. Thanks, Jae -- Jaemyoun Lee CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) E-mail : jaemy...@hanyang.ac.kr Website : http://cpslab.hanyang.ac.kr --- 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다. 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다. 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. This e-mail is intended only for the named recipient. Dissemination, distribution, forwarding, or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received it in error, please notify the sender by e-mail and completely delete it. Thank you for your cooperation. ---___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Register ceph daemons on initctl
Thanks! I solved it by the ceph-osd command. So... there is no a script to install Upstart, isn't it? Jae On Fri, Nov 18, 2016 at 3:26 PM 钟佳佳 <zhongjia...@haomaiyi.com> wrote: > if you built from git repo tag v10.2.3, > refers to links below from ceph.com > http://docs.ceph.com/docs/emperor/install/build-packages/ > > http://docs.ceph.com/docs/jewel/rados/operations/operating/#running-ceph-with-upstart > > if you built from ceph-10.2.3.tar.gz, it seems there's no debian stuff for > dpkg-buildpackage, > you shall ./configure and make yourself. > > I think it's right to copy the *.conf for upstart to /etc/init, > run: > initctl reload-configuration > initctl list | grep ceph > start ceph-osd id=XXX > ps aux | grep ceph-osd for the osd.X > > worse: > check the log, or > just run "ceph-osd --cluster=ceph -i X -f -d"(for foreground and > debug) to make sure if it has anything to do with the system init > > Good Luck > > the footer sucks > > > -- Original -- > *From: * "Jaemyoun Lee"<jaemy...@hanyang.ac.kr>; > *Date: * Fri, Nov 18, 2016 01:52 AM > *To: * "“ceph-us...@ceph.com”"<ceph-us...@ceph.com>; > *Subject: * [ceph-users] Register ceph daemons on initctl > > Dear all, > > I have a trouble for using Ceph. > When I built Ceph by source code from the official repo with v10.2.3, I > couldn't create a Ceph cluster. > > The stat of OSDs cannot be UP after they were activated. > I think the problem is Upstart because "make install" didn't copy conf > files to /etc/init/ > > # ls /etc/int/ | grep ceph* > # > > To solve it, I copied files in "src/upstart/" to /etc/init, and also > copied some script to /etc/init.d/. However, I failed. > > May you tell me how to add ceph daemons to initctl? > > Best regards, > Jae > > -- > > > 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 > 정보가 포함돼 있을 수 있습니다. > > 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, > 복사, 전송, 배포해서는 안 됩니다. > > 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. > > > This e-mail is intended only for the named recipient. > > Dissemination, distribution, forwarding, or copying of this e-mail by > anyone other than the intended recipient is prohibited. > > If you have received it in error, please notify the sender by e-mail and > completely delete it. Thank you for your cooperation. > -- > > --- 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다. 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다. 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. This e-mail is intended only for the named recipient. Dissemination, distribution, forwarding, or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received it in error, please notify the sender by e-mail and completely delete it. Thank you for your cooperation. --- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] creating a rbd
Thanks a lot. On Fri, Oct 7, 2016 at 2:19 PM Daleep Singh Bais <daleepb...@gmail.com> wrote: > Hi, > > Ceph uses other ports also for other daemons like OSD and MDS. Please > refer to > > http://docs.ceph.com/docs/jewel/rados/configuration/network-config-ref/ > > This might help you to resolve the issue. > > Thanks. > > > Daleep Singh Bais > > > On 10/07/2016 10:14 AM, Jaemyoun Lee wrote: > > Dear Laizer, > > Oh, I got it. the rbd was created successfully after I disabled the > firewall. > However, when the firewall is enabled and the 6789 port is allowed, the > authentication error is occurred. > > Is there any other port? > > On Fri, Oct 7, 2016 at 1:23 PM Lomayani S. Laizer <lomlai...@gmail.com> > wrote: > > Hello Lee, > Yes check can be firewall. Make sure port used by ceph are open > > -- > Lomayani > > On Fri, Oct 7, 2016 at 6:44 AM, Jaemyoun Lee <jaemy...@hanyang.ac.kr> > wrote: > > Dear Laizer, > > I did deploy the configuration and the key by '$ ceph-deploy admin > client-node' on admin-node. > jae@client-node$ ls /etc/ceph > ceph.client.admin.keyring ceph.conf rbdmap tmpoWLFTb > > > > > On Fri, Oct 7, 2016 at 12:33 PM Lomayani S. Laizer <lomlai...@gmail.com> > wrote: > > Hello Lee, > Make sure you have copied configuration and key for authentication > (ceph.client.admin.keyring) to client node. Looks it is an authentication > issue due to missing client key > > > -- > Lomayani > > On Fri, Oct 7, 2016 at 5:25 AM, Jaemyoun Lee <jaemy...@hanyang.ac.kr> > wrote: > > Hi, > I would like to create a rbd on client-node. > > I created a cluster on admin-node successfully. > jae@admin-node$ ceph health > HEALTH_OK > > A rbd was created on admin-node successfully. > jae@admin-node$ rbd created foo --size 1024 > > > However, when I created a rbd on client-node, the error was occurred: > jae@client-node$ rbd create foo --size 1024 > 2016-10-06 05:15:35.773242 7f3a52f367c0 0 monclient(hunting): > authenticate timed out after 300 > 2016-10-06 05:15:35.775497 7f3a52f367c0 0 librados: client.admin > authentication error (110) Connection timed out > > I guess the firewall is right because I can connect from client-node to > admin-node by ssh. > > May you talk me what is wrong? > > I referred the following site: > http://docs.ceph.com/docs/master/start/quick-rbd/ > > Best regards, > Jae > > -- > > > 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 > 정보가 포함돼 있을 수 있습니다. > > 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, > 복사, 전송, 배포해서는 안 됩니다. > > 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. > > > This e-mail is intended only for the named recipient. > > Dissemination, distribution, forwarding, or copying of this e-mail by > anyone other than the intended recipient is prohibited. > > If you have received it in error, please notify the sender by e-mail and > completely delete it. Thank you for your cooperation. > -- > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > > > 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 > 정보가 포함돼 있을 수 있습니다. > > 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, > 복사, 전송, 배포해서는 안 됩니다. > > 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. > > > This e-mail is intended only for the named recipient. > > Dissemination, distribution, forwarding, or copying of this e-mail by > anyone other than the intended recipient is prohibited. > > If you have received it in error, please notify the sender by e-mail and > completely delete it. Thank you for your cooperation. > -- > > > > -- > > > 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 > 정보가 포함돼 있을 수 있습니다. > > 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, > 복사, 전송, 배포해서는 안 됩니다. > > 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. > > > This e-mail is intended only for the named recipient. > > Dissemination, distribution, forwarding, or copying of this e-mail by > anyone other than the intended recipient is prohibited. > > If you have received it in error, please notify the sender by e-mail and > completely delete it.
Re: [ceph-users] creating a rbd
Dear Laizer, Oh, I got it. the rbd was created successfully after I disabled the firewall. However, when the firewall is enabled and the 6789 port is allowed, the authentication error is occurred. Is there any other port? On Fri, Oct 7, 2016 at 1:23 PM Lomayani S. Laizer <lomlai...@gmail.com> wrote: > Hello Lee, > Yes check can be firewall. Make sure port used by ceph are open > > -- > Lomayani > > On Fri, Oct 7, 2016 at 6:44 AM, Jaemyoun Lee <jaemy...@hanyang.ac.kr> > wrote: > > Dear Laizer, > > I did deploy the configuration and the key by '$ ceph-deploy admin > client-node' on admin-node. > jae@client-node$ ls /etc/ceph > ceph.client.admin.keyring ceph.conf rbdmap tmpoWLFTb > > > > > On Fri, Oct 7, 2016 at 12:33 PM Lomayani S. Laizer <lomlai...@gmail.com> > wrote: > > Hello Lee, > Make sure you have copied configuration and key for authentication > (ceph.client.admin.keyring) to client node. Looks it is an authentication > issue due to missing client key > > > -- > Lomayani > > On Fri, Oct 7, 2016 at 5:25 AM, Jaemyoun Lee <jaemy...@hanyang.ac.kr> > wrote: > > Hi, > I would like to create a rbd on client-node. > > I created a cluster on admin-node successfully. > jae@admin-node$ ceph health > HEALTH_OK > > A rbd was created on admin-node successfully. > jae@admin-node$ rbd created foo --size 1024 > > > However, when I created a rbd on client-node, the error was occurred: > jae@client-node$ rbd create foo --size 1024 > 2016-10-06 05:15:35.773242 7f3a52f367c0 0 monclient(hunting): > authenticate timed out after 300 > 2016-10-06 05:15:35.775497 7f3a52f367c0 0 librados: client.admin > authentication error (110) Connection timed out > > I guess the firewall is right because I can connect from client-node to > admin-node by ssh. > > May you talk me what is wrong? > > I referred the following site: > http://docs.ceph.com/docs/master/start/quick-rbd/ > > Best regards, > Jae > > -- > > > 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 > 정보가 포함돼 있을 수 있습니다. > > 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, > 복사, 전송, 배포해서는 안 됩니다. > > 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. > > > This e-mail is intended only for the named recipient. > > Dissemination, distribution, forwarding, or copying of this e-mail by > anyone other than the intended recipient is prohibited. > > If you have received it in error, please notify the sender by e-mail and > completely delete it. Thank you for your cooperation. > -- > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > > > 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 > 정보가 포함돼 있을 수 있습니다. > > 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, > 복사, 전송, 배포해서는 안 됩니다. > > 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. > > > This e-mail is intended only for the named recipient. > > Dissemination, distribution, forwarding, or copying of this e-mail by > anyone other than the intended recipient is prohibited. > > If you have received it in error, please notify the sender by e-mail and > completely delete it. Thank you for your cooperation. > -- > > > --- 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다. 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다. 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. This e-mail is intended only for the named recipient. Dissemination, distribution, forwarding, or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received it in error, please notify the sender by e-mail and completely delete it. Thank you for your cooperation. --- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] creating a rbd
Dear Laizer, I did deploy the configuration and the key by '$ ceph-deploy admin client-node' on admin-node. jae@client-node$ ls /etc/ceph ceph.client.admin.keyring ceph.conf rbdmap tmpoWLFTb On Fri, Oct 7, 2016 at 12:33 PM Lomayani S. Laizer <lomlai...@gmail.com> wrote: Hello Lee, Make sure you have copied configuration and key for authentication (ceph.client.admin.keyring) to client node. Looks it is an authentication issue due to missing client key -- Lomayani On Fri, Oct 7, 2016 at 5:25 AM, Jaemyoun Lee <jaemy...@hanyang.ac.kr> wrote: Hi, I would like to create a rbd on client-node. I created a cluster on admin-node successfully. jae@admin-node$ ceph health HEALTH_OK A rbd was created on admin-node successfully. jae@admin-node$ rbd created foo --size 1024 However, when I created a rbd on client-node, the error was occurred: jae@client-node$ rbd create foo --size 1024 2016-10-06 05:15:35.773242 7f3a52f367c0 0 monclient(hunting): authenticate timed out after 300 2016-10-06 05:15:35.775497 7f3a52f367c0 0 librados: client.admin authentication error (110) Connection timed out I guess the firewall is right because I can connect from client-node to admin-node by ssh. May you talk me what is wrong? I referred the following site: http://docs.ceph.com/docs/master/start/quick-rbd/ Best regards, Jae -- 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다. 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다. 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. This e-mail is intended only for the named recipient. Dissemination, distribution, forwarding, or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received it in error, please notify the sender by e-mail and completely delete it. Thank you for your cooperation. -- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com --- 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다. 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다. 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. This e-mail is intended only for the named recipient. Dissemination, distribution, forwarding, or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received it in error, please notify the sender by e-mail and completely delete it. Thank you for your cooperation. --- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] creating a rbd
Hi, I would like to create a rbd on client-node. I created a cluster on admin-node successfully. jae@admin-node$ ceph health HEALTH_OK A rbd was created on admin-node successfully. jae@admin-node$ rbd created foo --size 1024 However, when I created a rbd on client-node, the error was occurred: jae@client-node$ rbd create foo --size 1024 2016-10-06 05:15:35.773242 7f3a52f367c0 0 monclient(hunting): authenticate timed out after 300 2016-10-06 05:15:35.775497 7f3a52f367c0 0 librados: client.admin authentication error (110) Connection timed out I guess the firewall is right because I can connect from client-node to admin-node by ssh. May you talk me what is wrong? I referred the following site: http://docs.ceph.com/docs/master/start/quick-rbd/ Best regards, Jae --- 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다. 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다. 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다. This e-mail is intended only for the named recipient. Dissemination, distribution, forwarding, or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received it in error, please notify the sender by e-mail and completely delete it. Thank you for your cooperation. ---___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] no osds in jewel
Thanks for the feedback. I removed "ceph-deploy mon create + ceph-deploy gatherkeys." And my system disk is sde. As your opinion, the disk cannot be umounted when purgedata was run. Is it bug on Ubuntu 16.04? *$ ssh csAnt lsblk* NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:00 3.7T 0 disk ├─sda1 8:10 3.6T 0 part /var/lib/ceph/osd/ceph-0 └─sda2 8:20 5G 0 part sdb 8:16 0 3.7T 0 disk sdc 8:32 0 3.7T 0 disk sdd 8:48 0 3.7T 0 disk sde 8:64 0 238.5G 0 disk ├─sde1 8:65 0 512M 0 part /boot/efi ├─sde2 8:66 0 234G 0 part / └─sde3 8:67 0 4G 0 part [SWAP] sdf 8:80 0 3.7T 0 disk sdg 8:96 1 3.7T 0 disk sdh 8:112 1 3.7T 0 disk sdi 8:128 0 3.7T 0 disk sdj 8:144 0 3.7T 0 disk sdk 8:160 0 3.7T 0 disk *$ ceph-deploy purgedata csElsa csAnt csBull csCat* ... [ceph_deploy.install][INFO ] Distro info: Ubuntu 16.04 xenial [csAnt][INFO ] purging data on csAnt [csAnt][INFO ] Running command: sudo rm -rf --one-file-system -- /var/lib/ceph [csAnt][*WARNIN*] OSDs may still be mounted, trying to unmount them [csAnt][INFO ] Running command: sudo find /var/lib/ceph -mindepth 1 -maxdepth 2 -type d -exec umount {} ; [csAnt][*WARNIN*] umount: /var/lib/ceph/osd: not mounted [csAnt][*WARNIN*] umount: /var/lib/ceph/osd/ceph-0: target is busy [csAnt][*WARNIN*] (In some cases useful info about processes that [csAnt][*WARNIN*] use the device is found by lsof(8) or fuser(1).) [csAnt][INFO ] Running command: sudo rm -rf --one-file-system -- /var/lib/ceph [csAnt][*WARNIN*] rm: skipping '/var/lib/ceph/osd/ceph-0', since it's on a different device [csAnt][*ERROR* ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy][*ERROR* ] RuntimeError: Failed to execute command: rm -rf --one-file-system -- /var/lib/ceph *$ ssh csAnt sudo umount -f /dev/sda1* umount: /var/lib/ceph/osd/ceph-0: target is busy (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1).) Best regards, Jae On Tue, Jun 7, 2016 at 12:55 AM LOPEZ Jean-Charles <jelo...@redhat.com> wrote: > Hi, > > first I’f have one remark. You run both a ceph-deploy mon create-initial > then a "ceph-deploy mon create + ceph-deploy gatherkeys". Choose one or the > other not both. > > Then, I notice that you are zapping and deploying using drive /dev/sda > which is usually the system disks. So next question is: Is /dev/sda really > empty and available for OSDs > > I guess that what happened is that because sda is not free and empty, the > zap failed as it does not have the force option and then the old create > failed because sda could not be prepared/formatted. > > Regards > JC > > On Jun 5, 2016, at 10:22, Jaemyoun Lee <jaemy...@gmail.com> wrote: > > Hi, > > When I run below script to install Ceph (10.2.0), I met an error "no osds". > Hammer was installed by the script. > So I think I miss new thing, which was released since Hammer. > > Do you know what I miss? > > --- The script --- > #!/bin/sh > > set -x > > ceph-deploy new csElsa > echo "osd pool default size = 1" >> ceph.conf > ceph-deploy install csElsa csAnt csBull csCat > ceph-deploy mon create-initial > ceph-deploy mon create csElsa > ceph-deploy gatherkeys csElsa > ceph-deploy disk zap csAnt:sda > ceph-deploy disk zap csBull:sda > ceph-deploy disk zap csCat:sda > ceph-deploy osd create csAnt:sda csBull:sda csCat:sda > ceph-deploy admin csElsa csElsa csAnt csBull csCat > sudo chmod +r /etc/ceph/ceph.client.admin.keyring > ceph health > --- end --- > > --- The result of "ceph -w" --- > # I blocked the IP > jae@csElsa:~/git/ceph$ ceph -w > > cluster 8b2816e9-1953-4157-aaf7-95e9e668fe46 > health HEALTH_ERR > 64 pgs are stuck inactive for more than 300 seconds > 64 pgs stuck inactive > no osds > monmap e1: 1 mons at {csElsa=1xx.1xx.2xx.1:6789/0} > election epoch 3, quorum 0 csElsa > osdmap e1: 0 osds: 0 up, 0 in > flags sortbitwise > pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects > 0 kB used, 0 kB / 0 kB avail > 64 creating > > 2016-06-06 01:59:08.054985 mon.0 [INF] from='client.? > 1xx.1xx.2xx.1:0/115687' entity='client.admin' cmd='[{"prefix": "auth > get-or-create", "entity": "client.bootstrap-mds", "caps": ["mon", "allow > profile bootstrap-mds"]}]': finished > --- end --- > > Best regards, > Jae > > -- > Jaemyoun Lee > > CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) > E-mail :
[ceph-users] no osds in jewel
Hi, When I run below script to install Ceph (10.2.0), I met an error "no osds". Hammer was installed by the script. So I think I miss new thing, which was released since Hammer. Do you know what I miss? --- The script --- #!/bin/sh set -x ceph-deploy new csElsa echo "osd pool default size = 1" >> ceph.conf ceph-deploy install csElsa csAnt csBull csCat ceph-deploy mon create-initial ceph-deploy mon create csElsa ceph-deploy gatherkeys csElsa ceph-deploy disk zap csAnt:sda ceph-deploy disk zap csBull:sda ceph-deploy disk zap csCat:sda ceph-deploy osd create csAnt:sda csBull:sda csCat:sda ceph-deploy admin csElsa csElsa csAnt csBull csCat sudo chmod +r /etc/ceph/ceph.client.admin.keyring ceph health --- end --- --- The result of "ceph -w" --- # I blocked the IP jae@csElsa:~/git/ceph$ ceph -w cluster 8b2816e9-1953-4157-aaf7-95e9e668fe46 health HEALTH_ERR 64 pgs are stuck inactive for more than 300 seconds 64 pgs stuck inactive no osds monmap e1: 1 mons at {csElsa=1xx.1xx.2xx.1:6789/0} election epoch 3, quorum 0 csElsa osdmap e1: 0 osds: 0 up, 0 in flags sortbitwise pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects 0 kB used, 0 kB / 0 kB avail 64 creating 2016-06-06 01:59:08.054985 mon.0 [INF] from='client.? 1xx.1xx.2xx.1:0/115687' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.bootstrap-mds", "caps": ["mon", "allow profile bootstrap-mds"]}]': finished --- end --- Best regards, Jae -- Jaemyoun Lee CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) E-mail : jaemy...@hanyang.ac.kr Website : http://cpslab.hanyang.ac.kr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] fail OSD prepare
I am following the quick doc. It is successful until Adding the initial monitor. So I made the osd folder (/var/local/osd0, osd10, osd20) in each node (csAnt, csBull, csCat), and deployed to prepare the OSDs. But the below error was occurred. --- jae@csElsa:~$ ceph-deploy osd prepare csAnt:/var/local/osd0 csBull:/var/local/osd10 csCat:/var/local/osd20 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/jae/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.25): /usr/bin/ceph-deploy osd prepare csAnt:/var/local/osd0 csBull:/var/local/osd10 csCat:/var/local/osd20 [ceph_deploy][ERROR ] ConfigError: Cannot load config: [Errno 2] No such file or directory: 'ceph.conf'; has `ceph-deploy new` been run in this directory? --- Should I do something, which there is no the quick doc, before preparing the OSDs? -- Jaemyoun Lee CPS Lab. ( Cyber-Physical Systems Laboratory in Hanyang University) E-mail : jm...@cpslab.hanyang.ac.kr Homepage : http://cpslab.hanyang.ac.kr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Is OSDs based on VFS?
Hi all, I wonder that OSDs use system calls of Virtual File System (i.e. open, read, write, etc) when they access disks. I mean ... Could I monitor I/O command requested by OSD to disks if I monitor VFS? - Jae -- 이재면 Jaemyoun Lee CPS Lab. ( Cyber-Physical Systems Laboratory in Hanyang University) E-mail : jm...@cpslab.hanyang.ac.kr Homepage : http://cpslab.hanyang.ac.kr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is OSDs based on VFS?
Thanks for your rapid reply - Jae On Tue, Jul 22, 2014 at 1:29 AM, Gregory Farnum g...@inktank.com wrote: On Monday, July 21, 2014, Jaemyoun Lee jm...@cpslab.hanyang.ac.kr wrote: Hi all, I wonder that OSDs use system calls of Virtual File System (i.e. open, read, write, etc) when they access disks. I mean ... Could I monitor I/O command requested by OSD to disks if I monitor VFS? Yes. The default configuration stores data in a local filesystem and accesses it like any other sophisticated filesystem consumer. -Greg - Jae -- 이재면 Jaemyoun Lee CPS Lab. ( Cyber-Physical Systems Laboratory in Hanyang University) E-mail : jm...@cpslab.hanyang.ac.kr Homepage : http://cpslab.hanyang.ac.kr -- Software Engineer #42 @ http://inktank.com | http://ceph.com -- 이재면 Jaemyoun Lee CPS Lab. ( Cyber-Physical Systems Laboratory in Hanyang University) E-mail : jm...@cpslab.hanyang.ac.kr Homepage : http://cpslab.hanyang.ac.kr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is OSDs based on VFS?
Thanks for your rapid reply - Jae On Tue, Jul 22, 2014 at 1:28 AM, Kyle Bader kyle.ba...@gmail.com wrote: I wonder that OSDs use system calls of Virtual File System (i.e. open, read, write, etc) when they access disks. I mean ... Could I monitor I/O command requested by OSD to disks if I monitor VFS? Ceph OSDs run on top of a traditional filesystem, so long as they support xattrs - xfs by default. As such you can use kernel instrumentation to view what is going on under the Ceph OSDs. -- Kyle -- 이재면 Jaemyoun Lee CPS Lab. ( Cyber-Physical Systems Laboratory in Hanyang University) E-mail : jm...@cpslab.hanyang.ac.kr Homepage : http://cpslab.hanyang.ac.kr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-fuse couldn't be connect.
() 2014-07-16 17:12:54.202389 7f9a177fe700 10 client.4705 renew_caps() 2014-07-16 17:13:14.205054 7f9a177fe700 10 client.4705 renew_caps() 2014-07-16 17:13:16.164362 7f9a15ffb700 1 -- 192.168.122.166:0/3296 -- 192.168.122.106:6789/0 -- mon_subscribe({mdsmap=2+,monmap=2+}) v2 -- ?+0 0x7f99f8005030 con 0x7f9a23c0dd30 2014-07-16 17:13:16.165178 7f9a167fc700 1 -- 192.168.122.166:0/3296 == mon.0 192.168.122.106:6789/0 10 mon_subscribe_ack(300s) v1 20+0+0 (4226112827 0 0) 0x7f9a3c40 con 0x7f9a23c0dd30 2014-07-16 17:13:34.207622 7f9a177fe700 10 client.4705 renew_caps() 2014-07-16 17:13:54.209995 7f9a177fe700 10 client.4705 renew_caps() 2014-07-16 17:14:14.212277 7f9a177fe700 10 client.4705 renew_caps() 2014-07-16 17:14:34.214827 7f9a177fe700 10 client.4705 renew_caps() Thx~ - Jae On Wed, Jul 16, 2014 at 2:20 AM, Gregory Farnum g...@inktank.com wrote: On Tue, Jul 15, 2014 at 10:15 AM, Jaemyoun Lee jaemy...@gmail.com wrote: The output is nothing because ceph-fuse fell into an infinite while loop as I explain below. Where can I find the log file of ceph-fuse? It defaults to /var/log/ceph, but it may be empty. I realize the task may have hung, but I'm pretty sure it isn't looping, just waiting on some kind of IO. You could try running it with the --debug-ms 1 --debug-client 10 command-line options appended and see what it spits out. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com -- 이재면 Jaemyoun Lee E-mail : jaemy...@gmail.com Homepage : http://jaemyoun.com Facebook : https://www.facebook.com/jaemyoun ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph-fuse couldn't be connect.
Hi All, I am using ceph 0.80.1 on Ubuntu 14.04 on KVM. However, I cannot connect to the MON from a client using ceph-fuse. On the client, I installed the ceph-fuse 0.80.1 and added fuse. But, I think it is wrong. The result is # modprobe fuse (Any output was nothing) # lsmod | grep fuse (Any output was nothing) # ceph-fuse -m 192.168.122.106:6789 /mnt ceph-fuse[1905]: starting ceph client (at this point, ceph-fuse fell into an infinite while loop) ^C # What problem is it? My cluster like the follow: Host OS (Ubuntu 14.04) --- VM-1 (Ubuntu 14.04) -- MON-0 -- MDS-0 --- VM-2 (Ubuntu 14.04) -- OSD-0 --- VM-3 (Ubuntu 14.04) -- OSD-1 -- OSD-2 -- OSD-3 --- VM-4 (Ubuntu 14.04) -- it's for client. the result of # ceph -s on VM-1, which is MON, is # ceph -s cluster 1ae5585d-03c6-4a57-ba79-c65f4ed9e69f health HEALTH_OK monmap e1: 1 mons at {csA=192.168.122.106:6789/0}, election epoch 1, quorum 0 csA osdmap e37: 4 osds: 4 up, 4 in pgmap v678: 192 pgs, 3 pools, 0 bytes data, 0 objects 20623 MB used, 352 GB / 372 GB avail 192 active+clean # Regards, Jae -- 이재면 Jaemyoun Lee E-mail : jaemy...@gmail.com Homepage : http://jaemyoun.com Facebook : https://www.facebook.com/jaemyoun ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com